🆔FP: History of changes

Work in progress

Status Feature Passport

STATUS	OWNER	DATE
In proposal	Sofie	13 + 14/04/2022
In refinement - Design Research	Judith StraetemansSaska Pallayova	29/09/22
In refinement - Technical Research/Feedback
In development
In QA/ Testing
In Final state

Analysis

Linked Jira tickets

Linked JIRA's

EPIC = OP-943: History of changes

OP-871: How to find when (and by whom if possible) something got edited? This ticket describes the issues we had in case of double worship service and double central worship service. The comments in this ticket are merged into this Feature Passport, that's why ticket OP-871 has status GEREED.

OP-1224: Adding a modified and creation date for each bestuurseenheid (Requested in refinement ticket

Current state

• We (ABB users) don't have any information on any change event.

• Developers can archive data (data cannot be deleted due to Linked Open Data)

Goal

• It would be valuable to have this functionality for audit & educational purposes for internal ABB users.

• Data we'd like to include (needs to be refined more): data on the properties or classes that have an impact when they need to be archived due to incorrect data or other reasons.

  • Full name of the person who made a change

  • Timestamp: Date / hour

  • Action that was taken / fields that were changed

• Make sure other ABB-products can benefit from it as well

• Most important example: For easy and correct archiving it would be an improvement to know who created a person or an administrative unit in case of double registrations.

Usecases

• Audit & educational purposes

  • As OP-ABB-internal user, I want to be able to audit changes made that happened incorrectly.

  • As OP-ABB-internal user, I want to be able to educate or notify users that incorrectly made changes

Other potential usecases to explore

• Preventive system; push responsibilities to others

  • As ABB internal OR external user, I want to be alerted when someone else made a change to a page I created OR I want to follow.

• Super admin system without need for manual interventions

  • Ability to archive data by users with review possibilities of super admin users so developers don't need to be involved

• Other ABB users of different projects

  • As other product owner, I want to be able to build functionality into this back-end too.

🤩 Expectations

We have taken into account the advise from the developers in the comments of OP-871.

It is not realistic and overkill to have a history on all classes (see Past discussion points on full history in this FP).

Some bookkeeping on created resources might be a good idea

• created date

• created by

• modified date

• last modified by

on classes, but not on the separate fields

Classes in module 'Bestuurseenheden'

• 🔴 bestuurseenheid (or class organisation? > generalisation)

• 🔴 veranderingsgebeurtenissen

• 🟠 Gerelateerde organisaties

• 🟡 (vestiging)

• 🟡 bestuursorgaan

• 🟡 (betrokken lokale besturen)

Classes in Module 'Personen'

• 🔴 persoon

• 🔴 positie (or class Agent in positie?)

🕵️‍♂️Use Cases

See Use Cases on FP: Archive/Delete data

🤔 Past discussion points on full history

Past discussion on full history

1. Check audit logs?

• Answer by Niels We can check the audit logs, but this is very time-consuming (not a solution). They are only meant for exceptional cases (f.e. some kind of abuse) [23/11/2021 - Full comment of Niels]

2. Use Linked Data Event Streams?

• Answer by Aad LDES is not an option. It assumes you have a modified date. [28/11/2021 - Full comment of Aad]

3. How can we achieve this functionality? [29/11/2021 - Full question of Nordine]

1) via the front-end (through mu-cl-resource)

2) create the event using ember data every time a change we are interested in is made

• Answer by Niels It depends on the amount of governance that is required, are you interested in just the last change or every change? Given that we don’t keep history on the objects itself at the moment it seems overkill to keep track of the metadata for each change. My proposal was to add the three properties on the bestuurseenheid resource, so you’d have an idea who made the last change (and when). However if you need more it’s probably wise to have a look at the provenance ontology and implement that. I think it’s a bit of overkill here though, but could be wrong. [= Full comment]

4. To get an idea of the impact, how complex would it be to have an history of creation date and modified date on each field on the OP? [3/12/2021 - Full question of Yassin]

• Answer by Nordine [Full comment] For each field (property) Impact will be huge, from the backend side we have to

  1) listen to all delta messages

  2) add a “created date” and a “modified date” triples for each insert, insert/delete (update)

• Answer by Niels [Full comment] To have this on each field would definitely need to be stored in a separate database, it’s also an amount of data governance I have not seen often.

• Anwer by Aad [Full comment] If this (for whatever value of ‘this’) is really what we want to do, we can set something on the application adapter to save these few properties on first create. We could basically do the same if we want to have a series of events attached to an object. I’m not sure if this would really cause problems in the triplestore. We could guess and poc it.

  Again: please analyze first what is actually needed. This is a single request and it feels to me like this may as well just be a regular question. Most applications don’t have full tracking. If tracking exists, implementation varies widely. Analysis needed before poc.

Solution

Research requirements

• Set-up meetings with other PO's/PMs of Gelinkt Notuleren & Loket to research

  • Current solutions, developments & future plans how they track changes, both on user side as on back-end side

  • Goals & wishes they have if all is possible & with constraints

  • Feedback on our prototype

  • Whether their engineers would be willing to add changes to a global platform

Design solution & requirements

We will design a potential full fledge vision, focussed on the main usecase (auditing & educational purposes), but also including potential other future usecases to tease out reactions of other products.

• Back-end, stand-alone solution for all ABB-users

  • Ability to log-in based on the right permissions

  • Overview with a list of changes that can be filtered

    • Name of editor

    • Role of editor

    • Date change was made

    • Change made

    • Module & Field where the change was made

    • How data got into OP (Loket / Sharepoint / manual input)

    • Filtering possibilities

  • Editor page

    • Changes made

    • Email & contact details (to alert)

  • Reporting module

    • List view & aggregate (totals) with filters

    • Report (with charts)

    • CSV export of data with filters

  • Preventive measurements

    • Notification system

      • Ability to watch pages & be alerted when a change is made to a page

      • Automatic alert to person who created page

    • Ability to review changes

    • Ability to revert changes

• Front-end solution

  • Ability to easily see change history of organisations and people on the front-end if you have the right rights

Technical solution for all items

[7/12/2021 - Proposed solutions by Nordine in the comments]

Idea 1 (the simplest and the least “elegant”):

• We add createdDate, createdBy, updatedBy & updatedDate to all classes in domain.json (or we create a class with those properties that all class will inherits).

• Frontend will be responsible for managing these fields through ember data.

Idea 2:

• new microservice that uses delta notifier’s message to populate an elasticsearch index with the required fields (createdDate, createdBy, updatedBy & updatedDate , propertyUri, subjectUri).

Idea 3:

• Having this feature in the semtech stack out of the box. This makes sense if it becomes a requirement for other projects in the future.

Idea 4:

• we only add the properties createdDate, updatedDate to the class we are interested in. We will not know which property has been updated, and we won’t have an history of changes (only the latest change will be available).

Idea 5:

• We have more roles (acm/idm) and we limit WRITE access to certain types (e.g Only users with roles “Organization-Edit” can edit an Organization).

For idea 5, check FP: Roles and Rights

Design

This part of the feature passport is owned by the designer

User research

[If there is any user research preceding the wireframe mock-up stage, it needs to be documented here]

Mock-ups

[link to figma mockups + any explanation or extra documentation]

Technical

This part of the feature passport is owned by the technical team

[Information about the technical solutions for expectations that need it - e.g. using mu-search for showing all types of positions in one table.]

After the designer and/or the technical team finish their task, a meeting follows where the solutions are presented. The team exchanges feedback and amends the feature passport where necessary.