🆔FP: Archive/Delete data

Work in progress: This feature passport contains the URI strategy to archive or delete data.

Status Feature Passport

STATUS	OWNER	DATE
In proposal	Yassin (concept) Sofie reviewed (concept) Sofie (concept to FP)	before 08/04/2022 before 08/04/2022 08/04/2022 + 12/04/2022
In refinement - Design Research
In refinement - Technical Research/Feedback
In development
In QA/ Testing
In Final state

Yassin Boullauazan: when you reviewed this FP, the [Conception] Archiving page can be deleted as the information is merged into this FP.

EPIC = OP-944: Archiving of data

Analysis

What do we mean by archiving

1. Mandatory archiving of correct data

• 1a. No longer available in FE and BE. A copy is available in another very secure database. Archived data is removed from BE.

• 1b. No longer available in FE, but still available in BE.

= Legal retention period of contact and organisation information ('maximum bewaringstermijn')

• We are no longer allowed to use/show this data in the OP.

• We are no longer allowed to share this data with other products.

• Data that is not viewable anymore in the front-end, but can it still be in the database?

• Data can not be consumed via API.

• Can data be shown only via queries for investigation purposes (ISD).

TO DO: We need to discuss with 'team informatiebeheer' and DPO if it applies to any data that is stored via OP. Can we keep all the data permanently in the OP or are we obliged to archive some of it?

2. Archiving of incorrect data (hard delete)

= 'hard delete' of data that was incorrectly entered by end-users

• Data that is not viewable anymore in the front-end.

• Has a characteristic in the back-end, which makes it clear for API consumers of OP that the data should not be taken into account.

• If we run a query on the SPARQL endpoint, we need an attribute (or something else) to be able to include and exclude that it is archived (and can be filtered out in the query or afterwards manually).

3. Archiving of incorrect data = soft delete

= 'soft delete' of data that was incorrectly entered by end-users

• When we connect with external sources

Current state

Archiving of incorrect data can only be done by the developers via tickets in the epic OP-1170: Incident Handling.

1. Mandatory archiving of correct data

Not yet applicable to the current data in the database.

2. Archiving of incorrect data

Examples of incident tickets: (see details in use cases in this FP)

• Change status of administrative unit (status change was not part of a change event)

• Change date of the change event (= a non editable field)

• Delete mandataries in a governing body

• Delete board members in a governing body

• Delete a minister in a worship service

• Delete double administrative units

• Delete KBO number in a Representative body (temporary problem, we will be able to change this when the module 'ondernemingen en verenigingen' is launched')

Connection to FP History of Changes: We would like to know the history of the changes of the incorrect data so it's easier to investigate what happened and we can contact the right person for more information. Right now we are totally in the dark. (see issue OP-871)

For example: For easy and correct archiving it would be an improvement to know who created a person or an administrative unit in case of double registrations.

🤩Expectations

We should talk about the Use Cases and Discussion points to see what is realistic.

In general

• Do we have procedures/guidelines to execute this, taking into account all the relations mentioned in the use cases?

• How can we achieve this in an error-proof way?

• Is this only feasible for developers?

1. Mandatory archiving of correct data

• Context of worship services and connected persons:

  • Is this something that will happen in Loket and OP? BOTH

• Context of other administrative units and connected persons:

  • Is this something that will happen in Loket and OP? BOTH

• Context of organisations and associations and their connected persons?

  • This happens only in OP.

2. Archiving of incorrect data

• We want to limit the possibilities that mistakes can happen in OP. Later this is partially the responsibility of Loket as well. Can we take more actions in this context. (f.e. limit the number of persons who can add an administrative unit via an extra role?)

In this feature passport, we will tackle these future expectations:

[the analyst expresses the expectations that need to be met in order to finish this feature]

The analyst does not provide with solutions. This is the task of the technical and design team.

e.g.

❌ There should be a button in the bottom-left corner so the user can send an email

✅ The user needs the ability to send emails to customer service

After the analyst is done with this, there needs to be a meeting (this can become part of the BRM) where the analyst talks through the expectations with the technical and design team. The team can then decide which expectations are feasible for this feature and create tickets accordingly.

This means that if, for example, the technical team thinks that one of the expectations for this feature will take a lot longer than the analyst anticipated, the PO can still decide the work on that expectations as part of another feature passport (* this then needs to be amended in the expectations and a new feature passport can be created.)

Once the team has agreed on the expectations and the tickets have been created, acceptance criteria can be added on the tickets in jira.

This includes any dependencies, e.g.: Story: A user needs to be able to add a position to person Acceptance criteria:

• The position needs to be automatically added to the relevant organisation

• The position needs to be automatically added to the list of all the positions

🕵️‍♂️ Use Cases archiving incorrect data

1. Archiving an administrative unit (Bestuurseenheid)

Cause: Accidently creating a double

Frequency:

• Low - As maturity in use increases, I would estimate this would happen once every month.

• Current cases: at least 4

🔴 Combinatorial effects of archiving it (Very High!):

• It is related to a Bestuursorgaan, which is/can be related to mandates, which is linked to a person. (All the Bestuursorganen, Mandates would need to be archived too)

• It is related to a Site (All the sites would need to be archived too)

• It is related to a Bedienaar (for a worship service only): (All the Mandates would need to be archived too)

• It is related to a Betrokken Lokale Bestuur (this needs to be unlinked so that this relationship doesn’t need to be archived)

• It - can be – related to a (central) worship service and representative organ (this needs to be unlinked so that this relationship doesn’t need to be archived)

2. Archiving a minister (Bedienaar)

Cause: Accidently adding the wrong person as a bedienaar or an end-user adding person for testing purposes in Production.

Frequency:

• Low - As maturity in use increases, I would estimate this would happen once every month.

• Current cases: at least 1

🟡 Combinatorial effects of archiving it (Medium):

• The Minister position details data needs to be archived

• It is related to a Person. (The position that is being archived needs to be unlinked from the Person)

• It is related to a Bestuurseenheid (The position needs to be unlinked from the Bestuurseenheid)

3. Archiving a mandate/board member (Mandataris/Bestuurslid)

Cause: Accidently adding the wrong person as a 'mandataris/bestuurslid', or an end-user adding person for testing purposes in Production.

Frequency:

• Low - As maturity in use increases, I would estimate this would happen once every month.

• Current cases: at least 1

🟡 Combinatorial effects of archiving it (Medium):

• The Mandataris/Bestuurslid position details data needs to be archived

• It is related to a Person. (The position that is being archived needs to be unlinked from the Person)

• It is related to a Bestuursorgaan (The position needs to be unlinked from the Bestuursorgaan)

In Loket they solve this by making the end date the same as the start date.

6. Archiving a site (less common)

Cause: Accidently adding the wrong site, or an end-user adding person for testing purposes in Production.

For an organization it can occur that a vestiging (or a type vestiging) is no longer in use after date xx-xx-xxxx. This problem should best be covered with start- and end-dates.

🟡 Combinatorial effects of archiving it (Medium):

• The Site details data needs to be archived

• It is related to a Bestuurseenheid (The site needs to be unlinked from the Bestuurseenheid)

4. Archiving a Betrokken Lokale Bestuur

Cause: Accidently adding the wrong Betrokken Lokale Bestuur or a result due to a change event, or an end-user adding person for testing purposes in Production.

After a change event it can occur that a gemeente or provincie is no longer a Betrokken Lokale Bestuur after date xx-xx-xxxx. This problem should best be covered with start- and end-dates.

See ticket OP-1193: ON HOLD > Delete button not available in "Bewerk betrokken lokale besturen"

The 'Verwijder' button is only available when you add a line and before saving the change.

🟡 Combinatorial effects of archiving it (Medium):

• The Betrokken Lokale Bestuur details data (percentage, type) needs to be archived

• The sum of the financial percentages of the other Betrokken Lokale Besturen in the table needs to be 100%.

• It is related to a Bestuurseenheid (The Betrokken Lokale Bestuur with its relationship data needs to be unlinked from the Bestuurseenheid).

Sofie, 12/04/2022: I would change this to a common use case. End users have requested this. OK

5. Archiving a Person

Cause: Accidently adding a person that is not linked to a position or bestuurseenheid, or an end-user adding person for testing purposes in Production.

Frequency:

• Low - As maturity in use increases, I would estimate this would happen once every quarter.

🟡 Combinatorial effects of archiving it (Medium):

• The Mandataris/Bestuurslid position needs to be archived

• The Bedienaar position needs to be archived

• It is related to a Bestuurseenheid (The Betrokken Lokale Bestuur with its relationship data needs to be unlinked from the Bestuurseenheid)

• It is related to a Bestuursorgaan (The position needs to be unlinked from the Bestuursorgaan)

Sofie - 12/04/2022: We should add to every cause in the use cases: end-user adding an item for testing purposes in Production. OK

🤔 Discussion points

1. Can we handle the different forms of archiving?

2. In all forms of archiving we have to take into account the related/linked items

• Do the linked items also need to be archived (maybe not in all cases)?

• What if they are not archived?

3. Is it possible that the editor, PM/PO or helpdesk can do the archiving of wrong data (soft delete) themselves (temporarily so that it is no longer visible in the FE)? See example in hint by use case 3. Archiving a mandate/board member.

4. Are the steps to archive an item already documented?

5. Is adding a start and end date a solution?

6. What about the URI's, we can not delete URI's, are extra labels (archived) needed?

7. Check with Aad for best practices or archiving in other products.

Solution

The next two parts of the FP can [sometimes] be worked on simultaneously

Design

This part of the feature passport is owned by the designer

User research

[If there is any user research preceding the wireframe mock-up stage, it needs to be documented here]

Mock-ups

[link to figma mockups + any explanation or extra documentation]

Technical

This part of the feature passport is owned by the technical team

[Information about the technical solutions for expectations that need it - e.g. using mu-search for showing all types of positions in one table.]

After the designer and/or the technical team finish their task, a meeting follows where the solutions are presented. The team exchanges feedback and amends the feature passport where necessary.