Skip to content

Bulk Data Anonymization Workflow

Legacy reference, pending update

This guide is adapted from a customer-facing user manual for an older eXate version (v1.6.0, 2021). It's included as a starting point for GatorSet's bulk/ETL workflow documentation, not a finished guide. Before publishing, this needs: verification against the current product version and UI (menu names like "Static Data Management" and field labels may have changed), replacement of the illustrative screenshots (not available in the source material), and confirmation that the role names below still match eXate's current RBAC model. All customer-specific URLs and organisation references from the source document have been removed or genericised.

This guide walks through configuring and running a bulk anonymization job against a source database, from establishing the connection through to job completion. It uses three roles in a maker-checker workflow: an author who defines the job, a validator who approves it, and a data steward who runs it.

1. Establish a Connection String

Role: Data Steward

The first step is to establish the connection string to the source and target databases. Connections are managed through the eXate portal's connection management area.

  1. Navigate to the connections section of your eXate portal
  2. Click the "+" button to create a new connection
  3. Fill in the connection details:
    • Name: a name for this connection, used to identify it within your team
    • Database type: choose the connector type from the dropdown
    • Connection string: the database connection string
    • Vault storage type: choose the appropriate storage type for your environment
  4. Click Test Connection to confirm the details are correct
  5. Click Save

2. Define Attribute Groups

Role: Data Anonymization Author

Attribute groups define the anonymization rules that get applied during the job. Supported data types are String, Date, Integer, and Decimal.

  1. Navigate to Static Data Management → Attribute Groups in your eXate portal
  2. Click the "+" button to create a new group, give it a name, and submit
  3. Within a group, click "+" to add a new attribute (anonymization rule), or click an existing one to view or modify it

2.1 Data Type: Date

Choose "Date" as the attribute data type. You can apply pure randomisation, or preserve part of the date, for example keeping the Year or Month while randomising the rest, by setting Preserve Data to true and choosing the part to preserve.

2.2 Data Type: String

String tokenisation: choose "String" as the data type and define the maximum length of the anonymised value. For fields like email addresses, you can preserve part of the original value by setting Preserve Data to true and specifying which part.

List substitution: for lookup-based substitution rather than tokenisation, navigate to Static Data Management → List. Create a new list, fill in the lookup values (null values are supported), then reference that list from the relevant attribute's details.

2.3 Data Type: Decimal

Choose "Decimal" as the attribute data type and submit.

2.4 Data Type: Integer

Choose "Integer" as the attribute data type and submit.

3. Create a Manifest

Role: Data Anonymization Author

The Manifest defines the scope and configuration of the bulk job. Navigate to Manifest → Database Manifests in your eXate portal.

3.1 Create a New Manifest

  1. Click "+" to create a new manifest
  2. Fill in:
    • Database type: the source database type
    • Connection: the connection defined in step 1
    • Manifest name: a name for this manifest
    • Target: the target data type
  3. Click Connect to load the schema
  4. Select the schema, then click Get Schema Info to load table metadata
  5. Select the tables to bring into scope
  6. For each column you want anonymised, set the Attribute Type to the attribute defined in step 2
  7. Click Create Manifest

3.2 Update a Manifest

From the manifest list, click Details on the manifest you want to modify, make your changes, and click Update Manifest to save.

3.3 Add a Row Limit

Open the manifest's Details view, enable "Limit Rows," and specify the number of rows to anonymise for that table. Click Update Manifest to save.

3.4 Add a Filter

Open the manifest's Details view and click Add Filter on the relevant table. Choose the column, condition, and value. Click Add New Filter to add further conditions, then Submit to save the filter, and Update Manifest to save the manifest.

3.5 Target Database Write Behaviour

Open the manifest's Details view and choose the write operation:

  • Append: anonymised data is added to the existing target table
  • Truncate: anonymised data replaces the contents of the target table

4. Manifest Approval

Role: Data Anonymization Validator

  1. The author opens the manifest's Details view and clicks Submit for Approval
  2. The manifest status changes to Pending Approval. Authors cannot approve or reject their own submissions
  3. The validator opens the manifest's Details view and clicks Approve (or Reject to send it back for rework)
  4. Once approved, the manifest status becomes Live. A manifest can be reset to Draft from this state if it needs further changes

5. Run the Job

Role: Data Steward

  1. From the manifest list, click the Live status for the job you want to start, then Start Job
  2. Fill in the job configuration:
    • Source connection and target connection
    • Internal or external sharing: whether the data is being shared within your organisation or externally
    • Protection type: anonymise (irreversible), pseudonymise (reversible), or reconstruct (re-identify)
    • Consistent masking: whether masking should be consistent across runs, to support end-to-end chain testing
    • Snapshot date (seed): if consistent masking is enabled, the seed date. The same seed always produces the same anonymised output
    • Protect null values: whether null values should be preserved
    • Localisation / country code: jurisdiction-specific settings, where applicable
  3. Choose the tables in scope for this run and click Start
  4. Monitor progress under Recent Jobs, and click View Job for details. Status changes to Complete once the job finishes

Once complete, the anonymised data is fully loaded into the target data source. The platform does not persist business data itself.