Skip to content

Schema Classification

Welcome to the eXate Schema Classification Guide. This document explains how GatorAId automates data classification, how semantic embeddings power automated classification, and how to configure classification rules (including include/exclude terms and similarity thresholds) for maximum precision.

Core Classification Flows

The GatorAId classification system offers two primary flows tailored to different data sources:

1. JSON Schema (APIgator Flow)

  • Purpose: Scans a sample JSON message payload or an OpenAPI 3.0 Specification (OAS3) to locate and classify data.
  • Under the Hood: The system automatically flattens nested structures (for example, { "user": { "billing": { "post_code": "EC1A 1BB" } } } becomes user.billing.post_code) and performs classifications on these flattened path keys.

2. Database Schema (Datagator Flow)

  • Purpose: Scans database schemas to automatically inspect and classify tables and column names.
  • Under the Hood: Rather than requiring a sample payload, the database scanner retrieves the raw columns and field lists, treats them as flat keys, and runs them through the classification engine.

How Embeddings Power Classification

Matching exact strings (e.g., looking only for the exact word "postcode") is fragile. In real-world data, the same field might be called "postal_code", "zip_code", "zip", "pcode", or "p_code".

To solve this, eXate uses high-dimensional vector embeddings to capture the semantic meaning of fields. Natural language tokens are converted into dense mathematical vectors. The closer two vectors are in high-dimensional space, the more semantically similar the concepts are.

Word-Level Vector Space

  • Model Used: FastText (300-dimensional word/token vectors cached locally and stored in Redis).
  • When it is used: For general rule-based model matching across standard and non-sensitive attributes.
  • Process:
  • Path elements are tokenized.
  • FastText retrieves highly specific vectors for these words/segments (with intelligent fallback to sub-word n-grams for out-of-vocabulary terms like abbreviation combinations).
  • The system computes cosine distance between these tokens and the predefined attribute rules.
  • Matching similarity is calculated as: $$\text{Similarity} = 1 - \text{Cosine Distance}$$

How Models Work: Rules & Similarity Thresholds

Each eXate Model is a collection of classification definitions. Every Attribute (e.g., EMAIL_ADDRESS, POSTAL_CODE, NATIONAL_ID) in a model contains list-based rules that direct matching behaviour.

These rules rely on three key pillars: Include Terms, Exclude Terms, and the Similarity Threshold.

classDiagram
    class AttributeRule {
        +string attributeCode ("EMAIL_ADDRESS")
        +List includeTerms
        +List excludeTerms
        +float similarityThreshold (0.85)
    }

1. Include Terms (includeTerms)

  • Purpose: Positively identify matching paths.
  • Action: If the calculated similarity between a path/column token and an includeTerm is greater than or equal to the configured similarity threshold, it is marked as a potential match.
  • Example: For attribute POSTAL_CODE, an include term might be "postcode" with a similarity threshold of 0.85. The input field "postal_code" will be highly similar (e.g., 0.92) and triggers a positive match.

2. Exclude Terms (excludeTerms)

  • Purpose: Act as high-precision safety overrides to eliminate false positives.
  • Action: If a path/column token matches an excludeTerm with a similarity greater than or equal to its threshold, that attribute mapping is immediately discarded for that field.
  • Example: An attribute designed to detect personal email addresses (PERSONAL_EMAIL) might have an include term "email". However, if the field is actually "business_email", it should not be categorised as a personal email. Adding "business" to excludeTerms with a high similarity threshold ensures that "business_email" is safely filtered out and classified correctly.

3. Similarity Threshold (sim)

  • Purpose: Quantify the strictness of matching.
  • Action: A decimal value between 0.0 and 1.0 (typically configured around 0.8 to 0.9).
  • Include Trigger Condition: $\text{Cosine Similarity} \ge \text{Similarity Threshold}$
  • Exclude Trigger Condition: If an exclude term matches with $\text{Cosine Similarity} \ge \text{Similarity Threshold}$, the match is discarded.
  • Tuning Tip:
  • Higher values (e.g., 0.95): Require near-exact matches, reducing false positives but potentially missing highly abbreviated or slightly different names (high precision, lower recall).
  • Lower values (e.g., 0.70): Are more lenient and catch semantic synonyms, but are more likely to incorrectly group unrelated fields (high recall, lower precision).

Performance Optimisation

Standard word vectors are indexed in Redis and cached locally in an L1 memory cache. If a field name is unrecognized, the engine falls back to an n-gram lookup (splitting words into character segments of size 3-6) to construct a robust fallback vector. This guarantees that typos or custom acronyms still get accurately classified!