eXate Metrics & Billing System

Last Updated: February 2, 2026
Author: Shubham Dixit
Status: ✅ Production Ready

---

Table of Contents

1. Overview
2. Installation-Wide Reporting
3. API Reference
4. Data Models
5. Redis Key Structure
6. Configuration
7. IBM Integration
8. Testing Guide
9. Troubleshooting
10. Implementation Details
11. FAQ

---

1. Overview

1.1 System Purpose

The eXate Metrics & Billing System tracks and reports usage metrics across eXate installations for: - Billing: Generate usage reports for customer billing - Capacity Planning: Monitor usage patterns and trends - Compliance: Audit trail of data processing activities - IBM Cloud Pak Integration: Automated metering submissions to IBM

1.2 Architecture

The system comprises 3 microservices:

┌─────────────────────────────────────────────────────────────────┐
│                    EXATE METRICS SYSTEM                          │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌─────────────┐      ┌─────────────┐      ┌───────────────┐  │
│  │   APIgator  │────▶ │ AGGREGATOR  │────▶ │   POSTGRES    │  │
│  │  (Collect)  │      │  (Insight)  │      │  (Persist)    │  │
│  └─────────────┘      └─────────────┘      └───────────────┘  │
│         │                    │                                  │
│         │                    │                                  │
│    ┌────▼────────────────────▼────┐                           │
│    │         REDIS                 │                           │
│    │  - Raw Counters               │                           │
│    │  - PreComputed Snapshots      │                           │
│    │  - UUID Tracking              │                           │
│    │  - Distributed Locks          │                           │
│    └───────────────────────────────┘                           │
│                                                                  │
│                    ┌──────────────┐                            │
│                    │  IBM CLOUD   │ (Optional)                 │
│                    │   METERING   │                            │
│                    └──────────────┘                            │
└─────────────────────────────────────────────────────────────────┘

Service Responsibilities

Service	Role	Responsibilities
APIgator	Data Collection & Aggregation	• Collect raw metrics from API calls • Aggregate into time windows • Manage reporting workflow • Submit to IBM (if enabled)
Aggregator (Insight)	Persistence & API	• Store metrics in PostgreSQL • Provide query/reporting API • UUID-based idempotency • Track IBM submission status
Core	Business Logic	• Data processing operations • Generate metrics events • Publish to Redis

1.3 Key Design Principles

✅ Installation-Wide Processing

• All operations process ALL tenants in a single call
• No per-tenant API authentication required
• Single HTTP call to Insight for batch persistence
• Improved performance and reduced network overhead

✅ UUID-Based Idempotency

• Every snapshot has a unique UUID
• Permanent tracking in Redis (tenant:{id}:persisted SET)
• Retry-safe: Duplicate sends automatically skip persisted UUIDs
• Never cleared - permanent audit trail

✅ Closed-Window-Only Processing

• Only process windows where isClosed == true
• Prevents partial-month billing
• Guarantees complete data
• Mid-month safety: Open windows excluded automatically

✅ Batch Processing

• Single HTTP call to Insight API for all tenants
• Example: 100 tenants = 1 HTTP call (not 100)
• Bulk insert for database efficiency
• Reduced network latency

✅ Distributed Locking

• Redis-based locks prevent concurrent operations
• 10-minute TTL prevents deadlocks
• Automatic lock release on completion/error
• Returns HTTP 409 on lock conflict

---

2. Installation-Wide Reporting

2.1 Design Overview

The system implements installation-wide reporting with the following characteristics:

Aspect	Implementation
Scope	All tenants per call
Authentication	Not required for metrics endpoints
Preview Endpoint	GET /api/metrics/preview (installation)
Send Endpoint	POST /api/metrics/send (installation)
Insight API	Single batch call for all tenants
HTTP Calls	1 call for N tenants
Idempotency	UUID-based permanent tracking

2.2 Four Core Endpoints

Summary Table

Endpoint	Method	Purpose	Persistence	IBM Submission	Lock Required
/api/metrics/preview	GET	View unbilled usage	❌ No	❌ No	❌ No
/api/metrics/report	POST	Generate JSON (dry-run)	❌ No	❌ No	❌ No
/api/metrics/send	POST	Persist to database	✅ Yes	❌ No (manual)	✅ Yes
/api/metrics/clear	POST	Cleanup reported data	✅ Yes	N/A	✅ Yes

Workflow Diagram

┌──────────────────────────────────────────────────────────────────┐
│                    METRICS REPORTING WORKFLOW                     │
└──────────────────────────────────────────────────────────────────┘

1. PREVIEW (Read-Only)
   └▶ GET /api/metrics/preview
      └▶ Returns unbilled usage (no persistence)

2. GENERATE (Dry-Run)
   └▶ POST /api/metrics/report
      └▶ Returns report JSON (no persistence)

3. SEND (Persist)
   └▶ POST /api/metrics/send
      ├▶ Acquire distributed lock
      ├▶ Filter: closed windows only
      ├▶ Filter: unpersisted UUIDs only (idempotency)
      ├▶ Batch call to Insight API
      ├▶ Mark UUIDs as persisted (permanent)
      └▶ Release lock

4. CLEAR (Cleanup)
   └▶ POST /api/metrics/clear
      ├▶ Acquire distributed lock
      ├▶ Filter: closed + persisted windows only
      ├▶ Delete raw counters from Redis
      ├▶ Delete pre-computed snapshots
      └▶ Release lock

2.3 UUID-Based Idempotency

Redis Key Structure

Key: {prefix}:tenant:{tenantId}:persisted
Type: SET
Members: ["uuid-1", "uuid-2", "uuid-3", ...]
TTL: NEVER (permanent audit trail)

Flow Diagram

┌─────────────────────────────────────────────────────────────────┐
│                 UUID IDEMPOTENCY MECHANISM                       │
└─────────────────────────────────────────────────────────────────┘

FIRST SEND:
1. getPersistedUuids(tenantId) → []
2. Find snapshots → [snapshot-A, snapshot-B]
3. Filter: !persistedUuids.contains(uuid) → [snapshot-A, snapshot-B]
4. Persist to Insight → SUCCESS
5. addPersistedUuid(tenantId, "uuid-A")
6. addPersistedUuid(tenantId, "uuid-B")
7. Return: 2 reports sent

RETRY SEND (same snapshots):
1. getPersistedUuids(tenantId) → ["uuid-A", "uuid-B"]
2. Find snapshots → [snapshot-A, snapshot-B]
3. Filter: !persistedUuids.contains(uuid) → []
4. Return: 0 reports sent (idempotency)

Benefits

• ✅ Retry-Safe: Accidental double-clicks don't create duplicates
• ✅ Network Failure Recovery: Can retry without data loss
• ✅ Audit Trail: Permanent record of all sent reports
• ✅ Cross-Session Safe: Works across pod restarts

2.4 Closed-Window-Only Processing

Window States

┌──────────────────────────────────────────────────────────────────┐
│                    WINDOW LIFECYCLE                               │
└──────────────────────────────────────────────────────────────────┘

OPEN WINDOW (Current Period)
├─ isClosed: false
├─ Behavior: Accumulates metrics
├─ Reporting: EXCLUDED from all operations
└─ Example: February 2026 (during Feb 15, 2026)

CLOSED WINDOW (Past Period)
├─ isClosed: true
├─ Behavior: Read-only
├─ Reporting: INCLUDED in all operations
└─ Example: January 2026 (after Feb 1, 2026)

Safety Guarantees

Operation	Open Window	Closed Window
Preview	❌ Excluded	✅ Included
Report	❌ Excluded	✅ Included
Send	❌ Excluded	✅ Included
Clear	❌ Excluded	✅ Included

Code Implementation

// Get only closed windows
val closedWindows = getClosedWindows
  .filter(windowId => MetricsWindow.fromWindowId(windowId, granularity)
    .exists(_.isClosed))

// Filter snapshots to closed windows only
val snapshots = preAggregationService.getAllSnapshots(tenantId)
  .filter(s => closedWindows.contains(s.windowId))

2.5 Batch Insight API

The system uses a batch API approach for efficient persistence:

Per-Tenant Approach (Alternative Design)

// Per-tenant persistence - N HTTP calls
tenants.foreach { tenantId =>
  val metrics = buildMetrics(tenantId)
  restTemplate.postForObject(
    s"$insightUrl/api/reports/log",
    metrics,
    classOf[MetricsSnapshot]
  )
}

Current Batch Implementation

// Batch persistence - 1 HTTP call
val usageMetricsList: Seq[UsageMetrics] = allTenants.map(buildMetrics)

val response = restTemplate.exchange(
  s"$insightUrl/api/reports/log",
  HttpMethod.POST,
  new HttpEntity(usageMetricsList),
  new ParameterizedTypeReference[Seq[MetricsSnapshot]]() {}
)

Performance Comparison

Tenants	Per-Tenant Calls	Batch Calls	Improvement
10	10	1	10x faster
50	50	1	50x faster
100	100	1	100x faster

2.6 Distributed Locking

Lock Configuration

Key: {prefix}:metrics:lock
Value: "1"
TTL: 600 seconds (10 minutes)
Type: String with SETNX

Protected Operations

• POST /api/metrics/send - Prevents concurrent sends
• POST /api/metrics/clear - Prevents concurrent clears

Error Handling

try {
  val lockKey = s"$keyPrefix:metrics:lock"
  val acquired = redisCache.setex(lockKey, 600, "1") == "OK"

  if (!acquired) {
    throw new IllegalStateException(
      "Metrics operation is already in progress. " +
      "Lock will be released in approximately 10 minutes."
    )
  }

  // Perform operation...

} finally {
  redisCache.remove(lockKey) // Always release
}

HTTP Response on Conflict

HTTP/1.1 409 Conflict
Content-Type: application/json

{
  "error": "Metrics operation is already in progress on another instance.",
  "retryAfter": "10 minutes"
}

---

3. API Reference

3.1 GET /api/metrics/preview

Preview unbilled usage for all tenants (installation-wide).

Request

GET /api/metrics/preview HTTP/1.1
Host: apigator.exate.com

Response (200 OK)

{
  "tenantId": "installation",
  "disclaimer": "Installation-wide preview for 5 snapshots across 2 tenants",
  "previewPeriod": {
    "start": 1735689600000,
    "end": 1738367999999,
    "startLabel": "2025-01-01T00:00:00Z",
    "endLabel": "2025-01-31T23:59:59Z"
  },
  "dataFreshness": {
    "preComputedAt": 1738368000000,
    "preComputedAtLabel": "2025-02-01T00:00:00Z",
    "ageSeconds": 3600,
    "source": "PRE_COMPUTED"
  },
  "usage": {
    "jobTypes": {
      "masking": {
        "count": 15000,
        "percentage": 0
      },
      "pseudonymise": {
        "count": 8000,
        "percentage": 0
      },
      "reconstruct": {
        "count": 2000,
        "percentage": 0
      },
      "restrict": {
        "count": 5000,
        "percentage": 0
      }
    },
    "pathsProcessed": 50000
  },
  "counts": {
    "manifests": 100,
    "attributes": 250,
    "rulePacks": 10,
    "claims": 50,
    "dataUsageIds": 75,
    "dataSubjects": 30
  }
}

Response (200 OK - No Data)

{
  "tenantId": "installation",
  "disclaimer": "No closed windows available for preview",
  "previewPeriod": null,
  "dataFreshness": null,
  "usage": null,
  "counts": null
}

Behavior

• ✅ Read-only (no side effects)
• ✅ No authentication required
• ✅ Includes all closed windows
• ✅ Excludes open (current) windows
• ❌ No distributed lock

---

3.2 POST /api/metrics/report

Generate report JSON without persistence (dry-run).

Request

POST /api/metrics/report HTTP/1.1
Host: apigator.exate.com

Response (200 OK)

{
  "tenantId": "installation",
  "generatedAt": 1738368000000,
  "reportsGenerated": 2,
  "reports": [
    {
      "uuid": "pre-computed-uuid-1",
      "windowId": "202501",
      "period": "January 2025",
      "windowStart": 1735689600000,
      "windowEnd": 1738367999999,
      "status": "GENERATED",
      "ibmStatus": null
    },
    {
      "uuid": "pre-computed-uuid-2",
      "windowId": "202412",
      "period": "December 2024",
      "windowStart": 1733011200000,
      "windowEnd": 1735689599999,
      "status": "GENERATED",
      "ibmStatus": null
    }
  ],
  "summary": {
    "totalMasking": 15000,
    "totalPseudonymise": 8000,
    "totalReconstruct": 2000,
    "totalRestrict": 5000,
    "totalPathsProcessed": 50000,
    "totalManifests": 100,
    "totalAttributes": 250,
    "totalRulePacks": 10,
    "totalClaims": 50,
    "totalDataUsageIds": 75,
    "totalDataSubjects": 30
  }
}

Response (204 No Content)

No closed windows available to report.

Behavior

• ✅ Dry-run (no persistence)
• ✅ Shows what would be sent
• ✅ Uses pre-computed snapshot UUIDs
• ✅ Respects idempotency filter
• ❌ No distributed lock

---

3.3 POST /api/metrics/send

Persist reports to Insight API (installation-wide).

Request

POST /api/metrics/send HTTP/1.1
Host: apigator.exate.com
Authorization: Bearer eyJhbGc...

Response (200 OK)

{
  "tenantId": "installation",
  "generatedAt": 1738368000000,
  "reportsGenerated": 2,
  "reports": [
    {
      "uuid": "db-uuid-a1b2c3d4-e5f6-7890-abcd-ef1234567890",
      "windowId": "202501",
      "period": "January 2025",
      "windowStart": 1735689600000,
      "windowEnd": 1738367999999,
      "status": "PERSISTED",
      "ibmStatus": null
    },
    {
      "uuid": "db-uuid-f7e8d9c0-b1a2-3456-789a-bcdef0123456",
      "windowId": "202412",
      "period": "December 2024",
      "windowStart": 1733011200000,
      "windowEnd": 1735689599999,
      "status": "PERSISTED",
      "ibmStatus": null
    }
  ],
  "summary": {
    "totalMasking": 15000,
    "totalPseudonymise": 8000,
    "totalReconstruct": 2000,
    "totalRestrict": 5000,
    "totalPathsProcessed": 50000,
    "totalManifests": 100,
    "totalAttributes": 250,
    "totalRulePacks": 10,
    "totalClaims": 50,
    "totalDataUsageIds": 75,
    "totalDataSubjects": 30
  }
}

Response (204 No Content)

No pending reports to send (all UUIDs already persisted).

Response (409 Conflict)

{
  "tenantId": "installation",
  "generatedAt": 1738368000000,
  "reportsGenerated": 0,
  "reports": [],
  "summary": null
}

Behavior

• ✅ Persists to PostgreSQL via Insight
• ✅ UUID-based idempotency
• ✅ Distributed locking (10-min TTL)
• ✅ Marks UUIDs as persisted (permanent)
• ✅ Updates last_report timestamp
• ❌ No IBM submission (manual mode)

Idempotency Example

# First send
curl -X POST http://apigator/api/metrics/send
# Response: 2 reports sent

# Retry (within seconds)
curl -X POST http://apigator/api/metrics/send
# Response: 204 No Content (0 reports - idempotency)

---

3.4 POST /api/metrics/clear

Clear reported windows (installation-wide).

Request

POST /api/metrics/clear HTTP/1.1
Host: apigator.exate.com

Response (200 OK)

{
  "windowsCleared": 5,
  "message": "Cleared 5 windows"
}

Response (409 Conflict)

{
  "error": "Clear operation already in progress"
}

Behavior

• ✅ Distributed locking (10-min TTL)
• ✅ Only clears closed windows
• ✅ Only clears persisted UUIDs
• ✅ Deletes raw Redis counters
• ✅ Deletes pre-computed snapshots
• ❌ Does NOT delete persisted UUID set

Safety Checks

// Only clear if:
1. window.isClosed == true (closed window)
2. persistedUuids.contains(snapshot.uuid) (already sent)

// Never clear:
- Open windows
- Unpersisted snapshots
- UUID tracking sets

---

3.5 GET /api/metrics/status

Get system status and configuration.

Request

GET /api/metrics/status HTTP/1.1
Host: apigator.exate.com

Response (200 OK)

{
  "mode": "IBM",
  "granularity": "HOUR",
  "isLeader": true,
  "podId": "apigator-7d8f9c-xyz",
  "ibmMeteringEnabled": true,
  "precomputeEnabled": true,
  "precomputeCron": "0 0 * * * *",
  "reportCron": "0 0 * * * *",
  "persistenceEnabled": true
}

Fields

Field	Description	Values
mode	Operating mode	IBM or NON_IBM
granularity	Window size	MINUTE_15, HOUR, DAY, MONTH
isLeader	Leader election status	true or false
podId	Current pod identifier	String
ibmMeteringEnabled	IBM integration enabled	true or false
precomputeEnabled	Pre-aggregation enabled	true or false
precomputeCron	Pre-compute schedule	Cron expression or manual
reportCron	Report schedule	Cron expression or manual
persistenceEnabled	Persistence enabled	true or false

---

4. Data Models

4.1 PreComputedSnapshot (Redis)

Temporary snapshot created by hourly pre-aggregation.

case class PreComputedSnapshot(
  uuid: String,                    // Unique identifier (pre-generated)
  tenantId: Long,                  // Tenant identifier
  windowId: String,                // Window identifier (e.g., "202501")
  windowStart: Long,               // Window start timestamp (ms)
  windowEnd: Long,                 // Window end timestamp (ms)
  aggregatedAt: Long,              // When pre-computed (ms)
  metrics: Map[String, Map[String, Long]], // Nested metrics
  manifests: Long,                 // Count of manifests
  attributes: Long,                // Count of attributes
  rulePacks: Long,                 // Count of rule packs
  claims: Long,                    // Count of claims
  dataUsageIds: Long,              // Count of data usage IDs
  dataSubjects: Long               // Count of data subjects
)

Example

{
  "uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "tenantId": 12345,
  "windowId": "202501",
  "windowStart": 1735689600000,
  "windowEnd": 1738367999999,
  "aggregatedAt": 1738368000000,
  "metrics": {
    "jobtype": {
      "masking": 5000,
      "pseudonymise": 3000,
      "reconstruct": 1000,
      "restrict": 2000
    },
    "navigationtokenstoprocesscount": {
      "paths": 25000
    }
  },
  "manifests": 50,
  "attributes": 100,
  "rulePacks": 5,
  "claims": 20,
  "dataUsageIds": 30,
  "dataSubjects": 15
}

---

4.2 MetricsSnapshot (PostgreSQL)

Persistent record in database (managed by Insight service).

case class MetricsSnapshot(
  uuid: UUID,                      // Database-generated UUID
  tenantId: Long,                  // Tenant identifier
  windowId: String,                // Window identifier
  windowStart: Long,               // Window start timestamp (ms)
  windowEnd: Long,                 // Window end timestamp (ms)
  createdAt: Long,                 // When persisted (ms)
  status: String,                  // "PERSISTED"
  ibmStatus: Option[String],       // IBM submission status
  ibmSubmittedAt: Option[Long],    // IBM submission timestamp
  ibmLastError: Option[String],    // IBM error message
  metrics: Map[String, Map[String, Long]], // Nested metrics
  manifests: Long,                 // Count of manifests
  attributes: Long,                // Count of attributes
  rulePacks: Long,                 // Count of rule packs
  claims: Long,                    // Count of claims
  dataUsageIds: Long,              // Count of data usage IDs
  dataSubjects: Long               // Count of data subjects
)

IBM Status Values

Status	Description	Next Action
null	Not submitted (non-IBM mode)	None
PENDING	Queued for submission	Automatic submission
SUBMITTED	Successfully sent to IBM	None
FAILED	IBM API error	Manual retry or investigation

---

4.3 Relationship Diagram

┌─────────────────────────────────────────────────────────────────┐
│                    DATA MODEL LIFECYCLE                          │
└─────────────────────────────────────────────────────────────────┘

1. RAW COUNTERS (Redis)
   └─ Keys: {prefix}:{tenantId}:{windowId}:jobtype
      └─ Hash: {"masking": 100, "pseudonymise": 50}

2. PRE-COMPUTED SNAPSHOT (Redis)
   ├─ Created: Hourly cron
   ├─ UUID: Pre-generated
   ├─ Storage: {prefix}:snapshots:{windowId} (HASH)
   └─ TTL: Until cleared

3. PERSISTED SNAPSHOT (PostgreSQL)
   ├─ Created: On /send endpoint
   ├─ UUID: Database-generated (new)
   ├─ Storage: metrics_snapshots table
   └─ Permanent

4. UUID TRACKING (Redis)
   ├─ Key: {prefix}:tenant:{tenantId}:persisted (SET)
   ├─ Values: Pre-computed UUIDs
   └─ TTL: NEVER (permanent audit trail)

---

5. Redis Key Structure

5.1 Complete Key Reference

Key Pattern	Type	Purpose	TTL	Cleared By
{prefix}:active	ZSET	Active window tracking	Never	Window cleanup
{prefix}:precomputed	ZSET	Pre-computed window tracking	Never	Window cleanup
{prefix}:snapshots:{windowId}	HASH	Pre-computed snapshots	Never	/clear
{prefix}:window:{windowId}:tenants	SET	Tenant registry	Never	Window cleanup
{prefix}:window:{windowId}:granularity	STRING	Window granularity	Never	Window cleanup
{prefix}:{tenantId}:{windowId}:jobtype	HASH	Raw job type counters	Never	/clear
{prefix}:{tenantId}:{windowId}:navigationtokenstoprocesscount	HASH	Raw path counters	Never	/clear
{prefix}:tenant:{tenantId}:persisted	SET	Persisted UUID tracking	NEVER	NEVER
{prefix}:tenant:{tenantId}:last_report	STRING	Last report timestamp	Never	Manual
{prefix}:metrics:lock	STRING	Distributed lock	600s	Automatic

5.2 Key Examples

Active Windows

ZADD {prefix}:active 1735689600000 "202501"
ZADD {prefix}:active 1738368000000 "202502"

Pre-Computed Snapshots

HSET {prefix}:snapshots:202501 12345 '{"uuid":"...","tenantId":12345,...}'
HSET {prefix}:snapshots:202501 67890 '{"uuid":"...","tenantId":67890,...}'

Persisted UUIDs (Permanent)

SADD {prefix}:tenant:12345:persisted "uuid-1"
SADD {prefix}:tenant:12345:persisted "uuid-2"
SADD {prefix}:tenant:12345:persisted "uuid-3"
# Never expires, never cleared

Raw Counters

HSET {prefix}:12345:202501:jobtype masking 5000
HSET {prefix}:12345:202501:jobtype pseudonymise 3000

Distributed Lock

SETEX {prefix}:metrics:lock 600 "1"
# Expires after 10 minutes

---

6. Configuration

6.1 User-Facing Settings

METRICS_WINDOW_GRANULARITY

Controls window size for metrics aggregation.

Value	Description	Window Closes	Use Case
MINUTE_15	15-minute windows	After 15 minutes	High-frequency billing (IBM)
HOUR	1-hour windows	After 1 hour	Standard IBM billing
DAY	1-day windows	After midnight UTC	Daily reporting
MONTH	1-month windows	After month end	Monthly billing

Environment Variable:

METRICS_WINDOW_GRANULARITY=HOUR

Example Windows:

MINUTE_15:
  2025-01-01T00:00:00Z → 2025-01-01T00:14:59Z
  2025-01-01T00:15:00Z → 2025-01-01T00:29:59Z

HOUR:
  2025-01-01T00:00:00Z → 2025-01-01T00:59:59Z
  2025-01-01T01:00:00Z → 2025-01-01T01:59:59Z

MONTH:
  2025-01-01T00:00:00Z → 2025-01-31T23:59:59Z
  2025-02-01T00:00:00Z → 2025-02-28T23:59:59Z

IBM_METERING_ENABLED

Enables IBM Cloud Pak integration.

IBM_METERING_ENABLED=true

When Enabled: - ✅ Automated scheduler runs at granularity intervals - ✅ Reports submitted to IBM Metering API - ✅ IBM status tracked in database - ✅ Auto-cleanup after successful send

When Disabled: - ❌ Manual reporting only - ❌ No IBM submission - ❌ No automated scheduler

---

6.2 Mode Validation Matrix

Granularity	IBM Mode	Scheduler	Manual /send	IBM Submission
MINUTE_15	✅ Enabled	Every 15 min	✅ Allowed	✅ Yes
HOUR	✅ Enabled	Every hour	✅ Allowed	✅ Yes
DAY	✅ Enabled	⚠️ Not recommended	✅ Allowed	✅ Yes
MONTH	✅ Enabled	⚠️ Not recommended	✅ Allowed	✅ Yes
MINUTE_15	❌ Disabled	❌ Inactive	✅ Allowed	❌ No
HOUR	❌ Disabled	❌ Inactive	✅ Allowed	❌ No
DAY	❌ Disabled	❌ Inactive	✅ Allowed	❌ No
MONTH	❌ Disabled	❌ Inactive	✅ Allowed	❌ No

---

6.3 IBM-Specific Configuration

IBM Metering API

IBM_METERING_API_URL=https://metering-api.ibm.com/v1/metering
IBM_METERING_API_KEY=your-api-key
IBM_SERVICE_ID=exate-apigator

Scheduler Cron Expressions

# Hourly granularity (every hour at :00)
METRICS_REPORT_CRON=0 0 * * * *

# 15-minute granularity (at :00, :15, :30, :45)
METRICS_REPORT_CRON=0 0,15,30,45 * * * *

Pre-Compute Scheduler

# Run hourly
METRICS_PRECOMPUTE_CRON=0 0 * * * *
METRICS_PRECOMPUTE_ENABLED=true

---

7. IBM Integration

7.1 IBM Mode Overview

IBM mode enables automated submission to IBM Cloud Pak Metering API.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    IBM INTEGRATION FLOW                          │
└─────────────────────────────────────────────────────────────────┘

SCHEDULER (Cron)
  ├─ Runs at granularity intervals (e.g., hourly)
  ├─ Only on leader pod
  └─ Only if IBM_METERING_ENABLED=true
      │
      ├──▶ sendReports(submitToIbm = true)
      │      ├─ Persist to Insight API
      │      ├─ Submit each snapshot to IBM API
      │      └─ Update ibmStatus in database
      │
      └──▶ clearReportedWindows()
             └─ Auto-cleanup after success

---

7.2 Scheduler Details

Configuration

@Scheduled(cron = "#{@metricsReportCron}")
def runScheduledIbmReport(): Unit = {
  if (!IS_IBM_MODE) return
  if (!leaderElection.isLeader) return

  // Send with IBM submission
  val result = sendReports(submitToIbm = true)

  // Auto-cleanup
  if (result.reportsGenerated > 0) {
    clearReportedWindows()
  }
}

Leader Election

• Only the leader pod executes scheduler
• Prevents duplicate IBM submissions
• Leader determined by Redis-based election

Timing Examples

# Hourly granularity
METRICS_REPORT_CRON=0 0 * * * *
Runs at: 00:00, 01:00, 02:00, ..., 23:00

# 15-minute granularity
METRICS_REPORT_CRON=0 0,15,30,45 * * * *
Runs at: 00:00, 00:15, 00:30, 00:45, 01:00, ...

---

7.3 IBM Submission Flow

┌─────────────────────────────────────────────────────────────────┐
│                 IBM SUBMISSION DETAILED FLOW                     │
└─────────────────────────────────────────────────────────────────┘

1. PERSIST TO INSIGHT
   ├─ POST /api/reports/log
   ├─ Body: Seq[UsageMetrics]
   └─ Response: Seq[MetricsSnapshot] with DB UUIDs

2. FOR EACH SNAPSHOT
   ├─ submitUsage(precomputedSnapshot)
   │  ├─ IBM API: POST /v1/metering/usage
   │  ├─ Body: IBM-formatted metrics
   │  └─ Response: Success or Error
   │
   └─ UPDATE IBM STATUS
      ├─ Success: ibmStatus = "SUBMITTED"
      └─ Failure: ibmStatus = "FAILED", ibmLastError = error message

3. MARK AS PERSISTED
   └─ addPersistedUuid(tenantId, precomputedUuid)

IBM API Payload

{
  "serviceId": "exate-apigator",
  "tenantId": "12345",
  "startTime": 1735689600000,
  "endTime": 1738367999999,
  "metrics": [
    {
      "name": "masking_operations",
      "value": 5000,
      "unit": "operations"
    },
    {
      "name": "pseudonymise_operations",
      "value": 3000,
      "unit": "operations"
    }
  ]
}

---

7.4 IBM Status Tracking

Status Lifecycle

┌─────────────────────────────────────────────────────────────────┐
│                    IBM STATUS LIFECYCLE                          │
└─────────────────────────────────────────────────────────────────┘

NON-IBM MODE:
  ibmStatus: null
  ibmSubmittedAt: null
  ibmLastError: null

IBM MODE - SUCCESS:
  ibmStatus: "SUBMITTED"
  ibmSubmittedAt: 1738368000000
  ibmLastError: null

IBM MODE - FAILURE:
  ibmStatus: "FAILED"
  ibmSubmittedAt: null
  ibmLastError: "Connection timeout to IBM API"

Querying IBM Status

-- Find failed IBM submissions
SELECT uuid, tenant_id, window_id, ibm_last_error
FROM metrics_snapshots
WHERE ibm_status = 'FAILED';

-- Find successful submissions
SELECT uuid, tenant_id, window_id, ibm_submitted_at
FROM metrics_snapshots
WHERE ibm_status = 'SUBMITTED'
ORDER BY ibm_submitted_at DESC;

---

8. Testing Guide

8.1 Manual Testing Workflows

Test 1: Installation-Wide Preview

# Get preview of unbilled usage
curl -X GET http://localhost:8080/api/metrics/preview

# Expected: JSON with all closed windows
# Status: 200 OK

Test 2: Generate Report (Dry-Run)

# Generate report without persistence
curl -X POST http://localhost:8080/api/metrics/report

# Expected: JSON with report data
# Status: 200 OK or 204 (no data)

Test 3: Send Reports

# First send
curl -X POST http://localhost:8080/api/metrics/send \
  -H "Authorization: Bearer YOUR_TOKEN"

# Expected: 2+ reports sent
# Status: 200 OK

# Immediate retry (idempotency test)
curl -X POST http://localhost:8080/api/metrics/send \
  -H "Authorization: Bearer YOUR_TOKEN"

# Expected: 0 reports (idempotency)
# Status: 204 No Content

Test 4: Clear Reported Windows

# Clear after sending
curl -X POST http://localhost:8080/api/metrics/clear

# Expected: {"windowsCleared": N}
# Status: 200 OK

Test 5: Concurrent Send (Locking)

# Terminal 1
curl -X POST http://localhost:8080/api/metrics/send &

# Terminal 2 (immediately after)
curl -X POST http://localhost:8080/api/metrics/send

# Expected: Second call returns 409 Conflict

---

8.2 UUID Idempotency Verification

Redis Verification

# Check persisted UUIDs for tenant 12345
redis-cli SMEMBERS "{prefix}:tenant:12345:persisted"

# Expected output:
1) "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
2) "f7e8d9c0-b1a2-3456-789a-bcdef0123456"

Database Verification

-- Count reports per window
SELECT window_id, COUNT(*) as report_count
FROM metrics_snapshots
GROUP BY window_id, tenant_id
HAVING COUNT(*) > 1;

-- Should be 1 per window per tenant
-- Multiple reports = idempotency failure

Test Scenario

# 1. Send reports
curl -X POST /api/metrics/send
# Response: 2 reports sent

# 2. Check Redis
redis-cli SMEMBERS "{prefix}:tenant:12345:persisted"
# Should contain 2 UUIDs

# 3. Retry send
curl -X POST /api/metrics/send
# Response: 204 No Content (0 reports)

# 4. Check database
# Should still have 2 reports (no duplicates)

---

8.3 Closed-Window Logic Testing

Test Scenario: Mid-Month Generation

# Assume today is February 15, 2026
# Granularity: MONTH

# 1. Preview
curl -X GET /api/metrics/preview

# Expected windows:
# - January 2026: INCLUDED (closed)
# - February 2026: EXCLUDED (open)

# 2. Generate report
curl -X POST /api/metrics/report

# Expected reports:
# - January 2026: YES
# - February 2026: NO

Redis Verification

# Check window closure status
redis-cli ZRANGEBYSCORE "{prefix}:active" 0 +inf

# For each window, check if closed:
# January 2026: windowEnd < now() → closed
# February 2026: windowEnd >= now() → open

---

8.4 Distributed Locking Testing

Test 1: Lock Acquisition

# Acquire lock manually
redis-cli SETEX "{prefix}:metrics:lock" 600 "1"

# Try to send (should fail)
curl -X POST /api/metrics/send

# Expected: HTTP 409 Conflict

Test 2: Lock Release

# Start send operation
curl -X POST /api/metrics/send &

# Check lock exists
redis-cli GET "{prefix}:metrics:lock"
# Output: "1"

# Wait for completion
# Lock should be auto-released

Test 3: Lock Timeout

# Simulate stalled process
redis-cli SETEX "{prefix}:metrics:lock" 600 "1"

# Wait 10 minutes
sleep 600

# Try to send (should succeed)
curl -X POST /api/metrics/send

# Expected: HTTP 200 OK

---

9. Troubleshooting

9.1 Common Issues

Issue: No Closed Windows

Symptoms:

{
  "tenantId": "installation",
  "disclaimer": "No closed windows available for preview"
}

Causes: 1. Current window not closed yet 2. No data collected in previous windows 3. All windows already cleared

Solutions:

# Check active windows
redis-cli ZRANGEBYSCORE "{prefix}:active" 0 +inf

# Check if windows are closed
# For MONTH granularity: Wait until next month
# For HOUR granularity: Wait until next hour

---

Issue: 409 Conflict (Lock)

Symptoms:

HTTP/1.1 409 Conflict
{"error": "Metrics operation is already in progress"}

Causes: 1. Concurrent send/clear operation 2. Previous operation didn't release lock 3. Lock TTL not expired (10 minutes)

Solutions:

# Check lock status
redis-cli GET "{prefix}:metrics:lock"

# If stale (>10 min), delete manually
redis-cli DEL "{prefix}:metrics:lock"

# Retry operation
curl -X POST /api/metrics/send

---

Issue: Empty Preview

Symptoms: Preview returns valid response but all counts are 0.

Causes: 1. Pre-computed snapshots not generated yet 2. Snapshots already cleared 3. No metrics collected

Solutions:

# Check if snapshots exist
redis-cli HGETALL "{prefix}:snapshots:202501"

# Check pre-compute scheduler
curl -X GET /api/metrics/status

# Trigger pre-compute manually (if supported)
# Or wait for hourly cron

---

Issue: Duplicate Reports in Database

Symptoms:

SELECT window_id, tenant_id, COUNT(*)
FROM metrics_snapshots
GROUP BY window_id, tenant_id
HAVING COUNT(*) > 1;

Causes: 1. UUID tracking not working 2. Redis key corruption 3. Concurrent sends bypassing lock

Solutions:

# Verify UUID tracking
redis-cli SMEMBERS "{prefix}:tenant:12345:persisted"

# Check for lock bypassing (should never happen)
# Review application logs for errors

# Prevent future duplicates:
# - Ensure Redis connectivity
# - Verify lock mechanism
# - Check Insight API idempotency

---

Issue: IBM Submission Failures

Symptoms:

{
  "ibmStatus": "FAILED",
  "ibmLastError": "Connection timeout to IBM API"
}

Causes: 1. IBM API unreachable 2. Invalid credentials 3. Malformed payload

Solutions:

# Check IBM configuration
echo $IBM_METERING_API_URL
echo $IBM_METERING_API_KEY

# Test IBM connectivity
curl -X GET $IBM_METERING_API_URL/health \
  -H "Authorization: Bearer $IBM_METERING_API_KEY"

# Review failed submissions
SELECT uuid, ibm_last_error
FROM metrics_snapshots
WHERE ibm_status = 'FAILED';

---

9.2 Debug Commands

Redis Inspection

# List all metrics keys
redis-cli KEYS "{prefix}:*"

# Check active windows
redis-cli ZRANGEBYSCORE "{prefix}:active" 0 +inf WITHSCORES

# Check persisted UUIDs
redis-cli SMEMBERS "{prefix}:tenant:12345:persisted"

# Check snapshots
redis-cli HGETALL "{prefix}:snapshots:202501"

# Check lock status
redis-cli GET "{prefix}:metrics:lock"
redis-cli TTL "{prefix}:metrics:lock"

Database Queries

-- Recent reports
SELECT uuid, tenant_id, window_id, created_at, status, ibm_status
FROM metrics_snapshots
ORDER BY created_at DESC
LIMIT 10;

-- Reports by window
SELECT window_id, COUNT(*) as count, SUM(total_masking) as total_masking
FROM metrics_snapshots
GROUP BY window_id
ORDER BY window_id DESC;

-- IBM failures
SELECT uuid, tenant_id, window_id, ibm_last_error
FROM metrics_snapshots
WHERE ibm_status = 'FAILED';

Application Logs

# Search for metrics errors
kubectl logs apigator-pod | grep -i "metrics" | grep -i "error"

# Search for lock conflicts
kubectl logs apigator-pod | grep "409"

# Search for IBM failures
kubectl logs apigator-pod | grep "IBM submission failed"

---

10. Implementation Details

10.1 Service Responsibilities

APIgator

• File: MetricsPersistenceService.scala
• Responsibilities:
• Installation-wide reporting logic
• UUID idempotency management
• Distributed locking
• Batch Insight API calls
• IBM submission orchestration
• Window cleanup

Aggregator (Insight)

• File: ReportController.scala, ReportService.scala
• Responsibilities:
• Batch persistence endpoint
• UUID duplicate detection
• Database storage
• IBM status tracking

Core

• File: Various services
• Responsibilities:
• Metrics event publishing
• Raw counter increments
• Redis operations

---

10.2 Code Locations

Key Files

File	Path	Purpose
MetricsController	apigator/controllers/MetricsController.scala	REST endpoints
MetricsPersistenceService	apigator/services/metrics/MetricsPersistenceService.scala	Core logic
MetricsPreAggregationService	apigator/services/metrics/MetricsPreComputeService.scala	Hourly snapshots
IBMMeteringClient	apigator/clients/IBMMeteringClient.scala	IBM API
ReportController	aggregator/controllers/ReportController.scala	Insight API
ReportService	aggregator/services/ReportService.scala	Persistence logic

Key Methods

// MetricsPersistenceService.scala
def getPreview: PreviewResponse
def generateReport(): ReportGenerationResponse
def sendReports(submitToIbm: Boolean): ReportGenerationResponse
def clearReportedWindows(): Int
def runScheduledIbmReport(): Unit

// MetricsController.scala
def preview(): ResponseEntity[PreviewResponse]
def report(): ResponseEntity[ReportGenerationResponse]
def send(): ResponseEntity[ReportGenerationResponse]
def clear(): ResponseEntity[String]

// ReportController.scala (Insight)
def logMetricsBatch(metricsList: Seq[UsageMetrics]): Seq[MetricsSnapshot]

---

11. FAQ

Q1: Can I generate reports mid-month?

A: Yes, but only for closed windows.

Example (MONTH granularity): - Today: February 15, 2026 - January window: Closed → Included in report - February window: Open → Excluded from report

To include February, wait until March 1.

---

Q2: What happens if I retry a send operation?

A: UUID-based idempotency prevents duplicates.

Flow: 1. First send: Persists 5 reports, marks UUIDs as persisted 2. Retry send: Filters out persisted UUIDs, finds 0 new reports 3. Returns: 204 No Content (0 reports sent)

---

Q3: Should I call /clear before or after /send?

A: After /send. Always send first, then cleanup.

Recommended workflow:

# 1. Preview (optional)
curl -X GET /api/metrics/preview

# 2. Send
curl -X POST /api/metrics/send

# 3. Clear (after confirmation)
curl -X POST /api/metrics/clear

---

Q4: How do I check IBM submission status?

A: Query the database for ibm_status field.

-- Recent IBM submissions
SELECT uuid, tenant_id, window_id, ibm_status, ibm_submitted_at
FROM metrics_snapshots
WHERE ibm_status IS NOT NULL
ORDER BY created_at DESC;

-- Failed submissions
SELECT uuid, tenant_id, window_id, ibm_last_error
FROM metrics_snapshots
WHERE ibm_status = 'FAILED';

---

Q5: Can I manually trigger IBM submission?

A: Not directly. IBM submission is tied to: 1. Automated scheduler (if IBM_METERING_ENABLED=true) 2. Manual /send in IBM mode sends to Insight only

To manually submit to IBM: - Ensure IBM_METERING_ENABLED=true - Wait for scheduler to run - Or restart pod to trigger immediate run

---

Q6: What if lock doesn't release?

A: Lock has 10-minute TTL and auto-releases.

If stuck:

# Check TTL
redis-cli TTL "{prefix}:metrics:lock"

# If > 600 seconds, delete manually
redis-cli DEL "{prefix}:metrics:lock"

---

Q7: How long are UUIDs kept?

A: Forever. Persisted UUIDs are permanent audit trail.

# These are NEVER cleared
redis-cli SMEMBERS "{prefix}:tenant:12345:persisted"

To clean up (not recommended):

redis-cli DEL "{prefix}:tenant:12345:persisted"

---

Q8: Can I preview open windows?

A: No. Preview only shows closed windows.

Rationale: - Open windows have incomplete data - Prevents misleading billing estimates - Guarantees accurate reports

---

Q9: What's the difference between /report and /send?

Feature	/report	/send
Persistence	❌ No	✅ Yes
UUID Tracking	❌ No	✅ Yes
Distributed Lock	❌ No	✅ Yes
Use Case	Dry-run / preview	Production reporting

---

Q10: How do I troubleshoot missing reports?

Checklist: 1. ✅ Are there closed windows? redis-cli ZRANGEBYSCORE "{prefix}:active" 0 +inf 2. ✅ Are snapshots pre-computed? redis-cli HGETALL "{prefix}:snapshots:202501" 3. ✅ Are UUIDs already persisted? redis-cli SMEMBERS "{prefix}:tenant:12345:persisted" 4. ✅ Is lock blocking? redis-cli GET "{prefix}:metrics:lock" 5. ✅ Are there application errors? kubectl logs apigator-pod | grep -i error

---

Appendix A: Quick Reference

API Endpoints

GET  /api/metrics/preview  → View unbilled usage
POST /api/metrics/report   → Generate JSON (dry-run)
POST /api/metrics/send     → Persist reports
POST /api/metrics/clear    → Cleanup reported data
GET  /api/metrics/status   → System status

Redis Keys

{prefix}:active                           → Active windows (ZSET)
{prefix}:snapshots:{windowId}             → Pre-computed snapshots (HASH)
{prefix}:tenant:{tenantId}:persisted      → UUID tracking (SET, PERMANENT)
{prefix}:metrics:lock                     → Distributed lock (STRING, 10-min TTL)

Configuration

METRICS_WINDOW_GRANULARITY=HOUR           # MINUTE_15, HOUR, DAY, MONTH
IBM_METERING_ENABLED=true                 # true or false
METRICS_REPORT_CRON=0 0 * * * *           # Hourly

HTTP Status Codes

200 OK          → Success
204 No Content  → No data available
409 Conflict    → Lock held by another process
500 Error       → Server error

---

Document Version: 2.0
Last Reviewed: February 2, 2026
Next Review: March 2, 2026