Skip to content

feat(graph): add metadata storage and lookup for entity matching#480

Merged
mfiedorowicz merged 1 commit into
developfrom
feat-metadata-lookup
Feb 5, 2026
Merged

feat(graph): add metadata storage and lookup for entity matching#480
mfiedorowicz merged 1 commit into
developfrom
feat-metadata-lookup

Conversation

@mfiedorowicz
Copy link
Copy Markdown
Member

Add support for storing and querying entity metadata in graph_nodes table, enabling correlation-based entity matching with fallback mechanisms.

Key changes:

  • Add metadata JSONB column with GIN index for efficient lookups
  • Add content_hash column as last-resort matching fallback
  • Implement matching priority: diode_id -> source_match keys -> field-based -> content_hash
  • Store source_match.diode_id (equals externalID) for direct lookups
  • Preserve original client metadata in snapshots for audit trail

Database:

  • Migration 20260203000001: add metadata column
  • Migration 20260203000002: add content_hash column
  • New queries: FindNodeByMetadata, FindNodeByContentHash

Code quality:

  • Fix error shadowing bug in upsertNode
  • Remove unnecessary struct copy in maybeUpdateNodeSchema
  • Add ContentHashMatchConfidence constant
  • Increase matching package test coverage to 94%

🤖

Add support for storing and querying entity metadata in graph_nodes table,
enabling correlation-based entity matching with fallback mechanisms.

Key changes:
- Add metadata JSONB column with GIN index for efficient lookups
- Add content_hash column as last-resort matching fallback
- Implement matching priority: diode_id -> source_match keys -> field-based -> content_hash
- Store source_match.diode_id (equals externalID) for direct lookups
- Preserve original client metadata in snapshots for audit trail

Database:
- Migration 20260203000001: add metadata column
- Migration 20260203000002: add content_hash column
- New queries: FindNodeByMetadata, FindNodeByContentHash

Code quality:
- Fix error shadowing bug in upsertNode
- Remove unnecessary struct copy in maybeUpdateNodeSchema
- Add ContentHashMatchConfidence constant
- Increase matching package test coverage to 94%
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Feb 5, 2026

Go test coverage

STATUS ELAPSED PACKAGE COVER PASS FAIL SKIP
🟢 PASS 1.54s github.com/netboxlabs/diode/diode-server/auth 44.7% 42 0 0
🟢 PASS 1.12s github.com/netboxlabs/diode/diode-server/auth/cli 0.0% 0 0 0
🟢 PASS 1.03s github.com/netboxlabs/diode/diode-server/authutil 82.8% 5 0 0
🟢 PASS 0.12s github.com/netboxlabs/diode/diode-server/dbstore/postgres 0.0% 0 0 0
🟢 PASS 1.08s github.com/netboxlabs/diode/diode-server/entityhash 86.7% 16 0 0
🟢 PASS 1.22s github.com/netboxlabs/diode/diode-server/entitymatcher 83.0% 91 0 0
🟢 PASS 0.25s github.com/netboxlabs/diode/diode-server/errors 0.0% 0 0 0
🟢 PASS 1.29s github.com/netboxlabs/diode/diode-server/ingester 82.7% 25 0 0
🟢 PASS 1.13s github.com/netboxlabs/diode/diode-server/matching 94.1% 66 0 0
🟢 PASS 1.07s github.com/netboxlabs/diode/diode-server/migrator 70.4% 4 0 0
🟢 PASS 4.17s github.com/netboxlabs/diode/diode-server/netboxdiodeplugin 83.6% 40 0 0
🟢 PASS 0.17s github.com/netboxlabs/diode/diode-server/pprof 0.0% 0 0 0
🟢 PASS 2.66s github.com/netboxlabs/diode/diode-server/reconciler 58.9% 147 0 0
🟢 PASS 1.02s github.com/netboxlabs/diode/diode-server/reconciler/applier 85.7% 1 0 0
🟢 PASS 0.12s github.com/netboxlabs/diode/diode-server/reconciler/changeset 0.0% 0 0 0
🟢 PASS 1.10s github.com/netboxlabs/diode/diode-server/reconciler/differ 63.8% 6 0 0
🟢 PASS 1.02s github.com/netboxlabs/diode/diode-server/server 85.7% 14 0 0
🟢 PASS 1.01s github.com/netboxlabs/diode/diode-server/strcase 100.0% 24 0 0
🟢 PASS 1.04s github.com/netboxlabs/diode/diode-server/telemetry 28.0% 26 0 0
🟢 PASS 1.04s github.com/netboxlabs/diode/diode-server/telemetry/otel 91.7% 25 0 0
🟢 PASS 0.15s github.com/netboxlabs/diode/diode-server/tls 0.0% 0 0 0
🟢 PASS 1.01s github.com/netboxlabs/diode/diode-server/version 100.0% 2 0 0

Total coverage: 57.9%

Copy link
Copy Markdown
Contributor

@jajeffries jajeffries left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good 👍

@mfiedorowicz mfiedorowicz merged commit 1fb551e into develop Feb 5, 2026
8 checks passed
@mfiedorowicz mfiedorowicz deleted the feat-metadata-lookup branch February 5, 2026 09:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants