Skip to content
Closed
Show file tree
Hide file tree
Changes from 52 commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
65f8aee
fix: make index creation idempotent in entity hash lookup migration (…
mfiedorowicz Jan 19, 2026
83b7a57
chore: update CODEOWNERS (#465)
mfiedorowicz Jan 21, 2026
f84c30c
feat(matching): add fuzzy string matching package (#464)
mfiedorowicz Jan 21, 2026
1e9ab2d
feat(protograph): add entity mapping code generator (#466)
mfiedorowicz Jan 21, 2026
488ee9f
feat(db): add graph tables migration for entity storage (#467)
mfiedorowicz Jan 21, 2026
ff60ddc
feat(db): add SQLC queries and generated code for graph tables (#468)
mfiedorowicz Jan 22, 2026
f7f7d0f
feat(entitymatcher): add confidence-based entity matching package (#470)
mfiedorowicz Jan 23, 2026
343b589
feat(reconciler): add GraphBuilder for recursive entity extraction (#…
mfiedorowicz Jan 26, 2026
41bd6e5
feat(reconciler): integrate graph DB with feature flag (#472)
mfiedorowicz Jan 26, 2026
731ce14
feat: list entities rpc (#473)
mfiedorowicz Jan 28, 2026
1f56d16
chore: Implement pytest in Diode (#469)
manrodrigues Jan 28, 2026
32e7242
feat(reconciler): add CreateEntity RPC for synchronous entity creatio…
mfiedorowicz Jan 28, 2026
525476d
Chore: (OBS-1959) python linting (#475)
manrodrigues Jan 30, 2026
0ad24c0
feat: add DeviceConfig (#476)
mfiedorowicz Feb 3, 2026
85a84d0
add optional pprof listeners to servers
paulstuart Feb 4, 2026
eff7051
include supporting file
paulstuart Feb 4, 2026
c750b7f
use service configs to set auth endpoints
paulstuart Feb 4, 2026
f0b5662
restore removed return for early error
paulstuart Feb 4, 2026
caaa7ce
feat: add optional pprof listeners to servers (#478)
paulstuart Feb 4, 2026
1fb551e
feat(graph): add metadata storage and lookup for entity matching (#480)
mfiedorowicz Feb 5, 2026
85ec6f2
remove defaults and allow distinct env names for each pprof env var
paulstuart Feb 5, 2026
a4fb468
consistent naming for pprof var
paulstuart Feb 5, 2026
78216d9
fix: remove defaults for pprof env var (#481)
paulstuart Feb 5, 2026
410af18
fix erroneously updated doc
paulstuart Feb 5, 2026
dce5917
fix: docs had wrong info for pprof env var (#482)
paulstuart Feb 5, 2026
f42de3b
refactor: graph package (#486)
mfiedorowicz Feb 9, 2026
b56fb63
chore: remove CreateEntity and ListEntities RPCs from reconciler prot…
mfiedorowicz Feb 11, 2026
4b2bac5
refactor: graph pkg - Service reshape + ListNodes (#491)
mfiedorowicz Feb 13, 2026
d254bcd
chore(deps): bump go.opentelemetry.io/otel/sdk from 1.36.0 to 1.40.0 …
dependabot[bot] Mar 4, 2026
c8287b9
fix: prevent infinite recursion in updateRefsInData (#495)
mfiedorowicz Mar 5, 2026
73ce4c5
feat: make Redis password optional across all services (#496)
mfiedorowicz Mar 9, 2026
ad3972f
feat: reconciler pipeline optimizations (#497)
mfiedorowicz Mar 13, 2026
f38becc
ci: remove JFrog Artifactory publishing (#498)
marc-barry Mar 18, 2026
683a506
chore(deps): bump google.golang.org/grpc from 1.72.1 to 1.79.3 in /di…
dependabot[bot] Mar 19, 2026
7e77607
fix: PEL reclaim for resilient stream pending message processing (#500)
mfiedorowicz Mar 23, 2026
71a1251
feat: bulk write ingestion logs to reduce DB round-trips (#501)
mfiedorowicz Mar 23, 2026
a3b523e
fix: wire retryablehttp retry logic into NetBox HTTP client (#502)
mfiedorowicz Mar 24, 2026
352f76d
refactor: concurrent Redis stream consumer with heartbeat and PEL dra…
mfiedorowicz Mar 24, 2026
0502c5d
fix: bump Go from 1.25.4 to 1.26.1 (#506)
marc-barry Mar 25, 2026
0aa2d21
fix: upgrade Hydra from v25.4.0 to v26.2.0 (#505)
marc-barry Mar 25, 2026
e150865
feat: add container scanning for all diode services (#504)
marc-barry Mar 25, 2026
7b949fc
chore(deps): bump requests from 2.32.4 to 2.33.0 in /tests in the pip…
dependabot[bot] Mar 26, 2026
4b21095
fix: Preserve request-level metadata (#508)
MicahParks Mar 30, 2026
b4c0a15
revert: remove PEL reclaim and concurrent stream consumer logic (#509)
mfiedorowicz Mar 30, 2026
530af34
fix: exclude metadata fields from entity hash computation (#510)
mfiedorowicz Mar 30, 2026
daa9388
feat: add Source column to vulnerability scan tables (#512)
marc-barry Apr 1, 2026
b5c2d50
feat: snapshot metadata history with deduplication and time-based ret…
mfiedorowicz Apr 1, 2026
3c49471
Add marc-barry to CODEOWNERS (#515)
MicahParks Apr 3, 2026
f50fd06
chore(deps): bump go.opentelemetry.io/otel/sdk from 1.40.0 to 1.43.0 …
dependabot[bot] Apr 9, 2026
e220278
fix: gracefully handle fork PR comment permissions (#514)
marc-barry Apr 14, 2026
cc3e484
chore(deps): bump pytest from 8.4.1 to 9.0.3 in /tests in the pip gro…
dependabot[bot] Apr 15, 2026
18226fa
chore(deps): bump github.com/jackc/pgx/v5 from 5.7.5 to 5.9.0 in /dio…
dependabot[bot] Apr 17, 2026
871ca67
chore(deps): bump github.com/jackc/pgx/v5 from 5.9.0 to 5.9.2 in /dio…
dependabot[bot] May 10, 2026
8d353b8
fix: set MaxIdleConnsPerHost for NetBox HTTP client connection reuse …
mfiedorowicz May 11, 2026
8479b0c
perf: optimize entity hash with proto binary deterministic marshaling…
mfiedorowicz May 11, 2026
3972bbe
perf: optimize DB throughput — bulk COPY, pre-allocated IDs, unnest q…
mfiedorowicz May 11, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -1 +1 @@
* @jajeffries @leoparente @ltucker @mfiedorowicz @MicahParks
* @jajeffries @leoparente @manrodrigues @mfiedorowicz @MicahParks @grant-nbl @paulstuart @marc-barry
4 changes: 4 additions & 0 deletions .github/golangci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@ linters:
- revive
path: /*.go
text: 'package-comments: should have a package comment'
- linters:
- revive
path: /*.go
text: 'var-naming: avoid package names that conflict'
- path: /*.go
text: 'SA1019: filter.DataType is deprecated'
- path: /*.go
Expand Down
90 changes: 90 additions & 0 deletions .github/workflows/container-rescan.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
name: Scheduled Container Rescan

on:
schedule:
- cron: '0 6 * * *' # Daily at 06:00 UTC
workflow_dispatch:

jobs:
rescan:
name: Rescan ${{ matrix.app }}
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
actions: read
contents: read
security-events: write
strategy:
fail-fast: false
matrix:
app: [auth, ingester, reconciler]
env:
TRIVY_DISABLE_VEX_NOTICE: "true"

steps:
- name: Pull image
env:
APP: ${{ matrix.app }}
run: docker pull "docker.io/netboxlabs/diode-${APP}:latest"

- name: Scan for vulnerabilities
uses: aquasecurity/trivy-action@57a97c7e7821a5776cebc9bb87c984fa69cba8f1 # v0.35.0
with:
image-ref: docker.io/netboxlabs/diode-${{ matrix.app }}:latest
format: json
output: trivy-output.json
vuln-type: os,library
scanners: vuln,secret
severity: CRITICAL,HIGH,MEDIUM,LOW
ignore-unfixed: true
exit-code: "0"

- name: Build rescan summary
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0
env:
APP: ${{ matrix.app }}
with:
script: |
const fs = require('fs');
const app = process.env.APP;
const data = JSON.parse(fs.readFileSync('trivy-output.json', 'utf8'));
const vulns = (data.Results || []).flatMap(r =>
(r.Vulnerabilities || []).map(v => ({ ...v, target: r.Target }))
);

const SEVERITY_LABEL = {
CRITICAL: '🔴 **CRITICAL**',
HIGH: '🟠 **HIGH**',
MEDIUM: '🟡 MEDIUM',
LOW: '⚪ LOW'
};

let summary = `### Rescan Complete — diode-${app}\n\n`;
summary += `**Image:** \`docker.io/netboxlabs/diode-${app}:latest\`\n`;
summary += `**Scanned at:** ${new Date().toISOString().replace('T', ' ').split('.')[0]} UTC\n\n`;

if (vulns.length === 0) {
summary += '_No vulnerabilities found._\n';
} else {
summary += `| Source | Library | CVE | Severity | Installed | Fixed | Title |\n`;
summary += `|---|---|---|---|---|---|---|\n`;
for (const v of vulns) {
const title = (v.Title || '').replace(/\|/g, '\\|').substring(0, 80);
const cve = v.PrimaryURL ? `[${v.VulnerabilityID}](${v.PrimaryURL})` : v.VulnerabilityID;
const source = (v.target || '').replace(/\|/g, '\|');
summary += `| ${source} | ${v.PkgName} | ${cve} | ${SEVERITY_LABEL[v.Severity] || v.Severity} | ${v.InstalledVersion} | ${v.FixedVersion || 'N/A'} | ${title} |\n`;
}
}

fs.appendFileSync(process.env.GITHUB_STEP_SUMMARY, summary);

- name: Convert scan results to SARIF
if: "!cancelled()"
run: trivy convert --format sarif --output trivy-results.sarif trivy-output.json

- name: Upload SARIF to GitHub Code Scanning
if: "!cancelled()"
uses: github/codeql-action/upload-sarif@0d579ffd059c29b07949a3cce3983f0780820c98 # v4.32.6
with:
sarif_file: trivy-results.sarif
category: diode-${{ matrix.app }}-rescan
205 changes: 205 additions & 0 deletions .github/workflows/container-scan.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,205 @@
name: Container Scan

on:
workflow_dispatch:
pull_request:
paths:
- "diode-server/**"
- "!diode-server/docker/**"
- "!diode-server/Makefile"
- "!diode-server/README.md"
- ".github/workflows/container-scan.yaml"

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

env:
GO_VERSION: '1.25'

jobs:
build-and-scan:
name: ${{ matrix.app }}
runs-on: blacksmith-2vcpu-ubuntu-2404
timeout-minutes: 15
permissions:
actions: read
contents: read
pull-requests: write
security-events: write
strategy:
fail-fast: false
matrix:
app: [auth, ingester, reconciler]
env:
TRIVY_DISABLE_VEX_NOTICE: "true"

steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@b5ca514318bd6ebac0fb2aedd5d36ec1b5c232a2 # v3.10.0

- name: Set build info
env:
APP: ${{ matrix.app }}
run: |
echo ${GITHUB_SHA::7} > ./diode-server/version/BUILD_COMMIT.txt
LATEST_RELEASE=$(curl --silent "https://api.github.com/repos/${{ github.repository }}/releases/latest" | jq -r '.tag_name')
echo $LATEST_RELEASE > ./diode-server/version/BUILD_VERSION.txt

- name: Build image
uses: docker/build-push-action@1dc73863535b631f98b2378be8619f83b136f4a0 # v6.17.0
with:
context: diode-server
file: diode-server/docker/Dockerfile-build.${{ matrix.app }}
load: true
push: false
platforms: linux/amd64
cache-from: type=gha,scope=${{ matrix.app }}
cache-to: type=gha,scope=${{ matrix.app }},mode=max
build-args: |
SVC=${{ matrix.app }}
tags: diode-${{ matrix.app }}:scan

- name: Scan image for vulnerabilities
uses: aquasecurity/trivy-action@57a97c7e7821a5776cebc9bb87c984fa69cba8f1 # v0.35.0
with:
image-ref: diode-${{ matrix.app }}:scan
format: json
output: trivy-output.json
vuln-type: os,library
scanners: vuln,secret
severity: CRITICAL,HIGH,MEDIUM,LOW
ignore-unfixed: true
exit-code: "0"
list-all-pkgs: "true"

- name: Build scan summary
id: scan-summary
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0
env:
APP: ${{ matrix.app }}
with:
script: |
const fs = require('fs');
const app = process.env.APP;
const data = JSON.parse(fs.readFileSync('trivy-output.json', 'utf8'));
const vulns = (data.Results || []).flatMap(r =>
(r.Vulnerabilities || []).map(v => ({ ...v, target: r.Target }))
);

// auth copies oryd/hydra:v26.2.0 built with Go 1.26.0 which carries unfixed HIGH CVEs
// Tracked: https://github.com/ory/hydra/issues/4080
const blockingSeverities = new Set(app === 'auth' ? ['CRITICAL'] : ['CRITICAL', 'HIGH']);

const SEVERITY_LABEL = {
CRITICAL: '🔴 **CRITICAL**',
HIGH: '🟠 **HIGH**',
MEDIUM: '🟡 MEDIUM',
LOW: '⚪ LOW'
};

const counts = { CRITICAL: 0, HIGH: 0, MEDIUM: 0, LOW: 0 };
for (const v of vulns) {
if (counts[v.Severity] !== undefined) counts[v.Severity]++;
}

const hasBlocking = vulns.some(v => blockingSeverities.has(v.Severity));
core.setOutput('has_blocking', hasBlocking.toString());

const buildTable = (vulns) => {
if (vulns.length === 0) return '_No vulnerabilities found._\n';
let t = '| Source | Library | CVE | Severity | Installed | Fixed | Title |\n';
t += '|---|---|---|---|---|---|---|\n';
for (const v of vulns) {
const title = (v.Title || '').replace(/\|/g, '\\|').substring(0, 80);
const cve = v.PrimaryURL ? `[${v.VulnerabilityID}](${v.PrimaryURL})` : v.VulnerabilityID;
const source = (v.target || '').replace(/\|/g, '\|');
t += `| ${source} | ${v.PkgName} | ${cve} | ${SEVERITY_LABEL[v.Severity] || v.Severity} | ${v.InstalledVersion} | ${v.FixedVersion || 'N/A'} | ${title} |\n`;
}
return t;
};

const table = buildTable(vulns);
fs.appendFileSync(process.env.GITHUB_STEP_SUMMARY,
`## Vulnerability Scan Results — diode-${app}\n\n` + table
);

const artifact = JSON.stringify({ table, hasBlocking, counts });
fs.writeFileSync('scan-result.json', artifact);

- name: Post scan results as PR comment
continue-on-error: true # Fork PRs have read-only tokens and cannot post comments
if: github.event_name == 'pull_request'
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0
env:
APP: ${{ matrix.app }}
COMMIT_SHA: ${{ github.sha }}
with:
script: |
const fs = require('fs');
const app = process.env.APP;
const { table, hasBlocking } = JSON.parse(fs.readFileSync('scan-result.json', 'utf8'));

const heading = hasBlocking
? `## Vulnerability Scan: Failed — diode-${app}`
: `## Vulnerability Scan: Passed — diode-${app}`;

const commitSha = process.env.COMMIT_SHA;
const marker = `<!-- trivy-scan-diode-${app} -->`;
const body = `${marker}\n${heading}\n\n**Image:** \`diode-${app}:scan\`\n\n${table}\n*Commit: ${commitSha}*`;

const { data: comments } = await github.rest.issues.listComments({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
});
const existing = comments.find(c => c.body.includes(marker));
if (existing) {
await github.rest.issues.updateComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: existing.id,
body,
});
} else {
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
body,
});
}

- name: Fail on blocking vulnerabilities
if: steps.scan-summary.outputs.has_blocking == 'true'
env:
APP: ${{ matrix.app }}
run: |
echo "Blocking vulnerabilities found for diode-${APP}. Failing build."
exit 1

- name: Convert scan results to SBOM
if: "!cancelled()"
run: trivy convert --format cyclonedx --output sbom.cdx.json trivy-output.json

- name: Upload SBOM artifact
if: "!cancelled()"
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
with:
name: sbom-diode-${{ matrix.app }}-${{ github.sha }}
path: sbom.cdx.json
retention-days: 90

- name: Convert scan results to SARIF
if: "!cancelled()"
run: trivy convert --format sarif --output trivy-results.sarif trivy-output.json

- name: Upload SARIF to GitHub Code Scanning
if: "!cancelled()"
uses: github/codeql-action/upload-sarif@0d579ffd059c29b07949a3cce3983f0780820c98 # v4.32.6
with:
sarif_file: trivy-results.sarif
category: diode-${{ matrix.app }}
4 changes: 2 additions & 2 deletions .github/workflows/go-test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@ jobs:
- name: Setup Go
uses: actions/setup-go@v5
with:
go-version: '1.25.x'
check-latest: true
go-version-file: diode-server/go.mod
cache-dependency-path: diode-server/go.sum
- name: Run go build
run: go build ./...
- name: Install additional dependencies
Expand Down
5 changes: 2 additions & 3 deletions .github/workflows/golangci-lint.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,10 @@ jobs:
- name: Setup Go
uses: actions/setup-go@v5
with:
go-version: '1.25.x'
check-latest: true
go-version-file: diode-server/go.mod
cache-dependency-path: diode-server/go.sum
- name: Lint
uses: golangci/golangci-lint-action@e7fa5ac41e1cf5b7d48e45e42232ce7ada589601 #v9.1.0
with:
version: v2.6
working-directory: diode-server
args: --config ../.github/golangci.yaml
45 changes: 45 additions & 0 deletions .github/workflows/ruff-lint.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
name: Python - ruff lint

on:
push:
branches:
- "!release"
paths:
- "tests/**/*.py"
- "tests/pyproject.toml"
- ".github/workflows/ruff-lint.yaml"
pull_request:
paths:
- "tests/**/*.py"
- "tests/pyproject.toml"
- ".github/workflows/ruff-lint.yaml"

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

permissions:
contents: read

jobs:
ruff:
runs-on: ubuntu-22.04
timeout-minutes: 10
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: '3.12'
cache: 'pip'
cache-dependency-path: tests/requirements.txt
- name: Install dependencies
run: |
pip install -r tests/requirements.txt
- name: Run ruff check
run: |
make ruff-check
- name: Run ruff format check
run: |
make ruff-format-check
Loading
Loading