Skip to content

integration-guide.md

Latest commit

 

History

History
543 lines (423 loc) · 17.8 KB

integration-guide.md

File metadata and controls

543 lines (423 loc) · 17.8 KB

sentry-options

File-based configuration system for Sentry services. Provides validated, hot-reloadable options without database overhead.

Overview

sentry-options replaces database-stored configuration with git-managed, schema-validated config files. Services read options directly from mounted files with automatic hot-reload when values change.

Key benefits:

  • Fast reads - Options loaded in memory, file reads only on init and updates
  • Schema validation - Type-safe options with defaults
  • Hot-reload - Values update without pod restart (~1-2 min propagation)
  • Audit trail - All changes tracked in git

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         Build Time                              │
│  service repo (e.g., seer)                                      │
│    └── sentry-options/schemas/seer/schema.json  ──→  Docker image│
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│                         CI Pipeline                             │
│  sentry-options-automator repo                                  │
│    └── option-values/seer/default/values.yaml                   │
│              ↓                                                  │
│    sentry-options-cli (validates against schema)                │
│              ↓                                                  │
│    ConfigMap applied to cluster                                 │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│                         Runtime                                 │
│  Kubernetes Pod                                                 │
│    ├── /etc/sentry-options/schemas/seer/schema.json (image)     │
│    └── /etc/sentry-options/values/seer/values.json (ConfigMap)  │
│              ↓                                                  │
│    sentry_options client library                                │
│    - Validates values against schema                            │
│    - Polls for file changes (5s interval)                       │
│    - Returns defaults when values missing                       │
└─────────────────────────────────────────────────────────────────┘

Components

Component Location Purpose
Schemas Service repo (e.g., seer/sentry-options/schemas/) Define options with types and defaults
Values sentry-options-automator/option-values/ Runtime configuration values
CLI sentry-options-cli Fetches schemas, validates YAML, generates ConfigMaps
Client sentry_options (Python) Reads options at runtime with hot-reload

Integrating a New Service

Service teams are responsible for their service repo changes and adding entries to sentry-options-automator. The CI/CD infrastructure is pre-configured by the platform team.

Phase 1: Service Repo Changes

All these changes can be deployed together - the library uses schema defaults when values don't exist.

1. Create Schema

sentry-options/schemas/{namespace}/schema.json:

{
  "version": "1.0",
  "type": "object",
  "properties": {
    "feature.enabled": {
      "type": "boolean",
      "default": false,
      "description": "Enable the feature"
    },
    "feature.rate_limit": {
      "type": "integer",
      "default": 100,
      "description": "Rate limit per second"
    },
    "feature.enabled_slugs": {
      "type": "array",
      "items": {"type": "string"},
      "default": ["getsentry"],
      "description": "Which orgs to enable the feature for"
    }
  }
}

Supported types: string, integer, number, boolean, array, object

Example array option:

"feature.allowed_ids": {
  "type": "array",
  "items": {"type": "integer"},
  "default": [],
  "description": "List of allowed IDs"
}

Example object option:

"feature.config": {
  "type": "object",
  "properties": {
    "host": {"type": "string"},
    "port": {"type": "integer"},
    "label": {"type": "string", "optional": true}
  },
  "default": {"host": "localhost", "port": 8080},
  "description": "Service configuration"
}

Object fields must be primitives (string, integer, number, boolean). Fields are required by default; add "optional": true to allow omission. Nested objects are not supported.

Example array-of-objects option:

"feature.endpoints": {
  "type": "array",
  "items": {
    "type": "object",
    "properties": {
      "url": {"type": "string"},
      "weight": {"type": "integer"}
    }
  },
  "default": [],
  "description": "Weighted endpoints"
}

Namespace naming: The namespace directory must be either {repo} (exact match) or {repo}-* (prefixed). For example, in the seer repo: seer, seer-autofix, seer-grouping are valid; autofix alone is not.

Schema Evolution Rules

The CI enforces these rules when you modify schemas:

Change Allowed
Add new options ✅
Add new namespaces ✅
Remove options ✅ (CI blocks if still in use in automator)
Remove namespaces ❌
Change option types ❌
Change default values ❌

Breaking changes require a migration strategy (contact DevInfra).

2. Update Dockerfile

Schemas are baked into the Docker image so the client can validate values and provide defaults even before any ConfigMap is deployed.

# Copy schemas into image (enables validation and defaults)
COPY sentry-options/schemas /etc/sentry-options/schemas

ENV SENTRY_OPTIONS_DIR=/etc/sentry-options

3. Add Dependency

Python:

# pyproject.toml
dependencies = [
    "sentry_options>=0.0.11",
]

Rust:

# Cargo.toml
[dependencies]
sentry-options = "0.0.14"

4. Initialize and Use

Python:

from sentry_options import init, options

# Initialize early in startup
init()

# Get options namespace
opts = options('seer')

# Read values (returns schema default if ConfigMap doesn't exist)
if opts.get('feature.enabled'):
    rate = opts.get('feature.rate_limit')

Rust:

use sentry_options::{init, options};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    init()?;

    let opts = options("seer");
    if opts.get("feature.enabled")? == serde_json::json!(true) {
        let rate = opts.get("feature.rate_limit")?;
    }
    Ok(())
}

5. Override Options in Tests

Use override_options to temporarily replace option values in tests. Overrides are validated against the schema — unknown keys and type mismatches raise errors. Overrides are thread-local and won't apply to spawned threads.

Python:

Add a conftest.py to initialize options once per test session:

# conftest.py
import pytest
from sentry_options import init

@pytest.fixture(scope='session', autouse=True)
def _init_options() -> None:
    init()
from sentry_options import options
from sentry_options.testing import override_options

def test_feature_enabled():
    with override_options('seer', {'feature.enabled': True}):
        assert options('seer').get('feature.enabled') is True

# Nesting works — inner overrides restore to outer values on exit
def test_nested_overrides():
    with override_options('seer', {'feature.rate_limit': 50}):
        with override_options('seer', {'feature.rate_limit': 100}):
            assert options('seer').get('feature.rate_limit') == 100
        assert options('seer').get('feature.rate_limit') == 50

Rust:

init() is idempotent (safe to call from parallel test threads). override_options() returns a guard that restores values when dropped.

use sentry_options::testing::override_options;
use sentry_options::{init, options};
use serde_json::json;

#[test]
fn test_feature_enabled() {
    init().unwrap();
    let _guard = override_options(&[
        ("seer", "feature.enabled", json!(true)),
    ]).unwrap();

    let opts = options("seer");
    assert_eq!(opts.get("feature.enabled").unwrap(), json!(true));
    // guard dropped here — value restored
}

6. Test Locally

Before deploying, test your integration locally. The library automatically looks for a sentry-options/ directory in your working directory.

Remember: Namespace directories must be prefixed with your repo name (e.g., seer, seer-autofix).

# Create values file for your namespace
mkdir -p sentry-options/values/{namespace}
cat > sentry-options/values/{namespace}/values.json << 'EOF'
{
  "options": {
    "feature.enabled": true,
    "feature.rate_limit": 200
  }
}
EOF

# Run your service or test script
python -c "
from sentry_options import init, options
init()
opts = options('{namespace}')
print('feature.enabled:', opts.get('feature.enabled'))
print('feature.rate_limit:', opts.get('feature.rate_limit'))
"

The directory structure should be:

sentry-options/
├── schemas/{namespace}/schema.json   # Your schema (already created in step 1)
└── values/{namespace}/values.json    # Test values

To test hot-reload, modify values.json while your service is running - changes should be picked up within 5 seconds.

7. Add Schema Validation CI

.github/workflows/validate-sentry-options.yml:

name: Validate sentry-options schema

on:
  pull_request:
    paths:
      - 'sentry-options/schemas/**'

jobs:
  validate:
    uses: getsentry/sentry-options/.github/workflows/validate-schema.yml@0.0.15
    secrets: inherit
    with:
      schemas-path: sentry-options/schemas

Phase 2: sentry-options-automator Changes

Dependency: Schema must exist in service repo first (CI fetches it for validation).

Service teams only need to do 2 things here. CI/CD is pre-configured by platform team.

1. Register Service in repos.json

Add your service to repos.json:

{
  "repos": {
    "seer": {
      "url": "https://github.com/getsentry/seer",
      "path": "sentry-options/",
      "sha": "abc123def456..."
    }
  }
}

Fields:

  • url - GitHub repo URL
  • path - Path to schemas directory within the repo (contains {namespace}/schema.json)
  • sha - Commit SHA (pinned for reproducibility)

Important: When you update your schema, you must also update the SHA in repos.json to point to the commit containing the new schema.

2. Create Values File

option-values/{namespace}/default/values.yaml:

options:
  feature.enabled: true
  feature.rate_limit: 200

Region-specific overrides in option-values/{namespace}/{region}/values.yaml.

That's it! The CI will automatically validate your values against your schema, and the CD pipeline will deploy ConfigMaps to all regions on merge.

Updating Schema Workflow

When you update your schema in the service repo:

  1. Merge schema change to service repo (e.g., getsentry/seer)
  2. Get the merge commit SHA
  3. Update repos.json in sentry-options-automator with new SHA
  4. If values need to change, update them in same PR
Service Repo                    sentry-options-automator
───────────                    ─────────────────────────
PR: Update schema.json    →    PR: Update repos.json SHA
        ↓                              ↓
    Merge                          + Update values if needed
        ↓                              ↓
   (commit abc123)                  Merge
                                       ↓
                               CI validates values against new schema
                                       ↓
                               CD deploys new ConfigMaps

Phase 3: ops Repo Changes

Can happen anytime — pods start normally without the ConfigMap, falling back to schema defaults.

Add pod annotations to your deployment so the sentry-options injector automatically mounts the ConfigMap:

# deployment.yaml
spec:
  template:
    metadata:
      annotations:
        options.sentry.io/inject: 'true'
        options.sentry.io/namespace: {namespace}

The injector automatically adds the necessary volumes and volume mounts based on these annotations. No manual volume configuration is needed.

For multiple namespaces, use a comma-separated list:

options.sentry.io/namespace: seer-code-review,seer

Replace {namespace} with your actual namespace (e.g., seer). The appropriate target's values are deployed to each region's cluster.

ConfigMap Generation

The default/ directory contains base values inherited by all targets. Region directories contain overrides merged with defaults. Each namespace/target produces a ConfigMap named sentry-options-{namespace} with target-specific values:

option-values/
├── seer/
│   ├── default/values.yaml      → Base values (inherited, not deployed directly)
│   ├── us/values.yaml           → ConfigMap: sentry-options-seer-us (default + us merged)
│   └── de/values.yaml           → ConfigMap: sentry-options-seer-de (default + de merged)
└── relay/
    ├── default/values.yaml      → Base values (inherited, not deployed directly)
    └── us/values.yaml           → ConfigMap: sentry-options-relay-us (default + us merged)

What is a target? A target represents a deployment environment or region (e.g., us, de, s4s). Each target gets its own ConfigMap with values merged from default/ plus target-specific overrides. The mapping of targets to Kubernetes clusters is configured in the CD pipeline.

The CD pipeline iterates over each namespace and non-default target to generate and apply ConfigMaps.

CLI Usage

# Fetch schemas from repos.json config
sentry-options-cli fetch-schemas --config repos.json --out schemas/

# Validate values against schema
sentry-options-cli validate-values --schemas schemas/ --root option-values/

# Generate ConfigMap for ONE namespace/target (run per combination)
sentry-options-cli write \
  --schemas schemas/ \
  --root option-values/ \
  --output-format configmap \
  --namespace seer \
  --target us \
  --commit-sha "$COMMIT_SHA" \
  --commit-timestamp "$(date -u +%Y-%m-%dT%H:%M:%SZ)"

# Output: ConfigMap named "sentry-options-seer" (with us target values merged)

The CLI generates one ConfigMap per invocation by merging default/ values with the specified target's overrides. The CD pipeline calls it once per namespace/target combination (excluding default).

sentry-options-automator Structure

sentry-options-automator/
├── repos.json                     # Tracks schema sources (url, path, sha)
├── option-values/                 # Values for new system
│   └── {namespace}/
│       ├── default/
│       │   └── values.yaml
│       └── {region}/
│           └── values.yaml
├── options/                       # Legacy system (unchanged)
│   ├── default/
│   └── regions/
├── .github/workflows/
│   └── sentry-options-validate.yml
└── gocd/pipelines/
    ├── sentry-options.yaml        # Legacy pipeline
    └── new-sentry-options.yaml    # New system pipeline

Note: The reusable workflow for schema validation (validate-schema) is in the sentry-options repo, not sentry-options-automator.

Directory Structure (Runtime)

/etc/sentry-options/
├── schemas/                    # Baked into Docker image
│   └── {namespace}/
│       └── schema.json
└── values/                     # Mounted via ConfigMap
    └── {namespace}/
        └── values.json

Hot-Reload Behavior

  • Polling interval: 5 seconds
  • ConfigMap propagation: ~1-2 minutes (kubelet sync period)
  • Total latency: ConfigMap update → ~1-2 min → file update → ~5 sec → reload

No pod restart required when values change.

Observability

The library emits Sentry transactions on reload with:

  • reload_duration_ms - Time to reload values
  • generated_at - When ConfigMap was generated
  • applied_at - When application loaded values
  • propagation_delay_secs - Time from generation to application

What NOT to Put in sentry-options

Keep these as environment variables or secrets:

  • Database URLs, API keys, credentials
  • Infrastructure config (PORT, worker counts)
  • Sentry DSN

sentry-options is for feature flags and tunable parameters, not secrets.

Comparison with Legacy System

Aspect Legacy (getsentry) New (sentry-options)
Values location options/ in automator option-values/ in automator
Generation generate.py sentry-options-cli
Storage ConfigMap → DB sync ConfigMap → file mount
How consumed Django reads from DB Client reads from file
Automator pod Required Not needed

Development

# Setup
devenv sync

# Run tests
cargo test                    # Rust
pytest tests/                 # Python

# Lint
cargo clippy
pre-commit run --all-files