Implement Centralized Configuration Manager (YAML/TOML)

[sebastienrousseau]

Context

Pain001 currently relies solely on CLI arguments for configuration. As the project scales toward API and batch processing use cases, users need a way to define default settings without repeating CLI flags for every invocation.

Problem

Current limitations:

• Every CLI invocation requires full parameter set (template path, XSD path, message type, etc.)
• No way to define environment-specific defaults (dev/staging/prod)
• Hard to maintain consistent settings across multiple runs
• Difficult to version-control payment workflow configurations

Example Current Workflow (Repetitive)

python -m pain001 \
    -t pain.001.001.03 \
    -m pain001/templates/pain.001.001.03/template.xml \
    -s pain001/templates/pain.001.001.03/pain.001.001.03.xsd \
    -d payments.csv

# Every single invocation repeats these paths

Proposed Solution

Implement a hierarchical configuration system supporting YAML/TOML config files with CLI overrides.

Configuration Precedence (Lowest to Highest)

1. Built-in defaults (hardcoded in code)
2. Project config file (pain001.yaml in current directory)
3. User config file (~/.config/pain001/config.yaml)
4. Environment variables (PAIN001_MESSAGE_TYPE, etc.)
5. CLI arguments (highest precedence)

Example: pain001.yaml

# pain001.yaml - Project configuration
message_type: pain.001.001.03
defaults:
  template_dir: pain001/templates
  xsd_dir: pain001/templates
  output_dir: ./output

profiles:
  development:
    validation_strict: false
    log_level: DEBUG
  
  production:
    validation_strict: true
    log_level: INFO
    pii_masking: true

# Now CLI becomes simpler:
# python -m pain001 -d payments.csv --profile production

Acceptance Criteria

• New module: pain001/config/manager.py for configuration loading
• Support YAML and TOML formats (YAML preferred)
• Implement precedence chain (defaults → project → user → env → CLI)
• Add --config flag to specify custom config file path
• Add --profile flag to select profile from config
• Validate config schema on load (fail fast on invalid config)
• 100% backward compatibility (CLI-only workflows still work)
• Documentation: docs/configuration.rst with examples
• Add config schema file: pain001/config/schema.yaml

Implementation Plan

Phase 1: Config Loader (v0.0.47)

# pain001/config/manager.py
from pathlib import Path
from typing import Dict, Any, Optional
import yaml

class ConfigManager:
    def __init__(self):
        self.config = self._load_default_config()
    
    def load_from_file(self, path: Path) -> Dict[str, Any]:
        """Load and validate config from YAML/TOML."""
        pass
    
    def merge_with_env(self) -> None:
        """Override config with environment variables."""
        pass
    
    def get_profile(self, profile_name: str) -> Dict[str, Any]:
        """Get named profile settings."""
        pass

Phase 2: CLI Integration (v0.0.47)

• Update pain001/__main__.py to load config before parsing CLI args
• CLI args override config values (highest precedence)
• Add --show-config flag for debugging configuration

Phase 3: Schema Validation (v0.0.47)

# pain001/config/schema.yaml
message_type:
  type: string
  pattern: "^pain\\.001\\.001\\.(0[3-9]|1[0-1])$"
  required: true

defaults:
  template_dir:
    type: string
    required: false
  
profiles:
  type: object
  required: false

Impact

• Developer Experience: 90% reduction in CLI verbosity for repeated tasks
• Maintainability: Version-controlled configs for team workflows
• Flexibility: Environment-specific profiles (dev/staging/prod)
• Onboarding: New users can start with template configs

Dependencies

• Related: Make CLI interface more intuitive #105 (Make CLI more intuitive)
• Enables: Preset profiles for banks/regions #107 (Preset profiles for banks/regions)
• Requires: PyYAML or tomli package (user sign-off needed)

Package Dependency Note

NEW DEPENDENCY REQUIRED:

• Option 1: PyYAML (12.3M downloads/month, 1 CVE patched in latest)
• Option 2: tomli (Python 3.11+ has built-in TOML, tomli for 3.9-3.10)

Security Impact: Low risk (both packages widely used, actively maintained)

Justification: Existing tools (argparse) cannot solve config file hierarchies. YAML/TOML parsers are standard for config management.

👤 USER SIGN-OFF REQUIRED before adding dependency.

Effort Estimate

• Size: Medium (4-5 days)
• Coverage Impact: +0.8% (config manager tests)
• Breaking Changes: None (purely additive feature)

---

Priority Rationale: Medium priority (p2) - Quality-of-life improvement that doesn't block core functionality but significantly enhances UX for power users.