Performance Optimization

[behrangsa]

Summary

Optimize Samoid for fast startup and execution times to minimize impact on Git operations.

Performance Testing Strategy: To ensure consistent and reliable performance measurement, benchmarks should be moved from the main test pipeline (test.yml) to a dedicated performance pipeline (perf.yml). This separation provides:

• Parallel execution with functional tests for faster feedback
• Specialized environment with consistent hardware for reliable measurements
• Non-blocking results so performance issues don't prevent functional test feedback
• Better performance tracking with dedicated artifacts and historical comparison
• Flexible scheduling allowing nightly/release-branch-only performance validation

Acceptance Criteria

• AC8.1: Hook execution overhead less than 50ms
• AC8.2: Binary size under 10MB
• AC8.3: Memory usage under 50MB during execution
• AC8.4: Startup time under 100ms
• AC8.5: Efficient file system operations
• AC8.6: Minimize external dependencies
• AC8.7: Dedicated performance testing pipeline implemented
• AC8.8: Benchmark results tracking and comparison system

Details

AC8.1: Hook execution overhead less than 50ms

Detailed Specification

Definition: Hook execution overhead is the additional time Samoid adds to Git operations beyond the actual hook script execution time.

Measurement Method:

• Use cargo bench with the existing benchmark suite in samoid/benches/benchmark.rs
• Measure total time from samoid-hook binary start to hook script completion
• Subtract actual hook script execution time to isolate Samoid overhead
• Test with empty hook scripts to measure pure overhead
• Use system tools like hyperfine for external validation

Test Scenarios:

• Empty hook script (measures pure Samoid overhead)
• Simple echo command hook script
• Hook script that reads/writes small files
• Hook script with environment variable access

Verification:

• Run benchmarks on multiple platforms (Linux x64, macOS x64, Windows x64)
• Measure 100 consecutive executions and use median value
• Test both debug and release builds (requirement applies to release builds)
• Compare against baseline Husky performance

Success Criteria - Environment Specific:

Local Development Environment (Ryzen 4500U, 16GB RAM, Ubuntu 24.04):

• Median overhead ≤ 30ms across all test scenarios
• 95th percentile ≤ 45ms
• Zero executions exceed 60ms overhead

GitHub Actions Standard Runners (4 vCPU, 16GB RAM):

• Median overhead ≤ 50ms across all test scenarios
• 95th percentile ≤ 75ms
• Zero executions exceed 100ms overhead

GitHub Actions Larger Runners (if used for testing):

• Median overhead ≤ 25ms across all test scenarios
• 95th percentile ≤ 40ms
• Zero executions exceed 50ms overhead

AC8.2: Binary size under 10MB

Detailed Specification

Definition: Combined size of both samoid and samoid-hook release binaries after compilation and stripping.

Measurement Method:

• Build release binaries: cargo build --release
• Strip debug symbols: strip target/release/samoid target/release/samoid-hook
• Measure file sizes using ls -la or stat
• Test on multiple platforms to account for platform-specific differences

Individual Binary Limits:

• samoid (main CLI): ≤ 6MB
• samoid-hook (hook runner): ≤ 4MB
• Combined total: ≤ 10MB

Optimization Techniques to Verify:

• Link-time optimization (LTO) enabled
• Dead code elimination functional
• Minimal feature flags used for dependencies
• No unused dependencies included

Platform Requirements:

• Linux x86_64: ≤ 10MB combined
• macOS x86_64: ≤ 10MB combined
• Windows x86_64: ≤ 10MB combined
• Allow ±5% variance between platforms

Success Criteria: All release binaries meet size requirements across all supported platforms.

AC8.3: Memory usage under 50MB during execution

Detailed Specification

Definition: Peak RSS (Resident Set Size) memory consumption during typical hook execution scenarios.

Measurement Method:

• Use time -v on Linux or \usr\bin\time -l on macOS
• Monitor with valgrind --tool=massif for detailed analysis
• Measure both samoid init and samoid-hook execution

Test Scenarios:

• samoid init in empty Git repository
• samoid init in large repository (>10,000 files)
• samoid-hook executing simple hook script
• samoid-hook executing complex hook script with file operations
• Concurrent execution of multiple hook instances

Memory Components to Monitor:

• Heap allocations
• Stack usage
• Memory-mapped files
• Loaded shared libraries

Success Criteria:

• samoid init: ≤ 30MB peak RSS
• samoid-hook: ≤ 20MB peak RSS
• No memory leaks detected by Valgrind
• Memory usage scales linearly (not exponentially) with repository size

Verification Process:

• Run memory tests in CI/CD pipeline
• Test on systems with limited memory (1GB RAM)
• Use automated memory profiling tools

AC8.4: Startup time under 100ms

Detailed Specification

Definition: Time from binary invocation to ready-to-execute state (CLI parsed, dependencies loaded).

Measurement Method:

• Use hyperfine for precise timing measurements
• Measure time from process start to first meaningful output
• Test both cold start (first execution) and warm start (subsequent executions)

Components to Measure:

• Binary loading time
• Dynamic library loading
• CLI argument parsing
• Configuration file reading
• Environment variable processing

Test Commands:

• samoid --help (minimal operation)
• samoid init (full initialization)
• samoid-hook pre-commit (hook execution startup)

Environmental Factors:

• Cold start (no filesystem cache): ≤ 100ms
• Warm start (cached): ≤ 50ms
• Network filesystem (NFS): ≤ 150ms (allowance for network latency)
• Spinning disk storage: ≤ 120ms

Success Criteria:

• Median startup time ≤ 100ms across 100 executions
• 95th percentile ≤ 150ms
• No outliers exceeding 200ms

Verification:

• Test on various storage types (SSD, HDD, network)
• Test with different system loads
• Automated performance regression tests

AC8.5: Efficient file system operations

Detailed Specification

Definition: Minimize filesystem I/O operations and optimize access patterns for performance.

Optimization Requirements:

• Batch file operations where possible
• Use efficient file access patterns (sequential over random)
• Minimize redundant filesystem queries
• Implement appropriate filesystem caching

Metrics to Measure:

• Number of stat(), open(), read(), write() system calls
• Total bytes read/written during installation
• Filesystem cache hit ratio
• Directory traversal efficiency

Measurement Tools:

• strace on Linux to count system calls
• fs_usage on macOS for filesystem monitoring
• Custom instrumentation in debug builds

Performance Targets:

• samoid init: ≤ 50 filesystem operations total
• Hook file creation: ≤ 15 operations per hook
• Repository validation: ≤ 5 operations (.git detection)
• No redundant file existence checks

Test Scenarios:

• Fresh installation in empty repository
• Installation in repository with existing .samoid directory
• Installation with custom directory name
• Large repository (>10,000 files) handling

Success Criteria:

• Install operation filesystem efficiency comparable to cp command
• No O(n²) filesystem operations
• Proper error handling for filesystem failures
• Atomic operations where data consistency matters

AC8.6: Minimize external dependencies

Detailed Specification

Definition: Limit runtime and compile-time dependencies to essential libraries only.

Current Approved Dependencies (from Cargo.toml):

• Runtime Dependencies:

  • clap v4.5 (CLI argument parsing)
  • toml v0.8 (configuration file parsing)
  • serde v1.0 (serialization)
  • anyhow v1.0 (error handling)

• Development Dependencies:

  • tempfile v3.13 (testing)
  • criterion v0.5 (benchmarking)

Dependency Evaluation Criteria:

• Essential: Required for core functionality
• Mature: Stable API, active maintenance
• Minimal: Small footprint, few transitive dependencies
• Secure: No known vulnerabilities

Prohibited Dependencies:

• Async runtimes (tokio, async-std) - unnecessary complexity
• Heavy serialization libraries beyond basic TOML/JSON
• GUI frameworks
• Network libraries (no network operations required)
• Database libraries

Transitive Dependency Limits:

• Total dependency count (direct + transitive): ≤ 25
• No duplicate functionality across dependencies
• All dependencies must compile on target platforms without additional system libraries

Verification Process:

• Use cargo tree to audit dependency graph
• Regular security audits with cargo audit
• Monitor for dependency bloat in CI/CD
• Quarterly dependency review and updates

Success Criteria:

• Dependency count remains stable or decreases
• Build time ≤ 2 minutes from clean state
• All dependencies justify their inclusion with documentation
• Zero high-severity security vulnerabilities

AC8.7: Dedicated performance testing pipeline implemented

Detailed Specification

Definition: Create a separate GitHub Actions workflow (perf.yml) dedicated to performance testing and benchmarking.

Pipeline Requirements:

• Separate from functional tests: Run in parallel to test.yml, not blocking it
• Consistent environment: Use same runner type consistently for reliable measurements
• Multiple triggers:
  • push to main branches (master, develop)
  • pull_request to main branches
  • schedule for nightly performance monitoring
  • workflow_dispatch for manual execution
• Platform coverage: Initially Ubuntu latest (stable Rust), expand to other platforms as needed

Benchmark Execution:

• Run cargo bench --verbose with comprehensive output
• Use release builds only for accurate performance measurements
• Execute multiple iterations for statistical significance
• Capture both summary metrics and detailed timing data

Artifact Management:

• Store benchmark results as workflow artifacts
• Generate performance comparison reports for PRs
• Maintain historical performance data
• Export results in machine-readable format (JSON/CSV)

Integration Points:

• Comment performance comparisons on pull requests
• Fail workflow if performance regressions exceed thresholds
• Integration with existing benchmark infrastructure in samoid/benches/

Success Criteria:

• perf.yml workflow created and functional
• Benchmarks run independently of functional tests
• Performance results available within 10 minutes of trigger
• Historical performance tracking operational

AC8.8: Benchmark results tracking and comparison system

Detailed Specification

Definition: Implement automated system for tracking performance metrics over time and comparing results across commits/PRs.

Tracking Requirements:

• Historical Data: Store benchmark results with commit SHA, timestamp, and environment details
• Trend Analysis: Identify performance improvements and regressions over time
• Comparison Logic: Compare PR performance against base branch benchmarks
• Threshold Detection: Automatically flag significant performance changes

Metrics to Track:

• Hook execution overhead (median, 95th percentile)
• Binary sizes (samoid, samoid-hook)
• Memory usage (peak RSS)
• Startup time (cold/warm start)
• Filesystem operation counts

Reporting Features:

• PR Comments: Automated comments showing performance impact
• Performance Badges: Status indicators for performance health
• Regression Alerts: Notifications when performance degrades beyond thresholds
• Performance Reports: Periodic summaries of performance trends

Implementation Approach:

• Use GitHub Actions artifacts for result storage
• JSON format for structured benchmark data
• GitHub API integration for automated commenting
• Optional: External storage for long-term historical data

Alert Thresholds:

• Critical Regression: >20% performance degradation
• Warning: 10-20% performance degradation
• Improvement: >10% performance improvement
• Acceptable Variance: ±10% considered normal fluctuation

Success Criteria:

• Automated performance comparison on all PRs
• Performance trends visible and trackable
• Regression detection functional with appropriate sensitivity
• Performance data accessible and queryable

Priority: Medium

Effort: 4 story points
Phase: Construction

Source

Inherent requirement for hook systems - performance is critical for developer experience. User feedback and stakeholder requirements emphasize the need for minimal performance impact on Git operations.

Dependencies

• Depends on: Core Hook Installation System #1, Command Line Interface #2, Hook Execution Runtime #3, Comprehensive Git Hooks Support #4 (Core functionality must be complete)
• Requires: Comprehensive Test Infrastructure #5 (Test Infrastructure for benchmarking)
• See also: CI/CD Pipeline with GitHub Actions #11 (CI/CD Pipeline for automated performance testing)

Performance optimization should only begin after core functionality is complete and we have proper benchmarking infrastructure in place.