Map out entire ingestion flow and document as mermaid charts

[spacegoatai]

The current kalliste ingestion pipeline would benefit from comprehensive documentation showing the complete data flow from input to output.

Goals

• Create visual documentation of the entire ingestion pipeline
• Map all processing stages, decision points, and data transformations
• Document input/output formats at each stage
• Identify potential bottlenecks and optimization opportunities

Deliverables

• Mermaid flowchart(s) showing complete ingestion flow
• Documentation of each processing stage
• Input/output specifications for each component
• Performance characteristics and timing information

Benefits

• Better understanding of system architecture
• Easier onboarding for new developers
• Identification of optimization opportunities
• Foundation for future architectural decisions
• Debugging and troubleshooting reference

Implementation Notes

• Use mermaid syntax for charts (can be embedded in markdown)
• Consider creating separate charts for:
  • High-level overview
  • Detailed processing stages
  • Error handling and fallback paths
  • Configuration and parameter flow

This documentation will be essential for the video preprocessing integration and future pipeline enhancements.