Skip to content

Add non-blocking semantic indexing with Jobs Dashboard UI#19

Merged
brobertsaz merged 1 commit intobrobertsaz:mainfrom
illAssad:feat/background-indexing-jobs-ui
Dec 9, 2025
Merged

Add non-blocking semantic indexing with Jobs Dashboard UI#19
brobertsaz merged 1 commit intobrobertsaz:mainfrom
illAssad:feat/background-indexing-jobs-ui

Conversation

@illAssad
Copy link
Contributor

@illAssad illAssad commented Dec 8, 2025

Summary

This PR fixes a critical issue where semantic indexing blocks the entire server, making it unresponsive until indexing completes. The fix implements background processing with real-time progress tracking.

Problem

When running semantic indexing on large codebases (2000+ files), the server became completely unresponsive because the async def api_index_semantic endpoint called blocking functions (ingest_file() with Ollama embeddings) synchronously, blocking FastAPI's event loop.

Solution

  • Background Processing: Use FastAPI's BackgroundTasks to run semantic indexing in a separate thread
  • Job Tracking: Add INDEXING_JOBS dictionary with thread-safe progress updates
  • New Endpoints:
    • GET /api/jobs - List all indexing jobs
    • GET /api/jobs/{job_id} - Get specific job status
  • Jobs Dashboard UI: New React component with live progress bars, status indicators, and auto-refresh
  • Backward Compatibility: background=false parameter for sync mode when needed

Changes

  • mcp_server/server.py - Background worker, job tracking, new endpoints
  • mcp_server/claude_code_mcp.py - Updated tool schema with background parameter
  • frontend/src/components/JobsDashboard.tsx - New dashboard component
  • frontend/src/pages/MainApp.tsx - Added Jobs tab
  • tests/test_mcp_server.py - Updated tests

Testing

  • Server returns job_id in ~15ms (previously blocked for minutes)
  • Server stays responsive during indexing
  • Progress updates visible in real-time via Jobs tab
  • Completed jobs retained for 24 hours, auto-cleanup prevents memory leaks

Screenshots

The Jobs Dashboard shows:

  • Summary cards (Running, Queued, Completed, Failed)
  • Per-job progress bars with animations
  • Status badges with appropriate colors
  • Timestamps and duration tracking
  • Error messages for failed jobs

- Fix server blocking issue during semantic indexing by using FastAPI BackgroundTasks
- Add job progress tracking with real-time status updates
- Add /api/jobs and /api/jobs/{job_id} endpoints for monitoring
- Add JobsDashboard component with live progress bars and status indicators
- Add Jobs tab to MainApp for interactive monitoring
- Maintain backward compatibility with background=false option for sync mode
- Include automatic job cleanup to prevent memory leaks
@brobertsaz brobertsaz merged commit 046f189 into brobertsaz:main Dec 9, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Comments