Add MLX-VLM backend support for Apple Silicon

[aryasaatvik]

Summary

This PR adds MLX-VLM backend support for olmOCR, enabling efficient inference on Apple Silicon Macs (M1/M2/M3/M4) without requiring NVIDIA GPUs or cloud services.

Motivation

• Enable on-device inference for users with Apple Silicon Macs
• Reduce cloud inference costs for development and testing
• Provide privacy-focused local processing option
• Support users without access to NVIDIA GPU infrastructure

Changes

1. Backend Abstraction Layer (olmocr/backends.py) - 458 lines

• New abstract base class InferenceBackend for multi-backend support
• VLLMBackend implementation for NVIDIA GPUs
• MLXVLMBackend implementation for Apple Silicon
• Backend-specific request/response formatting
• Automatic server health checking and startup

Key architectural decision: Each backend handles its own API format differences:

• vLLM: /v1/chat/completions endpoint, pre-loads model with name "olmocr"
• MLX-VLM: /responses endpoint, lazy-loads model using actual path on first request

2. Pipeline Integration (olmocr/pipeline.py) - +145/-50 lines

• Integrated backend abstraction into main inference pipeline
• Backend-agnostic request building and response parsing
• Model download happens before server startup
• Correct model path handling for each backend

3. Configuration (olmocr/config.py) - +41 lines

• Added backend field: "vllm" (default) or "mlx-vlm"
• Added MLX-specific options: mlx_quantization, mlx_kv_bits
• Platform validation for MLX backend:
  • Checks for macOS (Darwin)
  • Validates Apple Silicon architecture (arm64/aarch64)
  • Verifies mlx-vlm package installation

4. Model Conversion Utility (olmocr/convert_to_mlx.py) - 218 lines

• CLI tool for converting HuggingFace models to MLX format
• Supports 4-bit and 8-bit quantization
• Configurable group size for quantization (default: 64)
• Platform validation and clear error messages
• Progress logging for multi-step conversion process

Usage:

python -m olmocr.convert_to_mlx allenai/olmOCR-2-7B-1025 \
  --output ~/models/olmocr-mlx --quantize 4 --group-size 64

5. Dependencies (pyproject.toml, uv.lock)

• Upgraded transformers: 4.55.2 → 4.57.0+
• Added MLX optional dependency: pip install olmocr[mlx]
• Requires mlx-vlm>=0.3.5
• Added CLI entry point: olmocr command

6. Documentation (docs/source/mlx-backend.md) - 482 lines

Comprehensive guide covering:

• System requirements and installation
• Quick start with pre-quantized models
• Configuration options and CLI usage
• Model selection (4-bit vs 8-bit trade-offs)
• Performance optimization tips
• Troubleshooting common issues
• API differences between backends
• Performance benchmarks on different Mac models

7. Gitignore (gitignore)

• Added workspace/* to ignore test pipeline output

Pre-quantized Models

Ready-to-use models available on HuggingFace:

• mlx-community/olmOCR-2-7B-1025-mlx-4bit (~2GB, fast)
• mlx-community/olmOCR-2-7B-1025-mlx-8bit (~4GB, better quality)

Usage Example

# Install with MLX support
pip install olmocr[mlx]

# Run with 4-bit quantized model
olmocr ~/workspace \
  --pdfs sample.pdf \
  --backend mlx-vlm \
  --model mlx-community/olmOCR-2-7B-1025-mlx-4bit

Testing

Tested on:

• macOS 15.2 (Sequoia)
• Apple M4 Pro
• MLX-VLM 0.3.5
• Pre-quantized 4-bit and 8-bit models

API Differences

Feature	vLLM	MLX-VLM
Endpoint	/v1/chat/completions	/responses
Model loading	Pre-load at startup	Lazy load on first request
Request format	OpenAI Chat Completions	OpenAI Responses
Response path	choices[0].message.content	output[0].content[0].text
Guided decoding	✅ Yes	❌ No (post-validation)
Default port	30024	8000

Limitations

• macOS + Apple Silicon only: MLX-VLM requires M-series chips
• No guided decoding: Responses validated after generation, may require retries
• Single GPU: No multi-GPU support (uses unified memory)

Breaking Changes

None - all changes are additive with safe defaults. Existing vLLM usage continues to work unchanged.

Next Steps

• Integration testing on different Mac models (M1/M2/M3)
• Benchmark performance comparison vs vLLM
• Consider adding MLX backend to CI/CD (if GitHub Actions supports M-series runners)

Related Issues

Addresses user requests for Apple Silicon support and cloud-free local inference.