FX-Store: High-Performance Time-Series Storage for Financial Data

Disclaimer: This is a toy project in progress out of curiosity, so it doesn’t work properly as desribed yet.

A production-grade, single-server solution for storing and querying foreign exchange market data, inspired by Databento's architecture but optimized for FX/HISTDATA workloads.

Table of Contents

• Architecture Overview
• Getting Started
• Performance Guide
  • Cache Optimization
  • Hardware Specifications (WSL)
  • Hardware Specifications (Linux)
• API Reference
• Data Format Specification
• Deployment Guide
• Benchmarks

Key Features

• Single-server 37 Gbps processing capability
• 10:1 compression with zstd
• Nanosecond precision timestamps
• Zero-copy query operations
• NUMA-aware memory management
• Lock-free concurrent access
• SIMD-accelerated filtering

Technology Stack

• Language: Rust
• Network: AF_XDP (kernel bypass)
• Storage: Memory-mapped files + zstd compression
• Concurrency: DashMap (lock-free hashmap)
• SIMD: AVX2 for filtering operations
• Time Sync: TSC + NTP calibration

Performance Highlights (Not Proven: Goal)

Metric	Value
Import Speed	5M records/sec
Query Speed	100M records/sec
Compression Ratio	10:1
P99 Latency	< 100μs
Memory Usage	1/10 of raw data

Quick Start

# Build
cargo build --release

# Import data
./fx-store import --symbol EURUSD --file data/EURUSD_2024.csv

# Query
./fx-store query --symbol EURUSD --start "2024-01-01" --end "2024-12-31"