Skip to content

RESULTS.md

Latest commit

 

History

History
121 lines (91 loc) · 4.68 KB

RESULTS.md

File metadata and controls

121 lines (91 loc) · 4.68 KB

ORM Benchmark Results

Date: December 2, 2025 PostgreSQL: 16-alpine (Docker) Python: 3.12+ Driver: asyncpg (native async)

Configuration

Setting Value
Iterations 100
Warmup Iterations 10
Bulk Insert Size 100 records
Test Authors 100
Test Books 200
Test Publishers 10

Summary

ORM Wins
TortoiseORM 10
SQLAlchemy 0

Winner: TortoiseORM - Faster in all benchmarks by 13-40%

Performance Comparison

Benchmark SQLAlchemy (ms) TortoiseORM (ms) Winner Difference
Single Insert 0.676 0.417 TortoiseORM 38.2% faster
Bulk Insert (100) 2.796 2.163 TortoiseORM 22.7% faster
Select by ID 0.708 0.427 TortoiseORM 39.6% faster
Select with Filter 0.798 0.635 TortoiseORM 20.5% faster
Select All (100) 0.988 0.767 TortoiseORM 22.4% faster
Select with JOIN 1.861 1.612 TortoiseORM 13.4% faster
Update Single 0.674 0.425 TortoiseORM 36.9% faster
Update Bulk 9.974 6.877 TortoiseORM 31.1% faster
Aggregate Count 0.961 0.698 TortoiseORM 27.4% faster
Aggregate GROUP BY 0.769 0.496 TortoiseORM 35.5% faster

Detailed Statistics

SQLAlchemy

Benchmark Mean (ms) Std Dev Min (ms) Max (ms) Iterations
Single Insert 0.676 0.043 0.610 0.889 100
Bulk Insert (100) 2.796 0.202 2.561 3.310 20
Select by ID 0.708 0.071 0.616 0.942 100
Select with Filter 0.798 0.057 0.731 0.956 100
Select All (100) 0.988 0.751 0.830 8.383 100
Select with JOIN 1.861 0.091 1.724 2.236 100
Update Single 0.674 0.058 0.599 0.923 100
Update Bulk 9.974 3.089 7.949 18.712 50
Aggregate Count 0.961 0.084 0.874 1.400 100
Aggregate GROUP BY 0.769 0.069 0.647 0.960 100

TortoiseORM

Benchmark Mean (ms) Std Dev Min (ms) Max (ms) Iterations
Single Insert 0.417 0.046 0.372 0.639 100
Bulk Insert (100) 2.163 0.055 2.069 2.280 20
Select by ID 0.427 0.030 0.395 0.576 100
Select with Filter 0.635 0.036 0.586 0.756 100
Select All (100) 0.767 0.034 0.718 0.877 100
Select with JOIN 1.612 0.114 1.431 1.930 100
Update Single 0.425 0.041 0.377 0.600 100
Update Bulk 6.877 0.566 6.151 9.184 50
Aggregate Count 0.698 0.062 0.626 0.961 100
Aggregate GROUP BY 0.496 0.038 0.452 0.645 100

Analysis

Key Observations

  1. TortoiseORM consistently outperforms SQLAlchemy across all benchmark categories when both use asyncpg driver.

  2. Insert Operations: TortoiseORM shows ~23-38% better performance for both single and bulk inserts. This suggests lower overhead in object creation and query building.

  3. Select Operations: TortoiseORM is 13-40% faster across different select patterns. The advantage is most pronounced in simple ID lookups (~40% faster).

  4. Update Operations: Single updates show ~37% improvement, bulk updates ~31% improvement with TortoiseORM.

  5. Aggregations: TortoiseORM performs 27-36% faster in aggregate operations.

  6. Variance: SQLAlchemy shows higher standard deviation in several benchmarks (especially Select All and Update Bulk), indicating less consistent performance.

Why TortoiseORM May Be Faster

  • Lighter Abstraction: TortoiseORM has a simpler ORM layer with less overhead
  • Optimized Query Building: Purpose-built for async operations from the ground up
  • Session Management: SQLAlchemy's session/unit-of-work pattern adds overhead compared to TortoiseORM's active record style

Considerations

  • Feature Set: SQLAlchemy offers more advanced features (complex queries, migrations with Alembic, multi-database support)
  • Ecosystem: SQLAlchemy has a larger ecosystem and more extensive documentation
  • Flexibility: SQLAlchemy provides more control over query generation and database interactions
  • These benchmarks measure raw ORM overhead - actual application performance depends on many other factors

Environment

PostgreSQL: 16-alpine (Docker container)
Host: macOS (darwin 24.6.0)
Database: Local Docker (no network latency)
Connection Pool: 10 connections, max overflow 20

Reproduce Results

# Start PostgreSQL and run benchmarks
make benchmark

# Run with more iterations for higher confidence
make benchmark-full