DTDE - Distributed Temporal Data Engine

Transparent horizontal sharding and temporal versioning for Entity Framework Core

📚 Documentation · 🚀 Quick Start · 💡 Samples · 📊 Benchmarks

Overview

DTDE is a NuGet package that adds transparent horizontal sharding and optional temporal versioning to Entity Framework Core applications. Write standard LINQ queries — DTDE handles data distribution, query routing, and result merging automatically.

// Standard EF Core LINQ - DTDE routes queries transparently
var euCustomers = await db.Customers
    .Where(c => c.Region == "EU")
    .ToListAsync();  // Automatically queries only EU shard

// Point-in-time queries for temporal entities
var ordersLastMonth = await db.ValidAt<Order>(DateTime.Today.AddMonths(-1))
    .Where(o => o.Status == "Completed")
    .ToListAsync();

✨ Key Features

Feature	Description
🔀 Transparent Sharding	Distribute data across tables or databases invisibly
⏱️ Temporal Versioning	Track entity history with point-in-time queries
🔗 Cross-Shard Transactions	ACID transactions spanning multiple database shards
🎯 Property Agnostic	Use ANY property names for sharding keys and temporal boundaries
📝 EF Core Native	Works with standard LINQ — no special query syntax required
⚡ Multiple Strategies	Date-based, hash-based, range-based, or composite sharding
🗄️ Hot/Warm/Cold Tiers	Support for data tiering across storage tiers
✅ Fully Tested	400+ unit and integration tests

📦 Installation

# All-in-one package (recommended)
dotnet add package Dtde.EntityFramework

# Or install individual packages
dotnet add package Dtde.Abstractions  # Core interfaces
dotnet add package Dtde.Core          # Core implementations
dotnet add package Dtde.EntityFramework  # EF Core integration

Requirements: .NET 8.0+ / 9.0+ / 10.0+, Entity Framework Core 8.0+ / 9.0+ / 10.0+

🚀 Quick Start

1. Define Your Entity

public class Customer
{
    public int Id { get; set; }
    public string Name { get; set; } = string.Empty;
    public string Region { get; set; } = string.Empty;  // Shard key
    public DateTime CreatedAt { get; set; }
}

2. Create Your DbContext

public class AppDbContext : DtdeDbContext
{
    public DbSet<Customer> Customers => Set<Customer>();

    public AppDbContext(DbContextOptions<AppDbContext> options) : base(options) { }

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        base.OnModelCreating(modelBuilder);

        modelBuilder.Entity<Customer>(entity =>
        {
            entity.ShardBy(c => c.Region);  // Enable sharding by Region
        });
    }
}

3. Configure Services

builder.Services.AddDbContext<AppDbContext>(options =>
{
    options.UseSqlServer(connectionString);
    options.UseDtde(dtde =>
    {
        dtde.AddShard(s => s.WithId("EU").WithShardKeyValue("EU").WithTier(ShardTier.Hot));
        dtde.AddShard(s => s.WithId("US").WithShardKeyValue("US").WithTier(ShardTier.Hot));
        dtde.AddShard(s => s.WithId("APAC").WithShardKeyValue("APAC").WithTier(ShardTier.Hot));
    });
});

4. Use It!

// Queries are automatically routed to correct shard(s)
var allCustomers = await _context.Customers.ToListAsync();  // Queries all shards
var euCustomers = await _context.Customers
    .Where(c => c.Region == "EU")
    .ToListAsync();  // Only queries EU shard

// Inserts are automatically routed based on shard key
_context.Customers.Add(new Customer { Name = "Acme Corp", Region = "US" });  // Routes to US shard
await _context.SaveChangesAsync();

🔀 Sharding Strategies

DTDE supports multiple sharding strategies to match your data access patterns:

Property-Based Sharding

Distribute data by a property value (region, tenant, category):

modelBuilder.Entity<Customer>(entity =>
{
    entity.ShardBy(c => c.Region);
});

Use cases: Multi-region deployments, GDPR compliance, data residency

Date-Based Sharding

Partition data by date for time-series workloads:

modelBuilder.Entity<Transaction>(entity =>
{
    entity.ShardByDate(t => t.TransactionDate, DateInterval.Month);
});

Use cases: Financial transactions, audit logs, metrics, event sourcing

Hash-Based Sharding

Even distribution across shards using consistent hashing:

modelBuilder.Entity<UserProfile>(entity =>
{
    entity.ShardByHash(u => u.UserId, shardCount: 8);
});

Use cases: High-volume data, preventing hotspots, horizontal scaling

Composite Sharding

Combine strategies for complex scenarios:

modelBuilder.Entity<Order>(entity =>
{
    entity.ShardBy(o => o.Region)
          .ThenByDate(o => o.OrderDate);
});

⏱️ Temporal Versioning

Track entity history and query data at any point in time:

Enable Temporal Tracking

modelBuilder.Entity<Contract>(entity =>
{
    entity.HasTemporalValidity(
        validFrom: nameof(Contract.ValidFrom),
        validTo: nameof(Contract.ValidTo));
});

Temporal Queries

// Current data
var current = await _context.ValidAt<Contract>(DateTime.UtcNow).ToListAsync();

// Historical data (as of a specific date)
var historical = await _context.ValidAt<Contract>(new DateTime(2024, 1, 1)).ToListAsync();

// All versions (bypass temporal filtering)
var allVersions = await _context.AllVersions<Contract>()
    .Where(c => c.ContractNumber == "C-001")
    .OrderBy(c => c.ValidFrom)
    .ToListAsync();

// Data within a date range
var rangeData = await _context.ValidBetween<Contract>(startDate, endDate).ToListAsync();

Temporal Operations

// Add with effective date
_context.AddTemporal(contract, effectiveFrom: DateTime.UtcNow);

// Create new version (closes old, opens new)
var newVersion = _context.CreateNewVersion(existing, changes, effectiveDate);

// Terminate (close validity)
_context.Terminate(contract, terminationDate: DateTime.UtcNow);

await _context.SaveChangesAsync();

🔄 Mixed Usage: Sharded + Regular Entities

DTDE works seamlessly alongside regular EF Core entities:

public class AppDbContext : DtdeDbContext
{
    public DbSet<Customer> Customers => Set<Customer>();   // Sharded
    public DbSet<Contract> Contracts => Set<Contract>();   // Temporal
    public DbSet<AuditLog> AuditLogs => Set<AuditLog>();   // Regular EF Core
}

Entity Configuration	Behavior
`ShardBy()` configured	Queries routed by shard key
`HasTemporalValidity()` configured	Temporal filtering applied
No special configuration	Standard EF Core entity
Direct `DbSet<T>` access	Bypasses DTDE filtering

🗄️ Shard Tiers

Organize shards by access frequency for cost optimization:

dtde.AddShard(s => s
    .WithId("2024-current")
    .WithTier(ShardTier.Hot)         // Frequently accessed, recent data
    .WithConnectionString(fastStorage));

dtde.AddShard(s => s
    .WithId("2023-archive")
    .WithTier(ShardTier.Cold)        // Archived, rarely accessed
    .WithConnectionString(cheapStorage)
    .AsReadOnly());                   // Prevent accidental writes

Tier	Description	Typical Storage
`Hot`	Frequently accessed, recent data	SSD, Premium SQL
`Warm`	Less frequently accessed	Standard SQL
`Cold`	Archived, rarely accessed	Archive storage
`Archive`	Long-term retention	Cold storage

⚙️ Configuration Options

Fluent API

options.UseDtde(dtde =>
{
    // Entity configuration
    dtde.ConfigureEntity<Order>(e => e.ShardByDate(o => o.OrderDate));

    // Shard definitions
    dtde.AddShard(s => s.WithId("primary").WithConnectionString("..."));

    // Load from configuration file
    dtde.AddShardsFromConfig("shards.json");

    // Default temporal context
    dtde.SetDefaultTemporalContext(() => DateTime.UtcNow);

    // Performance tuning
    dtde.SetMaxParallelShards(10);

    // Diagnostics
    dtde.EnableDiagnostics();
});

JSON Configuration (shards.json)

{
  "shards": [
    {
      "shardId": "2024-q4",
      "name": "Q4 2024 Data",
      "connectionString": "Server=...;Database=Data2024Q4",
      "tier": "Hot",
      "dateRangeStart": "2024-10-01",
      "dateRangeEnd": "2024-12-31",
      "isReadOnly": false,
      "priority": 100
    },
    {
      "shardId": "2024-archive",
      "name": "2024 Archive",
      "connectionString": "Server=...;Database=Archive2024",
      "tier": "Cold",
      "isReadOnly": true,
      "priority": 10
    }
  ]
}

📊 Performance Benchmarks

Comprehensive benchmarks comparing single table vs sharded approaches:

Test Environment

Component	Specification
OS	Windows 11 (10.0.26200.7462)
CPU	12th Gen Intel Core i9-12900H (14 cores, 20 threads)
Runtime	.NET 9.0.10, RyuJIT AVX2
SDK	.NET SDK 10.0.101
Database	SQLite (file-based, separate DBs per benchmark)
Framework	BenchmarkDotNet v0.15.0

Key Results

Query Type	Records	Single Table	Sharded	Improvement
Point Lookup (by ID)	100K	153.1 ns	145.7 ns	~Same
Date Range (1 month)	100K	16,257 µs	3,555 µs	4.6x faster
Region Scan (single)	100K	3,638 µs	1,774 µs	2.1x faster
Region Scan (all)	100K	5,535 µs	2,014 µs	2.7x faster
Count (all records)	100K	7,873 µs	1,330 µs	5.9x faster
Count (all records)	50K	3,387 µs	24.1 µs	141x faster

Concurrent Access Performance

Scenario	Threads	Single Table	Sharded	Improvement
Parallel Aggregation	4	1,668 µs	546 µs	3.1x faster
Parallel Aggregation	8	2,043 µs	696 µs	2.9x faster
Parallel Reads (diff regions)	4	1,822 µs	542 µs	3.4x faster
Parallel Reads (diff regions)	8	2,479 µs	758 µs	3.3x faster

Write Operations

Operation	Batch Size	Single Table	Sharded	Notes
Batch Update	100	414.5 µs	317.7 µs	23% faster
Batch Update	1000	388.8 µs	299.5 µs	23% faster
Batch Delete	100	326.8 µs	302.4 µs	~Same

Key insight: Sharded queries benefit significantly from partition pruning — queries that target a specific shard key value only scan relevant partitions. Concurrent workloads show even greater improvements due to reduced contention.

When to Use Sharding

✅ Good candidates:

Large datasets (millions+ rows)
Time-series / temporal data
Multi-tenant applications
Geographic distribution requirements
High write throughput needs
Hot/cold data patterns
Concurrent read-heavy workloads

⚠️ Consider carefully:

Small datasets (<100K rows)
Random access patterns
Complex cross-entity joins
Simple CRUD applications

# Run benchmarks
cd benchmarks/Dtde.Benchmarks
dotnet run -c Release

🏗️ Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Your Application                          │
├─────────────────────────────────────────────────────────────┤
│                   Dtde.EntityFramework                       │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │DtdeDbContext│  │Query Engine │  │   Update Engine     │  │
│  │  - ValidAt  │  │ - Rewriter  │  │  - VersionManager   │  │
│  │  - AllVersions│ │ - Executor  │  │  - ShardRouter     │  │
│  └─────────────┘  └─────────────┘  └─────────────────────┘  │
├─────────────────────────────────────────────────────────────┤
│                      Dtde.Core                               │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │  Metadata   │  │  Sharding   │  │     Temporal        │  │
│  │  Registry   │  │  Strategies │  │     Context         │  │
│  └─────────────┘  └─────────────┘  └─────────────────────┘  │
├─────────────────────────────────────────────────────────────┤
│                   Dtde.Abstractions                          │
│         Interfaces • Contracts • Exceptions                  │
└─────────────────────────────────────────────────────────────┘

📁 Project Structure

dtde/
├── src/
│   ├── Dtde.Abstractions/        # Core interfaces and contracts
│   ├── Dtde.Core/                # Core implementations
│   └── Dtde.EntityFramework/     # EF Core integration
├── tests/
│   ├── Dtde.Core.Tests/          # Unit tests
│   ├── Dtde.EntityFramework.Tests/
│   └── Dtde.Integration.Tests/
├── samples/                       # Sample applications
│   ├── Dtde.Sample.WebApi/       # Basic getting started
│   ├── Dtde.Samples.RegionSharding/   # Property-based sharding
│   ├── Dtde.Samples.DateSharding/     # Date-based sharding
│   ├── Dtde.Samples.HashSharding/     # Hash-based sharding
│   ├── Dtde.Samples.MultiTenant/      # Multi-tenancy
│   └── Dtde.Samples.Combined/         # Mixed strategies
├── benchmarks/
│   └── Dtde.Benchmarks/          # Performance benchmarks
└── docs/                         # Documentation (MkDocs)

💡 Sample Projects

Explore working examples for each sharding strategy:

Sample	Strategy	Use Case
Dtde.Sample.WebApi	Basic	Getting started
Dtde.Samples.RegionSharding	Property-based	Multi-region data residency
Dtde.Samples.DateSharding	Date-based	Time-series, audit logs
Dtde.Samples.HashSharding	Hash-based	Even data distribution
Dtde.Samples.MultiTenant	Tenant-based	SaaS multi-tenancy
Dtde.Samples.Combined	Mixed	Complex enterprise scenarios

🧪 Testing

# Run all tests (101 tests)
dotnet test

# Run with coverage
dotnet test --collect:"XPlat Code Coverage"

# Run specific test project
dotnet test tests/Dtde.Core.Tests/

📚 Documentation

Full Documentation — Complete guides, API reference, and tutorials
Quick Start Guide — Get running in 5 minutes
Sharding Guide — Deep dive into sharding strategies
Temporal Guide — Temporal versioning explained
API Reference — Complete API documentation
Architecture — Design decisions and architecture

🤝 Contributing

Contributions are welcome! Please read our Contributing Guide before submitting PRs.

# Clone and build
git clone https://github.com/yohasacura/dtde.git
cd dtde
dotnet build
dotnet test

📄 License

This project is licensed under the MIT License — see the LICENSE file for details.

🙏 Acknowledgments

Built with Entity Framework Core
Inspired by temporal database concepts and bi-temporal data modeling
Benchmarked with BenchmarkDotNet

Made with ❤️ for the .NET community

⭐ Star on GitHub · 🐛 Report Bug · 💬 Discussions

Name		Name	Last commit message	Last commit date
Latest commit History 24 Commits
.github		.github
assets		assets
benchmarks/Dtde.Benchmarks		benchmarks/Dtde.Benchmarks
docs		docs
samples		samples
src		src
tests		tests
.editorconfig		.editorconfig
.gitignore		.gitignore
.markdownlint.json		.markdownlint.json
CHANGELOG.md		CHANGELOG.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
Directory.Build.props		Directory.Build.props
Directory.Packages.props		Directory.Packages.props
Dtde.sln		Dtde.sln
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
global.json		global.json
mkdocs.yml		mkdocs.yml
requirements-docs.txt		requirements-docs.txt

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

DTDE - Distributed Temporal Data Engine

Overview

✨ Key Features

📦 Installation

🚀 Quick Start

1. Define Your Entity

2. Create Your DbContext

3. Configure Services

4. Use It!

🔀 Sharding Strategies

Property-Based Sharding

Date-Based Sharding

Hash-Based Sharding

Composite Sharding

⏱️ Temporal Versioning

Enable Temporal Tracking

Temporal Queries

Temporal Operations

🔄 Mixed Usage: Sharded + Regular Entities

🗄️ Shard Tiers

⚙️ Configuration Options

Fluent API

JSON Configuration (shards.json)

📊 Performance Benchmarks

Test Environment

Key Results

Concurrent Access Performance

Write Operations

When to Use Sharding

🏗️ Architecture

📁 Project Structure

💡 Sample Projects

🧪 Testing

📚 Documentation

🤝 Contributing

📄 License

🙏 Acknowledgments

About

Topics

Resources

License

Code of conduct

Contributing

Security policy

Uh oh!

Stars

Watchers

Forks

Releases 1

Sponsor this project

Uh oh!

Packages 0

Uh oh!

Uh oh!

Contributors

Uh oh!

Languages

Packages