Run benchmarks#

The benchmark suite compares zodb-json-codec against Python’s pickle module on both synthetic data and real ZODB FileStorage databases.

Prerequisites#

Always benchmark against a release build. Debug builds are 5-10x slower and do not reflect production performance.

maturin develop --release

Install the benchmark dependencies:

pip install "zodb-json-codec[bench]"

Benchmark commands#

Synthetic micro-benchmarks#

Tests individual type encode/decode paths with generated data:

python benchmarks/bench.py synthetic --iterations 5000

Real FileStorage scan#

Decodes every record from an actual ZODB FileStorage file:

# Decompress the bundled sample data first
gunzip benchmarks/bench_data/Data.fs.gz

python benchmarks/bench.py filestorage benchmarks/bench_data/Data.fs

Limit the number of records with --max-records N.

PG path comparison#

Compares the two PostgreSQL decode paths – decode_zodb_record_for_pg() (returns Python dict) vs. decode_zodb_record_for_pg_json() (returns JSON string directly):

python benchmarks/bench.py pg-compare --filestorage benchmarks/bench_data/Data.fs

Combined run#

Runs both synthetic and filestorage benchmarks and exports results to JSON:

python benchmarks/bench.py all --filestorage benchmarks/bench_data/Data.fs --output results.json

Generate benchmark data#

Creates a reproducible FileStorage from Wikipedia seed data:

python benchmarks/bench.py generate

Output defaults to benchmarks/bench_data/Data.fs.

Output options#

All benchmark commands accept:

--output FILE – export results as JSON
--format {table,json,both} – output format (default: table)

PGO builds, the way the wheels are built#

The release wheels are Profile-Guided Optimization (PGO) builds, so a local PGO build reproduces what users install. PGO is not simply faster for every path: until 1.7.0 it made the PG JSON pipeline on real records slower than a build without PGO (#52, a byte loop in the JSON string writer that the profile-guided build compiled worse; the performance page has the numbers before and after the fix). Measure both builds when a change touches the storage path.

1. install LLVM tools#

rustup component add llvm-tools

2. instrumented build#

Build with profiling instrumentation enabled:

RUSTFLAGS="-Cprofile-generate=/tmp/pgo-data" maturin develop --release

3. generate profiles#

Run all three benchmark modes so the profile covers the Python-dict paths and the PG JSON path (the one zodb-pgjsonb runs; a profile without it leaves the JSON writer without counts, and the optimizer treats it as cold):

python benchmarks/bench.py synthetic --iterations 5000
python benchmarks/bench.py filestorage benchmarks/bench_data/Data.fs
python benchmarks/bench.py pg-compare --iterations 2000 --filestorage benchmarks/bench_data/Data.fs

4. merge profile data#

llvm-profdata merge -o /tmp/pgo-data/merged.profdata /tmp/pgo-data/*.profraw

5. final PGO build#

RUSTFLAGS="-Cprofile-use=/tmp/pgo-data/merged.profdata" maturin develop --release

Now re-run the benchmarks against this optimized build for production-representative numbers.

CI regression detection#

The check subcommand is designed for CI pipelines. It runs synthetic benchmarks and fails with a non-zero exit code if performance has regressed:

python benchmarks/bench.py check --iterations 500