Changelog#

1.19.2 (unreleased)#

Bug fixes#

  • Fix: deferred startup DDL is no longer lost when PostgreSQL is unreachable (#144). _apply_pending_ddl() cleared its queue before connecting and put it back only when the startup lock timed out. If the lock connection failed, for example during a failover shortly after a pod start, the DDL was dropped until the next restart, and a column added by a state processor could stay missing (#105). Now any failure while connecting or taking the lock requeues the queue for the next transaction or poll. An entry that fails with OperationalError under the lock (lost connection, lock_timeout) is requeued on its own; other errors still drop it until the next startup.

  • Fix: the startup DDL path connects with a timeout (#145). The schema-version probe, the advisory lock session and the DDL statements connect directly, outside the pool, and had no connect_timeout. A primary that drops packets blocked the poll applying the DDL for the OS TCP timeout. These connections, and the DSN passed to deferred startup actions, now carry connect_timeout set to pool-timeout (rounded up, at least 2 seconds), unless the DSN or PGCONNECT_TIMEOUT already sets one.

1.19.1#

Bug fixes#

  • Fix: a failover no longer strands ZODB connections with their full cache (#142). ZODB polls in Connection.open() and in every transaction.abort(). While PostgreSQL was unreachable, poll_invalidations() raised OperationalError or PoolTimeout there. DB.open() then never returned the connection, and Zope’s publisher skipped close() after the failing abort(). A connection that is never closed is never freed, so every request in flight during a failover left its connection and its whole object cache in memory, and the process grew with every failover. Now poll_invalidations() returns None on any psycopg.OperationalError, which tells ZODB to invalidate its whole cache. The next access to the instance’s PostgreSQL connection, and every load() and load_multiple() before it consults the shared cache, repeats the poll first, so reads get a fresh snapshot. Other exceptions still propagate. The traceback is logged once per outage, later failures log one line each.

    OperationalError covers more than an unreachable server: it includes PoolTimeout, QueryCanceled (statement_timeout), lock and disk errors. A flushed cache is cheaper than a stranded connection in all of them, but it has costs:

    • After a failover each connection reloads its objects once.

    • If the pool is exhausted under load while PostgreSQL is healthy, the whole object cache is flushed, and the reloads add load while the pool is saturated.

    • If the poll query itself keeps failing (for example a persistent QueryCanceled), every transaction flushes the cache and every read fails.

New features#

  • Declare and test Python 3.15 support: trove classifier and CI test matrix. The test suite passes on 3.15.0 without code changes.

Documentation#

  • Fix the write path sequence diagram in the architecture explanation. The ; in BEGIN; pg_advisory_xact_lock(0) ends a Mermaid statement, so the diagram rendered as “Syntax error in text”; it is now written as #59;.

  • Move the reasoning behind the S3 timeout keys from the S3 how-to into the architecture explanation, turn the how-to section into concrete steps with a configuration example, mark the keys as added in 1.19, document ZODB_PGJSONB_IDLE_IN_XACT_TIMEOUT_MS, and list the zodb-s3blobs>=1.2.0 requirement in the dependency table.

1.19.0#

New features#

  • S3 timeouts and retries are configurable (#136). New <pgjsonb> keys s3-connect-timeout, s3-read-timeout, s3-download-connect-timeout, s3-download-read-timeout and s3-download-max-attempts are passed to zodb-s3blobs’ S3Client when set; unset keys keep zodb-s3blobs’ defaults. Blob downloads run in the request that needs the blob while its PostgreSQL transaction is open, so a stalled download used to block the request for zodb-s3blobs’ former 300 s read timeout, past idle_in_transaction_session_timeout, and end in a 500 (bluedynamics/zodb-s3blobs#25). zodb-s3blobs 1.2.0 shortens the download timeouts by default; these keys let a deployment tune them. The [s3] extra now requires zodb-s3blobs>=1.2.0.

1.18.0#

New features#

  • State processors can react to undo (#121). A processor may implement the optional undone(cursor, tid, restored, deleted, entries). It is called from undo() once per undone transaction, before the undo is queued or written, in the undo’s PostgreSQL transaction, so object_state still shows the pre-undo rows. tid and the zoid lists are ints; entries gives per zoid the revision the undone transaction wrote and the one the undo writes, so a processor can see what changed. An exception aborts the undo. Processors without the method are unaffected. Documented in the state processor API, together with a documentary IStateProcessor in zodb_pgjsonb.interfaces.

Bug fixes#

  • Undo of a transaction whose objects changed again afterwards failed with ValueError instead of resolving the conflict. undo() passed the arguments of tryToResolveConflict in the wrong order (a pickle where the undone serial belongs), so every undo that needed conflict resolution, for example of a BTrees.Length or a BTree bucket changed later, raised. It now passes them as FileStorage does.

Internal#

  • The undo queueing loop duplicated in PGJsonbStorage.undo() and PGJsonbStorageInstance.undo() now lives in undo.py.

Documentation#

  • Design for restoring state-processor columns on undo (#121): an optional undone() hook for state processors; plone.pgcatalog hides the affected objects and subtrees fail closed and recatalogs them afterwards. A more precise variant is in the backlog (#130, plone-pgcatalog#268). Design: docs/superpowers/specs/2026-10-12-undo-processor-columns-design.md.

  • Undo design amended: the subtree rule only fires when the undone transaction changed security or location at a container (conservative option B). In Plone every add and delete writes the container, so the original rule hid a whole site on undo of a delete in the portal root. The hook receives the before/after revisions as entries.

1.17.1#

Bug fixes#

  • Fix: a database failover could wedge every Zope process permanently (#126). ZODB creates new connections while holding its DB._lock, and the storage instance checked a connection out of the pool right there. When the pool was full, the opening thread waited up to pool-timeout while holding the lock, and every thread trying to return a connection waited for that lock. After a failover, piled-up readiness probes kept the convoy alive indefinitely: all pool connections sat idle (last statement COMMIT) and every request failed with PoolTimeout. The instance now checks out its connection on first use, outside the lock, and a failed poll_invalidations() returns its connection instead of keeping it.

    Known side effect: a PoolTimeout now surfaces in ZODB’s Connection.open() instead of in Connection.__init__. ZODB leaves such a half-opened Connection behind, and psycopg-pool keeps it reachable until the next connection is returned to the pool. If the process shuts down in that window, DB.close() raises KeyError (unregisterSynch of a never-registered connection). This only affects shutdown; Connection.open() not being exception-safe is a ZODB issue.

Documentation#

  • Correct the sizing rule for pool-max-size in the production guide: every open or idle ZODB connection holds a pool connection, so the pool must cover ZODB’s own pool-size plus the worker threads.

  • Replace MinIO with Garage in the Docker Compose example and the docs: MinIO is no longer maintained and its images are no longer published on Docker Hub. Garage runs in single-node mode and creates the zodb-blobs bucket and its access key on startup, so the createbucket bootstrap container is gone.

Tests#

  • Add benchmarks/failover_repro.py, a time-scaled reproduction of the #126 failover wedge, and regression tests for the lock inversion.

1.17.0#

Bug fixes#

  • Fix: a plain object write no longer wipes state-processor columns (#120). The batch writer wrote every registered extra column for every object, so a processor that returned None (“nothing to say about this object”) got its columns overwritten with NULL. With plone.pgcatalog this silently uncataloged content whenever an object was written without a full reindex, for example when opening the edit form (edit lock), on the sharing tab, or on a write followed by a partial reindex. Now the rule is per processor: None leaves its columns untouched, a dict writes all its columns (missing keys as NULL), {} clears them. This also holds with several registered processors. Rows wiped before this fix stay wiped; plone.pgcatalog ships a repair (bluedynamics/plone-pgcatalog#244). Undo still restores processor columns incorrectly (#121), but no longer sets them to NULL.

  • ExtraColumn.update_expr is now used in the ON CONFLICT clause. It was documented but ignored.

Documentation#

  • Add plone.registryfromenviron to the ecosystem dashboard under Tools & Libraries: it overrides plone.registry values from environment variables, which makes registry configuration 12-factor friendly. The same card was added to the duplicated dashboard in plone.observability, so both copies stay identical.

Internal#

  • Pin ruff to 0.16.7 in the QA workflow (and bump the ruff-pre-commit rev to match) instead of running unpinned uvx ruff, so ruff releases no longer change CI behavior on unrelated PRs. Part of the ecosystem-wide alignment (bluedynamics/plone-pgcatalog#217).

  • Lower ruff’s C901 max-complexity threshold from 15 to 13 as part of the ecosystem-wide complexity ratchet, refactoring the three functions above the new limit: _copyTransactionsFrom_parallel (15 → 10, worker clamp and the thread-local write worker become dedicated methods), _compute_undo (14 → 3, per-object undo split into simple/cascading/conflict helpers with the duplicated restore/delete entry dicts deduplicated), and the test_tid_ordering_w_commit override (14 → 9, optional getTid / lastInvalidations readers extracted). Behavior is unchanged.

  • Enable ruff’s cyclomatic-complexity check (C901, mccabe) with max-complexity = 15. src/ and tests/ pass as-is; benchmarks/ is exempted via per-file-ignores (CLI harness code, legitimately branchy).

1.16.2#

Bug fixes#

  • Fix packer crash KeyError: 0 on any pack that deletes blobs (#108). pack() read the RETURNING s3_key rows with tuple indexing (row[0]), but the storage’s admin connection is opened with row_factory=dict_row — the first deleted blob row raised KeyError: 0. Because the admin connection runs in autocommit, the DELETEs had already committed at that point: the pack completed on the database side while the collected S3 keys were lost, orphaning every deleted blob’s S3 object with no remaining record of its key. pack() now opens its cursor with an explicit dict_row factory and reads row["s3_key"], so it works regardless of how the caller’s connection was configured. The packer tests now run against both tuple- and dict-row connections, and the deprecated blob_history cleanup paths gained coverage.

  • Bump hynek/build-and-inspect-python-package from v2 to v3.0.1. Hatchling now emits Metadata-Version: 2.5, which the Twine bundled in v2 rejects with InvalidDistribution: '2.5' is not a valid metadata version — the release build failed before uploading anything. v3 ships Twine 7, which supports it.

Documentation#

  • Add cloud-vinyl and plone.observability to the ecosystem dashboard (docs/sources/ecosystem.md) and to the ecosystem navigation dropdown in the docs. cloud-vinyl joins the existing Deployment group; plone.observability introduces a new Observability group.

  • Make the benchmark harness variance-aware. Each cell is now reported as median ±CV% (coefficient of variation over the pooled samples) so noisy rows are obvious instead of hidden behind a single number, and a new --runs K flag pools several independent runs (each re-does its own setup, so it is safe for the write benchmarks). On a shared/desktop machine a CV of 10–40% is normal; the benchmarks/README.md documents running on a quiet machine and comparing ratios rather than absolute values. (Investigating apparent slowdowns showed no regression: current storage numbers match zodb-pgjsonb 1.13.1 on the same machine, and the only consistent cost is the ~20 µs connection-checkout liveness probe added with the 1.14.2 pool-leak fix.) #99

1.16.1#

Bug fixes#

  • Self-healing admin connection. The storage’s admin connection (self._conn, opened once in __init__) had no liveness guard: once it died — a CNPG switchover/failover, an operator kill, an idle timeout — every user raised psycopg.OperationalError: the connection is closed until the process was restarted, flooding logs via the metrics scrape (len(storage)/getSize()) and silently stopping the cache warmer. Admin operations now reconnect on a closed connection and retry once on OperationalError (a server-side kill is only detected client-side on first use). Routed paths: __len__, getSize, get_blob_stats, get_blob_histogram, history, undoLog, current_max_tid (keeps its degrade-to-None contract when the DB is really unreachable), new_oid, load/loadBefore/loadSerial, loadBlob, and the direct-use tpc write path (ensure-only before BEGIN; no retry mid-transaction, where a silent reconnect would drop the transaction block). pack gets an ensured connection but no retry. The cache warmer now obtains the connection through the storage’s healing accessor instead of holding a reference that goes stale on reconnect, and close() marks the storage so healing cannot resurrect connections during shutdown. #103

Documentation#

  • Document the GIL convoy under concurrency: why per-load wall time inflates when CPU-heavy request work runs alongside object loads in the same process, why it is a Python-runtime property rather than a storage issue, and the levers that reduce it (cache hits and prefetch to cut round-trips, keeping CPU work out of the request path, and sizing worker threads per process against replica count). Added to the performance explanation and the production how-to. #98

  • Reformat Python code fences in the documentation for ruff 0.16, which now formats markdown code blocks (the QA workflow always runs the latest ruff via uvx). No content changes. #103

1.16.0#

Features#

  • Add a per-connection _pg_query_count counter (PostgreSQL round-trips) next to the existing _pg_load_count (objects). load() bumps both by 1; load_multiple() bumps _pg_load_count by the batch size but _pg_query_count by 1 (one WHERE zoid = ANY(...) query), so _pg_load_count / _pg_query_count quantifies batching/prefetch. Plain int, no dependency; read best-effort by plone.observability. #96

Documentation#

  • Refresh the performance benchmarks and document how to reproduce them. Re-ran the whole suite – storage/zodb/pack/history (PostgreSQL 17.9, RelStorage 4.2.0) and the Plone workloads (Plone 6.2.1, RelStorage 4.3.0, zodb-json-codec 1.6.1) – and updated explanation/performance.md with current numbers, an honest machine-specific caveat, the two-tier L1/L2 cache description, and the optimization history through 1.15 (shared cache, cache warmer, connection-pool hardening, per-entry L2 gate, the ZODB prefetch hook). Add benchmarks/README.md documenting the prerequisites, how to run each subset (including a ready-to-run recipe for the Plone subset against the current Plone release), the methodology, and how to update the docs from a run.

  • Fix the Plone benchmark’s instancehome. bench.py plone generated a zope.conf pointing at a non-existent instancehome outside the checkout, so the subset failed with a ZConfig.DataConversionError. The instance home now lives in the run’s temp directory, so the Plone subset runs regardless of checkout layout.

1.15.0#

Features#

  • Implement the ZODB prefetch hook so batch loading actually works. Connection.prefetch() calls storage.prefetch(oids); zodb-pgjsonb did not implement it, so ZODB installed a no-op fallback and a result set was loaded one object at a time — N sequential setstate → load round-trips (the classic N+1). The hook now delegates to load_multiple, fetching the not-yet-cached oids in a single WHERE zoid = ANY(...) query that warms L1/L2, so the subsequent per-object loads are cache hits. A caller (for example a collection tile) that prefetches its result set turns N round-trips into one. Micro-benchmark (benchmarks/bench_prefetch.py), 151 objects: 151 → 1 round-trips; at a 20 ms per-query latency the load drops from ~3.1 s to ~24 ms.

  • Per-entry read gate for the shared L2 cache (#92). SharedLoadCache.get previously denied a connection all of L2 whenever its snapshot was behind the process-wide consensus_tid — so during any write burst a lagging reader fell through to PostgreSQL for every object, one round-trip at a time, and an otherwise-warm pod served cold-looking requests. The gate is now per entry: a cached entry is served when its committed TID is at or below the reader’s snapshot (unchanged since the snapshot, hence exactly the version the reader must see) and skipped when it is newer. A lagging reader keeps hitting L2 for every object unchanged since its snapshot and misses only genuinely newer objects. Correctness rests on the existing invariant that every commit drops its changed zoids from L2 at commit time, so a present entry always carries the object’s true latest TID; the write gate is unchanged. Verified with a held-snapshot integration test (a REPEATABLE READ connection never observes a concurrent post-snapshot commit through L2) and the existing no-stale-reads concurrency stress test.

Documentation#

  • Document the cache tiers as a new explanation page (explanation/caching.md): the ZODB object cache, the L1 per-connection load cache, the L2 process-wide shared cache and its consensus-TID gate, the startup cache warmer, and how to read the plone.zodb.load_l2_hits / load_pg_queries span attributes to tell a cold/gated cache from high per-load latency. Also correct the stale cache configuration in the reference and llms.txt: the current keys are cache-shared-mb (default 256) and cache-per-connection-mb (default 16); cache-local-mb is a deprecated alias. The cache-warmer keys are now documented in the reference.

1.14.3#

Features#

  • Per-connection load counters for optional observability (no new dependency). Each storage instance now exposes two plain-int counters — _l2_load_hits (objects served from the shared L2 cache) and _pg_load_count (objects fetched from PostgreSQL) — incremented in load() and load_multiple(). A soft-coupled observer (e.g. plone.observability) can read them via getattr(connection._storage, ...) to attribute per-request cache effectiveness (shared-cache hit ratio = l2 / (l2 + pg)), which pinpoints load-time outliers between “cold/gated cache” and “slow per-load latency”. zodb-pgjsonb itself imports no tracing library; the counters are dependency-neutral numbers.

1.14.2#

Bugfixes#

  • Stop leaking pool slots when a pooled connection dies server-side (#85). When PostgreSQL closed a pooled connection while it was idle (a pooler/CNPG idle-recycle, idle_in_transaction_session_timeout, or an operator), poll_invalidations() raised on the first use. During Connection.open() ZODB does not guard that call, so it stranded the connection — its pool slot was never returned. Under sustained write load (a bulk reindexObject() run) the pool filled with stranded slots until every getconn() hit PoolTimeout; all pods went 0/1 Ready with PostgreSQL nearly idle, and only a restart recovered. (The 1.14.1 release() fix, #81, was already correct — this is a separate, open-path leak.)

    Two changes make the read path self-heal instead of stranding:

    • The instance pool now validates liveness on checkout (check=ConnectionPool.check_connection), so getconn() never hands out a connection that was closed while idle in the pool.

    • _begin_read_txn() (run by poll_invalidations) now returns a connection found broken and retries once on a fresh one, so a connection that died while held by a reused ZODB connection is replaced rather than raising through Connection.open().

    A write transaction whose connection is killed mid-flight still fails that one transaction (a half-applied write cannot be resumed on a new connection), but the pool now recovers and the pod stays healthy.

1.14.1#

Bugfixes#

  • Don’t leak a pool slot when a connection is closed server-side (#81). _end_read_txn() ran COMMIT unguarded, so when PostgreSQL terminated an idle-in-transaction connection (e.g. via idle_in_transaction_session_timeout) the next read-only request raised through poll_invalidations (HTTP 500) and, worse, release() raised before putconn() — leaking the pool slot. After ~pool-max-size such leaks the pool was permanently exhausted (PoolTimeout) and the pod never became ready again, stalling rolling deploys. _end_read_txn() now clears its flag and swallows the failed COMMIT, and release() always returns the connection to the pool (the pool discards the dead one and opens a replacement).

  • Stop logging a spurious 42P01 ERROR on every startup (#82). In non-history-preserving mode _set_lz4_compression() issued ALTER TABLE object_history … for a table that never exists, which PostgreSQL logged as a server-side ERROR on every pod startup (red noise in CNPG dashboards) even though the client caught and rolled it back. Each table is now probed with to_regclass first and skipped if absent, so no failing statement is sent.

1.14.0#

Features#

  • Gate startup DDL behind a double-checked schema-version probe (#78). On a rolling deploy every replica used to enter _apply_pending_ddl(), take the startup_ddl_lock advisory lock, and re-run the full idempotent DDL batch even when the schema was already current — so all N pods connected and waited on the lock, saturating the DB during the migration window (downstream symptom: bluedynamics/plone-pgcatalog#136).

    Deferred DDL/actions now carry a version marker recorded in a new pgjsonb_schema_state table. A cheap SELECT runs before the lock, so replicas whose tagged work is already current bail out without ever contending for it; the check is repeated inside the lock to cover the race where another replica applies while we wait. The static get_schema_sql() block is gated automatically (its version is the sha256 of the DDL), and defer_startup_action() gained an optional version argument for deferred callables. Untagged work (version=None) still runs under the lock on every startup, as before.

    Set ZODB_PGJSONB_FORCE_DDL=1 to bypass the gate and force a full run (e.g. after a manual index drop).

Documentation#

  • Add cdk8s-plone to the ecosystem dashboard under a new Deployment group, covering Kubernetes deployment of Plone backend and frontend.

  • Add cdk8s-plone to the ecosystem navigation dropdown.

  • Add RELEASE.md documenting the tag-based (hatch-vcs) release process: finalize CHANGES.md, merge via release PR, tag vX.Y.Z, and publish a GitHub release to trigger the PyPI upload.

1.13.1#

Bugfixes#

  • Bound the per-instance blob materialization dir (#71). Each PGJsonbStorageInstance used to materialize every blob read via loadBlob() into its own tempfile.mkdtemp(prefix="zodb-pgjsonb-blobs-") directory with no size limit, cleaned only on a clean close(). On long-running pods this filled local/ephemeral disk (11–14 GB per pod observed) and triggered node DiskPressure; instances killed abnormally leaked their dir entirely. For S3-tiered blobs it was doubly wasteful — every blob was stored both in the unbounded dir and in the bounded S3BlobCache.

    Read blobs are now materialized through a single, process-wide bounded cache (LRU by access time) for both PG-bytea and S3 blobs:

    • When S3 is configured the existing S3BlobCache is reused as the materialization target (no more duplicate copy in the temp dir).

    • Without S3 a new LocalBlobCache (in zodb_pgjsonb.blob_cache) provides the same bound for PG-bytea blobs.

    Both are sized by the existing blob-cache-size ZConfig key (default 1 GB), which now applies with or without S3. The per-instance temp dir is retained only for transient write-staging. On startup the storage also sweeps orphaned zodb-pgjsonb-blobs-* dirs left by pre-1.13.1 workers (conservatively: never its own, only dirs older than an hour).

Tests#

  • Recovery conformance: robust destination DSN derivation. PGJsonbRecoveryHP built the destination DSN via DSN.replace("dbname=zodb_test", ...), which silently no-ops for any DSN whose dbname is not literally zodb_test (e.g. a ZODB_TEST_DSN with dbname=zodb), leaving source and destination on the same database and producing confusing transaction_log_pkey duplicate-key failures. The destination dbname is now derived from the source DSN (<src>_dst) via regex and asserted distinct, so the tests work with any libpq key=value DSN and a broken derivation fails loudly.

1.13.0#

Features#

  • PGJsonbStorage.clear_caches() and SharedLoadCache.clear(): public methods that drop all in-memory caches and reset the consensus TID. Intended for test harnesses that roll the database backwards out of band (e.g. restoring a snapshot between tests): the monotonic shared cache cannot otherwise recover and keeps serving object state at TIDs newer than the rolled-back database, causing spurious ConflictError on the next commit. Not needed in production, where TIDs never decrease.

  • Cache warmer herd mitigation (#59). Rolling Kubernetes deploys used to multiply DB primary CPU by the replica count for the warmer startup window, because every pod independently fired its full set of warmup SELECTs simultaneously. Four composable behaviors fix this, all wired through six new ZConfig keys:

    • cache-warm-delay (default 15s): baseline sleep before warmer starts. Moves the warmer out of the pod’s own cold-start window (Plone import, plone.pgcatalog schema check, ANALYZE chatter).

    • cache-warm-jitter (default 30s): additional random(0, jitter) sleep on top. Spreads arrival times across pods.

    • cache-warm-concurrency (default 2): maximum number of pods warming in parallel, cluster-wide. Enforced via a session-level PostgreSQL advisory lock semaphore (same pattern as the existing startup-DDL lock). Pods that miss a slot retry with jittered backoff until cache-warm-wait-max expires.

    • cache-warm-wait-max (default 300s): retry cap before giving up and skipping warmup (logged as WARNING).

    • cache-warm-batch-size (default 500): zoids per SELECT batch.

    • cache-warm-batch-pause (default 0.5s): sleep between batches. Lowers per-pod peak qps.

    Observable behavior change: warmer queries are now delayed by ~15–45s post-startup (delay + jitter), not immediate. To restore pre-1.13 behavior, set cache-warm-delay=0, cache-warm-jitter=0, cache-warm-batch-pause=0, cache-warm-concurrency=9999.

    Last-pod latency: for a 6-pod fleet with default settings, the last pod has a warm L2 cache approximately 45s after the deploy window. All pods serve traffic from T+0; this is the time-to-warm, not the time-to-ready.

    Design: docs/superpowers/specs/2026-05-29-cache-warmer-herd-mitigation-design.md.

1.12.0#

Features#

  • Process-wide SharedLoadCache replaces per-connection duplication of the pickle-bytes cache (#63). Each ZODB Connection still keeps a small L1 cache for lock-free hot reads, but shares a single L2 cache across all connections in the process. On typical Plone deployments this frees ~1 GB per pod (5 threads × 256 MB per-instance → 1 × 256 MB shared).

    Correctness is protected by a process-wide _consensus_tid that gates cache reads and writes: an instance holding a snapshot older than another instance’s last-polled TID is not allowed to read or write the cache. Any poll_invalidations atomically advances the consensus and invalidates the changed zoids.

  • Config migration (#63):

    • New cache-shared-mb (default 256): size of the process-wide shared cache.

    • New cache-per-connection-mb (default 16): size of the per-connection L1 cache.

    • cache-local-mb becomes a deprecation alias for cache-shared-mb. Existing deployments that set e.g. cache-local-mb=256 automatically get a single 256 MB shared cache instead of 256 MB per connection — no action required beyond the deprecation warning.

  • CacheWarmer now populates the shared cache directly (#63). Its private _warm_cache dict, get(), and invalidate() are removed; the only surviving public API is record() and warm().

Fixes#

  • Unbounded growth of _serial_cache (#62). The conflict-resolution cache was a plain dict with no eviction or clearing, so it grew monotonically for the life of the storage instance — on a long-running pod with 5 threads and moderate traffic, estimated ~3.8 GB leaked after 24 hours, directly driving memory pressure toward the pod limit.

    In history-preserving mode, _do_loadSerial retrieves old revisions from object_history directly; the serial cache is redundant. Swapped for a _NoopSerialCache that silently drops writes and always misses on reads. Zero memory cost in this mode.

    In history-free mode, the cache is only needed within a single transaction (conflict resolution consumes its base versions during tpc_vote). afterCompletion now clears the cache, bounding its lifetime to the enclosing transaction.

    Applies to both PGJsonbStorageInstance and the main PGJsonbStorage. No new config knob, no API change, no behavioural regression outside an edge case that was already broken (cross-transaction conflict resolution in history-free mode, where the base version is already gone from PG).

  • CacheWarmer hardening (#65).

    • The MAX(tid) lookup was duplicated in three places with three different error-handling policies. Consolidated behind a single _read_max_tid(conn) helper and a public PGJsonbStorage.current_max_tid() method that logs a warning and returns None on failure. The warmer skips warmup when the method returns None instead of installing a fabricated consensus of 0.

    • SharedLoadCache.set() now returns True on accept and False on rejection. SharedLoadCache.consensus_tid is exposed as a read-only property. Both are used by the warmer’s race-recovery path; existing instance.load / load_multiple callers ignore the new return value and are unaffected.

    • The warmer re-reads shared_cache.consensus_tid after its own poll_advance and uses that value as polled_tid for the subsequent set() loop — this fixes a startup race where a concurrent instance poll could advance consensus past the warmer’s sampled TID and cause every warmup write to be silently rejected. A WARNING is now logged when the entire warmup was rejected despite a non-empty result set.

    • Added a regression test that pins PGJsonbStorage._finish (direct-use write path) advancing shared consensus and invalidating changed zoids.

1.11.1#

  • Fix PK-index deadlock in CacheWarmer._flush between concurrent Waitress workers. Two workers with overlapping pending sets used to deadlock on the cache_warm_stats primary-key index when their INSERT ... ON CONFLICT DO UPDATE acquired row locks in opposing orders (set iteration order is not stable across processes). Zoids are now sorted before the upsert, giving a deterministic lock acquisition order.

1.11.0#

  • Implement IStorage.afterCompletion() on PGJsonbStorageInstance so the REPEATABLE READ read-snapshot transaction is committed at request end (after every transaction.commit/abort and on Connection.close()). Previously the read tx persisted across request boundaries until the connection was reused, leaving idle in transaction sessions with live virtualxids that blocked CREATE INDEX CONCURRENTLY for minutes-to-hours under load. Idempotent and exception-swallowing — a connection killed externally is logged and rebuilt on next use. Closes bluedynamics/plone-pgcatalog#118.

  • Set idle_in_transaction_session_timeout (default 60_000 ms, env-overridable via ZODB_PGJSONB_IDLE_IN_XACT_TIMEOUT_MS) on every connection from the instance pool. Defense in depth for any future leak path that bypasses afterCompletion (e.g. SIGKILL-ed worker, buggy plugin). Set to 0 to disable.

  • Serialize startup DDL across replicas via session-level PostgreSQL advisory lock. New zodb_pgjsonb.startup_locks module exposes startup_ddl_lock(dsn) context manager. _apply_pending_ddl wraps its body in the lock and requeues pending work on timeout. Lock wait timeout is 15 minutes by default, overridable via ZODB_PGJSONB_DDL_LOCK_TIMEOUT. Closes bluedynamics/plone-pgcatalog#108 (credit: @davisagli).

1.10.4#

  • Apply deferred processor DDL on first read, not just first write (#105). poll_invalidations() now calls _apply_pending_ddl() before starting the REPEATABLE READ snapshot. Fixes UndefinedColumn crash when a read-only request hits a column added by a state processor (e.g. meta) before any write transaction has occurred.

1.10.3#

  • Fix startup self-deadlock when processor DDL blocks against own REPEATABLE READ snapshot (#100). _apply_processor_ddl() now always defers DDL to the first write transaction (tpc_begin()), when the read snapshot has been committed and ACCESS SHARE released.

  • New defer_startup_action(callable, name) API for plugins to defer arbitrary startup work (e.g. index creation) to the first write transaction. Callables receive the DSN as argument.

1.10.2#

  • Fix instance load() not using prefetch refs expression (#40). PGJsonbStorageInstance.load() now uses the registered prefetch_refs_expr and prefetches referenced objects into the load cache — previously this only worked in direct-use mode.

  • Fix cache warmer _flush() atomicity. The decay UPDATE, score UPSERT, and low-score DELETE are now wrapped in an explicit BEGIN/COMMIT transaction, preventing inconsistent scores if the process is killed mid-flush.

  • Apply deferred processor DDL in direct-use storage path. Previously _apply_pending_ddl() only fired from instance tpc_begin(); main storage _begin() now also applies deferred schema changes.

  • BackgroundBlobSink: prune completed futures every 256 submits to bound memory during large migrations with many blobs.

  • Cache warmer: use threading.Event instead of bare bool for the warming-done flag (future-proofs for free-threaded Python / PEP 703).

  • Internal: extract storage.py (3700→1535 lines) into focused modules (batch.py, conflict.py, undo.py, serialization.py, migration.py, stats.py, instance.py). Deduplicate loadSerial between main storage and instance. Migration pipeline moved to CopyTransactionsMixin. Reduce _copyTransactionsFrom_parallel cyclomatic complexity from 57→17 by extracting WatermarkTracker, ProgressTracker, _create_blob_sink.

1.10.1#

  • Cache warmer: use actual AVG(state_size) from DB instead of hardcoded 2KB estimate (#51). With median object size of 163B, the warmer now targets ~40k objects instead of 3200 (at 64MB cache).

1.10.0#

  • Learning cache warmer with L2 warm cache (#48). Records which objects are loaded first after each startup, persists scores to PG with exponential decay, and pre-loads the highest-scored objects into a shared L2 warm cache on the next startup. Expected cold-start latency improvement: 5-14s → ~1-2s. Configurable via cache-warm-pct (default 10%) and cache-warm-decay (default 0.8). Set cache-warm-pct to 0 to disable.

1.9.6#

  • Fix startup DDL blocking rolling updates (#96). ALTER TABLE SET COMPRESSION and CREATE INDEX IF NOT EXISTS now use lock_timeout = '5s'. If blocked by old pods’ REPEATABLE READ connections, the DDL is silently skipped and retried on next startup.

1.9.5#

  • Fix new_oid() leaving main connection “idle in transaction” indefinitely (#45). The main storage connection now uses autocommit=True after schema init, so SELECTs (new_oid, load, history, stats) never open implicit transactions. Write paths that need transactions continue using explicit BEGIN/COMMIT.

1.9.4#

  • Enable LZ4 TOAST compression on JSONB and BYTEA columns (PG 14+). LZ4 decompresses ~10x faster than the default pglz at similar compression ratios. Applied automatically during schema init. Only affects new writes — existing rows keep pglz until rewritten. Silently skipped on PG < 14.

1.9.3#

  • Fix ZODB undo nullifying catalog columns (plone-pgcatalog #30). undo() now calls _process_state() on restored entries (same as restore() does), so catalog columns (path, idx, searchable_text, allowed_roles, etc.) are recomputed from the restored state instead of being left as NULL.

1.9.2#

  • Fix refs prefetch over-fetching (#40). Replace hardcoded class_mod blacklist with a pluggable SQL expression via register_prefetch_refs_expr(). The expression is included in the load() query as a conditional refs column. When None (default), no prefetch occurs — same behavior as pre-v1.9.0. Applications register their own expression (e.g. pgcatalog uses CASE WHEN idx IS NOT NULL THEN refs END to prefetch only for cataloged content objects).

1.9.1#

  • Fix refs prefetch over-fetching (#40). Only prefetch refs for content-type objects, not internal ZODB structures (PersistentMapping, OOBTree, etc.) whose refs cascade into massive over-fetching. v1.9.0 made cold-start 40-84% slower; this fix restores the benefit by limiting prefetch to objects that actually have useful sub-objects (annotations, workflows, etc.).

1.9.0#

  • Prefetch referenced objects on load() (#38). When an object is loaded, its refs column (annotations, sub-mappings, OOBTrees) is used to prefetch all directly referenced objects in a single load_multiple() call. Turns N+1 individual loads into 1 batch load. Default-on, no configuration needed. Cold-start page loads reduced by ~85% fewer roundtrips per object.

1.8.1#

  • Fix packer DELETE using NOT IN anti-join (#35). Replaced all NOT IN (SELECT zoid FROM reachable_oids) with NOT EXISTS (SELECT 1 FROM reachable_oids r WHERE r.zoid = ...). NOT IN builds a hash of millions of rows; NOT EXISTS uses an indexed anti-join that short-circuits. Pack on 4.4M objects went from 48+ minutes (incomplete) to expected minutes. Also added progress logging per phase.

1.8.0#

  • Add load_multiple(oids) method to PGJsonbStorageInstance for batch object loading (#34). Loads multiple objects in a single SELECT WHERE zoid = ANY() query instead of individual roundtrips. Checks _load_cache first, only queries misses. Caches all results.

1.7.3#

  • Fix: add lock_timeout = '30s' to deferred DDL application (_apply_pending_ddl). Previously, deferred DDL could block indefinitely at tpc_begin() during rolling deployments when old REPEATABLE READ sessions held ACCESS SHARE locks.

1.7.2#

  • Fix OID collisions across multiple application pods (#31). Replace BaseStorage’s in-memory OID counter with a PostgreSQL sequence (zoid_seq). new_oid() now calls nextval('zoid_seq'), guaranteeing uniqueness across all processes sharing the same database. The sequence is created automatically during schema initialization and synchronized with existing data on restart.

1.7.1#

  • Smooth ETA during parallel migration using exponential moving average instead of single-window rate. Eliminates wild jumps caused by S3 retries or variable transaction sizes.

1.7.0#

  • Add blob_mode parameter to copyTransactionsFrom() for decoupled S3 blob uploads during parallel migration. Three modes: "inline" (default, current behavior), "background" (S3 uploads in a background thread pool, PG writes continue independently), and "deferred:<manifest_path>" (write manifest file for later upload). Background mode significantly improves throughput when S3 latency is the bottleneck.

1.6.1#

  • Fix _stage_blob PermissionError when hard-linking blobs owned by another user (e.g. Docker’s uid 500). fs.protected_hardlinks blocks the link even on the same filesystem. Now falls back to using the source path directly.

1.6.0#

  • Support incremental parallel imports with watermark-based resume. copyTransactionsFrom(source, workers=N, start_tid=X) now accepts a start_tid parameter. A migration_watermark table tracks contiguous commit progress so interrupted parallel imports can resume safely without losing transactions to out-of-order worker commits. Fixes #24.

1.5.6#

  • Fix _stage_blob crash when source blobs and temp directory are on different filesystems. The hard-link fallback tried to unlink an already-removed placeholder, causing FileNotFoundError during copyTransactionsFrom with --workers.

1.5.5#

  • Parallel S3 blob uploads in _batch_write_blobs(). When a transaction contains multiple blobs destined for S3, they are now uploaded concurrently using a thread pool (up to 8 workers). Single-blob transactions skip the thread pool to avoid overhead. This significantly speeds up bulk operations like zodb-convert imports with many blobs per transaction.

1.5.4#

  • Add PGTestDB class in zodb_pgjsonb.testing for test database management with stackable snapshot/restore (COPY BINARY protocol). Mirrors DemoStorage’s push/pop semantics for per-test isolation.

  • Consolidate test fixtures into shared conftest.py (clean_db, storage, db, hp_storage, hp_db), removing ~210 lines of duplicated boilerplate.

1.5.3#

  • Add composite index (tid, zoid) on object_state to speed up poll_invalidations() queries. Previously required a full sequential scan on large tables. Existing databases get the index automatically on next startup. Fixes #19.

1.5.2#

  • Add blob storage statistics API: get_blob_stats() and get_blob_histogram() methods on PGJsonbStorage for querying blob count, size, per-tier (PG/S3) breakdown, and logarithmic size distribution.

  • Add Zope Control Panel integration: “Blob Storage” tab appears in Control Panel -> Database -> [dbname] when the storage is a PGJsonbStorage. Shows blob statistics, tier breakdown, and size distribution histogram. Previously in plone-pgcatalog.

1.5.1#

  • Parallel copy performance and observability improvements:

    • Backpressure via BoundedSemaphore(workers * 2) prevents unbounded blob temp file accumulation (critical for large blob databases).

    • Hard-link blob staging on same filesystem (instant, zero extra disk); cross-device: read source directly without copying.

    • Progress logging (every 10s) with elapsed time, read/written transaction counts, OID and blob counts, percentage, and ETA.

    • Abort immediately on worker errors with clear message (e.g. non-empty target database) instead of silently accumulating failures.

    • Log WARNING per missing source blob (with oid/tid) and ERROR summary at end.

    • ETA based on recent-window throughput instead of overall average (much more accurate, especially when early transactions are empty after pack).

    • Non-blocking drain loop with periodic “Still waiting” log instead of silent hang during executor.shutdown.

1.5.0#

  • Add parallel copyTransactionsFrom(source, workers=N) for faster migrations. Multiple worker threads write to PostgreSQL concurrently, bypassing the advisory lock serialization. The main thread reads from the source storage, decodes pickles, and orchestrates OID-level dependency tracking to guarantee correct write ordering. Default workers=1 preserves existing sequential behavior.

1.4.1#

  • Fix FileNotFoundError when blob-temp-dir is configured but the directory does not exist yet. The storage now auto-creates the directory on startup.

1.4.0#

  • Reduce default blob-threshold from 1MB to 100KB to reduce WAL pollution when S3 tiering is enabled.

  • Add history mode switching support: convert_to_history_free() and convert_to_history_preserving() methods on PGJsonbStorage for converting between history-free and history-preserving modes. HP→HF conversion drops history tables, cleans old blob versions, and removes orphaned transaction log entries. Startup warning logged when HP tables exist in HF mode.

  • Add progress logging to copyTransactionsFrom() for zodbconvert usage: per-transaction TID, record count, throughput (MB/s), and completion summary. [mamico] [#16]

  • Fix schema init blocking concurrent instances: skip DDL when core tables already exist, avoiding ACCESS EXCLUSIVE locks that conflict with REPEATABLE READ snapshots held by other processes. [#15]

1.3.0#

  • Direct JSON string decode path: Use decode_zodb_record_for_pg_json() from zodb-json-codec 1.4.0 — the entire pickle-to-JSON pipeline now runs in Rust with the GIL released. No intermediate Python dicts are created for the store path. 1.3x faster end-to-end on real-world data. [#14]

  • History-preserving optimization: Changed object_history from full dual-write to copy-before-overwrite model — only previous versions are archived before overwrite. Eliminated redundant blob_history table. Batch writes up to 33% faster, loadBefore 15% faster, undo 14% faster, ~50% less storage overhead for HP mode. [#13]

  • Require zodb-json-codec>=1.4.0.

1.2.2#

  • Fix blob migration: override copyTransactionsFrom for blob-aware copying. BaseStorage.copyTransactionsFrom only calls restore() and silently drops blob data during zodbconvert from FileStorage+BlobStorage. The new override detects blobs via is_blob_record() and uses restoreBlob() to migrate them. Blob files are copied (not moved) to preserve source storage integrity. [mamico, jensens] [#9, #10]

1.2.1#

Security review fixes (addresses #7):

  • PG-H1: Strengthen ExtraColumn / register_state_processor docstrings with security guidance for SQL expressions.

  • PG-H2: Replace unrestricted zodb_loads with _RestrictedUnpickler for legacy pickle extension data (blocks arbitrary code execution).

  • PG-H3: Fix DSN password masking regex to handle quoted passwords.

  • PG-M1: Add SECURITY NOTE to _batch_write_objects() about dynamic SQL from ExtraColumn.value_expr.

  • PG-M2: Expand _new_tid() docstring about advisory lock serialization.

  • PG-M3: Add pool_timeout parameter (default 30s) to prevent unbounded connection waits; exposed in ZConfig as pool-timeout.

  • PG-M4: Remove errors="surrogatepass" from _unsanitize_from_pg() to reject invalid surrogate bytes instead of silently accepting them.

  • PG-L1: Document PostgreSQL recursive CTE depth behavior in packer.

  • PG-L2: Add DSN format validation in ZConfig factory (config.py).

1.2.0#

  • Add finalize(cursor) hook to state processor protocol [#5] plone-pgcatalog needs to apply partial JSONB merges (idx || patch) for lightweight reindex operations (e.g. reindexObjectSecurit) without full ZODB serialization. The finalize(cursor) hook provides the extension point.

1.1.0#

  • Added pg_connection read-only property to PGJsonbStorageInstance, exposing the underlying psycopg connection for read queries that need to share the same REPEATABLE READ snapshot as ZODB loads (e.g. catalog queries in plone-pgcatalog).

  • Security hardening: validate ExtraColumn.name against SQL identifier pattern to prevent injection via state processor plugins.

  • Mask credentials in DSN before debug logging (_mask_dsn()).

  • Restrict blob file permissions to 0o600 (owner-only, was 0o644).

  • Narrow bare except Exception blocks to specific exception types.

1.0.1#

  • Fix FileNotFoundError when using blobs with transaction.savepoint() (e.g. plone.exportimport content import). Blob files are now staged to a stable location before the caller can delete them. [#1]

1.0.0#

Added#

  • State processor plugin system: Register processors that extract extra column data from object state during writes. This enables downstream packages (e.g. plone-pgcatalog) to write supplementary columns alongside the object state in a single atomic INSERT...ON CONFLICT statement.

    New public API:

    • ExtraColumn(name, value_expr, update_expr=None) dataclass — declares an extra column for object_state.

    • PGJsonbStorage.register_state_processor(processor) — registers a processor whose process(zoid, class_mod, class_name, state) method can pop keys from the state dict and return extra column data.

    Processors are called in store() after pickle-to-JSON decoding. Extra columns are included in the pipelined executemany() batch write during tpc_vote(), keeping everything in the same PostgreSQL transaction for full atomicity.

  • State processor DDL via get_schema_sql(): Processors can now optionally provide a get_schema_sql() method returning DDL statements (e.g. ALTER TABLE, CREATE INDEX). The DDL is applied using the storage’s own connection during register_state_processor(), avoiding REPEATABLE READ lock conflicts with pool connections.

Optimized#

  • Batch conflict detection: Conflict checks are now batched into a single SELECT ... WHERE zoid = ANY(...) query in tpc_vote() instead of individual per-object queries in store(). Eliminates N-1 SQL round trips per transaction while holding the advisory lock.

  • Prepared statements: Added prepare=True to hot-path queries (load, loadSerial, loadBefore) for faster repeated execution.

  • Removed GIN index on state JSONB: The jsonb_path_ops GIN index indexed every key-path and value, causing significant write amplification. With plone-pgcatalog providing dedicated query columns, direct state JSONB queries are no longer needed in production.

1.0.0a1#

Initial feature-complete release.

Added#

  • Core IStorage: Full two-phase commit (2PC) with ZODB.DB integration.

  • IMVCCStorage: Per-connection MVCC instances with REPEATABLE READ snapshot isolation and advisory lock TID serialization.

  • IBlobStorage: Blob support using PostgreSQL bytea with deterministic filenames for Blob.committed() compatibility.

  • S3 tiered blob storage: Configurable threshold to offload large blobs to S3 while keeping small blobs in PostgreSQL.

  • History-preserving mode: Dual-write to object_state and object_history with full IStorageUndoable support (undo, undoLog, undoInfo).

  • IStorageIteration / IStorageRestoreable: Transaction and record iteration for backup/restore tooling.

  • Conflict resolution: Inherited tryToResolveConflict with serial cache for correct loadSerial during conflict resolution.

  • ZConfig integration: %import zodb_pgjsonb with <pgjsonb> section for Zope/Plone deployments.

  • PG null-byte sanitization: Automatic \u0000 handling with @ns marker for JSONB compatibility.

Optimized#

  • LRU load cache: Configurable in-memory cache for load() calls.

  • Connection pooling: psycopg3 connection pool with configurable size.

  • Batch SQL writes: executemany() for pipelined store performance during tpc_vote().

  • Single-pass store: decode_zodb_record_for_pg for combined class extraction and state decoding.

  • Reduced default cache: 16 MB per instance (down from 64 MB).