Changelog

Unreleased

Fixed

• The query result cache no longer grows each Zope process by up to gigabytes. Brains wrote idx and meta into the row dicts they were built from, and those were the cache’s own rows: on the first metadata access a cached result kept the metadata of every object in it, tens of thousands for a site search, until the next catalog write. The cache now stores and hands out copies, so it holds only zoid and path. It is also bounded by rows, not only by entries: PGCATALOG_QUERY_CACHE_MAX_ROWS (default 100 000, roughly 32 MiB) over all entries, and results over PGCATALOG_QUERY_CACHE_MAX_ENTRY_ROWS (default 10 000) are not cached. The least recently used entry is evicted first, instead of the cheapest, which kept the large result sets in. The first metadata access now loads idx and meta only for the block of PGCATALOG_IDX_BATCH brains (default 500) holding the accessed brain, not for the whole result set, so a search page showing 20 of 27 000 hits no longer loads the metadata of all of them. The ZMI cache statistics show the cached rows and the results skipped as too large. Workaround on releases before this fix: PGCATALOG_QUERY_CACHE_SIZE=0. (#280)

• Undo on a history-preserving storage no longer leaves catalog data of the undone transaction behind. The restored objects kept the catalog columns of the undone transaction, and rows the undone transaction had changed without writing them were not touched at all: after undoing a sharing change, the folder’s content stayed findable for the user whose access was revoked; after undoing a move, the catalog showed the moved paths; after undoing a delete, the restored content stayed invisible. Now the undo itself hides the affected rows, in its own transaction: restored content loses all catalog data, and the rows below a deleted object, and below a restored one whose undone change could reach them, lose their allowedRolesAndUsers data, so no restricted search finds them. The latter is decided by comparing what the undone transaction wrote with what the undo writes back: any difference but modification_date and cmf_uid, a class change or a move counts. Undoing an add, a delete or a move therefore leaves the rest of the container’s content alone, while an undone edit of a container, even of its title, hides the rows below it until the follow-up is done. The known gaps (local roles kept outside the object by add-ons, changes on the Zope root) are listed in the how-to “Rebuild or reindex the catalog”. A follow-up recomputes the hidden rows from the restored content: at the end of the request that did the undo (up to PGCATALOG_UNDO_FOLLOWUP_LIMIT rows, default 2000), from the ZMI Catalog tab, with the new plone.pgcatalog.undo.process_queue(), and as the first step of maintenance.repair_uncataloged(). Rows the follow-up has not reached stay hidden; Zope warns at startup while any are queued. Requires zodb-pgjsonb >= 1.18.0 (its new undone() hook with both revisions of every undone object); with an older one, Zope warns at startup on a history-preserving storage. Sites that undid transactions on a history-preserving storage before this release have stale rows that nothing can detect: run one Clear and Rebuild after upgrading. #265

Tests

• New history-preserving variant of the PG test layer, plone.pgcatalog.testing.PGCATALOG_PG_HP_FIXTURE, used by test_undo_integration.py. Its snapshot includes object_history, pack_state and the undo queue, so an undo cannot pick a revision another test left behind. #265

1.0.0rc8 (2026-10-13)

Fixed

• A cross-container move (cut and paste into another folder) left the descendants of the moved folder with the allowedRolesAndUsers of their old location. Search and listings could show title, description and metadata to users who had lost access, or hide content that should be visible. The security reindex after a move looked the descendants up by their old path, which no longer exists, and skipped them silently. It now resolves them relative to the moved folder, checks that it found the right object and logs a warning for every descendant it cannot resolve. Present since 1.0.0b21. Upgrade profile version 6 reindexes allowedRolesAndUsers of all cataloged objects once to repair existing sites; it only rewrites the security index values and does not touch other index data or extracted text. Also available as maintenance.reindex_security. #271

Tests

• The move and rename optimization is now active in the test layers. plone.testing stacks the global component registries, and OFS registers its move event dispatchers in a lower level than the one install_move_handlers() worked on. The originals stayed active next to the wrappers, so every descendant was uncataloged, reindexed and written on rename and cut/paste, and the move tests passed through the slow path. The layers now replace the dispatchers in the registry that holds them and restore them on teardown; install_move_handlers() no longer registers the wrappers twice. New tests assert that descendants are not written on rename and cross-container move. #270

• The PG test layer no longer leaks one pool connection per test. Its testSetUp dropped the ZODB connections of the previous test with pool.clear(), which does not release them, so each one kept its zodb-pgjsonb instance and a connection of the instance pool (10 slots) until the layer was torn down. A class with ten committing tests hung for 30 s and failed with PoolTimeout. The split of TestRepairHardening that worked around it is undone. testTearDown now also returns the request-scoped connection, which IPubEnd returns in production but which stayed checked out in tests from the first cut and paste on. #261

• The PG test layer is set up once for test_pg_integration.py instead of once per test class. The classes now carry the layer attribute that zope.pytestlayer needs to keep a layer between classes; before, every class built a new test database and Plone site. The full test suite runs in about half the time (locally 4:58 → 2:22). As a side effect, zope.pytestlayer now runs these tests last.

• The same for PGCATALOG_INTEGRATION_TESTING: the 25 test classes that use it (in test_move_integration.py, test_plone_integration.py, test_observability.py and test_query_modifier.py) derive from the new plone.pgcatalog.testing.UsesIntegrationLayer and share one layer setup instead of one per class. Full suite locally 2:22 → 1:50.

• CI turns off fsync and synchronous_commit in both PostgreSQL service containers before the tests run. The databases are thrown away after the job, so durability buys nothing there.

• The development Compose setup (development/) does the same with a one-shot postgres-tuning service, which works with both the standard and the BM25 image. The full test suite runs about 28 % faster locally (100 s → 72 s). The price: an OS crash or power loss can corrupt the databases in that volume. example/ and tryout/ are unchanged.

1.0.0rc7 (2026-10-12)

Fixed

• Partial catalog updates are no longer lost when the transaction writes no ZODB object. reindexIndex() (and the ZMI “Reindex” action on the Indexes tab) reported the reindexed objects and changed nothing, and so did reindexObject(idxs=...) in scripts that wrote nothing else: the updates are applied when the storage votes, and without a write it never does. The pending data manager now applies them itself in that case, with the same SQL. #249

• Editing an item no longer breaks the order of its folder. Every reindex overwrote the item’s stored getObjPositionInParent rank with the dense position from the indexer, while folders that were reordered keep sparse ranks, so the item jumped to a wrong place in sort_on getObjPositionInParent listings and folder_contents drag & drop aborted with “Client/server ordering mismatch” until the next add, delete or move in that folder. Folders with a prepend ordering broke on every save of a new item. Every catalog write now schedules the folder’s rank sync, which rewrites only rows that are out of order. This also covers content recataloged by repair_uncataloged() and requeue_lost_extractions(). The upgrade step to profile version 5 resyncs all folders once. #245

• A request-scoped PostgreSQL connection that the server closed (database failover, pg_terminate_backend, idle timeout) is now returned to the pool. It used to be dropped without putconn(), which leaked one slot of zodb-pgjsonb’s instance pool per occurrence until every getconn() timed out. #246

• maintenance.requeue_lost_extractions() only recatalogs objects whose current type passes PGCATALOG_TIKA_CONTENT_TYPES. Images and other types excluded by a narrowed allowlist were rewritten on every run without anything being queued, because their old done jobs with empty text look like lost text. The skipped count is logged. #252

• Log an ERROR at startup when the database’s LC_CTYPE is C or POSIX. PostgreSQL’s text search parser then glues non-ASCII punctuation such as », – or … to the adjacent word, and full-text queries silently return incomplete results. initdb and CloudNativePG default to this when no locale is set. Startup continues. The production how-to now documents the requirement, the CloudNativePG setting and how to move an affected database. #228

• Planner statistics follow a bulk recatalog. clearFindAndRebuild() (and with it the catalog install and refreshCatalog(clear=1)) now commits its last batch and runs ANALYZE object_state, through the new maintenance.analyze_object_state(), which repair_uncataloged() uses too. At startup, object_state gets autovacuum_analyze_scale_factor = 0.01 and autovacuum_analyze_threshold = 5000, unless an operator set autovacuum_analyze_* options on it; the PostgreSQL default of 10 % let a full recatalog of 140k objects go unanalyzed, and search hung because the planner ignored the GIN indexes. #224

• The keyword vocabulary (plone.app.vocabularies.Keywords, tag and Subject autocomplete, Collection criteria) was empty when plone.subjects_of_navigation_root is on. Plone intersects the path index with each keyword via _apply_index, and the special indexes (path, effectiveRange, SearchableText) were returned unwrapped, so _apply_index silently found nothing. They now answer _apply_index from PostgreSQL, which fixes project subclasses of KeywordsVocabulary too. pgcatalog also overrides the Keywords vocabulary to list a section’s keywords in one query instead of one per keyword (PGKeywordsVocabulary; subclass it for other keyword indexes), using the new path argument of PGIndex.uniqueValues(). #234

Documentation

• New reference page “Overrides and replaced behaviour”: every place where plone.pgcatalog replaces Plone, Zope or add-on behaviour (event handlers, ZCML overrides, replaced persistent objects, index proxies) and the hooks it adds.

Tests

• Regression tests for renaming and moving (zodb-pgjsonb#122): folder_contents’ rename action on a folder and on a default page, cutting and pasting a folder into a folder that was never ordered, and moving an item between folders. Each checks every row involved, including source and target containers. With stock Plone these paths reindex everything they write, so they pass on zodb-pgjsonb 1.16.x too and guard the scenario rather than reproduce it.

1.0.0rc6 (2026-10-09)

Changed

• Replace MinIO with Garage in the Docker Compose setups (example/, development/, tryout/) and the docs: MinIO is no longer maintained and its images are no longer published on Docker Hub. Garage runs in single-node mode and creates the zodb-blobs bucket and its access key on startup, so the createbucket bootstrap container is gone.

Fixed

• Recatalog content that zodb-pgjsonb before the fix for bluedynamics/zodb-pgjsonb#120 left without catalog data. Any write of a cataloged object without a full reindex NULLed its catalog columns, for example opening the edit form (edit lock), the sharing tab, or a write followed by a partial reindex. The object stayed in the ZODB but vanished from listings and search, and partial reindexes skipped it. New maintenance.repair_uncataloged() walks the site without clearing the catalog and recatalogs only content whose row lacks catalog data. The upgrade step to profile version 4 runs it once; the how-to “Rebuild or reindex the catalog” shows how to run it from the command line with a dry run. Requires zodb-pgjsonb >= 1.17.0, so repaired content cannot be wiped again. The repair also checks the installed version at runtime, retries a batch once on a write conflict, and runs ANALYZE at the end; the upgrade step skips the site walk when no row can be damaged. With Tika enabled, the repair enqueues extraction for every repaired file, so the worker’s allowlist and the Tika image pin must be in place before the upgrade. #244

• A full reindex of a file (title edit, workflow transition, clearFindAndRebuild()) no longer drops its Tika-extracted text: the finished extraction job is queued again. The worker only finishes jobs that are still in processing, so a reindex during extraction cannot leave a job marked done without its text, and a database error while merging the text no longer leaves the job stuck in processing. Stopgap until #247. #244

• New maintenance.requeue_lost_extractions() finds files whose extracted text was already lost and queues their extraction again. Run it after the upgrade, once the worker’s allowlist and the Tika image pin are in place, and with Zope’s environment: it refuses to run without PGCATALOG_TIKA_URL. #244

• The catalog’s site walk (clearFindAndRebuild(), the repair) no longer takes quadratic time in the number of objects.

1.0.0rc5 (2026-10-05)

Added

• Extract via Tika’s /rmeta/text endpoint instead of /tika, and harvest a whitelist of Dublin Core metadata fields into searchable_text alongside the body text. Configurable with PGCATALOG_TIKA_METADATA_FIELDS, default dc:title,dc:description,dc:subject,dc:creator,meta:keyword, and TIKA_WORKER_MAX_EMBEDDED_RESOURCES, default 1000.

  Two measured reasons. /tika returns body text only, so an image without OCR yielded zero characters even when it carried an EXIF caption; dc:description now puts that caption into the index at metadata-only cost. And on Tika 4 /tika returns Markdown, so a link’s target was being tokenised into searchable_text: the sentence Siehe [die Akte](https://example.org/akte). produced five lexemes including example.org and /akte)., where the same sentence out of /rmeta/text produces two. tk:content carries no Markdown.

  The content key differs between Tika majors, tk:content on 4.x and X-TIKA:content on 3.x, and both are read so one worker build serves both. Word order changes relative to /tika, which interleaved each embedded file’s name with its content, so phrase proximity over searchable_text shifts; term content does not. #222

Fixed

• Normalise the MIME type before the Tika extraction gate. _should_extract compared the catalogued mime_type against the configured set by exact string, so text/plain; charset=utf-8 and APPLICATION/PDF never matched although Plone can hold both, and those blobs were silently never queued for extraction. The configured entries are normalised too, so a PGCATALOG_TIKA_CONTENT_TYPES value of Application/PDF now matches as well. Parameters are not simply stripped: Tika’s own supported-type set contains parameterised entries such as audio/ogg; codecs=opus, so the full normalised value is tried first and the bare type second, which keeps an explicit parameterised configuration entry meaningful. #222

• Defer Tika extraction jobs with backoff instead of failing them. Three attempts inside one second cannot bridge a Tika restart, so a pod restart or a deploy turned transient [Errno 111] Connection refused and [Errno 1] Operation not permitted errors into failed rows that needed a manual SQL reset: 102 of them in one reported run, 1703 in another. Connection-level failures, meaning httpx.ConnectError, httpx.ConnectTimeout and httpx.RemoteProtocolError, now re-queue on a 5/30/120 second ladder without spending one of the job’s three attempts, since a server that was not there had no opinion about the document. Every other exception keeps its previous meaning and still counts an attempt. Two new columns, not_before and deferrals, are added idempotently, and idx_teq_pending is recreated to lead on not_before because the dequeue now filters on it. Note that trg_notify_extraction fires on insert only, so a deferred job is picked up by the polling fallback rather than by a notification. #222

• Re-check the Tika content-type allowlist when the worker claims a job, not only when the job is queued. PGCATALOG_TIKA_CONTENT_TYPES used to be enforced at enqueue alone, so narrowing it left every already-queued row of the excluded types to be extracted anyway, and any requeue bypassed it entirely. On one production site 1144 of 1708 failed rows were images queued before images were excluded, so the obvious UPDATE ... SET status = 'pending' WHERE status = 'failed' would have fed them all back to Tika. A row whose type is not allowed is now marked skipped with the reason skipped: content-type-not-allowed: <type> before its blob is fetched. skipped is terminal and is not an error: a reset of stuck work should target failed only. Existing rows are not swept or migrated; only rows the worker claims are checked, and it never claims failed rows. The allowlist now has a single definition shared by the enqueue side and the worker, so the two cannot disagree about what is allowed or what “unset” means.

  Deployment note: the standalone pgcatalog-tika-worker reads the same PGCATALOG_TIKA_CONTENT_TYPES variable as the Zope processes and must be given the same value. If it is unset the worker falls back to the default allowlist, which includes image types, and logs a warning at startup saying so. The in-process worker inherits Zope’s environment and needs no change. #235

• PGCATALOG_TIKA_METADATA_FIELDS is now honoured by the in-process extraction worker too. It was read only by the standalone worker’s main(), so a site running the worker inside Zope with PGCATALOG_TIKA_INPROCESS always harvested the default metadata fields. The worker’s constructor now reads it, as it already did for PGCATALOG_TIKA_CONTENT_TYPES, which restores the package’s convention: PGCATALOG_* settings apply to Zope and the in-process worker, TIKA_WORKER_* settings to the standalone worker.

Documentation

• Add the design and implementation plan for bounded Tika renditions (#222), with the evidence they rest on rather than only the conclusions: real object_state fixtures under tests/fixtures/state/, real /parsers/details payloads from Tika 3.2.3 and 4.1.0 under tests/fixtures/tika/, and an OCR recall benchmark in benchmarks/tika_ocr_downscale.md.

  Three measured findings drove the design and are worth reading before changing any of it. Without OCR, Tika never decodes image pixels, so a 165 megapixel JPEG costs 0.2 s and tens of MiB at container limits from 512 MiB to 2 GiB; a byte-size guard is therefore the wrong instrument, since such a file weighs about 5 MB. Downscaling before OCR costs 3 percentage points of recall at a 4000 px cap, and recall is not monotonic in resolution, so raising a cap is not reliably an improvement. And OCR availability cannot be read off /parsers/details on Tika 4.x, where the stock and -full images report semantically identical payloads although only one of them OCRs.

  No production code changes.

• Document the Tika extraction behaviour that shipped for #222 and #235, which until now was described only in this changelog. The reference gains PGCATALOG_TIKA_METADATA_FIELDS, TIKA_WORKER_MAX_EMBEDDED_RESOURCES, the queue’s not_before and deferrals columns, and a table of queue status values including skipped. The how-to covers capping the Java heap below the container limit, pinning the Tika image by digest, choosing which metadata is indexed, giving the standalone worker the same allowlist as Zope, and recovering failed extractions safely. The explanation records why the worker uses /rmeta/text, why a missing Tika does not cost an attempt, and why the allowlist is checked twice.

  Corrects several statements that had become wrong. The image-indexing explanation claimed that Tika includes Tesseract and that photos become searchable by their visible text, which holds only for the -full image. The data flow still described PUT /tika. The queue schema listed UNIQUE(zoid, tid) where the table has UNIQUE(blob_zoid, tid), omitted the blob_zoid column entirely, and gave a stale index definition. Also fixes a broken anchor to the OCR section of the how-to.

1.0.0rc4 (2026-10-02)

Fixed

• searchResults() now honours AccessInactivePortalContent granted through a local or group role, not only site-wide. The permission was checked on the catalog tool alone, and that tool sits directly below the portal root, so a role assigned deeper never reached it. In practice that means Editor through a group on a Lineage subsite or a per-section editor group, and Owner, which Plone grants the permission to by default. For those users inactive content (expired, or not yet effective) silently vanished from folder contents, listings, collections and @@search alike; there was no error to go on, and only globally assigned roles still worked. Products.CMFPlone.CatalogTool falls back to a path-scoped check in exactly this case, and its allow_inactive() is now ported with its conservative semantics: the site-wide check first, then the permission on every object named by the query’s path, where one disallowed path denies the whole query, untraversable paths are ignored and a query without a path denies. Blank paths and path lists longer than the query builder’s limit of 100 deny too, rather than being traversed: the former add no path filter at all, the latter make the query fail later anyway. #232

• An explicit show_inactive=False now suppresses inactive content even for users who hold the permission, as CMFPlone documents it. The key was read with a False default, which cannot tell “not passed” from an explicit False, and the permission check then overrode it, so the documented escape hatch for admins did nothing. searchResults() also stops mutating the query dict it was handed: it merged the keyword arguments into that dict and popped show_inactive back out of it, so a caller reusing one dict for a second query silently got different filtering. #232

1.0.0rc3 (2026-10-02)

Fixed

• A brain now returns Missing.Value instead of None for a registered metadata column that is absent from idx, matching ZCatalog’s Catalog.recordify(). Rendering a Collection crashed with TypeError: can only concatenate str (not "NoneType") to str in plone.app.querystring’s results.pt, which evaluates 'state-' + item.review_state(). The column is legitimately absent for content whose type has an empty workflow chain, and Missing.Missing is built so that templates can call, concatenate and stringify it without checking, which is why ZCatalog never showed this. The silent half of the same bug: str(None) is 'None', so a string:${item/<field>} expression over an absent column rendered the literal text None into the page. Columns stored as JSON null still return None — ZCatalog stores a real None when the indexer returns one — and registered index-only names still return None, since ZCatalog brains never expose those at all. Completes the read side of #81. #230

Internal

• Correct the release process documentation. RELEASE.md claimed that pushing the tag triggers the release workflow and publishes to PyPI, and that pre-release tags go to Test PyPI instead. Neither is true: no trigger in release.yaml matches a tag push, publishing to PyPI happens on the release: published event, and Test PyPI receives a dev version from every successful CI run on main. Following the old text would leave a tag pushed and nothing on PyPI. Also document the Release <version> commit that dates the changelog section, which the process always had but the document never mentioned, and add a table of what each workflow trigger publishes where.

1.0.0rc2 (2026-09-30)

Fixed

• SearchableText no longer falls back to the simple text search configuration when Language is a list. ZCatalog passes Language as a list on multilingual sites (["de", "de-at"], or a record holding one), and stringifying it yielded "['de']", which pgcatalog_lang_to_regconfig mapped to simple. The query side then ran plainto_tsquery('simple', 'staufer') against a vector built with the german configuration ('stauf'), so stemmed body text never matched and only Title/Description hits came back — a site-wide search returned 3 results where ZCTextIndex returned 6. List, tuple and record values are now reduced to their language roots (de-at → de) and deduplicated. #225

• The SearchableText tsquery is now OR-ed across the simple configuration and the query’s language configurations, so a search can reach the whole weighted vector. Title and Description are indexed with simple (weights A and B) while the body text uses the document’s own language configuration (weight D), so a tsquery built for one configuration alone could never match both halves: with a language set, title-only hits were silently missed, and without one, stemmed body hits were. The combined tsquery folds into a single @@ operand and still resolves through one GIN bitmap index scan. #225

• A trailing * on a SearchableText, Title or Description term is honoured again as a ZCTextIndex prefix glob. plainto_tsquery treats the * as punctuation and drops it, so archit* degraded to an exact match on the truncated lexeme 'archit' and returned nothing where ZCTextIndex returned 2630 hits. Live and typeahead searches built for ZCTextIndex send "<term>*", so this affected every add-on following that convention, Plone’s own livesearch among them. Glob queries are now built with to_tsquery and a term:* operand per prefixed term, terms AND-ed as ZCTextIndex ANDs them. Each lexeme is wrapped in SQL quote_literal(), so PostgreSQL performs the escaping and raw input never reaches the tsquery parser — passing user text to to_tsquery directly is a syntax error for something as ordinary as foo bar. Queries without a * keep going through plainto_tsquery, leaving stop word and punctuation handling untouched. A leading * (suffix glob) still has no tsquery equivalent and is ignored. #226

Documentation

• Describe the query-time text search behavior accurately in the full-text search deep dive. The page stated that a query runs a single plainto_tsquery(pgcatalog_lang_to_regconfig(Language), search_text), which no longer holds: the tsquery is OR-ed across simple and the query’s language configurations. Add sections on why the query mixes configurations and on prefix globs, and document list-valued Language and the * glob in the query API reference. #225 #226

1.0.0rc1 (2026-09-15)

Changed

• Raise the trove classifier from “Development Status :: 3 - Alpha” to “Development Status :: 5 - Production/Stable”. After 73 betas hardened by months of production use the package enters the release-candidate phase for 1.0.0.

Documentation

• Document in the install guide that no uninstall profile exists: the install replaces portal_catalog and rebuilds it in PostgreSQL, so reverting would mean restoring the complete previous ZCatalog state. Deferred past 1.0 in #27.

• Fix vale style errors (spaced em dashes, informal spellings) in the resolve-relations-by-zoid how-to, which had never been vale-checked because the hook only runs on changed files.

Internal

• Pin ruff to 0.16.7 in the QA workflow and the ruff-pre-commit rev (was 0.15.0), realigning CI with what plain uvx ruff resolves locally, including the one-time mechanical fallout: ruff 0.16 now formats Python code blocks inside Markdown, reformatting 32 Markdown files (docs, plans, CHANGES). No Python source changes. #217

Fixed

• getObjPositionInParent no longer goes stale after a reordering. Plone never reindexes siblings when a container’s order changes (plone.folder’s GopipIndex is a fake index that ZCatalog resolves at sort time), so the stored snapshot pgcatalog sorts on went stale on every reorder and every delete — most visibly folder_contents showed the old order and its drag & drop aborted with “Client/server ordering mismatch”. A subscriber on IContainerModifiedEvent now snapshots the final ordering once per transaction (beforeCommitHook) and finalize() resyncs the stored values in the same PG transaction as the ZODB commit. Values are maintained as sparse LexoRank-style ranks — only their order is observable (the ZCatalog index is a StubIndex: not queryable, not metadata) — so a prepend or drag & drop rewrites one row and a delete rewrites none, instead of O(n) wide-tuple rewrites per operation in large folders. Sort SQL, indexes (idx_os_navtree included) and the query cache are unchanged, and existing dense positions are valid ranks (no migration). Profile upgrade step 2→3 runs maintenance.resync_gopip() once to heal folders that went stale before the fix. Known gap: IExplicitOrdering.orderObjects fires no event in plone.folder (no Plone UI path calls it). See docs/plans/gopip-resync-on-reorder.md. #216

Internal

• Add a workflow_dispatch trigger to the CI workflow so a run on main can be started manually. Needed after a GitHub Actions outage left the merge-commit CI run stuck in queue with no way to retrigger it (a startup_failure run cannot be retried, and CI previously only ran on push and pull_request events).

• Lower ruff’s C901 max-complexity threshold from 15 to 13 as part of the ecosystem-wide complexity ratchet, refactoring the two functions above the new limit: _enqueue_tika_jobs (15 → 5, wrapper-ref resolution and per-candidate enqueueing become dedicated methods) and importToolset (15 → 5, required-tool filtering, forbidden-tool deletion, and required-tool creation become module helpers). Behavior is unchanged.

• Refactor the three C901 complexity hotspots below the threshold and drop their # noqa: C901 markers (#212): extract_idx (23 → 1) now delegates to phase helpers for index extraction, metadata extraction, and @meta encoding; _process_sort (16 → 4) resolves each key via a dedicated sort-expression helper; _run_search (16 → 4) gains helpers for query-key stripping, change-counter lookup, SQL assembly, query logging, and total count extraction. Behavior is unchanged.

Fixed

• Bump hynek/build-and-inspect-python-package from v2 to v3.0.1. Hatchling now emits Metadata-Version: 2.5, which the Twine bundled in v2 rejects with InvalidDistribution: '2.5' is not a valid metadata version — the release build failed before uploading anything. v3 ships Twine 7, which supports it.

Added

• Add the LICENSE file with the full GNU General Public License v2.0 text. The packaging metadata already declared GPL-2.0-only, but the license text itself was missing from the repository, so GitHub reported “No license” and the terms could not be verified from the source tree alone. Fixes #208.

• Enable ruff’s cyclomatic-complexity check (C901, mccabe) with max-complexity = 15. Three existing hotspots are marked with targeted # noqa: C901 as visible refactor candidates (extract_idx 23, _process_sort 16, _run_search 16).

Documentation

• Add cloud-vinyl and plone.observability to the ecosystem navigation dropdown in the docs.

1.0.0b72 (2026-08-03)

Fixed

• Non-ICatalogAware objects no longer acquire their container’s index values into idx — most damagingly the UID, which made UID lookups ambiguous: uuidToObject could return a subsite’s robots.txt or lineage_registry instead of the subsite (417+ colliding rows on a production DB, the site root’s UID shared by 387 tool/FTI/workflow rows). CMFPlone registers the IndexableObjectWrapper adapter for (ICatalogAware, IPloneCatalogTool) only; wrap_object fell back to the raw acquisition-wrapped object for everything else, so getattr acquired UID, Creator, dates, … from the container. Classic portal_catalog never catalogs such objects, which is why the leak was invisible there. wrap_object now constructs the IndexableObjectWrapper directly when no adapter is registered — its aq_base guard turns acquired attributes into “not applicable” (#81), so only genuinely-owned values are extracted. #205

Added

• maintenance.remove_acquired_uids(conn) — one-shot data repair that strips idx->>'UID' from rows whose pickled state carries no own _plone.uuid attribute (the plone.uuid ownership marker). Run it once on databases written before this fix; alternatively a full clearFindAndRebuild() also heals all leaked idx values. #205

1.0.0b71 (2026-08-03)

Fixed

• b70 regression of #200: resolve_date_bound crashed with TypeError: can't compare offset-naive and offset-aware datetimes when a merged date-range list mixed tz-naive and tz-aware values — the original #200 query kept failing, one layer earlier. A stored querystring value like '2021-10-08 14:55' (no timezone, exactly what the collection widget stores) parses to a timezoneNaive() Zope DateTime whose asdatetime() is naive, while the merged afterToday partner is aware. The min/max ordering now uses a normalization key that treats naive datetimes as UTC (comparison only — the bind parameter stays as converted); the b70 tests missed this because they built DateTimes from strings with explicit timezone suffix. #203

1.0.0b70 (2026-08-03)

Fixed

• Date queries with range='min'/'max' and a list value no longer crash the whole search with InvalidDatetimeFormat (SQLSTATE 22007). plone.app.querystring merges two “after …” (or “before …”) rows on the same date index into {'query': [d1, d2], 'range': 'min'}; pgcatalog bound str(list) as a single timestamptz parameter. Now ZCatalog UnIndex semantics apply: min(values) for 'min', max(values) for 'max' — in the DateIndex, DateRecurringIndex, and FieldIndex range branches. A plain multi-value date query without range now ORs over the values (= ANY(...)) instead of binding the list as one scalar. #200

Changed

• Pin ruff in the QA workflow to the same version as the pre-commit hook (0.15.0). Unpinned uvx ruff picked up 0.16, which newly format-checks Python code blocks in Markdown files and turned CI red on 32 docs files no PR had touched.

1.0.0b69 (2026-07-03)

Added

• Built-in zoid (and oid) query operator: catalog(zoid=[...]) / catalog(oid=[...]) filters the object_state BIGINT primary key directly (the catalog rid is the zoid). It lets callers resolve a set of ZODB object ids to their security-filtered brains without waking the objects — e.g. zc.relation/intids: intid -> KeyReference.oid -> int.from_bytes(...) -> zoid, replacing an object-load N+1 in relation/back-reference resolution. oid accepts raw 8-byte ZODB oids (converted via int.from_bytes, matching extraction.obj_to_zoid); zoid accepts ints or digit strings; both take a scalar or a list. Filters the real primary-key column, so it is index-backed with no extra storage and no reindex, and composes with the normal query and the allowedRolesAndUsers security filter (searchResults). An empty set matches nothing (never the whole table). #197

Fixed

• The Slow Queries tab’s empty-state message now shows the actually-configured threshold (via manage_get_slow_query_threshold()) instead of hard-coding the PGCATALOG_SLOW_QUERY_MS default of 10 ms.

1.0.0b68 (2026-06-30)

Removed

• Drop the project-specific idx_os_cat_events_upcoming partial index. Its WHERE referenced show_in_sidecalendar (an AAF project field, not generic Plone) and it was never used by the planner. The sidecalendar slowness it targeted (#131) was resolved generically once path became a typed column (#132, accurate stats for the selective path predicate) and boolean queries were normalised. Existing installs self-heal via DROP INDEX IF EXISTS on the next deploy. #131

Changed

• Require zodb-pgjsonb>=1.14.0, the release that ships the startup-DDL schema-version gate the deferred index/ANALYZE actions are tagged for (bluedynamics/zodb-pgjsonb#78). The runtime still feature-detects the version= argument, so this only raises the floor for new installs.

• TestEnqueueUnit (Tika enqueue unit tests) now runs against a real dict_row cursor and the actual PG schema instead of a MagicMock cursor with hand-stubbed fetchall() rows. The mocked row shapes were the contract that drifted and shipped a KeyError to production (#124); driving the tests through psycopg means they can no longer diverge silently. Tests only. #125

Fixed

• Sort handling no longer silently drops or no-ops three sort keys (#157): sort_on=effectiveRange now sorts by the effective date (the range start) instead of being ignored; sort_on=SearchableText sorts by full-text relevance when a SearchableText term is queried (and logs a warning, rather than dropping silently, when there is no term); and sorting on a dedicated TEXT[] column (allowedRolesAndUsers, object_provides) — whose value lives in a column, not idx JSONB — is now ignored with a warning instead of emitting a NULL ORDER BY.

• The Tika worker now streams S3-tiered blobs straight from S3 into the Tika request body in chunks, instead of buffering the whole blob in memory (peak ~2–3× blob size). A single large PDF/image could OOM-kill a small worker pod (256Mi) into CrashLoopBackOff; streaming keeps the worker’s memory roughly constant regardless of blob size. The PG-bytea path (small inline blobs below the S3 tiering threshold) is unchanged. #189

• The Advanced-tab Update Catalog and Clear and Rebuild buttons now submit via POST instead of GET. The forms in catalogAdvanced.dtml had no method, so the (destructive) action stayed in the URL bar and a reload / Back / prefetch silently re-ran it — observed on production as a second full clear+rebuild kicked off by a browser reload. The handlers already redirect, so this yields a clean Post/Redirect/Get flow. #188

1.0.0b67 (2026-06-29)

Changed

• Version-tag the three deferred startup actions (ensure_text_indexes, ensure_field_indexes, analyze_object_state) so zodb-pgjsonb’s double-checked schema-version gate lets replicas skip the startup advisory lock when their work is already current. On a rolling deploy this stops every pod from queuing behind the lock — catalog indexes and ANALYZE now run once per schema- or registry-changing deploy instead of on every pod start. The index versions track the live IndexRegistry; the ANALYZE version tracks the catalog DDL. The version= argument is passed only when the installed zodb-pgjsonb supports it, so older releases keep working unchanged. bluedynamics/zodb-pgjsonb#78

• Raise the per-column statistics target for object_provides to 2000. The planner estimates TEXT[] array-containment via element MCEs; at the default target the MCE list is short, so rare / mid-frequency marker interfaces fall below the floor and get badly underestimated — flipping to Bitmap-AND plans that heap-scan hundreds of thousands of rows (queries on rare markers spiked to 20–30 s). The larger target doubles the MCE list and stabilises those plans; it is populated by the deferred ANALYZE (whose version tracks the catalog DDL, so it re-runs on the next deploy). #133

Documentation

• Correct the Tika OCR documentation: the minimal apache/tika image bundles no OCR engine, so plain images and scanned (image-only) PDFs index as empty text (the queue row still completes). OCR needs the -full image plus server-side tika-config.xml (OCR strategy + languages). The how-to now pins an explicit image version, adds an “OCR for images and scanned PDFs” section, and the misleading “includes Tesseract OCR” claims are removed. #183

Fixed

• The Tika worker HTTP timeout is now configurable via TIKA_WORKER_HTTP_TIMEOUT (default 120 s, was hard-coded). OCR of large multi-page scanned PDFs can exceed 120 s; the request then timed out and the queue row was marked failed. #183

• Skip portal_transforms for searchable NamedBlobFile fields when Tika is active. plone.app.dexterity.textindexer’s dynamic SearchableText indexer runs NamedfileFieldConverter (a synchronous portal_transforms call) on every field marked searchable; only the IFile indexer was overridden before (#41). A new TikaAwareNamedfileFieldConverter returns the filename and defers body-text extraction to the async Tika worker when PGCATALOG_TIKA_URL is set, and delegates to the stock converter otherwise. Registered only when plone.app.dexterity is installed. (The secondary-blob enqueue-coverage gap this surfaces is tracked in #184.) #114

• Full-text search now picks up the current site/negotiated language when the query has no explicit Language parameter, instead of falling back to the simple text-search configuration. On multilingual sites this means BM25 / tsvector ranking uses the right language config (stemming, stop words) for queries that don’t pass Language (e.g. plain SearchableText searches). The language is resolved from the request, the context, or the portal default via a small vendored helper (no plone.api dependency). #166

1.0.0b66 (2026-06-29)

Added

• Ship a partial idx_os_navtree index for the navigation-tree / navigation- portlet listing pattern (Language + portal_type + path filter + effectiveRange + nav-visibility flags). The index bakes the two highly selective boolean nav flags (is_default_page, exclude_from_nav) into its partial predicate and INCLUDEs the result/security columns, shrinking the candidate set to the nav-visible subtree before any JSONB heap filter. On a ~4M-row production catalog this navigation shape was the single biggest contributor to the slow-query log. Adapted to the current schema: references the typed path_depth column (no longer in idx JSONB). #130

• Ship a plone.observability IMetricProvider (pgcontent) that produces plone_content_total / plone_content_by_state content-count metrics via SQL for pg-catalog sites. Registered only when plone.observability is installed (zcml:condition) and emits only for sites backed by IPGCatalogTool (migration safe). #174

Documentation

• Add cdk8s-plone to the ecosystem navigation dropdown.

Fixed

• Catalog searches now ignore query keys that do not match any catalog index, matching ZCatalog semantics. plone.restapi’s @search forwards its whole request dict to the catalog — including control parameters like metadata_fields (e.g. from Volto’s ObjectBrowserWidget) — which are not indexes. Previously any such unmapped key fell through to a JSONB idx->>'key' filter that matched no row and returned an empty result set; these keys are now silently dropped before query building (and before the cache key is computed). #168

• The async Tika worker now forwards S3 credentials to boto3. New TIKA_WORKER_S3_ACCESS_KEY / TIKA_WORKER_S3_SECRET_KEY env vars are passed as aws_access_key_id / aws_secret_access_key. Previously the worker built the boto3 client without any credentials, so every blob tiered to S3 failed extraction with Unable to locate credentials (only inline-bytea blobs were extracted). When the vars are unset, boto3 still falls back to its default credential chain, so IAM-role / ambient-credential setups are unaffected. #178

• PGIndex no longer parametrizes the JSONB key name in its SQL (idx->>%(key)s). On PostgreSQL the prepared statement flipped to a generic plan after 5 executions, which could not match the idx->>'key' expression indexes and fell back to a sequential scan over object_state — on a large catalog this triggered a LockManager LWLock storm and a production outage. The validated key is now baked into the SQL text as a psycopg.sql.Literal (value stays parameterized), so each key gets its own plan and matches its expression index. All key-bearing sites in pgindex.py (get, keys, __len__, uniqueValues) are covered; an index whose name is not a safe identifier degrades to the unwrapped ZCatalog index instead of crashing catalog.Indexes[name]. #161

• pgcatalog-tika-worker no longer crashes with a bare ModuleNotFoundError when installed without the tika / tika-s3 extra. The console script is registered unconditionally, but httpx (and boto3 for the S3 blob path) ship only via those extras. The worker module now imports httpx defensively and the script fails fast with a clear hint pointing at the required extra; the S3 path raises an actionable error too. #171

1.0.0b65

Added

• Suggestions engine now emits bundles of indexes. The new Bundle data shape groups one or more related indexes together so the UI can present “apply these together” semantics for hybrid patterns. Single btree-composite suggestions render unchanged in the existing ZMI via a back-compat flatten in manage_get_slow_query_stats; the full bundle structure is additionally exposed under suggestions_bundles for the upcoming PR β JSON/JS UI.

• New templates: partial GIN (T4 — GIN with an AND-joined equality predicate baked into WHERE) and hybrid bundles (T5 — one btree composite plus one or more partial GINs working together). Triggered by the MIXED shape classification when a slow query has both btree-eligible and KEYWORD filters.

• Partial-predicate scoping is data-driven. A selectivity probe uses pg_stats.most_common_vals for common values (zero DB round-trip) and falls back to live SELECT COUNT(*) for long-tail values. Equality filters whose live selectivity is below 10 % (configurable via PGCATALOG_PARTIAL_SELECTIVITY_THRESHOLD env var) are baked into the partial index’s WHERE clause.

• Closes the canonical Plone slow-query gap from #122: portal_type + review_state + Subject + effective range + sort_on now produces a hybrid bundle — btree composite (portal_type, review_state, effective) + partial GIN on Subject scoped by portal_type='Event' AND review_state='published'.

Changed

• suggest_indexes() now returns list[Bundle] (was list[dict]). Signature gains an optional conn argument for selectivity probes; conn=None disables partial scoping and keeps bundle output identical to pre-α behavior. No call-site changes required in existing code that imports this function from outside the package.

Fixed

• Fix automatic language detection for BM25 ranking #164

• PG-backed test layer: reset zodb_pgjsonb’s process-wide SharedLoadCache between tests. The per-test restore() rolls the database backwards to a snapshot, but cacheMinimize() / pool.clear() only reset the ZODB connection caches — not the shared L2 cache that outlives the pool. That cache then served object state at TIDs newer than the rolled-back database, causing spurious ConflictError (on PLexicon during clearFindAndRebuild) in whichever rebuild test ran second. testSetUp now calls storage.clear_caches() after restore(); this requires zodb-pgjsonb >= 1.13.0 (the dependency floor is bumped accordingly).

1.0.0b64

Fixed

• _process_sort now emits idx->'{key}' (JSONB operator) instead of idx->>'{key}' (text operator) for the FieldIndex fallback. Text-cast sorting compared everything lexicographically, so a FieldIndex over a numeric attribute ranked "10" before "2". JSONB comparison is type-aware: numbers sort numerically, strings lexically, so a homogeneous FieldIndex now always sorts correctly regardless of value type. Affects any FieldIndex with numeric source data (counters, priorities, prices, weights). #158

1.0.0b63

Changed

• PGCatalogBrain no longer stores a reference to the catalog tool and no longer consults self._catalog.REQUEST for URL rendering. getURL uses zope.globalrequest.getRequest(); getObject and _unrestrictedGetObject resolve a traversal root lazily via a new _traversal_root() helper (getSite().getPhysicalRoot() first, getRequest().PARENTS[-1] as fallback). This keeps brains catalog-independent so callers can cache / pickle / re-queue them without dragging the Acquisition chain along. The catalog reference that the ZODB-prefetch batch needs has moved onto the transient CatalogSearchResults container, where it belongs.

  The catalog= keyword on PGCatalogBrain.__init__ is accepted but ignored — kept until the next major version for signature compatibility with direct callers.

1.0.0b62

Fixed

• _process_index now falls back to the correct handler for built-in Plone indexes when they are missing from the IndexRegistry — previously the miss fell through to _handle_field which emits idx->>'name', bypassing the dedicated typed columns and their indexes. On aaf-6 prod this produced 4-9 second seq-scans over a 450k-row object_state table for every folder-listing query, saturating the worker pool and causing intermittent Varnish backend-fetch errors. Resolution happens via a new _builtin_index_type(name) helper that combines two sources: the three Plone-native specials (path, effectiveRange, SearchableText) hardcoded because their SQL lives inside the handler, and every TEXT[]-typed ExtraIdxColumn (derived at dispatch time — currently allowedRolesAndUsers and object_provides, extensible via register_extra_idx_column). The correct handler then uses path / allowed_roles / object_provides / searchable_text columns and the DateRangeIndex composite clause. Explicit registry entries still win so addons can override behavior. Closes #154.

• _handle_keyword no longer crashes with TypeError: 'DateTime' object is not iterable when a caller passes a non-str, non-iterable value (e.g. a Zope DateTime or a Python datetime) as the query for a KeywordIndex. The old coercion assumed the value was either a str or iterable; anything else (DateTime, int, …) hit list(value) and raised. Values are now coerced via str() into a single-element list, matching the JSONB storage shape (keyword arrays always contain strings). Closes #152.

1.0.0b61

Fixed

• _field_range on FieldIndex no longer silently returns zero rows for numeric range queries. Two stacked bugs: [max, min] order from the caller was not normalized (produced always-false SQL), and idx->>'key' comparisons ran lexicographically on text — so '46.1' <= '5.0' <= '49.0' included values outside the range. _field_range now sorts min/max and casts idx->>'key' to ::numeric when the range values are int / float. String values (ISO dates etc.) keep text comparison. Affected aaf-6 prod map-widget bbox filter via collective.collectionfilter — closes #150.

• _CatalogCompat.getIndex and _CatalogIndexesView.__getitem__ no longer silently fall back to the raw ZCatalog index when they cannot find the catalog tool — that fallback returned empty BTrees and masked #143 / #146 for weeks. A new private helper _resolve_catalog tries three paths in order (__parent__ → Acquisition chain → zope.component.hooks.getSite().portal_catalog) and raises RuntimeError if all three fail.

• _CatalogCompat.indexes property now self-heals a missing __parent__ on first access via getSite().portal_catalog. First page render after deploy persists __parent__; no second upgrade-step click required on sites where the #139 upgrade ran before that fix landed. Zero-touch prod recovery.

Added

• PGIndex._apply_index(request, resultset=None) — ZCatalog- compatible low-level query entry point. Returns (IITreeSet(zoids), (index_name,)). Reuses _QueryBuilder._process_index so every registered IndexType (FIELD, KEYWORD, PATH, DATE, DATE_RANGE, UUID, TEXT, BOOLEAN, GOPIP) plus every IPGIndexTranslator utility works for free. No implicit security filtering — matches ZCatalog semantics; use catalog(**query) for secured results. Emits a DeprecationWarning once per caller site.

• _PGIndexMapping.__getitem__ / __len__ — round out the PG-backed mapping so Plone core callers (plone.app.uuid.utils, plone.app.vocabularies.Keywords) work against catalog._catalog.getIndex(name)._index without needing catalog.Indexes[name] acquisition.

• _PGIndexMapping.items() / values() raise NotImplementedError with guidance pointing at uniqueValues, _apply_index, and catalog(**query) as alternatives. No Plone-core caller uses them on a wrapped index; a concrete usecase can land in a future issue with server-side-cursor streaming.

• PGIndex._index property emits a DeprecationWarning on access — signals callers that the BTree-shaped API is an emulation and suggests the preferred pgcatalog-native alternatives.

Closes #146.

1.0.0b60

Fixed

• _PGIndexMapping (backing PGIndex._index) is now iterable and branches its SQL on the index type. plone.app.vocabularies.Keywords iterates index._index directly to populate the tag-autocomplete widget in the Plone edit form — the previous mapping had no __iter__ and its keys() coerced JSONB arrays to their text representation. Net effect on b59: typing into the Schlagwort field offered zero suggestions even when matching keywords existed. keys() / __iter__ now use the same UNION ALL expansion as uniqueValues for KEYWORD, and get() uses idx->key @> to_jsonb(value::text) so membership checks against a keyword actually match. Follow-up to #143.

1.0.0b59

Fixed

• PGIndex.uniqueValues() now branches on the wrapped index type. For IndexType.KEYWORD the JSONB value is a list of tags, so the SQL uses jsonb_array_elements_text to expand it into individual entries. Previously all index types went through idx->>key, which coerces a JSONB array to its JSON text representation — producing entries like '["Werkvortrag", "Tirol"]' instead of 'Werkvortrag' / 'Tirol'. Callers (the querystring composer vocabulary, plone.app.vocabularies.Keywords, collective.collectionfilter tag clouds, etc.) now see the distinct set of elements as expected.

  A defensive UNION ALL branch treats a scalar row under the same keyword key (corrupt/legacy data) as a single-value keyword so the query does not raise cannot extract elements from a scalar. _maybe_wrap_index passes the registered IndexType through to PGIndex so callers that build the wrapper directly keep getting the (correct) scalar path by default.

  Closes #143.

1.0.0b58

Fixed

• Register SanitizeRowsModifier as an IQueryModifier utility so that malformed Collection-query rows (missing or non-string i field) are dropped before plone.app.querystring.parseFormquery processes them. Upstream queryparser.py:73 otherwise sets query[None] = ... and the subsequent catalog(**parsedquery) in querybuilder._makequery fails at the Python level with TypeError: keywords must be strings — which the Collection edit widget cannot recover from. With the sanitizer in place the preview renders again, so editors can open their Collections and repair corrupted Subject / tag data through the UI.

  This is a defensive workaround for an upstream gotcha, not a fix for the underlying plone.app.querystring behavior. Closes #142.

1.0.0b57

Fixed

• PGCatalogBrain.getObject() now mirrors upstream Products.ZCatalog.CatalogBrains.AbstractCatalogBrain.getObject: the parent path is traversed unrestricted and only the final object is restrictedTraverse-checked. Previously the full path went through restrictedTraverse, so any intermediate container with stricter permissions than the leaf raised AccessControl.unauthorized.Unauthorized — even though the catalog filter had already authorized access to the target. Sites with a private parent folder publishing individual public items (the common “kalender/event-xyz” pattern) hit this on every anonymous render. Closes #141.

1.0.0b56

Fixed

• The v1->v2 upgrade step was silently no-op on production sites. GenericSetup invokes upgrade handlers with the portal_setup tool as the context, but _resolve_compat only understood the ImportContext shape (getSite()), so the setup-tool call path hit the return None, None branch and logged migrate_catalog_indexes: no _CatalogCompat found; skipping — while GenericSetup happily bumped the profile version to 2. Net effect: the persisted _CatalogCompat kept its legacy indexes attribute and the new indexes property then raised AttributeError for _raw_indexes, which Acquisition swallowed and replaced with the tool’s indexes() method — surfacing as 'function' object has no attribute 'keys' from catalog.indexes.keys() and 'method' object is not subscriptable from catalog.indexes[name].

  _resolve_compat now also walks aq_parent(context) to reach the Plone site when the context is the setup tool, so the normal ZMI manage_upgrades path migrates the state as intended.

• Made _CatalogCompat.indexes self-healing: if the legacy indexes attribute is still in __dict__ (unmigrated or fresh-install site that skipped v1), the property moves it to _raw_indexes on first access and marks the instance dirty. This avoids the Acquisition-swallowed AttributeError failure mode even when the upgrade step never ran.

  Closes #139.

1.0.0b55

Fixed

• Plone and addon code commonly reaches into the catalog via the non-API- conform pattern catalog._catalog.indexes[name] / .get(name) / .items(). Previously this returned the raw ZCatalog index objects with empty BTrees, so queries against them silently returned no results. _CatalogCompat.indexes is now a property returning a transient view that wraps each index with PGIndex (same behavior as catalog.Indexes[name]). Custom PATH-type indexes and other special indexes (idx_key=None) continue to be returned raw, since they have dedicated typed columns and don’t need PG-backed wrapping.

  Migration: GenericSetup profile bumped from v1 to v2. The upgrade step renames the persisted indexes attribute to _raw_indexes and sets __parent__ on the compat so aq_parent can reach the catalog tool through bare attribute access. Run Plone Site Setup → Add-ons → plone.pgcatalog → Upgrade on existing sites, or let the next runAllImportSteps on the default profile pick it up.

  Likely-affected callers include plone.base.utils.check_id (reserved-name check), plone.restapi.search.query.Query.get_index, plone.app.discussion, plone.app.referenceablebehavior, plone.volto, collective.collectionfilter, and collective.exportimport — per-package verification is recommended after upgrade.

  Based on prior prototyping by @thet on thet/indexes-wrapper. Closes #137.

1.0.0b54

Changed

• Stop duplicating path, path_parent, and path_depth between the typed columns on object_state and the idx JSONB. These three fields now live exclusively in their typed columns (path, parent_path, path_depth) — previously identical values were stored in both places, wasting ~10 % of JSONB storage and (more importantly) blocking the planner from collecting selectivity statistics on path-subtree filters. Indexes and extended statistics on these fields have been migrated to reference the typed columns directly. Custom PATH-type indexes (e.g. tgpath) are unaffected and continue to store their data in idx.

  Migration: Schema and writer changes are picked up automatically on startup (the eight affected indexes and three extended-statistics objects are reissued with idempotent DROP … IF EXISTS / CREATE … IF NOT EXISTS pairs). To strip the obsolete keys from existing JSONB on large catalogs, run:

  from plone.pgcatalog.migrations.strip_path_keys import run
  
  run(conn, batch_size=5000)
  

  Safe to run online, idempotent, batched. Issue #132.

1.0.0b53

Fixed

• Migration install handler silently dropped every DateRecurringIndex (DRI) — e.g. plone.app.event’s / bda.aaf.site’s general_start / general_end — when replacing a foreign portal_catalog with PlonePGCatalogTool. _snapshot_catalog correctly captured the stored attr_recurdef / attr_until attributes, but _build_extra had no DRI branch, so the restored extra namespace lacked the recurdef / until keys that DateRecurringIndex.__init__ reads. The constructor raised AttributeError, the outer try/except in _restore_from_snapshot swallowed it as a warning, and the index was never created — which meant extract_idx never indexed those fields, the IndexRegistry had no entry for them, and every site-wide Collection filtering on general_end returned zero results.

  Added the DRI translation in _build_extra, plus a roundtrip test that actually instantiates DateRecurringIndex with the built extra — the kind of assertion that would have caught this before it ever shipped. Issue #126.

  Existing deployments that migrated on an affected build have to re-add the missing indexes manually (the upgrade can’t recover them without the original catalog snapshot). Run:

  catalog = portal.portal_catalog
  
  
  class _Extra:
      pass
  
  
  extra = _Extra()
  extra.recurdef = "recurrence"
  extra.until = ""
  for name in ("general_start", "general_end"):
      if name not in catalog._catalog.indexes:
          catalog.addIndex(name, "DateRecurringIndex", extra)
  catalog.reindexIndex("general_start")
  catalog.reindexIndex("general_end")
  

Added

• Slow-query suggestions now produce covering composite indexes for the common portal_type + effectiveRange + sort_on=effective pattern (issue #122). The suggestion engine splits the legacy _NON_IDX_FIELDS into purpose-specific constants, expands effectiveRange to its effective date contributor, and appends the query’s sort_on field as a trailing btree composite column so the planner can skip the ORDER BY sort step.

1.0.0b52

Fixed

• CatalogStateProcessor._enqueue_tika_jobs indexed result rows by integer position (row[0], row[1]), but the request-scoped connection pool uses a dict_row factory, so every content save that produced an unresolved blob ref raised KeyError: 0 during tpc_vote (e.g. uploading a Dexterity Image). Switched to column-name access.

  Existing tests didn’t catch this because the integration tests opened their cursor with tuple_row and the unit tests mocked fetchall() with tuple rows — both diverged from production. Tests updated to use dict_row to match the real pool.

1.0.0b51

Added

• Extended PostgreSQL statistics for every default composite catalog index on object_state, so the planner has accurate joint-selectivity estimates for the expression pairs we actually index. mcv + dependencies for low-cardinality pairs (type + state, parent + type, type + effective, type + expires), dependencies only for path-pairs (high-cardinality paths make mcv wasteful; CMS content structure is typically wide-shallow, so dependency signal is what matters).

  Without these, PG’s per-column histograms treat the expressions as independent and underestimate joint selectivity, so the planner picks a composite-index scan and heap-filters thousands of tuples instead of doing a Bitmap-AND with the available GIN indexes. On a published-Event navigation query observed in production, this dropped query time from 911 ms to sub-100 ms.

  On existing installations a one-shot ANALYZE object_state runs on the first write transaction after upgrade so the new statistics take effect immediately rather than waiting for autovacuum. Idempotent via pg_stats_ext skip check.

  Issue #122 (PR 1 of 3 — engine refactor and EXPLAIN-driven coverage to follow).

1.0.0b50

Fixed

• release_request_connection now issues an explicit conn.rollback() before returning the connection to the pool. Otherwise an implicit transaction opened by a prior SELECT on the pool fallback path stays alive, holding a virtualxid that blocks CREATE INDEX CONCURRENTLY. Companion fix to bluedynamics/zodb-pgjsonb#58 (the storage-conn path). Closes #118.

• Suggested Indexes UI: detect already-applied suggestions with mixed-case field names (e.g. Language) by matching index names case-insensitively — PostgreSQL folds unquoted identifiers to lowercase. Also strengthen expression normalization (whitespace around ->>, iterative paren collapse, WHERE-anchored extraction) so generated and PG-stored indexdef forms compare equal. apply_index is now idempotent when a valid index with the same name already exists — returns success no-op instead of propagating the DuplicateTable error. Closes #119.

• Tika enqueue: resolve Dexterity NamedBlobFile / NamedBlobImage wrapper OIDs via a second-hop lookup through object_state, so the queue receives jobs for modern Dexterity File/Image content. Previously _enqueue_tika_jobs() only looked up the OIDs it found in the content’s state — which are the wrapper OIDs, not the inner ZODB.blob.Blob OIDs. The direct lookup returned zero rows and the enqueue silently skipped. Flat-state content (legacy/Archetypes- style, where the content state carries a direct ZODB.blob.Blob @ref) is unchanged. Closes #115.

• _handle_uuid now accepts list/tuple queries (uses = ANY(...)), matching _handle_field semantics. Previously a list query such as catalog.searchResults(UID=['f852...']) was stringified as str(['f852...']) → "['f852...']" and the JSONB ->> comparison never matched, so @@getVocabulary?name=plone.app.vocabularies.Catalog with a plone.app.querystring.operation.list.contains criterion on UID returned an empty vocabulary.

• catalog._catalog.getIndex(name) now returns a PGIndex wrapper with PG-backed _index and uniqueValues(), same as catalog.Indexes[name]. Previously it returned the raw ZCatalog index with empty BTrees, which broke:

  • plone.app.vocabularies.KeywordsVocabulary (empty Subject/Tags dropdowns).

  • Products.CMFPlone.browser.search.Search.types_list() (empty “Item type” filter in @@search).

  • plone.app.event.setuphandlers (DateIndex detection).

  • Other Plone code paths that bypass catalog.Indexes[name].

  Special indexes registered with idx_key=None (SearchableText, path, effectiveRange) are returned unwrapped so dedicated columns are used for them.

1.0.0b49

Added

• Log all catalog queries for debugging via PGCATALOG_LOG_ALL_QUERIES=1 (also accepts true/yes). Enabled queries are logged at INFO level with duration, SQL, params, and query keys. Params are truncated at 2000 chars to bound log size. The env var is re-checked on every query, so the setting can be toggled at runtime without a restart. See docs/how-to/debug-queries.md for details and a production-safety warning about logging user-supplied query values.

• Slow-query log format changed slightly: the prefix is now Slow SQL catalog query (%.2f ms) instead of Slow catalog query (%.1f ms) — log-aggregation grep patterns may need an update.

Fixed

• clearFindAndRebuild() now works on fresh installs and after refactorings that leave object_state.path empty. Previously the rebuild relied on a PG snapshot of WHERE path IS NOT NULL, so when the column was not yet populated only the Plone root was re-indexed. The rebuild now walks the ISiteRoot breadth-first regardless of PG state.

  The new traversal is still memory-flat: the BFS queue holds only path strings (not objects), and objects are ghosted by cacheMinimize() after every 500 commits. Also yields discussion items via the IConversation adapter when plone.app.discussion is installed, so comments on content are included in the rebuild.

• Stringify Boolean query values to JSON notation ('true'/'false') so queries against JSONB ->> comparisons match. Previously str(True) produced 'True' which never matched JSONB’s lowercase form, causing queries to return no results. Fix applied in query.py (all field handlers), pgindex.py (ZCatalog _index.get() compat), addons_compat/eeafacetednavigation.py (faceted search dispatch), and backends.py (text search backends). Helper renamed from _to_json_string to _bool_to_lower_str to match what it actually does.

• Gracefully handle missing meta column in _load_idx_batch() (#105). Falls back to SELECT zoid, idx if the column does not exist yet, preventing UndefinedColumn crash on first read after upgrade. Root cause fix is in zodb-pgjsonb 1.10.4 (poll_invalidations now applies deferred DDL before the read snapshot).

• Show index creation errors in red instead of green in ZMI (#104). manage_apply_index / manage_drop_index now redirect with index_error param on failure, rendered as Bootstrap alert-danger.

1.0.0b48

Fixed

• Fix startup warnings “security declaration for nonexistent method” for unsupported ZCatalog stubs (getAllBrains, searchAll, etc.). ObjectManager.__class_init__ calls InitializeClass at class creation time, so stub methods must be defined in the class body, not via post-hoc setattr.

Changed

• Extract @meta, object_provides, and allowedRolesAndUsers from idx JSONB into dedicated columns via generic ExtraIdxColumn mechanism. Reduces idx size by ~85% (from ~3.2 KB to ~400 B avg, below TOAST threshold). Run clear_and_rebuild after upgrading. (#98)

• object_provides queries now use a dedicated TEXT[] column with GIN index instead of JSONB containment.

• Removed _backfill_allowed_roles startup function (superseded by generic extraction mechanism).

1.0.0b47

Fixed

• Fix ValueError: Invalid path: '' when path query receives empty string. Empty/blank paths are now silently filtered, matching ZCatalog behavior.

1.0.0b46

Fixed

• Query cache: use catalog-specific change counter instead of MAX(tid) (#94). The cache was invalidated on every ZODB write (~2500/hour from ScalesDict alone), making it nearly useless (~28% hit rate). Now uses pgcatalog_change_seq which only increments on actual catalog writes (catalog_object, uncatalog, reindex, move). Expected hit rate 90%+ on typical sites.

1.0.0b45

Fixed

• Fix Tika queue never populated: content_type always None (#90). Removed broken extract_content_type() — IPrimaryFieldInfo can’t adapt the indexer wrapper and Dexterity items have no top-level content_type attribute. MIME type is now read directly from idx["mime_type"] (the standard Plone catalog index), which is reliably extracted by the IndexRegistry.

• Fix suggestion index existence check + dedicated KEYWORD fields (#92). _check_covered() now compares by index name (reliable) with normalized expression fallback. object_provides and Subject added to _DEDICATED_FIELDS — their existing GIN indexes make new suggestions useless.

1.0.0b44

Added

• Smart index suggestions in ZMI Slow Queries tab (#86). Replaces the naive _suggest_index() with field-type-aware suggestions using the IndexRegistry. Generates correct DDL per IndexType (btree expression, GIN, tsvector, composites). Detects already-covered fields (dedicated columns, existing indexes). Manual “Apply” button creates indexes via CREATE INDEX CONCURRENTLY. “Drop” button for removing suggestion indexes (idx_os_sug_*). On-demand EXPLAIN plans for slow queries. New suggestions.py module with pure suggestion engine + DB helpers.

1.0.0b43

Fixed

• Fix Tika enqueue: _collect_ref_oids() and the ANNOTATION_KEY fallback in CatalogStateProcessor.process() now handle JSON string state (from decode_zodb_record_for_pg_json). Previously state was assumed to be a dict, but the fast codec path returns a JSON string — so @ref markers were never found, no extraction jobs were enqueued, and Tika sat idle.

1.0.0b42

Fixed

• Handle Unix epoch floats/ints in ensure_date_param(). Callers like plone.app.textfield pass time.time() values as date query params. Now converts to datetime.fromtimestamp(value, tz=UTC). Fixes #82.

• Skip missing attributes instead of storing null in idx JSONB. Matches ZCatalog semantics: missing attribute = not indexed (key omitted), not “indexed as null”. Fixes #81.

• Use @> containment for single-value KeywordIndex queries instead of ?| overlap. The GIN index handles @> much better. object_provides queries: 2.4s to 650ms. Fixes #80.

1.0.0b41

Fixed

• Register refs prefetch expression for cataloged content objects. Uses CASE WHEN idx IS NOT NULL THEN refs END so only content objects (with catalog data) trigger prefetch. Requires zodb-pgjsonb >= 1.9.2.

1.0.0b40

Added

• Brain object prefetch via storage.load_multiple(). When iterating search results and calling getObject(), the first call in each batch prefetches up to 100 objects in a single SQL query, warming the storage cache for subsequent calls. Configurable via PGCATALOG_PREFETCH_BATCH environment variable (default 100, set to 0 to disable).

1.0.0b39

Fixed

• Log query cache TID lookup failures instead of silently swallowing them. Diagnoses why the query cache may not be populating.

1.0.0b38

Fixed

• Extract allowed_roles backfill from schema DDL into batched startup step (#65 Phase 2). Previously the backfill ran as a single UPDATE on 4.4M rows inside ACCESS EXCLUSIVE, blocking the entire database. Now processes 5000 rows per batch with autocommit and lock_timeout. Safe to re-run, idempotent, logs progress.

1.0.0b37

Fixed

• Prevent database lockup during rolling deployments (#65). _ensure_text_indexes() and _ensure_field_indexes() now set lock_timeout = '5s' to prevent indefinite blocking on ACCESS EXCLUSIVE locks from concurrent REPEATABLE READ sessions. Log level changed from error to warning since indexes are retried on next startup.

1.0.0b36

Added

• Process-wide query result cache with TID-based invalidation. Caches catalog query results in memory. Invalidated when MAX(tid) changes (any ZODB commit). Cost-based eviction keeps expensive queries in cache. Configurable via PGCATALOG_QUERY_CACHE_SIZE (default 200) and PGCATALOG_QUERY_CACHE_TTR (default 60s). Fixes #74.

• ZMI: Cache Status section on the Slow Queries tab showing hit/miss rate, entries, invalidations, TTR, and top cached queries by cost.

Fixed

• getCounter() fix: SELECT MAX(tid) returns column max, not tid. Added AS tid alias. Also fixes the test.

1.0.0b35

Performance

• Add partial index idx_os_cat_nav_visible for navigation listings (exclude_from_nav=false is only ~1.6% of rows). Verified on production: 261ms → 20ms (13×).

• Add partial index idx_os_cat_events_upcoming for calendar/event queries (portal_type=Event + show_in_sidecalendar=true + end date). Verified on production: 728ms → 33ms (22×).

• Mark pgcatalog_to_timestamptz() as PARALLEL SAFE to allow parallel query execution.

Fixed

• getCounter() now returns MAX(tid) from PostgreSQL instead of a persistent counter that was never incremented (always returned 0). This enables Plone’s cache invalidation (plone.memoize) for catalog-dependent caches like navigation trees. ~0.2ms via Index Only Scan, no ZODB write overhead.

1.0.0b34

Fixed

• DateRecurringIndex fields (e.g. start, end) now get auto-created btree expression indexes and appear in the ZMI Indexes tab. Added DateRecurringIndex to META_TYPE_MAP. Query builder now checks IPGIndexTranslator before IndexRegistry, so rrule query logic takes priority. Fixes #71.

1.0.0b33

Fixed

• Revert path_parent IN subquery for bounded-depth queries (#68). The subquery caused Nested Loop plans where PG repeated the allowed_roles GIN scan per parent path (615ms). Reverts to LIKE + path_depth which uses the path_depth_type composite index (85-300ms depending on cache state).

1.0.0b32

Fixed

• Optimize bounded-depth path queries: rewrite path LIKE + path_depth to path_parent IN (subquery) so PG can use the composite (path_parent, portal_type) index. Navigation tree queries drop from 630ms to ~77ms. Fixes #66.

1.0.0b31

Added

• Denormalize allowedRolesAndUsers into dedicated allowed_roles TEXT[] column with GIN index. Security filter queries now use allowed_roles && ARRAY[...] instead of JSONB decompression. Includes automatic backfill migration for existing databases. Navigation queries 85ms to 5-15ms, all queries benefit. Fixes #63.

1.0.0b30

Fixed

• Add dedicated GIN indexes for allowedRolesAndUsers, object_provides, and Subject keyword fields. The full-idx GIN index is too broad for ?| queries on individual keyword arrays. Dedicated indexes are much smaller and faster. object_provides queries drop from 850ms to sub-millisecond.

1.0.0b29

Fixed

• Use btree-friendly expressions instead of GIN containment for FieldIndex single-value, BooleanIndex, and UUIDIndex queries. Root cause of 3-4 second navigation queries on large sites. Navigation queries drop from 3900ms to <1ms.

• Fix Slow Queries ZMI tab crash with KeyError when PGCATALOG_SLOW_QUERY_MS is not set. Fixes #58.

• Python 3.14 CI compatibility. Fixes #57.

1.0.0b28

Added

• Auto-create btree expression indexes for custom CatalogIndex fields at startup. Only standard Plone fields have hardcoded indexes; custom fields (like general_end) now get indexes automatically based on the IndexRegistry. Date fields use pgcatalog_to_timestamptz() wrapper. Fixes #49.

1.0.0b27

Fixed

• Add composite indexes for common catalog query patterns. Without these, PG picks a single-column index and sequentially filters all indexed rows (3+ seconds per query, 30+ second page loads). With composite indexes: sub-millisecond. Fixes #50.

  New indexes:

  • (path_parent, portal_type) — folder listings, navigation

  • (path pattern, portal_type) — collections, search

  • (path pattern, path_depth, portal_type) — navigation tree

  • (portal_type, review_state) — workflow-filtered listings

  Indexes are created automatically on startup (idempotent DDL).

Added

• Slow catalog query logging: queries exceeding PGCATALOG_SLOW_QUERY_MS (default: 10ms) are logged as warnings and recorded in the pgcatalog_slow_queries table. Fixes #52.

• ZMI “Slow Queries” tab on portal_catalog: shows aggregated slow query patterns (count, avg/max duration, last seen) with suggested composite index DDL for frequent patterns. Includes a “Clear Stats” button.

1.0.0b26

Added

• ZMI: Tika status card on the Advanced tab showing URL, worker mode, configured content types, extraction queue stats (pending/processing/ done/failed), and IFile transform override status. Fixes #47.

1.0.0b25

Fixed

• reindexIndex(name) now re-extracts index values from ZODB objects instead of reshuffling existing JSONB values. Iterates all cataloged paths, loads via unrestrictedTraverse, extracts the requested index, and writes a JSONB merge update. Batched commits for memory. Fixes #43.

Added

• ZMI [reindex] button per index on the Indexes & Metadata tab with confirmation dialog. Calls new manage_reindexIndex endpoint.

• ZMI: confirmation dialogs on Advanced tab for “Update Catalog” and “Clear and Rebuild” buttons. Warns that operations may take a while and that Clear and Rebuild destroys catalog data temporarily. Fixes #44.

1.0.0b24

Changed

• clearFindAndRebuild now uses PG-driven iteration instead of ZopeFindAndApply. Queries object_state directly, filtering out known non-content classes (~96% of rows). No acquisition parent chains on the call stack means cacheMinimize() can ghost all objects — flat memory on large sites. Fixes #39.

Added

• Skip portal_transforms text extraction for IFile when PGCATALOG_TIKA_URL is set. The async Tika worker handles blob text extraction — no more synchronous pdftotext/wv calls or BFS graph traversal of the transform registry during indexing. Custom types with blob fields need their own override (see docs). Fixes #41.

1.0.0b23

Fixed

• Fix Tika worker queue never being populated. Content objects (File/Image) and their Blob sub-objects have different ZODB oids. The enqueue logic now extracts @ref oids from the content state to resolve the actual blob zoid in blob_state. The queue stores both zoid (content, for searchable_text update) and blob_zoid (for blob data fetch). Fixes #37.

1.0.0b22

Fixed

• Fix high memory usage during catalog rebuild. clearFindAndRebuild and refreshCatalog now commit every 500 objects, flushing dirty ZODB objects and pending catalog data so cacheMinimize() can actually reclaim memory. Previously, _p_changed = True on every indexed object prevented deactivation until the end of the (single) transaction.

1.0.0b21

Fixed

• Reduce memory usage during catalog rebuild. clearFindAndRebuild and refreshCatalog now deactivate ZODB objects after indexing and periodically call cacheMinimize() to keep RAM usage flat on large sites. Folderish objects are kept active during tree traversal to avoid redundant reloads.

1.0.0b20

Fixed

• Fix UID expression index using wrong case (idx->>'uid' instead of idx->>'UID'). JSONB keys are case-sensitive, so the old index was never used. Also add CREATE STATISTICS for UID selectivity so the query planner picks the correct index on large tables. Existing databases are migrated automatically on next startup. Fixes #28.

1.0.0b19

Removed

• Remove “Blob Storage” ZMI tab from portal_catalog. Blob storage statistics are now provided by zodb-pgjsonb >= 1.5.2 in the Zope Control Panel under Database management.

1.0.0b18

Fixed

• Fix computed index extraction (is_folderish, is_default_page, sortable_title, etc.) always returning null. IPGCatalogTool extended both ICatalogTool and IPloneCatalogTool, causing ICatalogTool to come first in the interface resolution order. CMFCore’s IndexableObjectWrapper (which does not resolve plone.indexer adapters) won over the plone.indexer wrapper. Fixed by extending IPloneCatalogTool only — ICatalogTool is already provided via IZCatalog.

1.0.0b17

Security

• CAT-Q1: Validate unknown query keys before SQL interpolation in _process_index() fallback path. Unregistered index names are now checked with validate_identifier() before being interpolated into JSONB field query expressions, preventing potential SQL injection via crafted query dict keys.

• CAT-S1: Replace f-string DDL in _ensure_text_indexes() with psycopg.sql.SQL/Identifier/Literal composition for defense-in-depth.

Changed

• CAT-P1: reindex_index() now uses a server-side cursor with batched fetches instead of loading all rows into memory at once. Progress is logged after each batch.

Fixed

• CAT-O1: Index/metadata extraction failures in extraction.py now emit log.debug() messages with field name and exception info instead of silently passing. Translator extraction failures are also logged.

• CAT-O2: Startup degradation (failed registry sync, failed text index creation) now logs at ERROR level with actionable context messages instead of WARNING/DEBUG.

• CAT-L1: Fallback connection pool (_fallback_pool from PGCATALOG_DSN env var) now registers an atexit close hook for clean shutdown.

• Install step now runs clearFindAndRebuild() after catalog replacement to index all existing content into PostgreSQL. Previously, content created before pgcatalog was installed (e.g. during Plone site creation) had no path/idx data, causing empty navigation and search results.

1.0.0b16

Added

• Add “Blob Storage” ZMI tab to portal_catalog showing blob statistics (total count, size, per-tier breakdown for PG/S3), a logarithmic size distribution histogram, and S3 tiering threshold visualization.

1.0.0b15

Fixed

• Protect PlonePGCatalogTool from being replaced during GenericSetup profile imports. CMFPlone’s baseline toolset.xml declares portal_catalog with CatalogTool; since PlonePGCatalogTool is a different class, the default importToolset deletes it, triggering an IObjectModifiedEvent cascade that raises KeyError: 'portal_catalog'. Added importToolset wrapper in overrides.zcml that skips portal_catalog when it is already a PlonePGCatalogTool.

1.0.0b14

Fixed

• Fix new objects not being indexed in PostgreSQL. ZODB assigns object IDs (_p_oid) during Connection.commit(), which runs after before_commit hooks (where the IndexQueue flushes). All new objects therefore have _p_oid=None at catalog_object() call time, causing the catalog to silently skip them. The fix stores pending catalog data directly in obj.__dict__ under the _pgcatalog_pending key when no OID is available yet; CatalogStateProcessor.process() pops and uses it during store() so the annotation is never persisted to the database. Fixes #27.

1.0.0b13

Fixed

• Preserve original Python types for metadata columns (e.g. brain.effective now returns a Zope DateTime object instead of an ISO string). Non-JSON-native metadata values (DateTime, datetime, date, etc.) are encoded via the Rust codec into idx["@meta"] at write time and restored on brain attribute access with per-brain caching. JSON-native values (str, int, float, bool, None) remain in top-level idx unchanged. Backward compatible — old data without @meta still works. Fixes #23.

1.0.0b12

Fixed

• Fix clearFindAndRebuild producing wrong paths (missing portal id prefix, e.g. /news instead of /Plone/news), indexing portal_catalog itself, and not re-indexing the portal root object. Now uses getPhysicalPath() for authoritative paths, aq_base() for identity comparison through Acquisition wrappers, and explicitly indexes the portal root before traversal (matching Plone’s CatalogTool). Fixes #21.

1.0.0b11

Fixed

• Fix example requirements.txt: use local editable path for pgcatalog-example instead of bare package name (not on PyPI). Fixes #18.

• Fix ZMI “Update Catalog” and “Clear and Rebuild” buttons returning 404. Added missing manage_catalogReindex and manage_catalogRebuild methods. Fixes #19.

• Fix clearFindAndRebuild indexing non-content objects (e.g. acl_users). Now filters for contentish objects only (those with a reindexObject method), matching Plone’s CatalogTool behavior. Fixes #20.

Changed

• uniqueValuesFor(name) is now a supported API (no longer deprecated). It delegates to catalog.Indexes[name].uniqueValues().

1.0.0b10

Changed

• Clean break from ZCatalog: PlonePGCatalogTool no longer inherits from Products.CMFPlone.CatalogTool (and transitively ZCatalog, ObjectManager, etc.). The new base classes are UniqueObject + Folder, providing a minimal OFS container for index objects and lexicons while eliminating the deep inheritance chain.

  This improves query performance by ~2x across most scenarios (reduced Python-side overhead from attribute lookups, security checks, and Acquisition wrapping) and write performance by ~5% (lighter commit path).

  A _CatalogCompat persistent object provides _catalog.indexes and _catalog.schema for backward compatibility with code that accesses ZCatalog internal data structures. Existing ZODB instances with the old _catalog (full Catalog object) continue to work without migration.

• ZCML override for eea.facetednavigation: Moved from <includeOverrides> inside configure.zcml to a proper overrides.zcml at the package root, loaded by Zope’s five:loadProductsOverrides. Fixes ZCML conflict errors when both eea.facetednavigation and plone.pgcatalog are installed.

Added

• eea.facetednavigation adapter: PGFacetedCatalog in addons_compat/eeafacetednavigation.py – PG-backed IFacetedCatalog that queries idx JSONB directly for faceted counting. Dispatches by IndexType (FIELD, KEYWORD, BOOLEAN, UUID, DATE) with IPGIndexTranslator fallback. Falls back to the default BTree-based implementation when the catalog is not IPGCatalogTool. Conditionally loaded only when eea.facetednavigation is installed.

• Deprecated proxy methods: search() proxies to searchResults() and uniqueValuesFor() proxies to Indexes[name].uniqueValues(), both emitting DeprecationWarning.

• Blocked methods: getAllBrains, searchAll, getobject, getMetadataForUID, getMetadataForRID, getIndexDataForUID, index_objects raise NotImplementedError with descriptive messages.

• AccessControl security declarations: Comprehensive Zope security matching ZCatalog’s permission model. Search ZCatalog on read methods (searchResults, __call__, getpath, getrid, etc.), Manage ZCatalog Entries on write methods (catalog_object, uncatalog_object, refreshCatalog, etc.), Manage ZCatalogIndex Entries on index management (addIndex, delIndex, addColumn, delColumn, getIndexObjects). setPermissionDefault assigns default roles (Anonymous for search, Manager for management). Private helpers (indexObject, reindexObject, etc.) declared private.

• DateRangeInRangeIndex support: Native IPGIndexTranslator for Products.DateRangeInRangeIndex overlap queries. Translates catalog({'my_idx': {'start': dt1, 'end': dt2}}) into a single SQL overlap clause (obj_start <= q_end AND obj_end >= q_start). Supports recurring events: when the underlying start index is a DateRecurringIndex with RRULE, uses rrule."between"() with duration offset for occurrence-level overlap detection. Auto-discovered at startup — no configuration needed. Allows dropping the Products.DateRangeInRangeIndex addon while keeping the same query API.

Fixed

• Addon index preservation: Installing plone.pgcatalog on a site with addon-provided catalog indexes (e.g. from collective.taxonomy, plone.app.multilingual, etc.) no longer silently drops those index definitions. The install step now snapshots all existing index definitions and metadata columns before replacing portal_catalog, then restores addon indexes after re-applying core Plone profiles. Removed toolset.xml in favour of a setuphandler-controlled replacement for correct timing.

1.0.0b9

Changed

• ZMI polish: All ZMI tabs now use Bootstrap 4 cards/tables matching Zope 5’s modern look (was old-style <table> layout with section-bar).

• Catalog tab (manage_catalogView): Replaced inherited ZCatalog BTree-based view with PG-backed version. Shows catalog summary (object count, index/metadata count, search backend with BM25/Tsvector status), path filter, and server-side paginated object table (20/page) with Previous/Next navigation. Object detail shows full idx JSONB and searchable text preview.

• Advanced tab (manage_catalogAdvanced): Simplified to only show Update Catalog and Clear and Rebuild actions. Removed ZCatalog-specific features (subtransactions, progress logging, standalone Clear Catalog) that don’t apply to PostgreSQL.

• Indexes & Metadata tab (manage_catalogIndexesAndMetadata): Merged the separate Indexes and Metadata tabs into one read-only view showing all registered indexes (name, type, PG storage location, source attrs) and metadata columns. Reflects the IndexRegistry rather than BTree counts (which were always 0).

• Removed tabs: Query Report, Query Plan (BTree timing), and the separate Indexes / Metadata tabs are hidden — replaced by PG-aware equivalents.

• Lexicon cleanup: setuphandlers.install() now removes orphaned ZCTextIndex lexicons (htmltext_lexicon, plaintext_lexicon, plone_lexicon) created by Plone’s catalog.xml — unused with PG-backed text search.

1.0.0b8

Changed

• Module split: config.py has been split into four focused modules: pending.py (thread-local pending store + savepoint support), pool.py (connection pool discovery + request-scoped connections), processor.py (CatalogStateProcessor), startup.py (IDatabaseOpenedWithRoot subscriber + registry sync). config.py is now a deprecation stub.

• Shared ensure_date_param(): Deduplicated date coercion utility from query.py and dri.py into columns.ensure_date_param().

• __all__ exports: Added explicit __all__ to pending.py, pool.py, processor.py, startup.py, columns.py, backends.py, interfaces.py.

• Top-level imports: Removed unnecessary deferred imports across catalog.py, processor.py, startup.py.

Added

• verifyClass/verifyObject tests for IPGIndexTranslator implementations.

• Shared query_zoids() test helper in conftest.py.

Security

Security review fixes (addresses #11):

• CAT-C1: Replace f-string DDL in BM25Backend.install_schema() with psycopg.sql.SQL/Identifier/Literal composition. Validate language codes against LANG_TOKENIZER_MAP allowlist + validate_identifier() on all generated column/index/tokenizer names.

• CAT-H1: Clamp sort_limit/b_size to _MAX_LIMIT (10,000) and b_start to _MAX_OFFSET (1,000,000) to prevent resource exhaustion.

• CAT-H2: Validate RRULE strings in DateRecurringIndexTranslator.extract() against RFC 5545 pattern and _MAX_RRULE_LENGTH (1,000) before storing.

• CAT-H3: Truncate full-text search queries to _MAX_SEARCH_LENGTH (1,000) to prevent excessive tsvector parsing.

• CAT-M1: Replace f-string SQL in clear_catalog_data() with psycopg.sql.Identifier for extra column names.

• CAT-M2: Add conn.closed guard in release_request_connection() to handle already-closed connections; document pool leak recovery in docstring.

• CAT-M3: Add defensive validate_identifier(index_name) in DateRecurringIndexTranslator.query().

• CAT-L1: Simplify error messages to not expose internal limit values.

• CAT-L2: Add rate limiting guidance note in searchResults() docstring.

• CAT-L3: Normalize double slashes in _validate_path().

1.0.0b7

Fixed

• sort_on now accepts a list of index names for multi-column sorting, matching ZCatalog’s API. sort_order can also be a list (one direction per sort key) or a single string applied to all keys.

• PGCatalogBrain.__getattr__ now distinguishes known catalog fields from unknown attributes. Known indexes and metadata columns return None when absent from idx (matching ZCatalog’s Missing Value behavior), while unknown attributes raise AttributeError. This enables CatalogContentListingObject.__getattr__ to fall back to getObject() for non-catalog attributes (e.g. content_type), and fixes PAM’s get_alternate_languages() viewlet crash on brain.Language.

• reindexIndex now accepts pghandler keyword argument for compatibility with ZCatalog’s manage_reindexIndex and plone.distribution. The argument is accepted but ignored (PG-based reindexing doesn’t need progress reporting). [#9]

• clearFindAndRebuild now properly rebuilds the catalog by traversing all content objects after clearing PG data. Previously only cleared without rebuilding.

• refreshCatalog now properly re-catalogs objects by resolving them from ZODB and re-extracting index values. Added missing pghandler parameter for ZCatalog API compatibility.

• Fixed ConnectionStateError on Zope restart when a Plone site already exists in the database. _sync_registry_from_db and _detect_languages_from_db now abort the transaction before closing their temporary ZODB connections.

• _ensure_catalog_indexes now checks for essential Plone indexes (UID, portal_type) instead of any indexes, preventing addon indexes from blocking re-application of Plone defaults.

• ZCatalog internal API compatibility: getpath(rid), getrid(path), Indexes["UID"]._index.get(uuid), and uniqueValues(withLengths=True) now work with PG-backed data. Uses ZOID as the record ID. This fixes plone.api.content.get(UID=...), plone.app.vocabularies content validation, and dexterity type counting in the control panel.

1.0.0b6

Added

• Relevance-ranked search results: SearchableText queries now automatically return results ordered by relevance when no explicit sort_on is specified. Title matches rank highest (weight A), followed by Description (weight B), then body text (weight D). Uses PostgreSQL’s built-in ts_rank_cd() with cover density ranking. No extensions required. Note: Requires a full catalog reindex after upgrade.

• Optional BM25 ranking via VectorChord-BM25 extension. When vchord_bm25 and pg_tokenizer extensions are detected at startup, search results are automatically ranked using BM25 (IDF, term saturation, length normalization) instead of ts_rank_cd. Title matches are boosted via combined text. Vanilla PostgreSQL installations continue using weighted tsvector ranking with no changes needed. Requires: vchord_bm25 + pg_tokenizer PostgreSQL extensions. Note: Full catalog reindex required after enabling.

• Per-language BM25 columns: each configured language gets its own bm25vector column with a language-specific tokenizer. Supports 30 Snowball stemmers (Arabic to Yiddish), jieba (Chinese), and lindera (Japanese/Korean). Configure via PGCATALOG_BM25_LANGUAGES environment variable (comma-separated codes, or auto to detect from portal_languages). Fallback column for unconfigured languages ensures BM25 ranking benefits for all content. Note: Changing languages requires full catalog reindex.

• SearchBackend abstraction: thin interface for swappable search/ranking backends. TsvectorBackend (always available) and BM25Backend (optional). Backend auto-detected at Zope startup.

• LANG_TOKENIZER_MAP in backends.py maps ISO 639-1 codes to pg_tokenizer configurations. Regional variants (pt-br, zh-CN) are normalized to base codes automatically.

• Estonian (et) added to language-to-regconfig mapping (supported by PG 17).

• Multilingual example: create_site.py zconsole script creates a Plone site with plone.app.multilingual (EN, DE, ZH), installs plone.pgcatalog, and imports ~800+ Wikipedia geography articles across all three languages with PAM translation linking. fetch_wikipedia.py fetches articles from en/de/zh Wikipedia with cross-language links. See example/README.md.

Fixed

• reindexObjectSecurity now works for newly created objects. unrestrictedSearchResults extends PG results with objects from the thread-local pending store (not yet committed to PG) for path queries. Previously, newly created objects were invisible to the path search in CMFCatalogAware.reindexObjectSecurity, so their security indexes (e.g. allowedRolesAndUsers) were never updated during workflow transitions in the same transaction.

• CatalogSearchResults now implements IFiniteSequence, enabling IContentListing adaptation in Plone’s search view.

• PGCatalogBrain now provides getId (property) and pretty_title_or_id() for compatibility with Plone’s Classic UI navigation and search templates. getId is a property (not a method) so brain.getId returns a string, matching standard ZCatalog brain behavior.

• PGCatalogBrain.__getattr__ returns None for missing idx keys instead of raising AttributeError, matching ZCatalog’s Missing Value behavior. Fixes PAM’s get_alternate_languages() viewlet crash on brain.Language.

• Unknown catalog indexes (e.g. Language, TranslationGroup from plone.app.multilingual) now fall back to JSONB field queries instead of being silently skipped. This enables PAM’s translation registration and lookup queries to work correctly.

• CJK tokenizer TOML format fixed: jieba (Chinese) and lindera (Japanese/Korean) now use the correct table syntax for pg_tokenizer’s pre_tokenizer configuration.

1.0.0b5

Added

• Add partial idx JSONB updates for lightweight reindex. [#6]

  • When reindexObject(idxs=[...]) is called with specific index names (e.g. during reindexObjectSecurity), extract only the requested values and register a JSONB merge patch (idx || patch) instead of full ZODB serialization + full idx column replacement

  • Avoids _p_changed = True and the associated pickle-JSON round-trip for every object in a subtree

  • Uses the new finalize(cursor) hook from zodb-pgjsonb to apply partial JSONB merges atomically in the same PG transaction

1.0.0b4

Added

• Language-aware full-text search: SearchableText now uses per-object language for stemming. The pgcatalog_lang_to_regconfig() PL/pgSQL function maps Plone language codes (ISO 639-1, 30 languages) to PostgreSQL text search configurations (e.g. "de" → german). Falls back to 'simple' for unmapped or missing languages. Non-multilingual sites are unaffected.

  Python mirror: columns.language_to_regconfig() for testing/validation.

• Title/Description text search: Title and Description queries now use tsvector word-level matching instead of exact JSONB containment. catalog(Title="Hello") now correctly matches "Hello World". Backed by GIN expression indexes with 'simple' config (no stemming).

• Automatic addon ZCTextIndex support: Addon-registered ZCTextIndex fields are automatically discovered at startup. GIN expression indexes are created dynamically by _ensure_text_indexes(), and queries use tsvector matching – zero addon code needed.

Fixed

• Title/Description query broken: Previously, querying Title or Description as ZCTextIndex used JSONB exact containment (idx @> '{"Title":"Hello"}'), which only matched exact values, not words within text. Now uses to_tsvector/plainto_tsquery for proper word-level matching.

1.0.0b3

Fixed

• Snapshot consistency: Catalog read queries now route through the ZODB storage instance’s PG connection, sharing the same REPEATABLE READ snapshot as load() calls. Previously, catalog queries used a separate autocommit connection that could see a different database state than ZODB object loads within the same request.

  New internal API:

  • pool.get_storage_connection(context) — retrieves the PG connection from context._p_jar._storage.pg_connection.

  • PlonePGCatalogTool._get_pg_read_connection() — prefers storage connection, falls back to pool for non-ZODB contexts (tests, scripts).

  CatalogSearchResults now accepts a conn parameter (was pool) for lazy idx batch loading, using the same connection directly.

1.0.0b2

Security

• SQL identifier validation: Added validate_identifier() in columns.py to reject unsafe SQL identifiers. All idx_key values in IndexRegistry and date_attr in DateRecurringIndexTranslator are now validated.

• Access control declarations: Added declareProtected for management methods (refreshCatalog, reindexIndex, clearFindAndRebuild) and declarePrivate for unrestrictedSearchResults on PlonePGCatalogTool.

• API safety: Renamed execute_query() to _execute_query() to mark as internal API. Capped path query list size to 100 (DoS prevention). Documented security contract for IPGIndexTranslator implementations.

Fixed

• Savepoint-aware pending store: The thread-local pending catalog data now participates in ZODB’s transaction lifecycle via ISavepointDataManager. Fixes two bugs: pending data not reverting on savepoint rollback, and stale pending data leaking across transactions after abort.

1.0.0b1 Initial release (2026-02-10)

Changed

• ZCatalog BTree write elimination: Removed super() delegation in indexObject(), reindexObject(), catalog_object(), and uncatalog_object(). All catalog data now flows exclusively to PostgreSQL via CatalogStateProcessor — no BTree/Bucket objects are written to ZODB. Content creation dropped from 175 ms/doc to 68.5 ms/doc (2.5x faster), making PGCatalog 1.13x faster than RelStorage+ZCatalog for writes.

Added

• Dynamic IndexRegistry: Replaced static KNOWN_INDEXES dict with a dynamic IndexRegistry that discovers indexes from ZCatalog at startup via sync_from_catalog(). Addons that add indexes via catalog.xml profiles are now automatically supported without code changes.

  • META_TYPE_MAP maps ZCatalog meta_types (FieldIndex, KeywordIndex, DateIndex, etc.) to IndexType enum values.

  • SPECIAL_INDEXES (SearchableText, effectiveRange, path) have dedicated PG columns and are excluded from idx JSONB extraction.

  • Registry entries are 3-tuples: (IndexType, idx_key, source_attrs), where source_attrs supports indexed_attr differing from index name.

  • Startup sync via _sync_registry_from_db() populates the registry from each Plone site’s portal_catalog before the first request.

• IPGIndexTranslator utility: Named utility interface for custom index types not covered by META_TYPE_MAP. Wired into query.py (query + sort fallback) and catalog.py (extraction fallback).

• DateRecurringIndex support: Built-in translator for Products.DateRecurringIndex (Plone’s start / end event indexes). Stores base date + RFC 5545 RRULE string in idx JSONB; queries use rrule_plpgsql (pure PL/pgSQL, no C extensions) for recurrence expansion at query time. Translators are auto-discovered from ZCatalog at startup – no manual configuration needed. Container-friendly: works on standard postgres:17 images without additional extensions.

• DDL via get_schema_sql(): CatalogStateProcessor now provides DDL through the get_schema_sql() method, applied by PGJsonbStorage using its own connection — no REPEATABLE READ lock conflicts during startup.

• Transactional catalog writes: catalog_object() sets a _pgcatalog_pending annotation on persistent objects. The CatalogStateProcessor extracts this annotation during ZODB commit and writes catalog columns (path, parent_path, path_depth, idx, searchable_text) atomically alongside the object state.

• PlonePGCatalogTool: PostgreSQL-backed portal_catalog replacement for Plone, inheriting from Products.CMFPlone.CatalogTool. Registered via GenericSetup toolset.xml.

• plone.restapi compatibility: CatalogSearchResults inherits ZTUtils.Lazy.Lazy for serialization; PGCatalogBrain implements ICatalogBrain for IContentListingObject adaptation.