Enable Tika text extraction#

Overview#

By default, Plone indexes text from rich-text fields (Title, Description, body) into searchable_text. Binary content—PDFs, Word documents, Excel spreadsheets, images—is not searchable because Plone cannot extract text from them.

Apache Tika is a stateless HTTP service that extracts text from over 1400 file formats. Optical character recognition (OCR) for images and scanned PDFs is available only with the -full Tika image and additional configuration—see OCR for images and scanned PDFs. When enabled, plone.pgcatalog enqueues binary content for asynchronous extraction via a PostgreSQL job queue. A background worker sends each blob to Tika and merges the extracted text into the object’s searchable_text tsvector (and BM25 columns, if active).

This feature is entirely opt-in. Without PGCATALOG_TIKA_URL, behavior is unchanged.

Step 1: start Apache Tika#

Docker Compose#

If you use the zodb-pgjsonb example setup, Tika is available as a profile:

docker compose --profile tika up -d tika

Production#

In production, Tika should run as a separate service (or sidecar container) accessible from the Zope/worker processes. Tika is stateless and needs no persistent storage. A single Tika instance handles concurrent requests from multiple workers.

Typical resource allocation: 512 MB–1 GB RAM. OCR (the -full image) is CPU- and memory-heavy and much slower per document; size the Tika service accordingly and raise TIKA_WORKER_HTTP_TIMEOUT (see below) so large scanned PDFs do not time out.

Step 2: configure environment variables#

Set PGCATALOG_TIKA_URL before starting Zope:

export PGCATALOG_TIKA_URL=http://localhost:9998

This single variable enables the entire extraction pipeline:

  • The queue table (text_extraction_queue) and merge function are created at startup

  • The CatalogStateProcessor starts enqueuing extraction jobs for objects with extractable binary content

Optional: customize content types#

By default, the following MIME types are sent to Tika:

  • application/pdf

  • application/msword

  • application/vnd.openxmlformats-officedocument.wordprocessingml.document

  • application/vnd.openxmlformats-officedocument.spreadsheetml.sheet

  • application/vnd.openxmlformats-officedocument.presentationml.presentation

  • application/vnd.oasis.opendocument.text

  • application/vnd.oasis.opendocument.spreadsheet

  • application/rtf

  • image/jpeg, image/png, image/tiff, image/webp, image/gif

Override with a comma-separated list:

export PGCATALOG_TIKA_CONTENT_TYPES=application/pdf,application/msword,image/jpeg

Step 3: start the extraction worker#

The worker dequeues jobs, fetches blobs, sends them to Tika, and writes extracted text back to PostgreSQL. Two modes are available:

Option A: in-process worker (development)#

Add a second environment variable to run the worker as a daemon thread inside the Zope process:

export PGCATALOG_TIKA_URL=http://localhost:9998
export PGCATALOG_TIKA_INPROCESS=true

The thread starts automatically on Zope startup. It shares nothing with Zope’s ZODB connections—it opens its own PostgreSQL connection and HTTP client. The thread is marked daemon=True, so it stops when Zope shuts down.

This mode is convenient for development but uses Zope’s process resources. For production, use the standalone worker.

Option B: standalone worker (production)#

Run the worker as a separate process or container:

export TIKA_WORKER_DSN="dbname=zodb host=localhost port=5432 user=zodb password=zodb"
export TIKA_WORKER_URL=http://tika:9998
pgcatalog-tika-worker

The standalone worker:

  • Connects directly to PostgreSQL (no Zope dependency)

  • Uses LISTEN/NOTIFY for instant wakeup on new jobs

  • Falls back to polling every TIKA_WORKER_POLL_INTERVAL seconds (default: 5)

  • Waits up to TIKA_WORKER_HTTP_TIMEOUT seconds for each Tika response (default: 120; raise it for OCR of large scanned PDFs)

  • Uses SELECT ... FOR UPDATE SKIP LOCKED for safe concurrent dequeuing

  • Handles SIGTERM/SIGINT for graceful shutdown

For S3-tiered blobs:

export TIKA_WORKER_S3_BUCKET=zodb-blobs
export TIKA_WORKER_S3_ENDPOINT_URL=http://minio:9000
export TIKA_WORKER_S3_REGION=us-east-1
export TIKA_WORKER_S3_ACCESS_KEY=...
export TIKA_WORKER_S3_SECRET_KEY=...

If TIKA_WORKER_S3_ACCESS_KEY / TIKA_WORKER_S3_SECRET_KEY are not set, the worker leaves credential resolution to boto3’s default provider chain (the standard AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables, ~/.aws/credentials, or an instance/IAM role).

See Configuration reference for the full list of worker environment variables.

OCR for images and scanned PDFs#

OCR is not part of the default Tika image. The minimal apache/tika image bundles no OCR engine, so plain images, photos, and scanned (image-only) PDFs are extracted as empty text: the queue row still completes as done, but no body lexemes are merged into searchable_text. The full-text index then silently lacks their content.

To enable OCR:

  1. Use the -full image, which ships Tesseract and ImageMagick:

    docker run -d --name tika -p 9998:9998 apache/tika:3.2.3.0-full

  2. Configure OCR server-side. The worker sends a plain PUT /tika with no OCR headers, so the OCR strategy and languages are set on the Tika service via a mounted tika-config.xml—for example OCR language deu+eng, and a PDF ocrStrategy of auto (or ocr_and_text) so image-only PDFs are run through OCR. See the Apache Tika OCR documentation.

  3. Budget for it. OCR is CPU- and memory-heavy and much slower per document. Raise TIKA_WORKER_HTTP_TIMEOUT (default 120 s) so large multi-page scans do not time out and get marked failed.

If you do not need OCR, the minimal image is the better choice: it is smaller, faster, and avoids the resource cost.

Step 4: rebuild the catalog#

A full reindex is needed to enqueue extraction jobs for existing objects:

Go to ZMI > portal_catalog > Advanced tab 2. Click “Clear and Rebuild”

Or via script:

catalog = portal.portal_catalog
catalog.clearFindAndRebuild()
import transaction

transaction.commit()

After the rebuild, the worker processes enqueued jobs. You can monitor progress:

-- Pending jobs
SELECT COUNT(*) FROM text_extraction_queue WHERE status = 'pending';

-- Completed jobs
SELECT COUNT(*) FROM text_extraction_queue WHERE status = 'done';

-- Failed jobs
SELECT * FROM text_extraction_queue WHERE status = 'failed';

Step 5: verify extraction#

Upload a PDF via Plone and wait a few seconds. Then query:

SELECT searchable_text::text
FROM object_state
WHERE path LIKE '%/my-uploaded-file';

The tsvector should contain terms extracted from the PDF content (at weight C), alongside the synchronous Title/Description terms (at weights A/B).

How it fits with BM25#

When BM25 is active, the merge function also updates per-language BM25 columns. Title gets 3x boosting (weight A), Description gets weight B, and extracted blob text gets weight C. This means a search for “quantum computing” ranks a document with “quantum computing” in the title higher than one that only mentions it in an attached PDF—exactly the right behavior.

See Tika text extraction architecture for a detailed architecture explanation.

Disabling extraction#

Remove PGCATALOG_TIKA_URL from the environment and restart Zope. The queue table remains but no new jobs are enqueued. Existing searchable_text values are preserved.

To clean up the queue table:

DROP TABLE IF EXISTS text_extraction_queue CASCADE;