How to configure S3 blob tiering#

This guide shows you how to configure tiered blob storage so that small blobs stay in PostgreSQL and large blobs go to S3.

Install the S3 extra#

pip install zodb-pgjsonb[s3]
# or
uv pip install zodb-pgjsonb[s3]

This installs the zodb-s3blobs dependency with the boto3 S3 client.

Add S3 configuration to ZConfig#

Add the S3 keys to your <pgjsonb> section in zope.conf:

%import zodb_pgjsonb

<zodb_db main>
    <pgjsonb>
        dsn dbname=zodb user=zodb host=localhost port=5432

        s3-bucket-name my-zodb-blobs
        s3-region us-east-1
        s3-prefix production/

        blob-threshold 100KB
        blob-cache-dir /var/cache/zodb-blobs
        blob-cache-size 1GB
    </pgjsonb>
</zodb_db>

S3 configuration keys#

Key	Default	Description
`s3-bucket-name`	none	S3 bucket name (setting this enables S3 tiering)
`s3-region`	none	AWS region name
`s3-endpoint-url`	none	S3 endpoint URL (for MinIO, Ceph, or other S3-compatible stores)
`s3-access-key`	none	AWS access key ID (uses boto3 credential chain if omitted)
`s3-secret-key`	none	AWS secret access key (uses boto3 credential chain if omitted)
`s3-use-ssl`	`true`	Enable SSL for S3 connections
`s3-prefix`	empty	S3 key prefix for namespace isolation

Blob storage keys#

Key	Default	Description
`blob-threshold`	`100KB`	Blobs larger than this go to S3
`blob-cache-dir`	auto	Local cache directory for S3 blobs
`blob-cache-size`	`1GB`	Maximum size of the local blob cache

Choose a blob-threshold strategy#

The blob-threshold setting controls how blobs are distributed between PostgreSQL and S3.

PG-only mode (no S3): Omit s3-bucket-name entirely. All blobs stay in PostgreSQL bytea columns regardless of size.

Tiered mode (default with S3): Set blob-threshold to a size like 100KB (the default). Blobs smaller than this stay in PostgreSQL for fast access. Blobs at or above this threshold go to S3.

S3-only mode: Set blob-threshold to 0. All blobs go to S3, keeping the PostgreSQL database compact.

Set up a local blob cache#

Set blob-cache-dir to a local directory for caching S3 blobs on disk. This avoids repeated S3 downloads for frequently accessed blobs.

Set blob-cache-size to the maximum disk space for the cache. The cache uses LRU eviction when the limit is reached.

For production, always configure a blob cache directory to reduce S3 latency and cost.

Use MinIO for development#

Start a MinIO container:

docker run -d --name minio \
  -p 9000:9000 -p 9001:9001 \
  -e MINIO_ROOT_USER=minioadmin \
  -e MINIO_ROOT_PASSWORD=minioadmin \
  minio/minio server /data --console-address ":9001"

Create a bucket:

mc alias set local http://localhost:9000 minioadmin minioadmin
mc mb local/zodb-blobs

Configure the storage to use MinIO:

<pgjsonb>
    dsn dbname=zodb user=zodb host=localhost port=5432

    s3-bucket-name zodb-blobs
    s3-endpoint-url http://localhost:9000
    s3-access-key minioadmin
    s3-secret-key minioadmin
    s3-use-ssl false

    blob-threshold 100KB
</pgjsonb>