How to migrate with zodbconvert#

This guide shows you how to migrate an existing ZODB database (FileStorage or RelStorage) to zodb-pgjsonb using zodb-convert or the Python API.

Write a zodbconvert configuration file#

Create a file named migrate.cfg:

From FileStorage#

<source>
    <filestorage>
        path /path/to/Data.fs
        blob-dir /path/to/blobstorage
    </filestorage>
</source>
<destination>
    %import zodb_pgjsonb
    <pgjsonb>
        dsn dbname=zodb_new user=zodb host=localhost port=5432
    </pgjsonb>
</destination>

From RelStorage#

<source>
    <relstorage>
        <postgresql>
            dsn dbname=zodb_old user=zodb host=localhost
        </postgresql>
    </relstorage>
</source>
<destination>
    %import zodb_pgjsonb
    <pgjsonb>
        dsn dbname=zodb_new user=zodb host=localhost port=5432
    </pgjsonb>
</destination>

Run the migration#

zodb-convert migrate.cfg

This copies all transactions sequentially, including blobs. The destination storage creates its schema automatically.

zodb-convert is a standalone tool that works with any ZODB-compatible storage. Install it with pip install zodb-convert (or uv pip install zodb-convert).

Run a parallel migration#

For large databases, use multiple worker threads for faster migration:

zodb-convert -w 4 migrate.cfg

This delegates to the destination storage’s copyTransactionsFrom(source, workers=4). The main thread iterates the source and decodes pickles; worker threads write to PostgreSQL concurrently. Set pool-max-size in your destination config to at least the number of workers plus one.

Python API#

If you need more control, use the Python API directly:

from zodb_pgjsonb.storage import PGJsonbStorage
from ZODB.FileStorage import FileStorage
from ZODB.blob import BlobStorage

source = BlobStorage("/path/to/blobstorage", FileStorage("/path/to/Data.fs"))
dest = PGJsonbStorage(
    dsn="dbname=zodb_new user=zodb host=localhost",
    pool_max_size=10,
)
dest.copyTransactionsFrom(source, workers=4)
dest.close()
source.close()

Resume an interrupted migration#

If a migration is interrupted (Ctrl-C, crash, network failure), resume with --incremental:

zodb-convert --incremental -w 4 migrate.cfg

During parallel copy, the storage tracks a watermark — the highest TID where all prior TIDs are also committed. On resume, iteration starts from the watermark (not lastTransaction()) to fill any gaps left by out-of-order worker commits. Already-committed transactions are skipped automatically. The watermark table is dropped on successful completion, adding zero overhead to non-incremental imports.

Python API#

from zodb_pgjsonb.storage import PGJsonbStorage
from ZODB.FileStorage import FileStorage
from ZODB.blob import BlobStorage
from ZODB.utils import p64, u64

source = BlobStorage("/path/to/blobstorage", FileStorage("/path/to/Data.fs"))
dest = PGJsonbStorage(
    dsn="dbname=zodb_new user=zodb host=localhost",
    pool_max_size=10,
)

# Determine where to resume from
start_tid = p64(u64(dest.lastTransaction()) + 1)
dest.copyTransactionsFrom(source, workers=4, start_tid=start_tid)
dest.close()
source.close()

Migrate blobs#

Blobs are migrated automatically by both zodbconvert and copyTransactionsFrom. The destination storage applies its blob tiering rules (PG bytea vs S3) based on the configured blob-threshold.

If you have S3 tiering configured on the destination, large blobs are uploaded to S3 during the migration.

Faster blob migration with S3#

When migrating large databases with many blobs to S3-backed storage, S3 upload latency can be the bottleneck. Two modes decouple S3 uploads from PostgreSQL writes:

Deferred mode#

Skips S3 uploads entirely during migration, writing a manifest file instead. Upload blobs later in a separate step:

# Step 1: Fast PG-only migration
zodb-convert -w 8 --deferred-blobs /tmp/blob-manifest.tsv migrate.cfg

# Step 2: Upload blobs from manifest
zodb-convert --upload-blobs /tmp/blob-manifest.tsv --dest-zope-conf zope.conf -w 16

This is useful when S3 is temporarily unavailable or when you want to run the PG import at maximum speed and handle S3 as a separate operational step.

Note: Temp blob files are preserved during deferred mode (they are needed for the upload step). Ensure the staging directory has sufficient disk space.

Python API#

# Background mode
dest.copyTransactionsFrom(source, workers=4, blob_mode="background")

# Deferred mode
dest.copyTransactionsFrom(source, workers=4, blob_mode="deferred:/tmp/manifest.tsv")

Verify the migration#

Connect to the destination database and check the object count:

SELECT count(*) FROM object_state;
SELECT count(*) FROM transaction_log;
SELECT count(*) FROM blob_state;

Open the database with ZODB and verify application data:

from zodb_pgjsonb.storage import PGJsonbStorage
import ZODB

storage = PGJsonbStorage(dsn="dbname=zodb_new user=zodb host=localhost")
db = ZODB.DB(storage)
conn = db.open()
root = conn.root()
print(list(root.keys()))
conn.close()
db.close()