Migrate a Plone site from FileStorage#

What we will build#

In this tutorial we will migrate an existing Plone site from FileStorage (the default Data.fs + blob directory) to zodb-pgjsonb, storing all object state as queryable JSONB in PostgreSQL.

By the end we will have a running Plone site backed entirely by PostgreSQL, with the original FileStorage preserved as a backup.

Prerequisites#

  • A working Plone 6 site using FileStorage (the default storage)

  • PostgreSQL 15+ running and accessible

  • Python 3.12+ with the same virtual environment as the Plone site

  • uv (recommended) or pip

Step 1: install zodb-pgjsonb#

Activate the virtual environment that our Plone site uses and install zodb-pgjsonb:

source .venv/bin/activate
uv pip install zodb-pgjsonb

This also installs zodbconvert, which is part of the ZODB package.

Step 2: prepare the PostgreSQL database#

Create a dedicated database for the ZODB data. Connect to PostgreSQL as a superuser and run:

psql -h localhost -p 5432 -U postgres

CREATE USER zodb WITH PASSWORD 'zodb';
CREATE DATABASE zodb OWNER zodb;
\q

Verify the connection:

psql -h localhost -p 5432 -U zodb -d zodb -c "SELECT version();"

We should see the PostgreSQL version string. zodb-pgjsonb creates all required tables (object_state, transaction_log, blob_state, etc.) automatically on first connection.

Step 3: stop the Plone site#

We must stop Plone before migrating to avoid writes to the source FileStorage during the copy. Use Ctrl+C in the terminal where Zope is running, or stop the process through your process manager (supervisord, systemd, etc.).

Step 4: write the zodbconvert configuration#

Create a file called migrate.cfg in the project directory:

<source>
    <filestorage>
        path /path/to/var/filestorage/Data.fs
        blob-dir /path/to/var/blobstorage
    </filestorage>
</source>

<destination>
    %import zodb_pgjsonb
    <pgjsonb>
        dsn dbname=zodb user=zodb password=zodb host=localhost port=5432
    </pgjsonb>
</destination>

Replace /path/to/var/filestorage/Data.fs and /path/to/var/blobstorage with the actual paths from the existing Zope configuration.

Tip

Look in the current zope.conf for the <filestorage> section to find the exact paths. A typical Plone installation uses instance/var/filestorage/Data.fs and instance/var/blobstorage.

Step 5: run the migration#

Run zodbconvert to copy all transactions from FileStorage to PostgreSQL:

.venv/bin/zodbconvert --clear migrate.cfg

The --clear flag ensures the destination database starts empty.

We should see progress output as transactions are copied:

Copying transactions from FileStorage to PGJsonbStorage
[00:15] 8,200 / 28,394 records (28.9%) | 547 rec/s | 12.3 MB/s
[00:30] 17,500 / 28,394 records (61.6%) | 620 rec/s | 11.8 MB/s
Copied 1,542 transactions with 28,394 records in 45.2s (628 rec/s, 12.3 MB/s)

Note

The migration speed depends on database size, network latency, and disk I/O. A typical Plone site with 50,000 objects migrates in under 5 minutes.

Parallel migration for large databases#

For large databases, zodb-pgjsonb supports parallel migration using multiple PostgreSQL connections. The zodbconvert CLI does not expose a --workers flag, so we use a short Python script instead:

Create a file called migrate_parallel.py:

from ZODB.FileStorage import FileStorage
from ZODB.blob import BlobStorage
from zodb_pgjsonb import PGJsonbStorage

# Open the source FileStorage (read-only)
fs = FileStorage("/path/to/var/filestorage/Data.fs", read_only=True)
source = BlobStorage("/path/to/var/blobstorage", fs)

# Open the destination with a larger connection pool
dest = PGJsonbStorage(
    dsn="dbname=zodb user=zodb password=zodb host=localhost port=5432",
    pool_max_size=8,
)

# Copy with 4 parallel workers
dest.copyTransactionsFrom(source, workers=4)

source.close()
dest.close()

Run the script:

python migrate_parallel.py

The main thread reads from FileStorage and decodes pickles. Worker threads write to PostgreSQL concurrently. OID ordering is preserved automatically.

We should see progress output with throughput and ETA:

Copying transactions with 4 parallel workers (total OIDs: 128,503)
[00:12] 15,200 / 128,503 OIDs (11.8%) | 1,267 obj/s | ETA 01:30
[00:24] 31,400 / 128,503 OIDs (24.4%) | 1,350 obj/s | ETA 01:12
[00:48] 72,100 / 128,503 OIDs (56.1%) | 1,502 obj/s | ETA 00:38
[01:38] All 8,421 transactions copied with 4 workers.

Tip

Set pool_max_size to at least the number of workers. Using 4 to 6 workers provides the best throughput for most deployments.

Step 6: update zope.conf#

Replace the <filestorage> section in zope.conf with a <pgjsonb> section:

%import zodb_pgjsonb

<zodb_db main>
    mount-point /
    cache-size 30000
    <pgjsonb>
        dsn dbname=zodb user=zodb password=zodb host=localhost port=5432
        blob-temp-dir ./instance/var/blobtemp
    </pgjsonb>
</zodb_db>

Create the blob temp directory:

mkdir -p instance/var/blobtemp

Note

Remove or comment out the old <filestorage> and <blobstorage> sections. Do not delete the original Data.fs and blob directory yet – keep them as a backup until we verify the migration.

Step 7: start and verify#

Start the Plone site:

.venv/bin/runwsgi instance/etc/zope.ini

We should see log output ending with:

INFO  [waitress:486] Serving on http://0.0.0.0:8081

Open the site in a browser and verify:

  1. Log in and browse existing content.

  2. Check that images and file downloads work (blob data).

  3. Create a new page to confirm writes work.

Verify with SQL#

Open a second terminal and connect to PostgreSQL:

psql -h localhost -p 5432 -U zodb -d zodb

Count the objects:

SELECT count(*) FROM object_state;

List the content types:

SELECT class_mod || '.' || class_name AS class,
       count(*) AS count
FROM object_state
GROUP BY 1
ORDER BY 2 DESC
LIMIT 10;

We should see the same objects that existed in the original FileStorage.

What we learned#

  • zodb-pgjsonb implements IStorageRestoreable, making it compatible with zodbconvert

  • The standard zodbconvert tool copies all transactions, including blobs

  • For large databases, the Python API copyTransactionsFrom(source, workers=N) uses parallel PostgreSQL connections for higher throughput

  • After migration, we swap the <filestorage> section for <pgjsonb> in zope.conf

  • All ZODB data is now queryable JSON in PostgreSQL

Next steps#