Upgrades, backups, retention
How to upgrade a self-hosted engine without losing data, what to back up and how to restore it, and how to choose how long each kind of data is kept.
On this page
- Upgrades
- What an upgrade may change
- Upgrade
- Read what changed
- Back up the database
- Get the new image and restart the engine
- Check
- Go back
- Backups
- What needs one
- Back up Postgres
- Restore
- Stop the engine
- Restore the dump
- Start the engine and check
- Process again what arrived after the backup
- If the bucket is lost
- Retention
- Change it
- How to choose
- What retention does not cover
- Related
Engine This page is for whoever keeps a self-hosted engine running after the first day. On Sypnia Cloud all three jobs are ours: we upgrade, we back up, and your plan sets the retention.
Upgrades
What an upgrade may change
The engine is built so that an upgrade inside version 1 is boring. These rules are part of its contract:
| Rule | What it means for you |
|---|---|
| Database migrations only add. | An older engine keeps working against a newer schema. You can migrate first, replace the engine after, and go back without touching the database. |
The API inside /v1 only adds. | Scripts, the CLI, the MCP server and your coding agent keep working. |
| Raw events keep fields the engine does not know. | A newer SDK never breaks an older engine. Upgrade the engine and your apps in any order. |
An unknown SYPNIA_… variable is a warning in sypnia doctor, not an error. | A setting that was renamed or removed does not stop an upgrade. The changelog names it. |
| Processing is safe to repeat. | A worker killed halfway through an upgrade loses nothing and counts nothing twice. |
A change that breaks one of these would be version 2, announced in the changelog, never a surprise inside version 1.
Which version you run: sypnia version, or the Sypnia-Version header on any API answer. The image tag ghcr.io/sypnia/sypnia:1 follows the latest release of version 1. To decide yourself when each upgrade happens, use an exact tag such as ghcr.io/sypnia/sypnia:1.0.0 and change it by hand.
Upgrade
At the end you will have: the new version running, the same data, and a backup taken just before.
AWS
Follow Update the engine on the AWS page: read the changelog, update the command and the template, run sypnia migrate, deploy. Take a snapshot of the database first:
aws rds create-db-snapshot \
--db-instance-identifier <DatabaseIdentifier> \
--db-snapshot-identifier sypnia-before-upgrade-$(date +%F)Ingest does not stop: the old functions answer until the new ones are ready.
Docker
Read what changed
The changelog says what each version changes and whether it asks anything of you.
Get the new image and restart the engine
Terminal docker compose pull sypnia docker compose up -d sypnia docker compose logs --tail 20 sypniaYou should seeThe engine applies its migrations as it starts, because
SYPNIA_AUTO_MIGRATEistrue, then says it is ready:Text schema 12 -> 13 applied 0013_… (0.3 s) sypnia 1.1.0 role=all listen=0.0.0.0:8080 ready https://sypnia.example.com
The engine is away for a few seconds while the container restarts. SDKs hold their events and try again after 1, 4 and 15 seconds, so a restart of that length loses nothing.
If you run several processes with SYPNIA_AUTO_MIGRATE set to false, migrate once before you restart any of them, then restart them one at a time:
docker compose run --rm sypnia sypnia migrateGo back
Start the previous version again: the previous image tag, or the previous template. Leave the database as it is. The older engine works against the newer schema, and what arrived in between is kept.
Do not restore a database backup to undo an upgrade. A restore is for a lost or damaged database, and it discards everything people did since the backup.
Backups
What needs one
| Data | Where | Can it be rebuilt? | Back it up? |
|---|---|---|---|
| Projects, ingest keys, API tokens, project settings, alert rules, webhooks, uptime checks | Postgres | No | Yes |
| What people did: problems resolved or ignored, notes, alerts acknowledged | Postgres | No | Yes |
| The agent’s analyses | Postgres | Only by running the agent again, which costs model calls | Yes |
| Problems’ counts, the session index, performance numbers, the trace index | Postgres | Yes, from events/ in the bucket, for as long as the events are kept | Comes with the same backup |
| Raw events, per-session files, replays, page snapshots | Bucket | No. They are the source | Your choice, see below |
| Source maps | Bucket | Yes: your CI uploads them again on the next build | No |
| The engine’s settings and the root token | Your .env file, or the stack’s parameters | No | Yes, in a password manager or a secret store |
So the rule is short: back up Postgres every day, and keep the settings somewhere safe. A database backup is small, because raw events are never in it.
The bucket is a different decision. A hosted store such as Amazon S3 or Cloudflare R2 already keeps several copies of every object; most people stop there. A MinIO on one disk, as in the Compose file, is one disk: if it dies, the raw events and the replays are gone, and Sypnia keeps only what Postgres holds. If that matters to you, copy the bucket somewhere else on a schedule, or use a hosted store.
Back up Postgres
AWS
The database the template creates is backed up by AWS every day and each backup is kept 7 days. You can also go back to any minute of those 7 days. Nothing to set up. To check:
aws rds describe-db-instances --db-instance-identifier <DatabaseIdentifier> \
--query "DBInstances[0].[BackupRetentionPeriod,LatestRestorableTime]"7 and a time a few minutes ago.If you brought your own Postgres, its backups are your provider’s or yours.
Docker
Add one line to the server’s crontab. It writes a dump every night at 03:00 and deletes dumps older than 14 days.
0 3 * * * cd /opt/sypnia && docker compose exec -T db pg_dump -U sypnia -Fc sypnia > backups/sypnia-$(date +\%F).dump && find backups -name 'sypnia-*.dump' -mtime +14 -deleteThen copy backups/ off the server, with whatever you already use. A backup that lives on the same disk as the database protects against a mistake, not against losing the disk.
Restore
At the end you will have: the database as it was at the backup, plus everything that can be rebuilt from the events that arrived since.
Process again what arrived after the backup
The bucket still holds every event. Tell the worker to read them again from the time of the backup:
Terminal docker compose exec sypnia sypnia aggregate --from 2026-03-12T03:00:00ZYou should seeText shop events 48211 -> 361 sessions, 4 problems (1 new), 22 routes 9.4 s done in 11.0 s, data through 2026-03-12T16:20:05ZIt lists the bucket for that range, so it also finds events the restored database has never heard of. Running it twice is safe.
What a restore cannot bring back is what people wrote after the backup:
- A problem resolved or ignored since then is open again.
- Ingest keys and API tokens created since then no longer exist. Apps that use them are refused: create them again and deploy.
- A problem first seen since then is new again, so it alerts again, and your webhook receives it a second time.
- Analyses written since then are run again when their problem grows, or when you ask.
Try a restore once, on a copy, before the day you need it.
If the bucket is lost
Postgres still has every problem, its sentence and its counts, and all the numbers. Sessions stay in the list and answer 410 expired when opened, because their files are gone. Point the engine at a new, empty bucket, run sypnia doctor, and new data flows. Source maps come back with your next build.
Retention
The engine deletes old data itself, once a day. You choose the number of days for each kind; the bucket needs no rule for it.
| Data | Default | Setting |
|---|---|---|
| Raw events, sessions with their timelines, traces, the daily Parquet files | 90 days | SYPNIA_RETENTION_EVENTS_DAYS (1 to 3650) |
| Replay recordings | 30 days | SYPNIA_RETENTION_REPLAYS_DAYS (not more than the events) |
| Page snapshots | 90 days | SYPNIA_RETENTION_SNAPSHOTS_DAYS |
| Hourly and daily numbers: performance, click maps, usage, uptime | 400 days | SYPNIA_RETENTION_NUMBERS_DAYS |
| Source maps | 90 days after the release was last seen | SYPNIA_RETENTION_SOURCEMAPS_DAYS |
| Problems: the sentence, the analysis, the status, the daily counts and one example | While the project exists | None |
| Projects, keys, tokens, settings | Until you delete them | None |
How it works:
- A session’s age is counted from its last event. Data is deleted within 24 hours of passing its age.
- The settings are for the whole engine. A self-hosted engine has no retention per project;
GET /v1/usagereturns the retention that applies. - A problem outlives the sessions it hit. That is what lets Sypnia say «came back» months later. Its linked sessions, replays and traces expire on their own schedule.
- After that, asking for expired data answers
410 expired, not an empty result.
Change it
Set the variable and restart the engine. On AWS, add the line to the template as in Change a setting.
SYPNIA_RETENTION_EVENTS_DAYS=30
SYPNIA_RETENTION_REPLAYS_DAYS=14- Shorter: everything older than the new number is deleted at the next daily cleanup. It cannot be undone, except from a backup.
- Longer: data is kept longer from now on. What was already deleted does not come back.
To run the cleanup now instead of waiting for the daily one:
sypnia aggregate --task retentionHow to choose
- Replays cost the most space and age the fastest: a recording is useful while the problem is fresh. 14 to 30 days is enough for most teams. See Storage layout.
- Events are what lets you rebuild the database and open an old session’s timeline. Keep them at least as long as your oldest database backup.
- Numbers are small and hold nothing about people. Keeping them a year lets you compare this month with the same month last year.
- Personal data. Replays, sessions and the names your app sent with
identify()are personal data. Keep them only as long as you can explain. To delete one person or one session sooner, see Deleting data.
What retention does not cover
- Your backups. A database dump keeps what was deleted afterwards until the dump itself is deleted. Do not keep backups longer than the data they contain is meant to live.
- Logs. The engine writes logs to standard output and never logs bodies, keys or tokens. How long they are kept is up to whatever collects them: Docker, CloudWatch (30 days in the AWS template), your own collector. The logs of the proxy in front of the engine hold IP addresses.
- Bucket versions. If you turned on versioning, old versions outlive the engine’s deletes until a rule of yours removes them.
Related
- Where each kind of data is and how big it gets.
- Remove a person, a session or a project before retention does.
- What each version changes.