Scaling and cost
What a self-hosted engine costs each month for a small and a medium project, with the arithmetic, what makes the bill move, the first limits you meet, and when to go from one process to several or to functions.
On this page
- What makes the bill move
- Two example projects
- On AWS
- On one server with Docker
- Which shape costs less
- The first limits you meet
- Database connections
- From one process to several
- Stop migrating at start
- Start one process per role
- Send each path to its process
- From processes to functions
- What is not known yet
- Related
Engine This page is for whoever pays for a self-hosted engine, or must say what it will cost before it exists. On Sypnia Cloud you pay a plan instead, and capacity is our job: see Plans and limits.
What makes the bill move
In the order you will feel them:
| Driver | Grows with | What you can change |
|---|---|---|
| The database | Almost nothing: it is a fixed monthly price, and at small sizes the largest line. | Bring a Postgres you already pay for. Any Postgres 15 or later works, with a database of its own. |
| Writes to the bucket | Recorded sessions (one write per 10 seconds of recording), sessions (their files are written again as they grow), and deploys (one write per source map). Not events: those are grouped into segments. | SYPNIA_REPLAY_SAMPLE_RATE for new projects, or each project’s own settings.replay.sample_rate. Upload source maps only for environments someone looks at. |
| Storage in the bucket | Events × days kept, and recordings × days kept. Recordings are about three quarters of it. | SYPNIA_RETENTION_REPLAYS_DAYS, SYPNIA_RETENTION_EVENTS_DAYS. |
| Compute | On AWS, requests and the worker’s seconds. On a server, nothing until the server is full. | The shape you deploy. |
| The model | Problems that are new, grow or come back. Not traffic. | SYPNIA_AI_MAX_ANALYSES_PER_DAY, SYPNIA_AI_WEEKLY_SUGGESTIONS, and the model you choose. |
| Logs | Requests, if you log each one. | Keep SYPNIA_LOG_LEVEL at info. |
| Data leaving your provider | Watching replays: each one is downloaded by the browser that plays it. | Nothing to set. It is the size of the replays you watch. |
Two of these come from experience, measured on the system this engine comes from:
- Writes cost more than storage. Of USD 3.81 of object storage in 12 days, USD 2.73 was writes: 552 thousand of them, most from source maps uploaded on every deploy of a development environment. The same account stored 93 GB for about USD 2 a month.
- One deploy of a large app is 965 source maps and 109 MB. Twenty deploys a month are 19,300 writes and 2 GB.
To cap what a project can cost you whatever it sends, set SYPNIA_QUOTA_EVENTS_PER_MONTH and SYPNIA_QUOTA_REPLAYS_PER_MONTH. At the limit the SDKs are told to send only failures, an alert is created at 80 % and at 100 %, and nothing is dropped silently. See Sampling and quotas.
Two example projects
| A month | Small | Medium |
|---|---|---|
| Sessions | 30,000 (1,000 a day) | 300,000 |
| Events | 4 million (130,000 a day) | 40 million |
| Batches sent to ingest (30 a session) | 0.9 million | 9 million |
| Replay chunks (15 % of sessions recorded × 22 chunks) | 0.1 million | 1 million |
| In the bucket once retention has filled | About 8 GB | About 80 GB |
| In Postgres | About 1.5 GB | About 2.5 GB |
Small is the size measured on the system this engine comes from: 110 to 150 thousand records and about a thousand sessions a day. Medium is ten times that. The sizes come from Storage layout.
On AWS
The line-by-line arithmetic, with the prices used, is in Deployment shapes. Summed by kind, in USD a month:
| Small | Medium | What it is | |
|---|---|---|---|
| Functions | 3.31 | 15.26 | Requests to ingest, and the seconds ingest and worker run |
| Queue | 0.60 | 4.00 | One send per batch, and the worker’s receives |
| Bucket: writes, reads, storage | 3.52 | 14.34 | Writes are 2.10 and 10.50 of it |
| Logs | 0.50 | 2.00 | Successes sampled at 1 % |
| The address | 0.00 | 0.00 | Inside the provider’s free allowance of 10 million requests a month |
| The engine, without Postgres | ≈ 8 | ≈ 36 | |
| Postgres | ≈ 14 | 25 to 50 | The smallest managed instance with 1 GB of memory and 20 GB of disk; 2 to 4 GB of memory at medium |
| Total | ≈ 22 | ≈ 60 to 85 |
Read three things from it:
- At small sizes you are paying for a database. If you already have a Postgres with room, the engine is about USD 8 a month.
- Receiving is nearly free. A million batches cost less than 40 cents to accept.
- An idle engine costs almost nothing but the database. That makes this shape a good fit for a staging engine, or for traffic that comes in bursts.
AWS’s free allowance for functions, 400,000 GB-seconds and a million requests a month, covers almost all the function lines of the small project.
Not in these totals:
| Extra | About | When |
|---|---|---|
| The model | Your provider’s price for a few thousand tokens in and at most two thousand out, per analysis. At most 100 analyses a day per project by default. | When you connect one. Measured on the system this comes from: a weekly run of suggestions cost USD 0.08 to 0.20. |
| A public address for the database | USD 3.65 a month (0.005 an hour × 730) | When the template creates the database. |
| Data leaving AWS | USD 0.09 per GB after the first 100 GB a month | When you watch replays or read a lot through the API. |
| A NAT gateway | USD 33 a month, plus traffic | Only if you move the functions into a private network yourself. The template does not. |
| A web firewall on the address | From USD 6 a month | Only if you add one to protect the Console. |
For reference, the design of the system this engine comes from estimated its own bill at USD 15 to 25 a month at one million events a day, with a database it shared with its app, so the database is not in that number.
On one server with Docker
A server has no price per request. You pay for its size, and the question is what size.
| What needs memory | Budget |
|---|---|
| The engine: two projects being processed (2 × 256 MB), the ingest buffer (32 MiB), the source-map cache (200 MB) | 1 GB is comfortable, 2 GB leaves room |
| Postgres | 1 GB for the small project, 2 to 4 GB for the medium one |
| MinIO | 1 GB |
| The system and Caddy | 0.5 GB |
One CPU core is enough for the engine at the small size: its work is waiting on the bucket and the database, not computing.
| Small | Medium | |
|---|---|---|
| Server | 2 vCPU, 4 GB of memory | 4 vCPU, 8 GB of memory |
| Disk | 40 GB: 8 GB of bucket, 1.5 GB of database, source maps, room to grow | 160 GB: 80 GB of bucket, 2.5 GB of database, room to grow |
Example price, on demand in us-east-1 | An arm64 instance of that size at USD 0.0336 an hour × 730 = 24.50, plus 40 GB of disk × 0.08 = 3.20 | USD 0.0672 an hour × 730 = 49.10, plus 160 GB × 0.08 = 12.80 |
| Total | ≈ USD 28 | ≈ USD 62 |
Providers that sell fixed-price servers are often cheaper for the same size; a commitment of a year lowers the on-demand price by about a third. With a hosted bucket instead of MinIO, take the bucket line of the AWS table (USD 3.52 or 14.34) and a smaller disk.
One server is also one point of failure. If it stops, SDKs retry for a short time and then drop what they hold, and nothing alerts you, because the engine that would is the one that stopped. Watch /healthz from somewhere else.
Which shape costs less
Per million events, the AWS engine lines come to about USD 2 at the small size (8 ÷ 4) and USD 0.90 at the medium one (36 ÷ 40), and stay near that as traffic grows. A server’s cost per event falls until the server is full, then jumps to the next size.
- Small or bursty traffic: functions. You pay for what arrives.
- Large and steady traffic: servers, with a hosted bucket.
- In between: the two totals above are within a few dollars of each other. Choose by what you would rather operate, not by price.
We have no measured crossover. It will be published here once the engine has run at both sizes.
The first limits you meet
Each limit has a sign you can see without guessing.
| Sign | What it means | What to do |
|---|---|---|
Ingest answers 503 with Retry-After: 2 | More than 32 MiB of events are waiting to be written: the bucket is slow, or the process has no CPU left. | A faster disk under MinIO, or a hosted bucket. Then run ingest apart. |
data_through in GET /v1/summary is more than 10 minutes old, or an «Engine failing» alert | The worker cannot keep up. Problems still appear on time for a while; numbers fall behind first. | Raise SYPNIA_PROCESSING_CONCURRENCY (each step up needs 256 MB more), or run a second worker. |
The API answers 503 with Retry-After | The database has no free connection. | See Database connections. |
| A project’s hour is marked failed and named in an «Engine failing» alert | One project sends more in an hour than a pass may hold in memory. | Give the worker more memory, or set a quota for that project. |
| The disk fills | Recordings, almost always. | SYPNIA_RETENTION_REPLAYS_DAYS, then sypnia aggregate --task retention. |
| Everything is slow at the same moments each day | The daily job (Parquet, retention, partitions) at 02:30 UTC shares the machine. | More memory, or a separate worker. |
Budgets the design must meet, not measurements: a busy hour of 60,000 events and 1,000 sessions for one project is processed in about 40 MB of memory, and ten times that, 600,000 events and 10,000 sessions, in about 110 MB. The arithmetic is in Aggregation. On AWS, 40 copies of the ingest function at 50 ms a request accept about 800 batches a second; the medium project sends 3.5 a second.
Database connections
This is the limit people meet first with functions, and the system this engine comes from met it too: 13 refused connections in two minutes, when every copy of a function opened its own pool.
The total is SYPNIA_DATABASE_MAX_CONNECTIONS times the number of processes, or copies of a function, that run at once.
| Shape | Arithmetic | Total |
|---|---|---|
| One process | 10, the default | 10 |
| Three processes by role | 3 × 10 | 30 |
| The AWS template | ingest: 40 copies × 1. api: 20 copies × 2. worker: 2 copies × 3 | At most 86 |
A managed Postgres with 1 GB of memory accepts about 100 connections, so the template’s defaults fit the smallest instance with little to spare. When the API starts answering 503:
- give the database more memory, which raises its limit, or
- put a connection pooler in front of it, or
- lower the number of copies in the template, if you would rather have a slower API than a larger database.
When the database refuses a connection, the engine answers 503 with Retry-After at once. It does not queue and it does not fall over.
From one process to several
At the end you will have: ingest, the API and the worker as separate processes against the same database and bucket, so each can be sized and restarted alone.
Start one process per role
The same image and the same settings, with a different role each:
Terminal sypnia serve --role ingest --listen 0.0.0.0:8081 sypnia serve --role api --listen 0.0.0.0:8082 sypnia serve --role workerYou should seeEach process prints its role on its first line, such assypnia 1.0.0 role=ingest listen=0.0.0.0:8081, and ends withready.Send each path to its process
In your proxy, under the one base URL:
Path Goes to /v1/ingest/*,/v1/otlp/*,/v1/sdk.jsThe ingestprocessesEverything else: /v1/*,/mcp,/consoleThe apiprocessYou should seecurl -i <your base URL>/healthzanswers200, andsypnia doctorpasses when run next to any of the processes.
What changes when you split:
- Run as many
ingestprocesses as you need. They share nothing but the bucket. - Two workers are safe. They never count anything twice; the second one mostly shortens the wait after a restart.
- Rate limits are counted per process, so the real limit is the setting times the number of processes.
- Timing. With ingest and the worker apart, the worker finds a new segment on its next pass, at most 30 seconds later, not at once. A new problem still appears within 2 minutes, and numbers within 10.
From processes to functions
Move to the AWS template when traffic comes in bursts that a server sized for the average cannot take, when you would rather not patch and watch servers, or when most of the day the engine has nothing to do.
Both shapes read the same bucket layout and the same tables, so moving is pointing the other shape at the same data: deploy the template with DatabaseUrl set to your database, copy the bucket to the one the stack created, then change the DNS record. The way back is the same.
What is not known yet
- How long the functions really run. The function lines above assume 50 ms for an ingest request and 0.5 to 2 seconds for a worker wake-up.
- How large the Parquet files are. They are budgeted at the size of the raw events.
- How much one process takes before it must be split. The signs above tell you when; the number will be published once measured.
Related
- The arithmetic line by line, and how each shape works inside.
- Where the gigabytes are.
- Every setting named on this page.