# Scaling and cost

> What a self-hosted engine costs each month for a small and a medium project, with the arithmetic, what makes the bill move, the first limits you meet, and when to go from one process to several or to functions.

Applies to: the Sypnia engine (open source, self-hosted). Status: at launch.  
Source: https://docs.sypnia.com/self-hosting/scaling-and-cost/

For agents: Cost and capacity of a self-hosted Sypnia engine: cost drivers and the setting that controls each, estimated monthly totals for a small project (4 million events a month, about USD 22 on AWS) and a medium one (40 million, about USD 60 to 85), server sizing for Docker, the signs that a deployment is at its limit, database connection arithmetic, how to split one process into roles, and when to use the AWS functions shape. All figures are estimates or measurements from the system the engine comes from, labelled.

**\[Engine]** This page is for whoever pays for a self-hosted engine, or must say what it will cost before it exists. On Sypnia Cloud you pay a plan instead, and capacity is our job: see [Plans and limits](https://docs.sypnia.com/cloud/plans-and-limits).

> **Warning: Estimates, not measurements of this engine**
>
> The engine is not released, so nothing here was measured on it. Every figure is one of three things, and says which: a **budget** the design must meet, an **estimate** built from published prices with its arithmetic, or a figure **measured on the system this engine comes from**. Prices are the on-demand ones published for `us-east-1` when this was written. Redo the sums with today's prices and your own traffic.

## What makes the bill move

In the order you will feel them:

| Driver                         | Grows with                                                                                                                                                                                             | What you can change                                                                                                                                           |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **The database**               | Almost nothing: it is a fixed monthly price, and at small sizes the largest line.                                                                                                                      | Bring a Postgres you already pay for. Any Postgres 15 or later works, with a database of its own.                                                             |
| **Writes to the bucket**       | Recorded sessions (one write per 10 seconds of recording), sessions (their files are written again as they grow), and deploys (one write per source map). Not events: those are grouped into segments. | `SYPNIA_REPLAY_SAMPLE_RATE` for new projects, or each project's own `settings.replay.sample_rate`. Upload source maps only for environments someone looks at. |
| **Storage in the bucket**      | Events × days kept, and recordings × days kept. Recordings are about three quarters of it.                                                                                                             | `SYPNIA_RETENTION_REPLAYS_DAYS`, `SYPNIA_RETENTION_EVENTS_DAYS`.                                                                                              |
| **Compute**                    | On AWS, requests and the worker's seconds. On a server, nothing until the server is full.                                                                                                              | The shape you deploy.                                                                                                                                         |
| **The model**                  | Problems that are new, grow or come back. Not traffic.                                                                                                                                                 | `SYPNIA_AI_MAX_ANALYSES_PER_DAY`, `SYPNIA_AI_WEEKLY_SUGGESTIONS`, and the model you choose.                                                                   |
| **Logs**                       | Requests, if you log each one.                                                                                                                                                                         | Keep `SYPNIA_LOG_LEVEL` at `info`.                                                                                                                            |
| **Data leaving your provider** | Watching replays: each one is downloaded by the browser that plays it.                                                                                                                                 | Nothing to set. It is the size of the replays you watch.                                                                                                      |

Two of these come from experience, measured on the system this engine comes from:

- **Writes cost more than storage.** Of USD 3.81 of object storage in 12 days, USD 2.73 was writes: 552 thousand of them, most from source maps uploaded on every deploy of a development environment. The same account stored 93 GB for about USD 2 a month.
- **One deploy of a large app is 965 source maps and 109 MB.** Twenty deploys a month are 19,300 writes and 2 GB.

To cap what a project can cost you whatever it sends, set `SYPNIA_QUOTA_EVENTS_PER_MONTH` and `SYPNIA_QUOTA_REPLAYS_PER_MONTH`. At the limit the SDKs are told to send only failures, an alert is created at 80 % and at 100 %, and nothing is dropped silently. See [Sampling and quotas](https://docs.sypnia.com/sdks/sampling-and-quotas).

## Two example projects

| A month                                               | Small                     | Medium       |
| ----------------------------------------------------- | ------------------------- | ------------ |
| Sessions                                              | 30,000 (1,000 a day)      | 300,000      |
| Events                                                | 4 million (130,000 a day) | 40 million   |
| Batches sent to ingest (30 a session)                 | 0.9 million               | 9 million    |
| Replay chunks (15 % of sessions recorded × 22 chunks) | 0.1 million               | 1 million    |
| In the bucket once retention has filled               | About 8 GB                | About 80 GB  |
| In Postgres                                           | About 1.5 GB              | About 2.5 GB |

Small is the size measured on the system this engine comes from: 110 to 150 thousand records and about a thousand sessions a day. Medium is ten times that. The sizes come from [Storage layout](https://docs.sypnia.com/self-hosting/storage-layout#how-big-it-gets).

## On AWS

The line-by-line arithmetic, with the prices used, is in [Deployment shapes](https://docs.sypnia.com/architecture/deployment-shapes#what-it-costs-on-aws). Summed by kind, in USD a month:

|                                  | Small    | Medium         | What it is                                                                                         |
| -------------------------------- | -------- | -------------- | -------------------------------------------------------------------------------------------------- |
| Functions                        | 3.31     | 15.26          | Requests to `ingest`, and the seconds `ingest` and `worker` run                                    |
| Queue                            | 0.60     | 4.00           | One send per batch, and the worker's receives                                                      |
| Bucket: writes, reads, storage   | 3.52     | 14.34          | Writes are 2.10 and 10.50 of it                                                                    |
| Logs                             | 0.50     | 2.00           | Successes sampled at 1 %                                                                           |
| The address                      | 0.00     | 0.00           | Inside the provider's free allowance of 10 million requests a month                                |
| **The engine, without Postgres** | **≈ 8**  | **≈ 36**       |                                                                                                    |
| Postgres                         | ≈ 14     | 25 to 50       | The smallest managed instance with 1 GB of memory and 20 GB of disk; 2 to 4 GB of memory at medium |
| **Total**                        | **≈ 22** | **≈ 60 to 85** |                                                                                                    |

Read three things from it:

1. **At small sizes you are paying for a database.** If you already have a Postgres with room, the engine is about USD 8 a month.
2. **Receiving is nearly free.** A million batches cost less than 40 cents to accept.
3. **An idle engine costs almost nothing but the database.** That makes this shape a good fit for a staging engine, or for traffic that comes in bursts.

AWS's free allowance for functions, 400,000 GB-seconds and a million requests a month, covers almost all the function lines of the small project.

Not in these totals:

| Extra                             | About                                                                                                                                             | When                                                                                                             |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| The model                         | Your provider's price for a few thousand tokens in and at most two thousand out, per analysis. At most 100 analyses a day per project by default. | When you connect one. Measured on the system this comes from: a weekly run of suggestions cost USD 0.08 to 0.20. |
| A public address for the database | USD 3.65 a month (0.005 an hour × 730)                                                                                                            | When the template creates the database.                                                                          |
| Data leaving AWS                  | USD 0.09 per GB after the first 100 GB a month                                                                                                    | When you watch replays or read a lot through the API.                                                            |
| A NAT gateway                     | USD 33 a month, plus traffic                                                                                                                      | Only if you move the functions into a private network yourself. The template does not.                           |
| A web firewall on the address     | From USD 6 a month                                                                                                                                | Only if you add one to [protect the Console](https://docs.sypnia.com/self-hosting/console#protect-it).           |

For reference, the design of the system this engine comes from estimated its own bill at USD 15 to 25 a month at one million events a day, with a database it shared with its app, so the database is not in that number.

## On one server with Docker

A server has no price per request. You pay for its size, and the question is what size.

| What needs memory                                                                                                | Budget                                                   |
| ---------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- |
| The engine: two projects being processed (2 × 256 MB), the ingest buffer (32 MiB), the source-map cache (200 MB) | 1 GB is comfortable, 2 GB leaves room                    |
| Postgres                                                                                                         | 1 GB for the small project, 2 to 4 GB for the medium one |
| MinIO                                                                                                            | 1 GB                                                     |
| The system and Caddy                                                                                             | 0.5 GB                                                   |

One CPU core is enough for the engine at the small size: its work is waiting on the bucket and the database, not computing.

|                                         | Small                                                                                                | Medium                                                       |
| --------------------------------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
| Server                                  | 2 vCPU, 4 GB of memory                                                                               | 4 vCPU, 8 GB of memory                                       |
| Disk                                    | 40 GB: 8 GB of bucket, 1.5 GB of database, source maps, room to grow                                 | 160 GB: 80 GB of bucket, 2.5 GB of database, room to grow    |
| Example price, on demand in `us-east-1` | An arm64 instance of that size at USD 0.0336 an hour × 730 = 24.50, plus 40 GB of disk × 0.08 = 3.20 | USD 0.0672 an hour × 730 = 49.10, plus 160 GB × 0.08 = 12.80 |
| **Total**                               | **≈ USD 28**                                                                                         | **≈ USD 62**                                                 |

Providers that sell fixed-price servers are often cheaper for the same size; a commitment of a year lowers the on-demand price by about a third. With a hosted bucket instead of MinIO, take the bucket line of the AWS table (USD 3.52 or 14.34) and a smaller disk.

One server is also one point of failure. If it stops, SDKs retry for a short time and then drop what they hold, and nothing alerts you, because the engine that would is the one that stopped. Watch `/healthz` from somewhere else.

## Which shape costs less

Per million events, the AWS engine lines come to about USD 2 at the small size (8 ÷ 4) and USD 0.90 at the medium one (36 ÷ 40), and stay near that as traffic grows. A server's cost per event falls until the server is full, then jumps to the next size.

- **Small or bursty traffic:** functions. You pay for what arrives.
- **Large and steady traffic:** servers, with a hosted bucket.
- **In between:** the two totals above are within a few dollars of each other. Choose by what you would rather operate, not by price.

We have no measured crossover. It will be published here once the engine has run at both sizes.

## The first limits you meet

Each limit has a sign you can see without guessing.

| Sign                                                                                          | What it means                                                                                             | What to do                                                                                            |
| --------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| Ingest answers `503` with `Retry-After: 2`                                                    | More than 32 MiB of events are waiting to be written: the bucket is slow, or the process has no CPU left. | A faster disk under MinIO, or a hosted bucket. Then [run ingest apart](#from-one-process-to-several). |
| `data_through` in `GET /v1/summary` is more than 10 minutes old, or an «Engine failing» alert | The worker cannot keep up. Problems still appear on time for a while; numbers fall behind first.          | Raise `SYPNIA_PROCESSING_CONCURRENCY` (each step up needs 256 MB more), or run a second worker.       |
| The API answers `503` with `Retry-After`                                                      | The database has no free connection.                                                                      | See [Database connections](#database-connections).                                                    |
| A project's hour is marked failed and named in an «Engine failing» alert                      | One project sends more in an hour than a pass may hold in memory.                                         | Give the worker more memory, or set a quota for that project.                                         |
| The disk fills                                                                                | Recordings, almost always.                                                                                | `SYPNIA_RETENTION_REPLAYS_DAYS`, then `sypnia aggregate --task retention`.                            |
| Everything is slow at the same moments each day                                               | The daily job (Parquet, retention, partitions) at 02:30 UTC shares the machine.                           | More memory, or a separate worker.                                                                    |

Budgets the design must meet, not measurements: a busy hour of 60,000 events and 1,000 sessions for one project is processed in about 40 MB of memory, and ten times that, 600,000 events and 10,000 sessions, in about 110 MB. The arithmetic is in [Aggregation](https://docs.sypnia.com/architecture/aggregation). On AWS, 40 copies of the ingest function at 50 ms a request accept about 800 batches a second; the medium project sends 3.5 a second.

## Database connections

This is the limit people meet first with functions, and the system this engine comes from met it too: 13 refused connections in two minutes, when every copy of a function opened its own pool.

The total is `SYPNIA_DATABASE_MAX_CONNECTIONS` times the number of processes, or copies of a function, that run at once.

| Shape                   | Arithmetic                                                            | Total      |
| ----------------------- | --------------------------------------------------------------------- | ---------- |
| One process             | 10, the default                                                       | 10         |
| Three processes by role | 3 × 10                                                                | 30         |
| The AWS template        | `ingest`: 40 copies × 1. `api`: 20 copies × 2. `worker`: 2 copies × 3 | At most 86 |

A managed Postgres with 1 GB of memory accepts about 100 connections, so the template's defaults fit the smallest instance with little to spare. When the API starts answering `503`:

- give the database more memory, which raises its limit, or
- put a connection pooler in front of it, or
- lower the number of copies in the template, if you would rather have a slower API than a larger database.

When the database refuses a connection, the engine answers `503` with `Retry-After` at once. It does not queue and it does not fall over.

## From one process to several

**At the end you will have:** ingest, the API and the worker as separate processes against the same database and bucket, so each can be sized and restarted alone.

### 1. Stop migrating at start

Several processes must not migrate at the same moment. Set this for all of them, and run `sypnia migrate` once in each deploy, before they start.

```bash
SYPNIA_AUTO_MIGRATE=false
```

### 2. Start one process per role

The same image and the same settings, with a different role each:

```bash
sypnia serve --role ingest --listen 0.0.0.0:8081
sypnia serve --role api    --listen 0.0.0.0:8082
sypnia serve --role worker
```

**You should see:**Each process prints its role on its first line, such as `sypnia 1.0.0  role=ingest  listen=0.0.0.0:8081`, and ends with `ready`.

### 3. Send each path to its process

In your proxy, under the one base URL:

| Path                                         | Goes to                |
| -------------------------------------------- | ---------------------- |
| `/v1/ingest/*`, `/v1/otlp/*`, `/v1/sdk.js`   | The `ingest` processes |
| Everything else: `/v1/*`, `/mcp`, `/console` | The `api` process      |

**You should see:**`curl -i <your base URL>/healthz` answers `200`, and `sypnia doctor` passes when run next to any of the processes.

What changes when you split:

- **Run as many `ingest` processes as you need.** They share nothing but the bucket.
- **Two workers are safe.** They never count anything twice; the second one mostly shortens the wait after a restart.
- **Rate limits are counted per process**, so the real limit is the setting times the number of processes.
- **Timing.** With ingest and the worker apart, the worker finds a new segment on its next pass, at most 30 seconds later, not at once. A new problem still appears within 2 minutes, and numbers within 10.

## From processes to functions

Move to the [AWS template](https://docs.sypnia.com/self-hosting/deploy-aws) when traffic comes in bursts that a server sized for the average cannot take, when you would rather not patch and watch servers, or when most of the day the engine has nothing to do.

Both shapes read the same bucket layout and the same tables, so moving is pointing the other shape at the same data: deploy the template with `DatabaseUrl` set to your database, [copy the bucket](https://docs.sypnia.com/self-hosting/storage-layout#move-to-another-bucket) to the one the stack created, then change the DNS record. The way back is the same.

## What is not known yet

- How long the functions really run. The function lines above assume 50 ms for an ingest request and 0.5 to 2 seconds for a worker wake-up.
- How large the Parquet files are. They are budgeted at the size of the raw events.
- How much one process takes before it must be split. The signs above tell you when; the number will be published once measured.

## Related
