berth, a placement estimator for inference engineers

The placement premium

The same workload can cost multiples more on one chip than another. And the premium is not a constant: it moves with the model, the context length, the batch size, and the latency target. That variance is the reason berth exists. Not a lookup table, but an estimator you run per workload, across every chip and provider you could place it on.

Here is one placement we measured end to end. For Llama-3-8B decode, an L40S is 1.6x cheaper than an H100 PCIe at matched concurrency. The H100 is faster, but if your p99 target does not require that speed, you are paying 1.6x for headroom you will never use.

Llama-3-8B decode, $/Mtok, list prices
Concurrency	L40S $/Mtok	H100 PCIe $/Mtok	Premium of pricier choice
batch 1	6.07	9.91	1.63x
batch 8	0.885	1.40	1.59x
batch 32	0.317	0.535	1.69x

Physics, not a black box

Every estimate is a closed-form roofline you can audit by hand. Decode is a race between compute and memory bandwidth. Prefill is compute plus a fixed scheduling floor. Concurrency is serial batch admission. No learned model stands between you and the number, and every prediction ships with the term that produced it.

On real silicon it holds: decode latency predicted to 4.2 percent on an H100 PCIe and 10.6 percent on an L40S, using spec-sheet priors with no tuning, against a 15 percent gate published before the first run.

Full validation results, including what failed

Estimate

You describe a workload; berth returns dollars per million tokens, time to first token, time per output token, and the placement premium: how much you would save on the cheapest silicon that still meets your p99.

Install and run

pip install berth-placement
berth estimate --model llama3-8b --batch 32

The distribution is berth-placement; the command is berth. Every line of output is tagged MEASURED or prior, so a prediction is never mistaken for an observation.

Apache-2.0. The core is standard-library Python. Nothing leaves your machine unless you choose to send a trace.

sounding: check the estimate against your own hardware

The measurement harness that produced the validation data

berth is an estimator. It predicts from a model, and an estimate you cannot check is just an assertion. So berth ships sounding, the same measurement harness that produced its own validation data. You use the estimator; the harness is how we earned your trust in it.

sounding drives an OpenAI-compatible completion server, vLLM or SGLang, with streaming requests across a grid of batch sizes, prompt lengths and output lengths, and records time to first token and time per output token per cell. Standard-library Python, so the box you rent needs nothing beyond a Python interpreter and a running model server.

Run sounding

python -m bench.sounding --base-url http://localhost:8000 \
    --silicon h100-pcie --model llama3-8b \
    --model-id meta-llama/Meta-Llama-3-8B --out traces.jsonl

python -m bench.fit_overhead traces.jsonl
python -m bench.validate     traces.jsonl --prefill-overhead-ms <fitted>

The fixed prefill floor is a property of one accelerator, driver, server and config, and is not predictable from a spec sheet: it measured 74.6 ms on the L40S and 54.6 ms on the H100 PCIe. Silicon profiles ship 0.0 and every run fits its own. Do not carry someone else's.

No GPU handy? --mock runs the whole path from sweep to error report against the analytical model plus noise. Mock traces are stamped as such and can never enter the corpus.

Every record carries source, either measured or mock, written by the harness. A mixed file is refused, and CI rejects any contributed trace that cannot prove which it is. A mock trace and a hardware trace are otherwise indistinguishable on disk, and the corpus is worth exactly as much as that distinction.

Full verify-and-contribute loop in the docs

Check us with data

Profile one real workload. If berth's number is useful, run a sounding and send the anonymised trace so the public estimate gets more accurate for everyone serving that workload shape. Disputes that arrive with traces attached outrank everything else.

The corpus is worth exactly as much as the provenance behind it.