GPU compute markets

Self-host or API

What running an open model on rented GPUs costs, next to the cheapest API for it.

Your setup

Pick a model and the precision to run it at. Optionally, enter the throughput you expect, in tokens per second across all requests.

Loading…

How this is worked out

  • Memory: every parameter at the chosen precision, plus 20% for the context cache and serving software. Mixture-of-experts models must hold every expert, not only those active per token.
  • Setups: a single GPU that holds the model, from any source, or a whole machine of up to eight GPUs that does, from the sources that describe machines. GPUs in separate machines are not combined: they cannot share a model at usable speed. Only offers you can rent now count.
  • API price: the cheapest provider OpenRouter lists for the model, blended as three input tokens to one output.
  • Break-even: the tokens per second a setup must produce, around the clock, to cost the same per token as that API. Idle hours raise the real cost of self-hosting; APIs charge only for what is used.