Inference

Whether a model fits, and how fast it decodes. Predicted from the memory side, with bandwidth measured on real hardware.

Model

Weights

GPU

Context

GPUs

Does not fitLlama 3.1 70B, Q4, 4K context on RTX 4090

Memory

43.9 GB needed of 24 GB

Weights 39.9 GB
KV cache 1.3 GB
Runtime 2.8 GB

Decode

Measured: –tokens/s; No verified runs for this class yet
Published peak: 23tokens/s; 1,008 GB/s

Own one? Measure it and this estimate switches to real bandwidth.

Fastest GPUs for this model

API

The same estimate, as JSON. Public, no key, open to any origin.

Estimate/api/v1/inference/estimate
Models/api/v1/inference/models
GPUs/api/v1/inference/gpus

Parameters and limits

Request

curl "https://www.hbmrwa.xyz/api/v1/inference/estimate?model=llama-3.1-70b&quant=q4&gpu=nvidia-rtx-4090&ctx=4096"

Response

{
  "model": {
    "id": "llama-3.1-70b",
    "name": "Llama 3.1 70B",
    "params_b": 70.6,
    "active_params_b": 70.6,
    "max_ctx": 131072,
    "source": "https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct"
  },
  "quant": {
    "id": "q4",
    "name": "Q4_K_M",
    "bits_per_weight": 4.85
  },
  "gpu": {
    "id": "nvidia-rtx-4090",
    "name": "RTX 4090",
    "count": 1,
    "vram_gb": 24,
    "bandwidth_gbps": {
      "measured": null,
      "spec": 1008
    },
    "runs": 0
  },
  "ctx": 4096,
  "batch": 1,
  "fits": false,
  "vram_needed_gb": 43.9,
  "vram_available_gb": 24,
  "memory": {
    "weights_gb": 39.9,
    "kv_cache_gb": 1.25,
    "overhead_gb": 2.81
  },
  "decode_tok_s": {
    "measured": null,
    "spec": 22.8
  },
  "batch_tok_s": 22.8,
  "source": "Published peak bandwidth",
  "runs": 0,
  "best_gpus": [
    "Instinct MI300X",
    "H200 SXM",
    "H100 SXM",
    "A100 SXM 80GB",
    "H100 PCIe"
  ],
  "best": [
    {
      "id": "amd-instinct-mi300x",
      "name": "Instinct MI300X",
      "vram_gb": 192,
      "decode_tok_s": 120.1,
      "source": "spec",
      "runs": 0
    },
    {
      "id": "nvidia-h200-sxm",
      "name": "H200 SXM",
      "vram_gb": 141,
      "decode_tok_s": 108.7,
      "source": "spec",
      "runs": 0
    },
    {
      "id": "nvidia-h100-sxm",
      "name": "H100 SXM",
      "vram_gb": 80,
      "decode_tok_s": 75.9,
      "source": "spec",
      "runs": 0
    },
    {
      "id": "nvidia-a100-sxm-80gb",
      "name": "A100 SXM 80GB",
      "vram_gb": 80,
      "decode_tok_s": 46.2,
      "source": "spec",
      "runs": 0
    },
    {
      "id": "nvidia-h100-pcie",
      "name": "H100 PCIe",
      "vram_gb": 80,
      "decode_tok_s": 45.3,
      "source": "spec",
      "runs": 0
    }
  ]
}

How it's calculated

Memory: Weights (parameters × bits per weight) plus the KV cache (layers × 2 × KV heads × head size × 2 bytes, per token of context, per sequence) plus a runtime reserve of 0.75 GB per GPU and 5 % for activations and fragmentation. GB are binary, as printed on the card.
Decode: Generating a token streams the active weights and the whole KV cache through memory once, so tokens per second is at most bandwidth ÷ those bytes. It is a ceiling: real runtimes land below it.
Bandwidth: The class median read of HBM's verified runs when the class has any, the published peak otherwise. Both are shown; the API's source field says which one ranks.
Mixture of experts: Every expert must fit in memory, but a token reads only the active ones.
Several GPUs: Naive tensor parallel: memory pools, bandwidth adds up, less 10 % per doubling for synchronisation.
Unified memory: Apple silicon lends the GPU 75 % of memory, integrated graphics 50 %. The API takes vram to override it.
Left out: Prefill, which is compute-bound; speculative decoding; runtime-specific kernels; hourly cost.