Silicon, priced honestly.
Every chip, every watt.

The register of inference and training hardware — RTX 5090, H200, B200, MI300X — with real FP16 TFLOPS, VRAM, memory bandwidth, and the cloud hourly you actually see on Vast.ai, RunPod and AWS. Numbers reproduced against the same quantisation, reported with latency and p95.

One page per accelerator.

X vs Y, same workloads.

Accelerators, ranked.

FP16 TFLOPS is the blunt instrument — useful for ordering, useless for predicting any one workload. Read it next to VRAM, bandwidth, and the hourly price you can actually rent the card for.

Tier

Consumer · Datacenter

Metric

FP16 dense · higher is better

Sources

NVIDIA · AMD datasheets

Updated

April 2026

Eleven workloads. One card at a time.

The same card is fast at LLM serving and slow at diffusion — it depends on the bottleneck. Each panel is a real benchmark, each figure at right is the current best number we have reproduced on a consumer RTX 5090 or cheapest-cloud equivalent.

Cloud, on-prem, or edge.

Three honest lanes for ML hardware in 2026. The right answer is usually two of them — cloud for the bursts, on-prem for the grind; edge for the field.

Consumer cards, specified.

The GPUs most ML engineers can actually buy and put in a workstation. The RTX 5090 is the first consumer card with 32 GB; the used 3090 is the first consumer card to stop being expensive.

How much memory, really.

Weights in FP16 = 2 bytes × parameters. A 7B model takes ~14 GB, a 70B ~140 GB. INT4 quantisation cuts this 4× with small quality loss — which is the only reason a 70B runs on a 4090.

Fine-tuning needs 3–4× more memory for gradients, optimiser state, and activations. 128k-context serving adds tens of GB of KV cache on top.

The hourly spread.

Aggregators undercut hyperscalers on the same silicon by 2–3×. The gap is widest on the H100: $1.65/hr on Vast.ai, $6.88/hr on AWS.

Spot and community instances may be interrupted. Hyperscaler per-GPU equivalents are derived from multi-GPU instance list prices.

Under a grand, and the break-even.

Used RTX 3090 remains the best sub-$1k ML card — 24 GB is enough for a 70B at 4-bit. The RTX 5060 is the cheapest new card if VRAM needs stay under 8 GB.

Break-even assumes $0.15/kWh, realistic utilisation during usage hours, and the cheapest available cloud rate at the same specification.

Unified memory, at a cost.

M-series chips trade throughput for envelope: you get 128–192 GB of unified memory on a single quiet workstation, but training is typically 3–5× slower than an equivalent NVIDIA card.

MLX beats PyTorch MPS for Apple-optimised inference. No NVLink-style multi-GPU scaling; not recommended for production training.

Five watts, and an LLM on chip.

For robotics, phones and factory cameras, the relevant question stops being "how many tokens per second on an H100?" and starts being "how many tokens per second under five watts, offline, with no round-trip to a datacenter?"

Hailo-10H · Jetson Orin · Apple Neural Engine · Qualcomm AI Hub. Quantised LLMs, vision backbones, and on-device transcription — read the register on /embedded-ai.

Read next, around the register.