TL;DR
- Rent the H100 only when it finishes the job faster than the price ratio between the two cards; otherwise the A100 is cheaper per run.
- On LowCostGPU that ratio is $1.680 against $0.468 per hour, while BF16 LoRA and QLoRA runs see only 1.7-2.2×.
- Both cards hold 80 GB, so fit never decides; the H100 adds 3.35 versus 2.0 TB/s of bandwidth and FP8.
- FP8 full fine-tunes see 2.5-3× and tip the decision to the H100; without FP8 enabled you pay for bandwidth alone.
The break-even rule
Both cards hold 80 GB, so the choice is never about fit — it is about dollars per finished run. Today the H100 SXM costs $1.680/hr and the A100 SXM4 $0.468/hr: a ratio of 3.59×. The rule that settles every job:
rent the H100 if speedup(H100 / A100) > 3.59, otherwise rent the A100
Speedups below that line make the A100 cheaper per run even though the H100 is faster; above it the H100 is both faster and cheaper. The ratio moves with the weekly price re-check — the number above is live.
What actually differs
| A100 SXM4 80 GB | H100 SXM 80 GB | Why it matters for fine-tuning | |
|---|---|---|---|
| Memory | 80 GB HBM2e | 80 GB HBM3 | Same capacity — identical model fit |
| Bandwidth | 2.0 TB/s | 3.35 TB/s | Optimizer steps and attention are bandwidth-bound: ~1.6× from this alone |
| BF16 tensor (dense) | 312 TFLOPS | 990 TFLOPS | Compute-bound layers scale; small batches rarely saturate either |
| FP8 | — | Transformer Engine | The H100's real weapon: 2.5–3× on full fine-tunes that use it |
| NVLink | 600 GB/s | 900 GB/s | Multi-GPU FSDP all-gathers; matters past 2× cards |
| Price today | $0.468/hr | $1.680/hr | Ratio 3.59× — the break-even line |
Five fine-tuning jobs, costed
Indicative wall-clock on the A100 and the speedup most teams measure on an H100 for the same recipe. The cost columns use today's on-demand prices; interruptible capacity halves both — fine-tunes checkpoint, so use it.
| Job | Setup | A100 hours | H100 speedup | A100 cost | H100 cost | Cheaper |
|---|---|---|---|---|---|---|
| QLoRA, Llama 3.1 8B, 10k pairs | BF16 compute, 4-bit base | 1.5 h | 1.8× | $0.70 | $1.40 | A100 (50% less) |
| LoRA, Llama 3.1 70B (4-bit base) | single card, 48 GB used | 3 h | 2.0× | $1.40 | $2.52 | A100 (44% less) |
| Full fine-tune, 8B, BF16 | 8× node, FSDP | 6 h | 2.2× | $22.46 | $36.65 | A100 (39% less) |
| Full fine-tune, 8B, FP8 (Transformer Engine) | 8× node, FSDP | 6 h | 3.0× | $22.46 | $26.88 | A100 (16% less) |
| DPO / RLHF pass, 8B | policy + reference | 2.5 h | 2.4× | $1.17 | $1.75 | A100 (33% less) |
The pattern is consistent: adapter methods in BF16 favour the A100; anything that engages FP8 or runs long enough for bandwidth to dominate favours the H100. Wall-clock is the tie-breaker — an H100 run that costs 10% more but finishes a day earlier is usually the right call for a deadline.
Decision table
| You are doing | Rent | Because |
|---|---|---|
| QLoRA / LoRA on ≤13B, BF16 | RTX 4090 or A100 PCIe | Fits in 24–80 GB; speedups under the break-even line |
| LoRA on 70B (4-bit base), single card | A100 SXM4 | Bandwidth-bound but 2× speedup stays under 3.6× |
| Full fine-tune, BF16, multi-GPU | A100 SXM4 unless deadline-bound | ≈2.2× speedup is close to the line — pick by wall-clock |
| Full fine-tune with FP8 (TE, NeMo, TorchAO) | H100 SXM | 2.5–3× speedup beats the price ratio outright |
| Sequence lengths ≥32k or big micro-batches | H100 SXM (or H200) | Memory bandwidth and, on the H200, 141 GB |
Making either card cheaper
- Go interruptible. Every job above checkpoints; the flat −50% turns an A100 into $0.234/hr and an H100 into $0.840/hr.
- Enable FP8 before renting an H100. Without it you are paying for bandwidth only; with it the H100 earns its price.
- Tune on 1×, train on 8×. Debug the config on a single card at one-eighth the burn rate, then scale.
- Datasets and checkpoints on a volume at $0.08/GB/month — the trainer instance stays disposable.
- Compare the neighbours. H100 SXM vs A100 SXM4 and the PCIe pair keep live numbers; the QLoRA walkthrough has the exact axolotl config.
Put the numbers to work
Every price in this guide is our live rate — fixed, ≥30% under the market median, billed per second. Deploy the exact setup above from the console in about 30 seconds, paid in crypto, no card and no KYC.
