Training vs Inference
How AI learns and how it answers — the two phases every ML system goes through, not just chatbots
ChatGPT answers in seconds — yet it took months to "learn." How can both be true?
If training an AI takes months and costs a fortune in compute, why does it respond to your question in under a second? The answer lies in two phases that never overlap: Training and Inference.
This explainer applies to every type of ML system — not just chatbots. Whether it's Face ID on your phone, a spam filter in Gmail, or a music recommendation engine, every ML model goes through exactly these two phases.
A Michelin chef and a driving licence — two ways to see it
The head chef experiments with recipes — tastes, adjusts, discards, repeats — hundreds of times. Takes months, burns through ingredients. The recipe book isn't finalised until opening night.
The recipe is fixed. A customer orders. The chef follows the recipe and plates in 5 minutes. Same recipe, every time. No experimenting between orders.
Training = Lessons, practice, tests — time-consuming, costly, done once (or rarely). | Inference = Every delivery run you do after passing — you don't re-learn to drive each morning.
Switch between phases — Training vs Inference
Learning
The model sees millions of examples, makes predictions, compares them to ground truth, calculates how wrong it was (loss), then adjusts internal values (weights). Repeat until good enough.
Serving
Receives user input, flows it through the frozen weights (forward pass), returns output. Repeatable millions of times per day without changing weights at all.
Every ML system uses these two phases — not just chatbots
Here are three familiar systems — an image classifier, a text classifier, and an LLM — all using the same pattern.
Face unlock on your phone (Image Classifier)
Apple (Face ID), Google (Face Unlock) and others trained on millions of face images for weeks in large data centres.
Every time you unlock your phone — under 1 second, on the chip inside the device. Weights unchanged.
Email spam filter (Text Classifier)
Email companies trained on billions of labelled spam/ham messages with user feedback over years.
Every incoming email is classified within milliseconds — no retraining per message.
ChatGPT / Claude / Gemini (LLM)
Trained on vast internet text for months. Estimated cost: hundreds of millions of dollars (estimate — exact figures not publicly confirmed).
Every question you type — answered in seconds. Same frozen weights for every user globally.
The same pattern holds for music recommendation engines, airport face-detection gates, and real-time language translation — every ML system separates learning from serving.
How much does Training vs Inference actually cost?
| Dimension | Training | Inference |
|---|---|---|
| Time | Weeks to months | Milliseconds to seconds |
| GPU / TPU | Very large cluster (thousands of chips) | Moderate (per request) |
| Energy bill | Very high (total run) | Low (per request) |
| Data input | Large labelled dataset | User's prompt / input |
| Weights | Updated every batch | Read-only (frozen) |
| Frequency | Infrequent (retrain cycles) | Millions of times per day |
5 things people commonly get wrong
Click each one to reveal the truth.
✗ "The AI learns while we chat with it"
✗ "Inference doesn't use GPUs"
✗ "You train a model once and it's done forever"
✗ "Fine-tuning = training from scratch"
✗ "Longer training always means a better model"
6 terms you need before reading AI news
Click any term to see its definition.
Test your understanding — 4 questions
Remember just these three things
- 1 Training = learning phase — weights update continuously, resource-intensive, happens infrequently (retrain cycles)
- 2 Inference = serving phase — weights are frozen, fast, runs millions of times per day without changing anything
- 3 Both phases apply to every ML system — not just chatbots. Face ID, spam filters, music recommendations — all go through Training then Inference