The Machine Learning Lifecycle: From Idea to Production

From notebook to product

Training a model that scores well on a test set is the start of the work, not the end. A model in a real product needs data arriving reliably, a way to serve predictions quickly, checks that it still works next month, and people who can fix it when it does not.

The discipline of building and running machine learning systems reliably is called MLOps, by analogy with DevOps in software engineering. This lesson walks through the whole lifecycle:

Framing the problem

Every project starts with questions that have nothing to do with neural networks:

1. What decision will the model support? "Flag payments for a person to review" is a clearer goal than "detect fraud".
2. What would a simple baseline achieve? Always predicting the most common answer, or a rule an expert would write. A model has to beat it to be worth building.
3. Which errors cost more? Missing a fraudulent payment (a false negative) and blocking a good customer (a false positive) have very different costs.
4. How will success be measured? Choose the metric before training, so it cannot be bent to fit the results.

With rare events, accuracy misleads badly. If 1% of payments are fraud, a "model" that always answers "not fraud" is 99% accurate and completely useless. Precision (of the payments flagged, how many were fraud) and recall (of the frauds, how many were flagged), from Level 4, tell the real story, and moving the decision threshold trades one against the other.

Data pipelines and leakage

Data is usually the largest part of the work: collecting it, labelling it, cleaning it, and turning it into features, every time new data arrives. A data pipeline does those steps automatically and in the same way every time, and every version of the dataset is stored, so that any result can be reproduced.

The most dangerous data mistake is leakage: information reaching the model in training that it will not have when it is used for real. The model looks brilliant in testing and fails in use.

For anything that changes over time, the test set must come after the training data, exactly as it will in real use.

Experiments and tuning

A model has hyperparameters, settings chosen before training rather than learned: the learning rate, batch size, number of layers, dropout and more. Finding good ones is an experiment, and every run's settings, data version, code version and results should be logged, so that the best model can be rebuilt exactly.

• Grid search tries every combination from a few values of each hyperparameter.
• Random search samples combinations at random. A 2012 study showed it usually wins for the same budget, because when only a few hyperparameters really matter, a grid wastes its runs repeating the same few values of them.
• Bayesian optimisation uses the results so far to choose the most promising next trial.

The validation set is used to choose between runs; the test set is touched once, at the very end, or it stops being a fair test.

Training at scale

Large models need many accelerators, GPUs or Google's TPUs, working together. The three main ways to split the work can be combined:

Mixed precision training does most arithmetic in 16-bit numbers, roughly halving memory and doubling speed, while keeping a 32-bit master copy of the weights so small updates are not lost. Long runs save checkpoints regularly, because with thousands of GPUs, something fails every day.

Deployment

A trained model is deployed so that it can make predictions:

• Batch prediction scores everything at once on a schedule, such as overnight product recommendations.
• Online prediction answers each request as it arrives, so latency, the time per answer, matters as much as accuracy.

New models are introduced carefully. In shadow mode the new model runs alongside the old one without its answers being used, so they can be compared. A canary release sends a small share of real traffic to the new model first. An A/B test splits users between old and new and measures what really matters, such as fraud caught or complaints received. And there is always a quick way to roll back.

Monitoring and drift

The world changes after a model is trained, and its performance can quietly decay:

• Data drift: the inputs change. A banking app launches and attracts younger customers, so ages shift.
• Concept drift: the relationship between inputs and the right answer changes. Fraudsters invent new tricks, so old patterns stop working.

Production systems watch the inputs, the predictions and, when labels eventually arrive, the accuracy, and raise an alert when they move. A common measure for one feature is the Population Stability Index (PSI), which compares its distribution now with its distribution in training: below 0.1 is usually treated as stable, and above 0.25 as a serious shift. Drift triggers investigation and retraining on fresh data, which is why the pipeline must be automatic.

Model cards and datasheets

Two short documents make models safer to use and to hand over:

• A model card (proposed in 2019) describes a model: what it is for and not for, how it was trained, its performance overall and for different groups of people, its known limitations, and ethical considerations.
• A datasheet for datasets (2018) does the same for data: why it was collected, how, by whom, what it contains, who is represented, and how it should and should not be used.

Writing them forces the questions that prevent harm, such as "does this work as well for everyone?", to be asked before release rather than after.

Hands-on: engineering checks

Why accuracy misleads

import random

random.seed(5)
# 10,000 card payments, about 1% fraudulent. Each has a "risk score" from a simple model.
payments = []
for _ in range(10000):
    fraud = random.random() < 0.01
    score = random.gauss(0.7 if fraud else 0.3, 0.15)
    payments.append((score, fraud))

def evaluate(name, predict):
    tp = sum(1 for s, f in payments if predict(s) and f)
    fp = sum(1 for s, f in payments if predict(s) and not f)
    fn = sum(1 for s, f in payments if not predict(s) and f)
    tn = len(payments) - tp - fp - fn
    accuracy = (tp + tn) / len(payments)
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    print(f"{name:28} accuracy {accuracy:6.1%}  precision {precision:6.1%}  recall {recall:6.1%}  "
          f"(caught {tp}, missed {fn}, false alarms {fp})")

evaluate("always say 'not fraud'", lambda s: False)
for threshold in (0.5, 0.6, 0.7):
    evaluate(f"flag if score > {threshold}", lambda s, t=threshold: s > t)

The "model" that never flags anything is nearly 99% accurate. Which threshold is best depends on the cost of a missed fraud against the cost of a false alarm.

Detecting drift

import math, random

random.seed(8)
def histogram(values, edges):
    counts = [0] * (len(edges) - 1)
    for v in values:
        for i in range(len(edges) - 1):
            if edges[i] <= v < edges[i + 1]:
                counts[i] += 1
                break
    total = sum(counts)
    return [max(c / total, 1e-4) for c in counts]      # avoid log(0)

def psi(expected, actual):
    # Population Stability Index: how far a feature's distribution has moved
    return sum((a - e) * math.log(a / e) for e, a in zip(expected, actual))

edges = [0, 20, 30, 40, 50, 60, 200]                    # customer age bands
training = [random.gauss(38, 10) for _ in range(5000)]
same = [random.gauss(38, 10) for _ in range(5000)]
younger = [random.gauss(31, 9) for _ in range(5000)]   # a new app brings in younger customers

base = histogram(training, edges)
for name, live in [("this month, same customers", same), ("after the new app launch", younger)]:
    value = psi(base, histogram(live, edges))
    verdict = "stable" if value < 0.1 else "watch closely" if value < 0.25 else "drifted: investigate and retrain"
    print(f"{name:28} PSI = {value:.3f}  -> {verdict}")

Grid search against random search

import math, random

def validation_score(learning_rate, dropout):
    # A made-up stand-in for "train a model and measure it": the learning rate matters a lot,
    # dropout hardly at all. The best learning rate is about 0.003.
    return 0.9 - 0.08 * (math.log10(learning_rate) + 2.5) ** 2 - 0.01 * (dropout - 0.3) ** 2

grid = [(lr, d) for lr in (0.0001, 0.001, 0.01) for d in (0.1, 0.3, 0.5)]
random.seed(1)
rand = [(10 ** random.uniform(-4, -2), random.uniform(0.1, 0.5)) for _ in range(9)]

for name, trials in [("grid search", grid), ("random search", rand)]:
    best = max(trials, key=lambda t: validation_score(*t))
    tried = sorted({round(math.log10(lr), 2) for lr, _ in trials})
    print(f"{name:13}: {len(tried)} different learning rates tried; best score {validation_score(*best):.4f} "
          f"at learning rate {best[0]:.5f}")

Both searches ran nine trials, but the grid only ever tried three learning rates.

Try this

1. Change the fraud rate from 1% to 0.1%. What happens to the accuracy of "always say not fraud", and to precision at each threshold?
2. Make the new customers only slightly younger, a mean age of 36. Is the PSI above or below 0.1?

Level check

• Explain why the machine learning code is a small part of a real system
• Describe the stages of the machine learning lifecycle
• Frame a problem: decision, baseline, costs of errors and metrics
• Explain why accuracy misleads on imbalanced data, using precision and recall
• Give examples of data leakage and explain how to avoid them
• Compare grid search and random search
• Describe data, tensor and pipeline parallelism, and mixed precision
• Describe safe ways to deploy a new model
• Explain data drift and concept drift, and how they are detected
• Describe what a model card and a datasheet contain

Key terms you must be able to define

MLOps

The practice of building, deploying and running machine learning systems reliably.

Baseline

A simple approach, such as always predicting the most common answer, that a model must beat.

Data pipeline

Automated steps that collect, clean and transform data the same way every time.

Data leakage

Information in training that will not be available when the model is used, making test results too good.

Hyperparameter

A setting chosen before training, such as the learning rate, rather than learned.

Random search

Trying hyperparameter combinations sampled at random, usually better than a grid for the same budget.

Data parallelism

Training copies of a model on different examples on several GPUs and averaging their gradients.

Mixed precision

Training mostly with 16-bit numbers while keeping a 32-bit master copy of the weights.

Canary release

Sending a small share of real traffic to a new model before switching everyone over.

Data drift

A change over time in the inputs a model receives.

Concept drift

A change over time in the relationship between the inputs and the correct output.

Population Stability Index

A measure of how far a feature's distribution has shifted since training.

Model card

A short document describing a model's purpose, training, performance for different groups and limitations.

Flashcards

Go further: notes for future AI engineers

Reproducibility

A result you cannot reproduce is a result you cannot trust or fix. Reproducing a training run needs the same code version, data version, hyperparameters, random seeds and software library versions, which is why experiment trackers log all of them automatically. Even then, GPU arithmetic is not always exactly repeatable, because adding numbers in a different order gives slightly different rounding, so teams also check that results are stable across several seeds.

Feedback loops

Some models change the very data they will be retrained on. A recommendation system that only shows popular items gathers clicks only on popular items, which makes them look even better next time. A predictive policing model that sends officers to certain areas records more crime there. Breaking such loops needs deliberate exploration, the idea from Level 13, or careful correction of the training data.

Misconceptions to avoid