Capstone: Build, Test and Explain Your Own AI

Your mission

Fourteen levels have taken you from the history of AI to transformers, generative models, agents and the engineering of real systems. The capstone puts it together: design, build, test and explain an AI system of your own, the way a professional would.

You will produce:

1. A working model, in Python, that does something useful or interesting.
2. An evaluation: how well it works, compared with a baseline, with its mistakes analysed.
3. A model card: what it is for, how it was built, how well it works and for whom, and its limitations.
4. A short presentation or demo for someone else.

The certificate for this course unlocks when you score at least 70% on every lesson's quiz. The capstone is how you prove the skills to yourself, and to anyone you show it to.

Choose a project

Pick a project you care about, small enough to finish. Some ideas, with the levels they draw on:

A good first project has data you can get or make yourself, a clear way to measure success, and a simple baseline to beat.

Plan it on a canvas

Before writing code, fill in a project canvas. It takes ten minutes and saves days.

The most important box is success: decide now how you will measure whether the model works, and what the baseline scores, so you cannot fool yourself later.

Eight milestones

1. Define: the problem, the users and the success measure.
2. Data: collect or create it, label it, split it into training, validation and test sets, and look at it.
3. Baseline: the simplest possible approach, such as always predicting the most common class.
4. First model: the simplest model that could work. Get it running end to end before improving anything.
5. Evaluate: measure it on the validation set, with a confusion matrix, not only accuracy.
6. Improve: look at the mistakes, form a hypothesis, change one thing, and measure again. Keep a log.
7. Check responsibly: who could be harmed, does it work equally well for different users, what personal data does it touch?
8. Present: test once on the test set, write the model card, and show it to someone.

Worked project: a spam classifier from scratch

Here is a complete small project, every step in plain Python: data, features, a logistic regression model trained by gradient descent, evaluation on unseen messages, and an explanation of what it learned.

import math, random

# 1. Data: messages labelled 1 for spam and 0 for genuine ("ham")
messages = [
    ("win a free phone now claim your prize", 1), ("you have won cash text claim now", 1),
    ("urgent your account is locked click the link", 1), ("free entry to win a holiday reply yes", 1),
    ("cheap loans approved today no checks", 1), ("claim your free gift card now", 1),
    ("congratulations you won a prize call now", 1), ("limited offer win cash click here", 1),
    ("your parcel is waiting pay the fee at this link", 1), ("earn money fast from home click now", 1),
    ("final notice claim your refund now", 1), ("free ringtones text win to claim", 1),
    ("are we still meeting at four", 0), ("can you send me the homework for maths", 0),
    ("dinner is at seven see you then", 0), ("thanks for the help with the project", 0),
    ("what time does the match start", 0), ("i left my coat at your house", 0),
    ("happy birthday hope you have a great day", 0), ("can you pick up milk on the way home", 0),
    ("the bus is late i will be ten minutes", 0), ("did you finish reading the book", 0),
    ("see you at football practice tomorrow", 0), ("mum says dinner is ready", 0),
    ("free tickets for the school play this friday", 0), ("call me when you get this", 0),
]
random.seed(7)
random.shuffle(messages)
train, test = messages[:20], messages[20:]

# 2. Features: which words appear (a bag of words)
vocab = sorted({w for text, _ in train for w in text.split()})
def features(text):
    words = set(text.split())
    return [1.0 if w in words else 0.0 for w in vocab]

# 3. Model: logistic regression, trained by gradient descent
w = [0.0] * len(vocab)
b = 0.0
def predict(text):
    z = b + sum(wi * xi for wi, xi in zip(w, features(text)))
    return 1 / (1 + math.exp(-z))

for epoch in range(300):
    for text, label in train:
        x = features(text)
        error = predict(text) - label                 # gradient of the cross-entropy loss
        w = [wi - 0.1 * error * xi for wi, xi in zip(w, x)]
        b -= 0.1 * error

# 4. Evaluate on messages the model has never seen
tp = fp = fn = tn = 0
for text, label in test:
    guess = 1 if predict(text) >= 0.5 else 0
    tp += guess == 1 and label == 1
    fp += guess == 1 and label == 0
    fn += guess == 0 and label == 1
    tn += guess == 0 and label == 0
    flag = "" if guess == label else "   <- wrong"
    print(f"{predict(text):.2f}  {'spam' if guess else 'ham ':4}  {text}{flag}")
print(f"test accuracy {(tp + tn) / len(test):.0%}: caught {tp} spam, missed {fn}, false alarms {fp}")

# 5. Explain: the words the model learned to trust most
ranked = sorted(zip(w, vocab))
print("most spammy words:", ", ".join(v for _, v in ranked[-6:][::-1]))
print("most genuine words:", ", ".join(v for _, v in ranked[:6]))

Error analysis

The model missed "cheap loans approved today no checks". Why? None of those words appeared in a spam message in its training set, so it had no evidence either way. That one mistake suggests the most useful next step: more varied training data, not a bigger model. It also shows the limits of so small a test set: six messages cannot tell you the real accuracy to within more than a few tens of per cent.

Its model card

Check it responsibly

Before anyone relies on your project, work through these questions, and write the answers into your model card:

• Who will use it, and who will it affect who is not using it?
• What happens when it is wrong, in each direction? Which mistake is worse?
• Does it work equally well for different groups of people, accents, languages or devices?
• What personal data does it collect, keep or learn from, and is all of it needed?
• Could someone misuse it, or attack it with crafted inputs?
• Can you explain an individual decision to the person it affects?
• Is there a person who can override it?

Present it

A good presentation of an AI project answers five questions in order:

1. What problem, and for whom?
2. What did you build, in one picture?
3. How well does it work, compared with the baseline, and where does it fail? Show real mistakes.
4. What would you do next, and why?
5. What should people be careful about, from your model card?

Share your code with a short README explaining how to run it. A project that someone else can run and understand is worth far more than a slightly more accurate one that nobody can.

Everything you have learned

From here, the paths divide:

• Machine learning engineer: build and run models in products. Next: a deep learning library such as PyTorch, cloud GPUs, and Level 14's engineering in practice.
• Researcher: invent new methods. Next: the mathematics underneath, linear algebra, probability and calculus, and reading papers.
• AI product designer: decide what AI should do for people. Next: user research and design, and Level 15's questions.
• Policy and law: shape how society governs AI. Next: law, ethics and economics, with the technical understanding you now have.

Whichever path you take: keep building small things, read widely, test everything, and stay curious about both what AI can do and what it should do.

Level check

• Choose a project with available data, a success measure and a baseline
• Fill in a project canvas
• Follow the eight milestones from definition to presentation
• Build, train and evaluate a model from scratch, with a confusion matrix
• Analyse errors and choose the most useful next step
• Carry out a responsible AI check
• Write a model card
• Present a project clearly, including its limitations

Key terms you must be able to define

Capstone project

A final project that brings together everything a course has taught.

Project canvas

A one-page plan of a project's problem, users, data, success measure, model, risks and demo.

Baseline

The simplest approach a model must beat to be worth building.

End to end

A version of the whole system that runs from raw input to final output, however simple.

Error analysis

Studying a model's mistakes to find patterns and decide what to improve.

Ablation

Removing one part of a system to measure how much it contributes.

Model card

A short document describing a model's purpose, data, performance, limitations and risks.

README

A file explaining what a project is and how to run it.

Reproducible

Able to be run again by someone else to get the same results.

Flashcards

Go further: notes for future AI engineers

Ablations: what really matters?

Once your model works, remove one ingredient at a time, a feature, a layer, a cleaning step, and measure how much worse it gets. These ablation studies show which parts of a system earn their place. Researchers include them in almost every paper, because they turn "it works" into "it works because of this".

From toy to real

The worked project's 26 messages make the ideas clear but are far too few for a real filter. A real version would use thousands of labelled messages, split by date so the test set comes after the training data, handle misspellings and links, and be evaluated on the cost of each kind of error. The structure, though, would be exactly the same: data, features, model, evaluation, error analysis, model card.

Misconceptions to avoid