Datasets, Splits and Bias: Good Data In, Good AI Out

Where data comes from

"Garbage in, garbage out" is older than AI, and nowhere truer. A model can only learn what its data contains.

Personal data is protected by law. In the UK, the UK GDPR and the Data Protection Act 2018 require a lawful basis for using it, data minimisation and security, and give people rights over data about them.

Training, validation and test sets

A model must be judged on data it has never seen, or you only learn how well it memorised. So the data is split before training starts.

If you keep tweaking the model until the test score looks good, the test set has quietly become a second validation set, and the final score is too optimistic. It should be used once, at the end.

Data leakage

Leakage is when information that would not be available in real use sneaks into training. Common causes:

• The same item, or a near copy, in both training and test sets
• Several photos of the same patient split across sets, so the model recognises the patient rather than the illness
• Features from the future, such as using "was refunded" to predict "will complain"
• Scaling or cleaning using statistics from the whole dataset

Shuffle before splitting, but never shuffle time series: train on the past and test on the future, as the model will be used.

Cleaning data

Real data is messy. Data scientists often say cleaning takes most of their time.

Class imbalance and the accuracy trap

Suppose 1 card payment in 100 is fraud. A "model" that always answers not fraud is right 99% of the time and catches no fraud at all.

When one class is rare, accuracy mostly measures the imbalance. Better measures, explained fully in Level 4:

• Recall: of all the real fraud, how much did we catch?
• Precision: of everything we flagged, how much was really fraud?

Fixes for imbalance

• Measure with precision and recall, not accuracy alone
• Oversample the rare class or undersample the common one
• Give mistakes on the rare class a bigger weight during training
• Collect more examples of the rare class

Bias: how unfairness gets into data

A model learns the patterns in its data, including the unfair ones. Bias can enter at every stage.

What engineers do about it

• Check who and what is missing from the data before training
• Audit by subgroup: a good overall accuracy can hide a much worse result for one group
• Document datasets with a datasheet (what was collected, how, by whom, and its gaps), an idea proposed by Timnit Gebru and colleagues in 2018
• Keep people involved in high-stakes decisions, and let those affected challenge them

Hands-on: splitting, auditing and the accuracy trap

A reproducible split

import random

random.seed(7)                   # the same "random" shuffle every run
data = [f"photo_{i:02}" for i in range(20)]
random.shuffle(data)

n = len(data)
train = data[:int(0.7 * n)]
valid = data[int(0.7 * n):int(0.85 * n)]
test = data[int(0.85 * n):]
print(len(train), "training:", train[:4], "...")
print(len(valid), "validation:", valid)
print(len(test), "test:", test)
print("Any overlap?", bool(set(train) & set(test)))

The accuracy trap

labels = ["ok"] * 990 + ["fraud"] * 10
predictions = ["ok"] * len(labels)          # a lazy model: never says fraud

accuracy = sum(p == y for p, y in zip(predictions, labels)) / len(labels)
caught = sum(p == y == "fraud" for p, y in zip(predictions, labels))
print(f"Accuracy: {accuracy:.1%}")
print(f"Fraud caught: {caught} of {labels.count('fraud')}  (recall {caught / labels.count('fraud'):.0%})")

Cleaning a messy table

rows = [
    {"name": "Ava", "age": 15, "score": 72},
    {"name": "Ben", "age": None, "score": 65},
    {"name": "Ava", "age": 15, "score": 72},
    {"name": "Cai", "age": 16, "score": None},
    {"name": "Dee", "age": 160, "score": 81},
]

seen, clean = set(), []
for r in rows:
    key = tuple(r.values())
    if key in seen:
        print("Duplicate removed:", r["name"])
        continue
    seen.add(key)
    clean.append(r)

for r in clean:
    if r["age"] is not None and not 10 <= r["age"] <= 19:
        print("Suspicious age for", r["name"], "->", r["age"])

scores = sorted(r["score"] for r in clean if r["score"] is not None)
median = scores[len(scores) // 2]
for r in clean:
    if r["score"] is None:
        r["score"] = median
        print("Filled missing score for", r["name"], "with the median", median)

An audit by subgroup

# How often a face unlock model was right, for two groups of users
results = {"group A": (950, 1000), "group B": (42, 50)}

total_right = sum(r for r, n in results.values())
total = sum(n for r, n in results.values())
print(f"Overall accuracy: {total_right / total:.1%}")
for group, (right, n) in results.items():
    print(f"  {group}: {right / n:.1%} from {n} people")

An overall score of 94.5% looks excellent, until the audit shows one group gets 84%: about one failure in six. The smaller group barely moves the average, which is exactly why averages hide unfairness.

Activity: who is missing?

A company trains a voice assistant on 10,000 hours of speech recorded by its own staff in one city. In groups, list who is likely to be under-represented, what would go wrong for them, and what data you would collect to fix it.

Level check

• Describe four sources of training data and a risk of each
• Explain the purpose of the training, validation and test sets
• Explain data leakage and give two ways it happens
• Describe how to deal with missing values, duplicates and outliers
• Explain why accuracy misleads on imbalanced data
• Describe three types of bias with a real example of each, and how engineers reduce them

Key terms you must be able to define

Training set

The data a model learns its parameters from.

Validation set

Data held back to choose settings and compare models during development.

Test set

Data kept unseen until the end, used once for an honest estimate of performance.

Data leakage

Information that would not be available in real use getting into training, making results look better than they are.

Class imbalance

When one class is far rarer than another, such as fraud among payments.

Outlier

A value far from the rest, which may be an error or a genuine and important extreme.

Label noise

Training examples whose labels are wrong.

Sampling bias

Data that does not represent the population the system will be used on.

Historical bias

Data that accurately records an unfair past, which a model then learns to repeat.

Measurement bias

Using a feature that is a poor stand-in for the thing that really matters.

Datasheet for a dataset

A document describing how a dataset was made, what it contains, and its gaps and intended uses.

Flashcards

Go further: notes for future AI engineers

Data-centric AI

For years research competed on better models for fixed datasets. Andrew Ng and others have argued for data-centric AI: hold the model fixed and systematically improve the data. In practice, cleaning labels, adding examples of the cases the model gets wrong and making the data consistent often helps more than a cleverer architecture.

Professional teams also version their data the way they version code, so every model can be traced back to exactly which data trained it. When a model misbehaves months later, that record is the first thing anyone asks for.

Misconceptions to avoid