Regression: Predicting Numbers with a Line

The problem: predict a number

Ten students recorded their hours of revision and their test score. Can we predict the score of a student who revises for 4.5 hours?

The simplest model is a straight line:

• w is the weight, or slope: how many extra marks each hour of revision is worth.
• b is the bias, or intercept: the predicted score for zero hours.

These two numbers are the model's parameters. Learning means finding the values of w and b that make the line fit the data best. Once we have them, prediction is one multiplication and one addition, which is why linear models are still everywhere: they are fast, cheap and easy to explain.

Measuring error: residuals and mean squared error

For each student, the residual is the actual score minus the predicted score. In the diagram above, residuals are the coloured vertical sticks between each point and the line.

To score a line we need one number for all the residuals together. Adding them does not work, because positive and negative residuals cancel out. So we square them first:

The MSE of these three is (13.7 + 2.9 + 0.6) ÷ 3 ≈ 5.7.

Why square?

• Squares are never negative, so errors cannot cancel.
• Squaring punishes big errors much more than small ones: an error of 10 costs 100, but two errors of 5 cost only 50.
• The maths of squares is smooth, which makes the best line easy to find.

Two relatives: the root mean squared error (RMSE) is the square root of the MSE, back in the original units (marks), and the mean absolute error (MAE) averages the sizes of the residuals without squaring, so it is less affected by a few big misses.

Finding the best line: least squares

The least squares line is the one with the smallest possible MSE. For a single feature there is an exact formula:

Here x̄ and ȳ are the mean of the hours and the mean of the scores, and Σ means "add up over every student". The top of the fraction measures how hours and scores vary together; the bottom measures how much the hours vary on their own. The formula for b guarantees the line passes through the point (x̄, ȳ).

For the ten students this gives w ≈ 6.52 and b ≈ 33.2: every hour of revision is worth about six and a half marks, and a student who did none would be predicted 33.

Drag a point far from the others in the lab above and watch the line swing towards it. Squaring makes least squares very sensitive to outliers.

More features: planes and hyperplanes

House prices depend on more than one thing. With several features, the model gives each one its own weight:

With two features the model is no longer a line but a flat plane floating above the floor of a 3D graph, and least squares tilts it to pass as close as possible to all the points.

With more than two features the plane becomes a hyperplane, which we cannot picture but which the maths handles exactly the same way. A model with 500 features has 500 weights and one bias.

Reading the weights

If w₁ = 2,100, each extra square metre adds about £2,100 to the predicted price, with the other features held fixed. A negative weight on distance means homes further from the station are predicted to be cheaper. Weights can only be compared with each other if the features were scaled first (Level 3).

Curves, underfitting and overfitting

Real relationships are often curved. A trick keeps the model linear in its parameters while letting it bend: add powers of a feature as new features, such as hours² and hours³. This is polynomial regression.

But flexibility is dangerous.

An overfitted model has memorised its training examples, including their random noise, so it fails on new data. This tension is the bias-variance trade-off: simple models are consistently wrong (high bias), while very flexible models change wildly with each sample of data (high variance).

Regularisation

Regularisation adds a penalty for large weights to the error being minimised, so the model only uses big weights when the data really demands them. Ridge regression penalises the sum of squared weights; lasso penalises the sum of their sizes and tends to set useless weights to exactly zero. The same idea, called weight decay, is used to train neural networks.

Judging a regression model

R² compares the model with the laziest possible baseline: always predicting the mean. R² = 0 means no better than the mean, R² = 1 means perfect, and a negative R² means worse than just guessing the average.

Finally, plot the residuals. If they show a pattern, such as a curve or a funnel shape, the model has missed something systematic.

Hands-on: build a regression model from scratch

Least squares, by hand

hours = [1, 2, 2, 3, 4, 5, 6, 6, 7, 8]
score = [38, 45, 50, 52, 61, 64, 70, 75, 78, 86]

n = len(hours)
mean_x = sum(hours) / n
mean_y = sum(score) / n
w = (sum((x - mean_x) * (y - mean_y) for x, y in zip(hours, score))
     / sum((x - mean_x) ** 2 for x in hours))
b = mean_y - w * mean_x
print(f"Model: score = {w:.2f} x hours + {b:.2f}")
print(f"Prediction for 4.5 hours: {w * 4.5 + b:.1f}")

predicted = [w * x + b for x in hours]
mse = sum((y - p) ** 2 for y, p in zip(score, predicted)) / n
mae = sum(abs(y - p) for y, p in zip(score, predicted)) / n
baseline = sum((y - mean_y) ** 2 for y in score) / n      # MSE of always guessing the mean
print(f"MSE {mse:.2f}, RMSE {mse ** 0.5:.2f}, MAE {mae:.2f}")
print(f"Baseline MSE {baseline:.2f}, so R squared = {1 - mse / baseline:.3f}")

Overfitting, caught in the act

A "memoriser" predicts the score of whichever training student revised the most similar number of hours. On its own training data it is perfect. Watch what happens on students it has not seen.

train = [(1, 38), (2, 45), (3, 52), (5, 64), (6, 70), (7, 78), (8, 86)]
test = [(2, 50), (4, 61), (6, 75)]

def fit_line(data):
    n = len(data)
    mx = sum(x for x, y in data) / n
    my = sum(y for x, y in data) / n
    w = sum((x - mx) * (y - my) for x, y in data) / sum((x - mx) ** 2 for x, y in data)
    return w, my - w * mx

w, b = fit_line(train)

def line(x):
    return w * x + b

def memoriser(x):
    nearest = min(train, key=lambda pair: abs(pair[0] - x))
    return nearest[1]

def mse(model, data):
    return sum((y - model(x)) ** 2 for x, y in data) / len(data)

for name, model in [("line", line), ("memoriser", memoriser)]:
    print(f"{name:10} training MSE {mse(model, train):6.2f}   test MSE {mse(model, test):6.2f}")

The memoriser's training error is zero and its test error is several times the line's. A perfect training score is a warning sign, not a triumph.

Try this

1. Add a student who revised 10 hours and scored 20, then refit. How far does the line move?
2. Change the memoriser to average its two nearest students. Does it overfit less?

Level check

• Write the equation of a linear model and explain what w and b mean
• Calculate residuals and the mean squared error for a few points
• Explain why errors are squared, and what RMSE, MAE and R² measure
• Use the least squares formulas, and explain why the line passes through the mean point
• Describe a model with several features as a plane or hyperplane, and interpret a weight
• Explain underfitting, overfitting and how regularisation helps

Key terms you must be able to define

Regression

Predicting a number, such as a price or a score.

Weight (slope)

The number a feature is multiplied by in a linear model; how much the prediction changes per unit of the feature.

Bias (intercept)

The constant added in a linear model; the prediction when every feature is zero.

Residual

The actual value minus the predicted value for one example.

Mean squared error (MSE)

The average of the squared residuals, the most common measure of regression error.

Least squares

Choosing the parameters that make the sum of squared residuals as small as possible.

R²

The fraction of the variation in the target explained by the model, compared with always predicting the mean.

Hyperplane

The flat surface a linear model forms when it has more than two features.

Overfitting

Learning the noise in the training data as well as the pattern, so the model does badly on new data.

Underfitting

A model too simple to capture even the main pattern.

Regularisation

A penalty on large weights that discourages overfitting.

Flashcards

Go further: notes for future AI engineers

The normal equation, and why we still use gradient descent

With many features, least squares has a matrix formula, the normal equation: w = (XᵀX)⁻¹Xᵀy, where X holds one row per example and y the targets. Ridge regression simply adds a multiple of the identity matrix before inverting, which also makes the maths stable.

So why do engineers train even linear models with gradient descent? Inverting a matrix gets very expensive as the number of features grows, roughly with its cube; datasets can be too large to fit in memory at once; and almost every other model, including every neural network, has no formula at all. Gradient descent works for all of them, a few examples at a time.

Correlation is not causation

A regression weight measures association, not cause. Ice cream sales predict drownings because hot weather drives both. Answering "what would happen if we changed x?" needs experiments or the tools of causal inference, a field of its own.

Misconceptions to avoid