Everything Is Numbers: Turning the World into Data

Examples, features and labels

A model never sees a cat, a song or a sentence. It sees numbers. The first job in any AI project is turning the world into a dataset: a table of numbers a model can learn from.

• Each row is one example (also called a sample, instance or data point).
• Each column is a feature: one measurable property of the examples.
• In supervised learning one extra column holds the label, the answer the model must learn to predict.

Kinds of feature

Models only do arithmetic, so every kind must end up as numbers. Choosing and building good features is called feature engineering, and before deep learning it was most of the work.

Images are grids of numbers

A greyscale image is a grid of pixels, each a brightness from 0 (black) to 255 (white): one byte each. The famous MNIST dataset of handwritten digits uses images 28 pixels square, so each digit is 784 numbers.

A colour image stores three numbers per pixel, one each for red, green and blue. Stack the three grids and you have a block of numbers: height × width × 3 channels.

That last row is why video models need so much computing power, and why images are usually shrunk before training.

Sound, text and categories as numbers

Sound

A microphone measures air pressure thousands of times a second. CD quality audio takes 44,100 samples a second, each a number. Speech models often turn the sound into a spectrogram first: a picture of which frequencies are loud at each moment, which can then be treated like an image.

Text

Every character already has a number (its Unicode code point: "A" is 65). Models usually work with bigger pieces called tokens, each given an ID from a fixed vocabulary. Level 9 shows how modern models split text into tokens and turn each one into a list of meaningful numbers.

Categories: one-hot encoding

You cannot just number categories, because the numbers imply an order and a size that are not there. If red = 1, green = 2 and blue = 3, a model may conclude blue is "three times" red, or that green is the average of red and blue.

One-hot encoding gives each category its own column, with a 1 in the column that applies and 0 everywhere else:

For ordinal categories, where the order is real (small, medium, large), plain numbers 0, 1, 2 are fine.

Scalars, vectors, matrices and tensors

AI engineers describe data by its shape.

"Tensor" is the general word for all of them, which is where the name of the TensorFlow library comes from. Models process data in batches, many examples at once, so there is nearly always an extra batch dimension at the front.

The dot product: the most important sum in AI

Multiply two vectors position by position and add up the results:

A dot product measures how much two vectors point the same way. Large and positive: similar direction. Zero: unrelated. Negative: opposite. Every artificial neuron computes a dot product, and so does every attention score in a transformer. Graphics processors are built to do billions of them a second.

Scaling features

Features are often on wildly different scales. House size might run from 50 to 300 square metres while bedrooms run from 1 to 6. Any method that measures distance will then be dominated by the feature with the biggest numbers, and gradient descent (Level 5) trains slowly and unevenly.

Two standard fixes

Standardisation copes better with outliers, because one huge value would squash every other min-max value towards 0.

Hands-on: turning things into numbers

An image is a list of lists

# A 7 by 7 greyscale image: 0 is black, 255 is white
image = [
    [  0,   0, 255, 255, 255,   0,   0],
    [  0, 255,   0,   0,   0, 255,   0],
    [255,   0, 255,   0, 255,   0, 255],
    [255,   0,   0,   0,   0,   0, 255],
    [255,   0, 255, 255, 255,   0, 255],
    [  0, 255,   0,   0,   0, 255,   0],
    [  0,   0, 255, 255, 255,   0,   0],
]

def show(img):
    for row in img:
        print("".join("##" if p > 127 else ". " for p in row))

show(image)
flat = [p for row in image for p in row]      # what a simple network sees
print("Shape 7 x 7 becomes a vector of", len(flat), "numbers:", flat[:10], "...")
print("Scaled to 0..1:", [p / 255 for p in flat[:10]], "...")
inverted = [[255 - p for p in row] for row in image]
show(inverted)

One-hot encoding, scaling and distance

import math

def one_hot(value, categories):
    return [1 if c == value else 0 for c in categories]

colours = ["red", "green", "blue"]
for c in colours:
    print(f"{c:6} -> {one_hot(c, colours)}")

# Three houses: (size in square metres, bedrooms)
A, B, C = (100, 2), (102, 5), (130, 2)
print("Unscaled: A to B", round(math.dist(A, B), 2), " A to C", round(math.dist(A, C), 2))

# Min-max scale each feature using ranges from the training data
size_lo, size_hi, bed_lo, bed_hi = 50, 300, 1, 6
def scale(h):
    return ((h[0] - size_lo) / (size_hi - size_lo), (h[1] - bed_lo) / (bed_hi - bed_lo))

sA, sB, sC = scale(A), scale(B), scale(C)
print("Scaled:   A to B", round(math.dist(sA, sB), 2), " A to C", round(math.dist(sA, sC), 2))

def dot(u, v):
    return sum(a * b for a, b in zip(u, v))
print("Dot product [1, 2, 3] . [4, 5, 6] =", dot([1, 2, 3], [4, 5, 6]))

Unscaled, house B looks almost identical to A, because a 2 square metre difference is tiny next to the 30 square metre gap to C, and the 3 extra bedrooms barely register. Once scaled, the extra bedrooms count properly and C becomes the nearest. A nearest neighbour model would give different answers before and after scaling.

Activity: design the features

A school wants to predict whether a student will be late tomorrow. In a group, list ten features you could collect, say what kind each one is, how you would turn it into numbers, and which ones it would be unfair or intrusive to use. Then do the same for predicting whether someone will skip a song.

Level check

• Describe a dataset in terms of examples, features and labels
• Explain how a colour image is stored as numbers, and work out its size from its shape
• Explain why categories need one-hot encoding, and when plain numbers are fine
• Name and give the shape of a scalar, vector, matrix and tensor
• Calculate a dot product and say what it measures
• Explain why features are scaled and the difference between min-max and standardisation

Key terms you must be able to define

Dataset

A collection of examples, usually a table with one row per example and one column per feature.

Example

One row of a dataset: one house, one image, one email. Also called a sample or data point.

Feature

One measurable property of the examples, used as an input to the model.

Feature engineering

Choosing, building and transforming features so a model can learn from them.

Pixel

One point in an image, stored as a brightness or as red, green and blue values from 0 to 255.

Channel

One layer of an image's values, such as its red, green or blue values.

One-hot encoding

Turning a category into a vector with a 1 in its own position and 0 everywhere else.

Tensor

A block of numbers with any number of dimensions; scalars, vectors and matrices are tensors of rank 0, 1 and 2.

Dot product

The sum of the products of two vectors' matching entries, measuring how much they point the same way.

Normalisation

Rescaling a feature into a fixed range, usually 0 to 1, with (x − min) ÷ (max − min).

Standardisation

Rescaling a feature to mean 0 and standard deviation 1 with (x − mean) ÷ standard deviation.

Flashcards

Go further: notes for future AI engineers

The curse of dimensionality

Every feature adds a dimension. With 784 pixel features, each MNIST digit is a point in 784-dimensional space, and high-dimensional spaces behave strangely. The volume grows so fast that data becomes sparse: to keep the same density of examples, doubling the number of features can require squaring the amount of data. Distances also become less meaningful, because most points end up roughly the same distance apart.

This is why engineers reduce dimensions (Level 4 mentions principal component analysis), choose features carefully, and why deep learning's ability to learn compact features from raw data was such a breakthrough. A network's hidden layers are, in effect, automatic feature engineering.

Misconceptions to avoid