F201 TA3: Big Data with Machine Learning and AI
What machine learning actually is, the three kinds of learning, how a model is trained and tested, and how to tell whether it works.
AI, machine learning and deep learning
These three are often used as synonyms and are not.
| Term | What it means |
|---|---|
| Artificial intelligence | The broad field of making systems perform tasks that appear to need intelligence. Includes rule based systems with no learning at all |
| Machine learning | Systems that improve at a task by finding patterns in data rather than by being explicitly programmed with rules |
| Deep learning | Machine learning using neural networks with many layers, which is what made image, speech and language processing work |
They nest: deep learning is part of machine learning, which is part of AI.
Why big data and machine learning arrived together
A learning algorithm needs examples, and its performance improves with the number of good ones. Two things made modern machine learning practical at the same moment: enough data, because systems began logging everything, and enough compute, because GPUs could do the arithmetic in parallel. Neither alone was sufficient, which is why techniques invented decades earlier only became useful recently.
The traditional approach against the learned one
To detect spam with rules, a programmer writes conditions: these words, these senders, these patterns. Every new trick needs a new rule, written by a human, forever. With machine learning, the system is given a large set of messages already labelled spam or not spam, finds the features that distinguish them, and is retrained as the tricks change. The rules are no longer written; they are derived, which is both the power and the problem, because nobody chose them and they can be hard to inspect.
"Machine learning is a system that learns patterns from data in order to make predictions or decisions, improving with more examples, rather than following rules written explicitly by a programmer."
The three kinds of learning, and the tasks
| Type | What it is given | What it learns | Examples |
|---|---|---|---|
| Supervised | Labelled examples: inputs with the right answer | To predict the label for new inputs | Spam detection, credit scoring, image classification, price prediction |
| Unsupervised | Unlabelled data | Structure that was not specified: groups, oddities, dimensions | Customer segmentation, anomaly detection, topic discovery |
| Reinforcement | An environment, actions and rewards | A policy of actions that maximises long run reward | Game playing, robotics, routing, energy control |
The tasks within supervised learning
- Classification predicts a category: spam or not, which of ten digits, will churn or will not
- Regression predicts a number: a price, a demand, a time to failure
And within unsupervised learning
- Clustering groups similar records, without being told the groups
- Anomaly detection finds records unlike the rest, which is how fraud and faults surface
- Dimensionality reduction compresses many variables into a few that carry most of the signal
The vocabulary of a model
| Term | Meaning |
|---|---|
| Feature | An input variable the model uses |
| Label | The correct answer, in supervised learning |
| Feature engineering | Creating useful inputs from raw data, which often matters more than the algorithm |
| Training set | The data the model learns from, typically most of it |
| Validation set | Held back to tune the model and compare options |
| Test set | Held back, untouched, to estimate real performance once |
| Overfitting | Learning the training data including its noise, so performance collapses on new data |
| Underfitting | Too simple to capture the real pattern, so it is poor on everything |
| Hyperparameter | A setting chosen before training, such as depth or learning rate |
A model tested on what it learned from will look excellent and tell you nothing. The test set has to be data the model has never seen, and it is used once, at the end. Testing repeatedly on it turns it into another training set.
- The input layer: one node per feature of the record.
- A hidden layer: each node combines every input, with its own weights.
- A second hidden layer: it combines what the first layer produced.
- The output layer: one number, here the probability of fraud.
- Training adjusts every weight, over and over, to reduce the error.
Measuring whether a model works
The confusion matrix
For a classifier, every prediction falls into one of four cells.
| Predicted positive | Predicted negative | |
|---|---|---|
| Actually positive | True positive (TP) | False negative (FN), a miss |
| Actually negative | False positive (FP), a false alarm | True negative (TN) |
The measures that come out of it
| Measure | Formula | The question it answers |
|---|---|---|
| Accuracy | (TP + TN) / all | What proportion of predictions were right? |
| Precision | TP / (TP + FP) | When it says positive, how often is it right? |
| Recall | TP / (TP + FN) | Of the real positives, how many did it find? |
| F1 score | the harmonic mean of precision and recall | A single figure balancing the two |
Why accuracy alone is misleading
Suppose 1 in 100 transactions is fraudulent. A model that predicts "not fraud" every single time is 99% accurate and catches nothing at all. Its recall is zero. On an imbalanced problem, accuracy measures the imbalance rather than the model, which is why precision and recall are reported together.
Precision against recall is a business decision
Tightening a model to raise precision lowers recall and vice versa. Which matters more depends entirely on the cost of each error:
- Cancer screening: a missed case is catastrophic, a false alarm means a further test. Favour recall.
- Spam filtering: a missed spam is an annoyance, a blocked job offer is serious. Favour precision.
- Fraud blocking: blocking a real customer's card loses them; missing a fraud costs money. The threshold is a commercial judgement, and it is usually set so a human reviews the uncertain middle.
For regression, the equivalent measures are mean absolute error and root mean squared error, in the units of the thing being predicted, which is why an error of "£38 per order" communicates better than any coefficient.
Applications across industries
| Sector | Application |
|---|---|
| Retail | Recommendation, demand forecasting, dynamic pricing, churn prediction |
| Finance | Fraud detection, credit scoring, anti money laundering, trading |
| Healthcare | Image diagnosis, triage, drug discovery, readmission risk |
| Manufacturing | Predictive maintenance, visual quality inspection, yield optimisation |
| Transport | Routing, demand prediction, autonomous driving, maintenance |
| Energy | Load forecasting, grid balancing, fault prediction |
| Agriculture | Yield prediction, disease detection from imagery, irrigation control |
| Public sector | Demand planning, fraud detection, triage of casework |
"Recall matters more here, because a missed fraudulent transaction is an unrecoverable loss while a false positive costs one customer service call, so the threshold should be set low and the borderline cases sent for human review." Measure, reason, and the operational consequence.
Step 1 of 5 Split the data first, before looking at any of it.
Classifier scorer
Move the threshold and watch precision and recall trade against each other. This tool needs JavaScript, so turn it on to use it. Everything you need to answer the exam question is explained in the text above.
Step 1 of 4 Underfitting: too simple to capture the real shape. Poor on everything.
Key terms you must be able to define
- Artificial intelligence
- The field of making systems perform tasks that appear to require intelligence.
- Machine learning
- Systems that learn patterns from data rather than following explicitly programmed rules.
- Deep learning
- Machine learning using neural networks with many layers.
- Supervised learning
- Learning from labelled examples to predict labels for new data.
- Unsupervised learning
- Finding structure in unlabelled data, such as clusters or anomalies.
- Reinforcement learning
- Learning a policy of actions through rewards and penalties from an environment.
- Classification
- Predicting which category something belongs to.
- Regression
- Predicting a numerical value.
- Clustering
- Grouping similar records without being told the groups in advance.
- Feature
- An input variable used by a model.
- Label
- The known correct answer attached to a training example.
- Training data
- The data a model learns from.
- Test data
- Held-back data, never seen in training, used to estimate real performance.
- Overfitting
- Learning the training data including its noise, so performance on new data collapses.
- Confusion matrix
- A table of true and false positives and negatives for a classifier.
- Precision
- Of the cases predicted positive, the proportion that really are.
- Recall
- Of the real positive cases, the proportion the model found.
- Accuracy
- The proportion of all predictions that were correct.
Flashcards
Say the definition out loud before you flip the card. Retrieving it from memory is what builds recall, reading it again is not.
Reach the top grade on this topic
This topic area rewards arithmetic and judgement together. Be able to compute precision and recall from a confusion matrix, state what each means in the scenario's own terms, and then say which one the organisation should optimise and what that costs. An answer that defines the terms without applying them to the scenario stays in the lower bands.
Mistakes that cost marks here
- Using AI, machine learning and deep learning as synonyms.
- Quoting accuracy on an imbalanced problem without mentioning precision and recall.
- Evaluating a model on its training data.
- Confusing overfitting with underfitting.
- Describing clustering as supervised. Nobody provides the groups.
- Treating a model's output as fact rather than as a prediction with an error rate.
A model that predicts a rare disease by always answering 'no' can be 99% accurate and completely useless. That single observation is why accuracy is never reported alone, and it is the most reliable exam question in this topic area.
Knowledge check
Exam-style questions
Five written questions in the style of the real paper. Write a full answer, then let the marker check it against the mark scheme.
- LSupervised learning uses labelled data to predict a known target
- LUnsupervised learning finds structure in unlabelled data
Supervised learning is trained on labelled examples, where each input has a known correct answer, and learns to predict that answer for new data. Unsupervised learning is given unlabelled data and finds structure within it, such as clusters of similar records or anomalies, without being told what to look for.
- LPrecision is calculated as 180/200
- LPrecision is stated as 90%
- LRecall is calculated as 180/220
- LRecall is stated as about 82%
- LPrecision is identified as more important for a spam filter
- LJustified by the cost of the error
Precision is true positives divided by all predicted positives: 180 / (180 + 20) = 180 / 200 = 90%. Recall is true positives divided by all actual positives: 180 / (180 + 40) = 180 / 220 = 81.8%. For a spam filter, precision matters more. A false negative means one piece of spam reaches the inbox, which costs the user a moment's irritation and nothing else. A false positive means a legitimate email is hidden in a folder the user may never check, and if that email is a job offer, an invoice or a medical appointment the consequence is serious and the user is not even aware anything was lost. A spam filter is therefore tuned to be cautious about declaring spam, accepting a lower recall in exchange for very high precision.
- LOverfitting is learning the training data including its noise
- LThe consequence on unseen data
- LOne valid mitigation
- LIt is explained
- LA second valid mitigation, explained
Overfitting happens when a model learns the training data too closely, capturing not only the real pattern but also the noise and the accidents of that particular sample. The symptom is a model that performs almost perfectly on the data it was trained on and markedly worse on data it has never seen, because it has effectively memorised rather than generalised. One way to reduce it is to use a simpler model or fewer features, since a model with less capacity cannot fit the noise; a shallow decision tree, or a pruned one, is forced to find the broad pattern. A second is to train on more data, because noise does not repeat while the real pattern does, so additional examples dilute the accidental features. Alongside both, holding back a validation set and stopping training when validation performance stops improving detects overfitting as it begins rather than after the model is deployed.
- LData and features are identified
- LIt is identified as supervised classification with labels from history
- LTrain, validation and test split is described
- LAppropriate evaluation measures are chosen
- LRecall is prioritised with a reason
- LBias in the training data is addressed
- LExplainability and clinical oversight are addressed
- LLegal, ethical or monitoring issues with a judgement
The task is supervised classification, and the labels already exist in the hospital's own history: for every past discharge it is known whether the patient was readmitted within thirty days. Features would come from the records the hospital already holds, such as age, primary and secondary diagnoses, length of stay, number of previous admissions, prescribed medication, and whether the patient lives alone, with feature engineering likely to matter more than the choice of algorithm. The data would be split into training, validation and test sets, and the split should be by time rather than at random, training on earlier years and testing on the most recent, because that is the only way to estimate how the model will behave on future patients rather than on contemporaries of its training data. Evaluation must not rest on accuracy. Readmission is a minority outcome, so a model predicting that nobody is readmitted would score well and be worthless; the hospital needs the confusion matrix and both precision and recall. Recall should be prioritised, because a missed high risk patient is a patient who comes back seriously unwell, while a false positive costs a follow-up phone call, so the threshold should be set to catch cases and the resulting workload planned for. Three issues then need settling before any use. The first is bias: if historic care was less effective for a deprived or minority group, the model learns that pattern as though it were clinical fact, so performance has to be measured separately for each group rather than only overall, and an unequal error rate treated as a defect. The second is explainability: a clinician cannot act on a score with no reason attached, and the hospital cannot defend a decision it cannot explain, so a model whose contributing factors can be surfaced is worth more here than a slightly more accurate opaque one. The third is legal and operational: this is special category health data, which means a Data Protection Impact Assessment, transparency with patients, and no solely automated decision with significant effects. Taken together, the model should be deployed as decision support that flags patients for a clinician to review, never as an automated discharge or intervention decision, with its performance and its error rates by group monitored continuously and the model retrained as practice and population change.