F201 TA3: Big Data with Machine Learning and AI

1 State the difference between supervised and unsupervised learning. 2

One mark for each of the following, to a maximum of 2.

• Supervised learning uses labelled data to predict a known target
• Unsupervised learning finds structure in unlabelled data

Model answer. Supervised learning is trained on labelled examples, where each input has a known correct answer, and learns to predict that answer for new data. Unsupervised learning is given unlabelled data and finds structure within it, such as clusters of similar records or anomalies, without being told what to look for.

2 A spam filter was tested on 1,000 emails. It correctly identified 180 spam emails, wrongly flagged 20 legitimate emails as spam, and missed 40 spam emails. Calculate its precision and recall, and explain which matters more for a spam filter. 6

One mark for each of the following, to a maximum of 6.

• Precision is calculated as 180/200
• Precision is stated as 90%
• Recall is calculated as 180/220
• Recall is stated as about 82%
• Precision is identified as more important for a spam filter
• Justified by the cost of the error

Model answer. Precision is true positives divided by all predicted positives: 180 / (180 + 20) = 180 / 200 = 90%. Recall is true positives divided by all actual positives: 180 / (180 + 40) = 180 / 220 = 81.8%. For a spam filter, precision matters more. A false negative means one piece of spam reaches the inbox, which costs the user a moment's irritation and nothing else. A false positive means a legitimate email is hidden in a folder the user may never check, and if that email is a job offer, an invoice or a medical appointment the consequence is serious and the user is not even aware anything was lost. A spam filter is therefore tuned to be cautious about declaring spam, accepting a lower recall in exchange for very high precision.

3 Explain what overfitting is and describe two ways it can be reduced. 5

One mark for each of the following, to a maximum of 5.

• Overfitting is learning the training data including its noise
• The consequence on unseen data
• One valid mitigation
• It is explained
• A second valid mitigation, explained

Model answer. Overfitting happens when a model learns the training data too closely, capturing not only the real pattern but also the noise and the accidents of that particular sample. The symptom is a model that performs almost perfectly on the data it was trained on and markedly worse on data it has never seen, because it has effectively memorised rather than generalised. One way to reduce it is to use a simpler model or fewer features, since a model with less capacity cannot fit the noise; a shallow decision tree, or a pruned one, is forced to find the broad pattern. A second is to train on more data, because noise does not repeat while the real pattern does, so additional examples dilute the accidental features. Alongside both, holding back a validation set and stopping training when validation performance stops improving detects overfitting as it begins rather than after the model is deployed.

4 A hospital is considering a machine learning model to predict which patients are at risk of readmission within 30 days. Discuss how such a model would be developed and evaluated, and the issues the hospital should consider before using it. 8

One mark for each of the following, to a maximum of 8.

• Data and features are identified
• It is identified as supervised classification with labels from history
• Train, validation and test split is described
• Appropriate evaluation measures are chosen
• Recall is prioritised with a reason
• Bias in the training data is addressed
• Explainability and clinical oversight are addressed
• Legal, ethical or monitoring issues with a judgement

Model answer. The task is supervised classification, and the labels already exist in the hospital's own history: for every past discharge it is known whether the patient was readmitted within thirty days. Features would come from the records the hospital already holds, such as age, primary and secondary diagnoses, length of stay, number of previous admissions, prescribed medication, and whether the patient lives alone, with feature engineering likely to matter more than the choice of algorithm. The data would be split into training, validation and test sets, and the split should be by time rather than at random, training on earlier years and testing on the most recent, because that is the only way to estimate how the model will behave on future patients rather than on contemporaries of its training data. Evaluation must not rest on accuracy. Readmission is a minority outcome, so a model predicting that nobody is readmitted would score well and be worthless; the hospital needs the confusion matrix and both precision and recall. Recall should be prioritised, because a missed high risk patient is a patient who comes back seriously unwell, while a false positive costs a follow-up phone call, so the threshold should be set to catch cases and the resulting workload planned for. Three issues then need settling before any use. The first is bias: if historic care was less effective for a deprived or minority group, the model learns that pattern as though it were clinical fact, so performance has to be measured separately for each group rather than only overall, and an unequal error rate treated as a defect. The second is explainability: a clinician cannot act on a score with no reason attached, and the hospital cannot defend a decision it cannot explain, so a model whose contributing factors can be surfaced is worth more here than a slightly more accurate opaque one. The third is legal and operational: this is special category health data, which means a Data Protection Impact Assessment, transparency with patients, and no solely automated decision with significant effects. Taken together, the model should be deployed as decision support that flags patients for a clinician to review, never as an automated discharge or intervention decision, with its performance and its error rates by group monitored continuously and the model retrained as practice and population change.