MskProdComputing
Back to the topic

AI · 14.1

The Machine Learning Lifecycle: From Idea to Production

Section A: 10 multiple choice questions, 10 marks. Section B: 3 written questions, 13 marks. Total 23 marks.

Name

Class

Date

Mark  / 23

Section A

Circle the letter of the correct answer. One mark each.

  1. 1What did the 2015 "hidden technical debt" paper show about real machine learning systems?1
    1. AMachine learning systems need no maintenance
    2. BThe model code is a small part of the whole system
    3. CThe model code is most of the work
    4. DData collection is fully automatic
  2. 21% of payments are fraud. What is the accuracy of a model that always predicts "not fraud"?1
    1. A0%
    2. B1%
    3. C50%
    4. D99%
  3. 3Which is an example of data leakage?1
    1. AUsing a feature only recorded after the outcome happened
    2. BTraining on more examples
    3. CTesting on data from after the training period
    4. DNormalising features using only the training data
  4. 4Why is a time-based split used for data that changes over time?1
    1. AIt removes the need for a validation set
    2. BThe model must be tested on data from after its training data
    3. CIt stops the model overfitting
    4. DIt makes the test set larger
  5. 5Why does random search usually beat grid search for the same number of trials?1
    1. AIt tries many more values of the hyperparameters that matter
    2. BIt always trains each model much faster
    3. CIt needs no validation data to compare runs
    4. DIt only ever tests the default settings
  6. 6In data parallelism, what does each GPU hold?1
    1. AOnly the test data, for checking the model
    2. BHalf of each weight matrix in every layer
    3. CA full copy of the model, trained on different examples
    4. DOne layer of the model, passing results along
  7. 7Why is mixed precision used in training?1
    1. AIt cuts memory use and speeds training up
    2. BIt removes the need for a GPU
    3. CIt makes the model more accurate on every task
    4. DIt stops the model overfitting
  8. 8What is a canary release?1
    1. ARetraining a model every night
    2. BDeleting the old model straight away
    3. CSending a small share of real traffic to a new model first
    4. DReleasing a model with no testing
  9. 9Fraudsters invent a new trick, so the old patterns no longer predict fraud. What is this?1
    1. AA canary release
    2. BData leakage
    3. COverfitting
    4. DConcept drift
  10. 10What should a model card include?1
    1. AThe names and contact details of its users
    2. BThe model's source code and its weights
    3. CA complete copy of the full training dataset
    4. DPerformance for different groups and known limitations

Section B

Answer in the space provided. The number of marks tells you how many separate points to make.

  1. 1ExplainA bank wants a model to flag fraudulent card payments. Explain how it should frame the problem and choose its metrics.5
  2. 2ExplainExplain the difference between data drift and concept drift, and how a team can detect and respond to them.4
  3. 3DescribeDescribe two ways of splitting the training of a large neural network across many GPUs.4

Mark scheme

Section A

1B2D3A4B5A6C7A8C9D10D

Section B

1 A bank wants a model to flag fraudulent card payments. Explain how it should frame the problem and choose its metrics. 5

One mark for each of the following, to a maximum of 5.

  • Define the decision, e.g. flag payments for review
  • Compare with a simple baseline
  • Fraud is rare, so accuracy is misleading
  • Use precision and recall (or similar)
  • Weigh the different costs of false negatives (missed fraud) and false positives (blocked customers)
  • Choose the threshold to balance those costs, and fix metrics before training

Model answer. The bank should first define the decision the model supports, such as flagging payments for a person to review, and measure what a simple baseline, such as an expert's rules, already achieves. Because fraud is rare, accuracy is misleading: a model that never flags anything could be 99% accurate. Instead it should use precision, the share of flagged payments that are really fraud, and recall, the share of frauds that are caught. A missed fraud and a wrongly blocked customer have different costs, so the bank should decide which matters more and choose the decision threshold to balance them, fixing these metrics before training so that the results cannot be bent to fit.

2 Explain the difference between data drift and concept drift, and how a team can detect and respond to them. 4

One mark for each of the following, to a maximum of 4.

  • Data drift: the distribution of the inputs changes
  • Concept drift: the relationship between inputs and the correct output changes
  • Detected by monitoring inputs, predictions and accuracy, e.g. PSI, with alerts
  • Respond by investigating and retraining on fresh data

Model answer. Data drift is a change in the inputs the model receives, such as customers becoming younger after a new app launches, while concept drift is a change in the relationship between the inputs and the right answer, such as fraudsters inventing new tricks so that old patterns no longer indicate fraud. A team detects them by monitoring the input distributions, for example with the Population Stability Index, along with the predictions and, once labels arrive, the accuracy, with alerts when they move too far. It responds by investigating the cause and retraining the model on fresh data through its automated pipeline.

3 Describe two ways of splitting the training of a large neural network across many GPUs. 4

One mark for each of the following, to a maximum of 4.

  • Data parallelism: each GPU holds a full copy of the model
  • trained on different examples, with gradients averaged
  • Tensor or pipeline parallelism: the model itself is split across GPUs
  • Each GPU holds part of each matrix, or a group of layers, passing results along

Model answer. In data parallelism, every GPU holds a full copy of the model and trains it on a different part of each batch; after each step their gradients are averaged so that all copies stay identical, which lets the batch scale up. When the model is too big for one GPU, the model itself is split: in pipeline parallelism each GPU holds a different group of layers and passes its activations on to the next, while in tensor parallelism each layer's large weight matrices are divided so that several GPUs share the work of a single layer.