MskProdComputing
Back to the topic

AI · 15.2

Capstone: Build, Test and Explain Your Own AI

Section A: 10 multiple choice questions, 10 marks. Section B: 2 written questions, 10 marks. Total 20 marks.

Name

Class

Date

Mark  / 20

Section A

Circle the letter of the correct answer. One mark each.

  1. 1What should a good first capstone project have?1
    1. AA dataset of millions of examples
    2. BData you can get, a success measure and a baseline
    3. CThe largest model you can find
    4. DA finished product on its first day
  2. 2Why build a baseline before the real model?1
    1. ATo know whether the model is actually any better
    2. BBecause baselines are always more accurate
    3. CBecause the model cannot train without one
    4. DTo make the test set bigger
  3. 3When should the test set be used?1
    1. ABefore training, to choose the features
    2. BNever: it is only for documentation
    3. COnce, at the end, after all choices are made
    4. DAfter every change, to pick the best version
  4. 4The spam model missed "cheap loans approved today no checks". What is the most useful next step?1
    1. ARaise the learning rate
    2. BUse a model with many more layers
    3. CAdd more varied spam messages to the training data
    4. DRemove the test set
  5. 5Why is a test set of six messages a problem?1
    1. AIt makes the model overfit
    2. BThe accuracy measured on it is very unreliable
    3. CSix messages take too long to test
    4. DModels cannot be tested on short messages
  6. 6What is error analysis?1
    1. AStudying the model's mistakes to decide what to improve
    2. BCounting how many errors the code has
    3. CRemoving all the wrong predictions from the results
    4. DAdding errors to the training data
  7. 7Which belongs in a model card?1
    1. AYour passwords
    2. BEvery training message in full
    3. COnly the best result you achieved
    4. DThe model's limitations and risks
  8. 8What does "end to end" mean when building a first model?1
    1. AThe model is trained on all the data at once
    2. BEvery part is as accurate as possible
    3. CA simple version of the whole system runs from input to output
    4. DThe project is finished
  9. 9What does an ablation study measure?1
    1. AHow large the dataset is
    2. BHow much each part contributes, by removing it
    3. CHow many users the model has
    4. DHow fast the model trains
  10. 10Which question belongs in a responsible AI check?1
    1. AWhich programming language is fastest?
    2. BWhich colour should the website be?
    3. CHow many lines of code does it have?
    4. DDoes it work equally well for different groups of people?

Section B

Answer in the space provided. The number of marks tells you how many separate points to make.

  1. 1DescribePlan a capstone project that builds a model to sort customer emails into "complaint", "question" and "praise". Describe the main stages.6
  2. 2ExplainExplain why error analysis and a model card make an AI project more trustworthy.4

Mark scheme

Section A

1B2A3C4C5B6A7D8C9B10D

Section B

1 Plan a capstone project that builds a model to sort customer emails into "complaint", "question" and "praise". Describe the main stages. 6

One mark for each of the following, to a maximum of 6.

  • Define the problem, users and a success measure (e.g. accuracy/F1 per class, compared with a baseline)
  • Collect and label example emails, checking label quality
  • Split into training, validation and test sets
  • Build a baseline (e.g. most common class or keyword rules)
  • Train a model (e.g. bag of words with logistic regression) end to end
  • Evaluate with a confusion matrix and analyse errors; improve
  • Responsible check and model card; present

Model answer. First I would define the problem: sorting emails so staff can answer complaints quickly, measured by the precision and recall for each class compared with a baseline that always predicts the most common class. Next I would collect several hundred real or realistic emails, label them carefully, and split them into training, validation and test sets. I would build the baseline, then a simple model, such as logistic regression on a bag of words, running end to end. I would evaluate it on the validation set with a confusion matrix, study the misclassified emails to decide what to improve, and change one thing at a time. Finally I would check privacy, since emails contain personal data, test the final model once on the test set, write a model card listing its limitations, and present it.

2 Explain why error analysis and a model card make an AI project more trustworthy. 4

One mark for each of the following, to a maximum of 4.

  • Error analysis reveals patterns in the model's mistakes
  • It shows what to improve, e.g. more varied data rather than a bigger model
  • A model card documents purpose, data, performance and limitations
  • So users know when not to rely on it and can use it safely / honesty builds trust

Model answer. Error analysis looks closely at the model's actual mistakes, revealing patterns such as missing spam that uses words never seen in training, which shows what would really improve it, more varied data rather than a bigger model, and shows the builder understands its behaviour. A model card records the model's purpose, what it must not be used for, its data, its performance overall and for different groups, and its limitations and risks. Together they let users know when the model can be relied on and when it cannot, which makes it safer to use, and honesty about weaknesses makes the whole project more credible.