F201 TA4: Legal and Ethical Issues in Data Management
What changes legally and ethically when the dataset is enormous and a model is making the decisions: bias, transparency, profiling and accountability.
Data protection at big data scale
The law is the same law as in F200. What changes is that big data practice pushes against the principles rather than fitting neatly inside them.
| Principle | Where big data strains it |
|---|---|
| Purpose limitation | The whole premise of a data lake is keeping data for purposes not yet identified |
| Data minimisation | Models improve with more features, which pulls directly against collecting less |
| Storage limitation | Training sets are kept to allow reproducibility and retraining |
| Transparency | It is hard to tell somebody what you will infer when you do not know yet |
| Accuracy | A prediction about a person is not a fact about them, yet it is stored like one |
| Accountability | A derived model is harder to audit than a query |
Inference is processing
If a model infers that a customer is probably pregnant, has a health condition or holds a religious belief, the organisation is processing special category data even though nobody entered it and the customer never disclosed it. That the information is inferred rather than collected makes no difference to the duties attached to it, and this is the single point in this topic area most often missed.
Automated decision making and profiling
Under the UK GDPR a person has the right not to be subject to a solely automated decision with legal or similarly significant effects, such as a credit refusal, an insurance price or a job rejection, unless a specific condition applies. Where such processing is permitted, the safeguards are meaningful human involvement, an explanation of the logic, and a route to contest the outcome. "The system decided" is not a defence.
Consent at scale
Consent must be freely given, specific and informed, and withdrawable. None of that fits a model trained on everybody's data at once, because a specific consent cannot be given to an unspecified future inference, and withdrawal is almost impossible to honour once the data has been absorbed into a trained model. This is why serious organisations rely on a documented legitimate interests assessment, with a real balancing test, rather than on a consent they could not evidence.
Bias, fairness and transparency
Where bias actually comes from
| Source | How it happens |
|---|---|
| Historic bias | The training data records past human decisions, including unfair ones |
| Sampling bias | A group is under-represented, so the model performs worse for them |
| Measurement bias | The chosen proxy is not the thing you care about: arrests are not crime, clicks are not interest |
| Label bias | The labels were applied by people with their own assumptions |
| Aggregation bias | One model applied to groups that genuinely behave differently |
| Deployment bias | A model used for a purpose or population it was never trained for |
| Feedback loops | The model's own output becomes tomorrow's training data and the bias compounds |
Removing the protected characteristic from the features does not remove the bias, because other fields act as proxies for it: postcode, school, first name, purchase history. A model can discriminate effectively without ever seeing the attribute, and under the Equality Act the outcome is what matters, not the intent or the feature list.
Testing for it
Measure performance per group, not just overall. If recall is 88% for one group and 61% for another, the headline accuracy has concealed a system that works for some people and not others. Then decide what fairness means for this application, because the definitions conflict mathematically and you cannot have all of them at once: equal error rates, equal selection rates, or equal treatment of similar individuals.
Transparency and explainability
- Explainability: can the factors behind an individual decision be stated?
- Interpretability: can a person understand how the model works overall?
- The black box problem: the most accurate models are often the least inspectable, which is a genuine trade-off, not a technical gap waiting to be closed
- The right to an explanation: where an automated decision significantly affects someone, they are entitled to meaningful information about the logic involved
A slightly less accurate model whose reasoning can be explained and audited is frequently the better choice, because a decision that cannot be explained cannot be defended, corrected or appealed.
- The training data is unbalanced: 18 apples and only 2 pears.
- The model copies the patterns it finds in its data.
- So it is right 98% of the time for apples but only 40% for pears.
Ownership, security and accountability
Who owns data
- Personal data is not owned by the organisation holding it; the individual has rights over it regardless
- Database right protects a database that required substantial investment to assemble, which is what makes scraping someone else's dataset an infringement
- Derived data and models trained on other people's data raise ownership questions the law is still settling
- Contracts usually decide ownership of data generated by a product in use, and customers rarely read that clause
- Open data licences permit reuse on stated conditions, and those conditions are enforceable
Security at scale
The impact of a breach scales with what you kept. The specific big data risks:
- Concentration: one lake holds what used to be in twenty systems, so one failure exposes everything
- Re-identification: datasets individually anonymous become identifying when joined
- Misconfiguration: the public storage bucket, which causes more exposure than any attack
- Insider access: analysts with broad access to the whole estate
- Supply chain: a compromised vendor or library reaching your pipeline
- Model attacks: extracting training data from a model, or poisoning the training set to change its behaviour
Surveillance, autonomy and manipulation
- Workplace monitoring: keystrokes, location, productivity scores, and the question of proportionality
- Behavioural advertising: profiles built across sites and apps, mostly invisible to the person profiled
- Dark patterns: interfaces designed so that the privacy-protecting choice is the hard one
- Price discrimination: different prices for different people, from inferred willingness to pay
- Filter bubbles: optimisation for engagement narrowing what a person is shown
- Chilling effects: people behaving differently because they believe they are observed
Accountability
Somebody has to be answerable. In practice that means: a named owner for each model and dataset; a documented impact assessment before deployment; a human in the loop for consequential decisions; monitoring for drift and for unequal outcomes; an appeal route for the person affected; and an audit trail showing what the model did and on what data.
Identify the specific harm and who bears it. Name the law or principle engaged. Weigh the benefit honestly rather than dismissing it. Reach a conclusion with a safeguard attached. A balanced answer that decides outscores a one-sided one every time.
Step 1 of 4 The raw record: obviously identifying, and obviously sensitive.
Key terms you must be able to define
- Inferred data
- Information a model derives about a person, which is personal data even though it was never disclosed.
- Profiling
- Automated processing to evaluate or predict aspects of a person's behaviour or characteristics.
- Automated decision making
- A decision made by a system without meaningful human involvement.
- Algorithmic bias
- Systematic unfairness in a model's outputs, usually inherited from its training data.
- Proxy variable
- A field that stands in for a protected characteristic, such as postcode for ethnicity.
- Explainability
- The ability to state the factors behind an individual automated decision.
- Black box model
- A model whose internal reasoning cannot practically be inspected.
- Re-identification
- Linking anonymised data back to individuals, often by joining datasets.
- Database right
- Protection for a database whose contents required substantial investment to assemble.
- Dark pattern
- An interface designed to steer users away from the choice that protects them.
- Data poisoning
- Deliberately corrupting training data so a model behaves as the attacker intends.
- Model drift
- The decline in a model's performance as the world it was trained on changes.
Flashcards
Say the definition out loud before you flip the card. Retrieving it from memory is what builds recall, reading it again is not.
Reach the top grade on this topic
Top band answers are balanced and decisive. State the benefit, state the harm, name the law or principle, then conclude with a safeguard. Answers that only condemn, or only enthuse, stay in the middle bands however much detail they contain.
Mistakes that cost marks here
- Thinking inferred information escapes data protection law. Inference is processing.
- Claiming that removing the protected characteristic removes the bias. Proxies remain.
- Reporting only overall accuracy, which hides unequal performance between groups.
- Treating 'the algorithm decided' as an explanation or a defence.
- Assuming aggregated data is automatically anonymous.
- Writing a one-sided answer. The marks are for weighing both sides and concluding.
A recruitment model trained on a decade of a company's own hiring decisions learned to downgrade applications that mentioned women's colleges. Nobody programmed that. It was in the historic data, and the model treated the past as a specification.
Knowledge check
Exam-style questions
Five written questions in the style of the real paper. Write a full answer, then let the marker check it against the mark scheme.
- LOne valid source
- LA second, different source
Historic bias, where the training data records past unfair human decisions, and sampling bias, where a group is under-represented in the training data.
- LInferred information relates to an identifiable person
- LIt therefore meets the definition of personal data
- LInferring it is itself processing
- LA consequence, such as special category duties or rights applying
Personal data is any information relating to an identified or identifiable living person, and that definition does not depend on how the information arose. A model's inference that a particular customer is likely to be pregnant, in financial difficulty or of a given religious belief relates directly to that individual, so it is personal data in exactly the way a declared field would be. Generating the inference is itself processing, which needs a lawful basis and must be transparent. The consequences are substantial: where the inference concerns health, religion or sexual orientation it is special category data requiring an additional condition, the individual can request access to it and dispute its accuracy, and because a prediction is not a fact the organisation must be careful not to present it as one.
- LPurpose limitation and the original collection purpose are addressed
- LTransparency is addressed
- LSpecial category data by inference is addressed
- LAutomated decision making rights are addressed
- LDiscrimination or proxy variables are addressed
- LAccuracy and the harm of a wrong inference are addressed
- LSpecific safeguards are proposed
- LA judgement weighing the insurer's interest against the harm
The insurer has a genuine interest here, since pricing risk accurately is what insurance is, and a model that prices more precisely can lower premiums for lower-risk customers. But the proposal has serious problems. The data was not collected for this: posts and purchases were made for social and commercial purposes, and reusing them to price insurance is a different and incompatible purpose, which engages purpose limitation and means the insurer cannot simply help itself to data that happens to be accessible. Transparency compounds it, because customers would have no realistic idea that their shopping history was affecting their premium, and a privacy notice that technically mentions it does not make the processing fair. The inferences themselves are the legal centre of the problem: shopping and social data are highly revealing, and a model will infer pregnancy, mental health, alcohol consumption and religious observance whether or not it is asked to, which means the insurer would be processing special category data and needs a specific condition for doing so. Because the outcome, a price or a refusal, has a significant effect on the individual, the rules on solely automated decision making apply, so there must be meaningful human involvement, an explanation of the logic and a route to contest the result. Discrimination is the ethical core. Removing protected characteristics from the model does not help, because postcode, store, brand and vocabulary are proxies, and under the Equality Act it is the outcome that matters: a model that prices higher for an ethnic group or a disabled customer is unlawful however innocent the feature list looks. Accuracy makes it worse rather than better, because a wrong inference, a gift bought for someone else, a parody account, produces a real financial penalty the customer cannot see, understand or appeal. The safeguards would have to be substantial: a Data Protection Impact Assessment before anything is built; the feature set cut to data the customer knowingly provided for insurance purposes; explicit exclusion of inferred special category data; testing of error and pricing rates per protected group with unequal outcomes treated as a defect; human review and a published appeal route for every adverse decision; and an audit trail allowing any individual price to be reconstructed and explained. On balance, the social media element should not proceed at all, because the purpose mismatch and the unavoidable inference of sensitive characteristics cannot be safeguarded away, while a narrow use of data the customer supplies for the policy itself can be defended provided the safeguards above are real rather than documentary.