F201 TA1: The Scope of Managing Big Data
What makes data big, where it comes from, and the effect it has on organisations, individuals and whole sectors.
What makes data big
Big data is not simply a large file. The usual definition is a set of characteristics, the Vs, and the point of them is that each one breaks a different conventional tool.
| Characteristic | What it means | What it breaks |
|---|---|---|
| Volume | Terabytes to petabytes, growing continuously | Single machine storage and processing |
| Velocity | Arriving continuously, needing a fast response | Batch processing on a nightly schedule |
| Variety | Structured, semi-structured and unstructured together | A fixed relational schema |
| Veracity | Uncertain accuracy, gaps, noise, duplicates, bias | The assumption that stored data is correct |
| Value | The business worth that has to be extracted | The assumption that collecting is the same as benefiting |
Variability and visualisation are sometimes added. The examinable point is that any one of these alone can make a dataset unmanageable with ordinary tools, which is the working definition of big data: data that cannot be stored, processed or analysed with conventional single-machine methods.
Volume, in proportion
| Unit | Scale |
|---|---|
| Kilobyte | A short document |
| Megabyte | A photograph |
| Gigabyte | A film |
| Terabyte | A desktop hard drive |
| Petabyte | A thousand terabytes: a large organisation's data estate |
| Exabyte | A thousand petabytes: a global platform |
Not "a lot of data", but "data whose volume, velocity or variety makes it impossible to process with conventional tools on a single machine, so distributed storage and processing are required."
Where big data comes from
| Source | What it produces | Characteristic it stresses |
|---|---|---|
| Transactional systems | Every order, payment, booking, click to buy | Volume |
| Web and app telemetry | Page views, taps, scroll depth, session paths | Volume and velocity |
| Social media | Posts, reactions, comments, images, networks | Variety and veracity |
| Sensors and IoT | Temperature, vibration, location, energy use | Velocity and volume |
| Machine and system logs | Every request, error and state change | Volume |
| Media | Images, audio, video, CCTV | Volume and variety |
| Mobile devices | Location, movement, app use | Velocity and privacy risk |
| Open data | Census, weather, transport, health, geography | Value through combination |
| Third party and purchased data | Market, credit, demographic datasets | Veracity and licensing |
Why combining sources is where the value is
Each source answers a narrow question. Joined, they answer a different kind of question entirely: weather data plus transaction data explains a sales dip; transport data plus footfall data decides where a shop goes; sensor data plus maintenance records predicts a failure before it happens. The combination is also where the privacy risk concentrates, because joining datasets is exactly what makes individuals re-identifiable.
- At the bottom, the sources: every system that generates data.
- The lake holds it raw, in whatever shape it arrived.
- Transformation cleans, types, joins and deduplicates it.
- The warehouse holds modelled data, ready to be queried.
- And only then can a dashboard be trusted by the people reading it.
The impacts of big data
On organisations
| Impact | Detail |
|---|---|
| Better decisions | Evidence instead of instinct, and measurement of whether a change worked |
| Personalisation | Recommendations, pricing and content tailored per customer |
| Operational efficiency | Routing, stock, staffing and energy optimised against real demand |
| Predictive maintenance | Replacing a part before it fails rather than after |
| New products | Data itself becoming the product, sold as insight or as a service |
| Risk and fraud detection | Patterns across millions of transactions that no reviewer could see |
| Cost | Storage, compute, specialist staff and governance, all ongoing |
| Skills shortage | The scarce resource is people who can use it, not the data |
| Risk concentration | A single breach now exposes far more than it would have done |
On individuals
The benefits are real: services that work better, prices that reflect demand, health research at a scale that was impossible, free products funded by advertising. So are the costs: surveillance as a default condition, profiling that affects credit and insurance, filter bubbles, price discrimination, and decisions made about a person by a system that cannot explain itself.
On society and sectors
- Healthcare — population screening, drug discovery, outbreak detection
- Transport — live routing, congestion charging, autonomous vehicles
- Retail — demand forecasting, dynamic pricing, supply chains that react in hours
- Finance — credit scoring, fraud detection, algorithmic trading
- Energy — smart grids balancing supply and demand minute by minute
- Public services — targeting support, planning capacity, detecting fraud
- Employment — some roles automated away, others created, and a persistent shortage of the new ones
Most organisations store far more than they analyse. Value is a characteristic that has to be extracted deliberately, and the storage bill arrives whether or not anybody ever asks the data a question.
- The benefits are real: better decisions, personalisation, efficiency, predictive maintenance and fraud detection.
- So are the costs and risks: ongoing cost, scarce skills, concentrated risk, surveillance and bias.
- Every sector uses big data in its own way.
Key terms you must be able to define
- Big data
- Data whose volume, velocity or variety prevents processing by conventional single-machine methods.
- Volume
- The sheer quantity of data, typically terabytes to petabytes.
- Velocity
- The speed at which data arrives and must be processed.
- Variety
- The range of data types and structures held together.
- Veracity
- The uncertainty, noise and bias present in the data.
- Value
- The business worth that has to be deliberately extracted from data.
- IoT
- The Internet of Things: networked sensors and devices generating continuous data.
- Telemetry
- Automatically captured measurements of how a system or product is used.
- Predictive maintenance
- Using sensor data to replace a component before it fails.
- Personalisation
- Tailoring content, recommendations or pricing to an individual.
- Petabyte
- A thousand terabytes, roughly a million gigabytes.
Flashcards
Say the definition out loud before you flip the card. Retrieving it from memory is what builds recall, reading it again is not.
Reach the top grade on this topic
Define big data by what it breaks, not by how large it is, and then tie each characteristic to a consequence. "High velocity means the data has to be processed as a stream, because by the time a nightly batch ran the fraudulent transaction would already have been settled" is a full mark sentence.
Mistakes that cost marks here
- Defining big data as 'a lot of data' with no reference to the limits of conventional processing.
- Naming the Vs without saying what each one causes.
- Confusing volume with velocity.
- Treating veracity as a minor issue. Big data is usually dirtier than small data, not cleaner.
- Assuming more data automatically means better decisions.
- Listing benefits to organisations and ignoring the effects on individuals.
A single modern jet engine generates more data on one flight than the entire airline industry produced in a year in the 1970s. Almost none of it is ever looked at by a human; it exists so that a model can notice a bearing starting to fail.
Knowledge check
Exam-style questions
Five written questions in the style of the real paper. Write a full answer, then let the marker check it against the mark scheme.
- LOne valid characteristic
- LA second, different characteristic
- LA third, different characteristic
Volume, velocity and variety.
- LVelocity means data arrives continuously
- LBatch processing runs on a schedule, so results are late
- LStream processing handles each event as it arrives
- LA consequence or example where the delay matters
High velocity means the data arrives continuously rather than as a file at a fixed time, so there is no natural point at which a complete set exists to be processed. Traditional batch processing runs on a schedule, typically overnight, which means every result is as old as the last run; for data arriving every second that delay is the whole problem rather than an inconvenience. Stream processing handles each event as it arrives, applying rules or a model in flight and acting within milliseconds. The difference matters wherever the value of the answer decays: a fraudulent card transaction has already been authorised and settled by the time a nightly batch notices it, and a vibration pattern indicating a bearing failure is useful before the bearing fails, not the next morning.
- LA benefit relating to demand forecasting or stock
- LIt is explained with the mechanism
- LA benefit relating to personalisation or marketing
- LIt is explained with the mechanism
- LA third benefit, such as pricing, layout or efficiency
- LIt is explained with the mechanism
- LA genuine risk is identified
- LThe risk is explained and a mitigation or judgement offered
The first benefit is demand forecasting. Joining till data to weather feeds lets the chain predict, store by store, that a forecast of 24 degrees on a Friday raises barbecue and salad sales by a known proportion, so orders are placed before demand arrives rather than after; that cuts both empty shelves and waste on short-life products, which is where a supermarket's margin actually sits. The second is personalisation. Loyalty data shows what each household buys and how often, so offers can be targeted at products a customer already buys or is likely to switch to, which gives a far higher redemption rate than a uniform discount and costs less in lost margin. The third is operational: combining transaction data with supply chain data identifies which lines sell in which stores at which times, which informs range decisions, shelf layout and staffing, and lets distribution be planned against real rather than assumed demand. The risk that comes with all of this is privacy, and it is concentrated precisely where the value is. Individually, a weather feed and a basket are innocuous; joined to a loyalty card they produce a detailed and sensitive record of a household, capable of revealing pregnancy, illness, religious observance and financial difficulty as inferences the customer never disclosed. That creates both a legal exposure, since inferred health information is special category data and the processing must be transparent and minimised, and a trust exposure, because customers who feel watched stop using the card and the data disappears. The chain should therefore be explicit in its privacy notice about what it infers and why, hold the minimum it needs with a real retention period, keep identified data tightly access controlled, and avoid using inferences in ways a customer would find intrusive even where doing so would be lawful.