Data Analytics F201 Paper A: Big Data and Machine Learning

Volume, velocity and variety.

Vehicle telemetry arrives continuously rather than as a file that is complete at a known moment, so there is no natural point at which a batch could run over a finished set. A batch process scheduled overnight means every answer is as old as the last run, and for this data that is the whole problem rather than an inconvenience: an engine fault pattern, a refrigeration unit losing temperature or a vehicle leaving its route are all worth knowing within seconds and worth nothing the following morning. The company needs stream processing, where each event is assessed as it arrives and an alert is raised while somebody can still act on it.

Scaling up means replacing a machine with a more powerful one: more processors, more memory, more storage in the same box. Scaling out means adding more machines that work together as a cluster, with the data partitioned across them. Scaling up is simpler and it stops, because there is a largest machine available and no budget moves that ceiling, which is precisely where big data begins. The cost of scaling out is consistency: with data replicated across nodes, different replicas can briefly disagree after a write, so many distributed stores offer only eventual consistency, and every application has to be examined to see whether it can tolerate that. A vehicle's live position can; a fuel card balance cannot.

The growth is accelerating rather than steady: the total roughly doubles each year, from 12 terabytes in 2021 to 19, 34, 68, 141 and 310, so the annual increase itself is growing, which on a linear axis gives the characteristic upward curve shown. Continuing that pattern gives an estimate of roughly 600 to 650 terabytes for 2027. The caution is that this is an extrapolation of a pattern with no stated cause: it assumes the same doubling continues, and growth of this kind usually stops for an ordinary reason, a retention policy, a budget ceiling, a change to what is logged, or simply the end of whatever rollout drove it. The estimate is useful for planning capacity headroom, not as a forecast to commit money against.

Instead of pulling a dataset to a machine that will analyse it, the query itself is sent to the machines that already hold the partitions, each one processes its own share locally and in parallel, and only the small partial results travel back to be combined. It matters at this scale because data has weight: moving hundreds of terabytes across a network takes days or weeks, so the transfer, rather than the processing, becomes the limit. Sending a query that is a few kilobytes and receiving a result that is a few kilobytes avoids that entirely, which is the organising idea behind every distributed processing framework.

The technique is clustering, and it is unsupervised learning, because no labels were supplied: nobody told the algorithm which customers belonged in which group, so the structure was discovered in the data rather than predicted against a known answer. The plot shows three clearly separated groups. One has few visits and a low average basket, which is the occasional small customer. One has many visits but still a low to moderate basket, which is the frequent top-up customer. One has a moderate number of visits with a high basket value, which is the smaller group that is probably worth the most per customer. The separation between them is wide enough that the groups are not an artefact of where the boundaries were drawn.

Precision is true positives divided by everything predicted positive: 96 / (96 + 304) = 96 / 400 = 0.24, which is 24%. Recall is true positives divided by everything that actually was positive: 96 / (96 + 24) = 96 / 120 = 0.80, which is 80%.

A recall of 80% means the model finds four fifths of the genuine fraud, which is substantial, though the 24 cases it missed are real losses and the company should know what they have in common. The precision of 24% is the figure that shapes how the model can actually be used: of every hundred transactions it flags, roughly seventy six are perfectly genuine, so the 400 alerts contain 304 false alarms. If each alert blocks a card, the company annoys three honest customers for every fraud it stops; if each is reviewed by a person instead, that is 400 reviews to find 96 cases. Neither is fatal, but it means the model belongs in a review queue rather than wired directly to an automatic block, and raising precision should be the next piece of work.

The model is overfitting. The training error falls continuously across all twelve epochs, from 0.62 down to 0.04, which looks like steady improvement. The validation error tells a different story: it falls to a minimum of about 0.28 at around epoch six and then turns and climbs steadily to 0.53, worse than it was at epoch three. From that turning point the model is no longer learning the pattern, it is learning the particular noise of the training set, and its performance on anything it has not seen is getting worse while its apparent performance improves. Two remedies: use early stopping, halting training at the epoch where validation error bottomed out and keeping that version rather than the final one; and reduce the model's capacity or train on more data, since a simpler model has less room to fit noise and more examples dilute it.

Historic bias, where the training data records past unfair human decisions, and sampling bias, where a group is under-represented in the training data.

Removing a protected characteristic removes the label, not the information. Other fields in the data correlate with it: a postcode, a depot, years of service, which licence categories somebody holds or what shifts they work can all stand in for age or nationality closely enough for the model to reproduce the same pattern without ever seeing the attribute. The result is a model that discriminates in effect while appearing neutral on inspection, and under the Equality Act 2010 it is the outcome that matters rather than the intent or the feature list. The only way to know is to measure: compare the model's error and selection rates for each group separately, and treat a gap between them as a defect rather than as a statistic.

PUE is total facility power divided by the power delivered to the IT equipment. Site A: 2,400 / 1,500 = 1.6, so 60% is spent on cooling and losses on top of the computing itself. Site B: 1,980 / 1,800 = 1.1, which is close to best practice. Two changes for Site A: raise the inlet temperature towards Site B's 27 degrees, since modern equipment is rated for it and cooling to 18 degrees is where most of that 60% overhead is going; and deal with the 310 idle servers out of 900, by consolidating those workloads through virtualisation and decommissioning the hardware, which removes power drawn for no work at all. Beyond those two, moving from a 20% to a higher renewable share would cut emissions without cutting consumption, and extending the three year refresh cycle would reduce the embodied carbon and e-waste.

The infrastructure problem is one of rate rather than size. Every vehicle reporting each second produces a continuous stream that no overnight batch can serve, and the volumes compound quickly: the growth chart already shows the company's holdings doubling annually before this expansion. The design should therefore push work to the edge. A unit in each vehicle and warehouse applies threshold rules locally, so a fault raises an alert within milliseconds and keeps doing so when the vehicle is out of coverage, and aggregates the rest into minute level summaries with the mean, minimum, maximum and spread. Full resolution data is buffered and uploaded only around an event, which cuts the transmitted volume by a large factor and matters particularly for vehicles on mobile data, where bandwidth is both limited and metered. Centrally, the summaries belong in a time series store built for timestamped measurements, with the raw event windows in object storage as a lake for model training; alerts run as stream processing because their value decays in seconds, while model retraining runs as batch because it does not. Capacity planning should assume the growth continues, and a retention policy should be written at the start rather than after the bill arrives, since storing everything forever is both an expense and, under storage limitation, a compliance problem. The ethical issues are at least as significant and are easier to get wrong. Telemetry that monitors a vehicle also monitors the person driving it: speed, braking, breaks taken, time at each stop. That is personal data about an identifiable employee, so the company needs a lawful basis, a clear policy saying exactly what is collected and what it will and will not be used for, and it should consult drivers rather than announce it. The difference between a system that predicts a failing alternator and one that scores drivers is a decision about how the same data is used, and if drivers believe it is the second they will work to the measure rather than to the job. Model limitations need stating honestly too: a breakdown predictor will raise false alarms, which cost a wasted inspection, and miss failures, which cost a roadside recovery, so the threshold is a commercial judgement and the output should prompt a maintenance decision rather than automatically take a vehicle off the road. Performance will also drift as the fleet ages and vehicles are replaced, so it must be monitored and the model retrained. Of all of this, the governance of driver data is what the company should get right first. The technical problems are well understood and have standard answers; the trust of the people being measured is the part that cannot be bought back once it is lost, and a workforce that resents the system will find ways to defeat it long before the engineering becomes the limiting factor.