Data Analytics F201 Paper C: Big Data and Machine Learning

Veracity is how far the data can be trusted: how much noise, error, duplication and bias it contains. On a streaming platform a clear example is a title left playing to an empty room, or on a shared household account where one profile is used by three people, both of which are recorded as a genuine view by a single person and are not.

The video and audio files themselves, and the free text of subscriber reviews and support messages.

Batch processing handles a large volume of accumulated data on a schedule, which suits work whose answer is not needed immediately: retraining the recommendation model overnight, or calculating the month's royalty payments. Stream processing handles each event as it arrives, which suits work whose value decays within seconds: detecting that playback quality has collapsed in one region and rerouting traffic, or spotting that one account is streaming from six countries at once. The distinction is not about volume but about how quickly the answer stops being useful.

The growth is accelerating, not steady: the total roughly doubles each year, from 12 terabytes in 2021 to 19, 34, 68, 141 and 310 in 2026, so each year adds more than all previous years combined did a short time ago. The consequence is that planning cannot be done by scaling up. A single machine large enough for last year is not large enough for next year, so the platform must scale out onto a cluster with the data partitioned and replicated, and accept the consistency trade that comes with it. The second consequence is that migration becomes impractical: moving 300 terabytes between providers over any ordinary link takes weeks and, in the cloud, incurs egress charges that turn an architectural decision into a financial one. Choices made now are harder to reverse each year.

Eventual consistency means that after a write, different replicas of the data may briefly disagree, and the guarantee is only that they will converge in time rather than that every reader sees the change immediately. On this platform it is perfectly acceptable for a view count or a trending list: if one region shows a figure a few seconds behind another, nothing is harmed and the alternative, coordinating every replica before acknowledging a play, would make the service slower for everyone. It is not acceptable for the subscription and billing state. If a cancellation reaches one replica and not another, the subscriber is charged after cancelling or keeps access after their final payment, and for parental controls the consequence is worse: a restriction set by a parent must take effect everywhere at once, not shortly.

Precision is TP / (TP + FP) = 210 / (210 + 90) = 210 / 300 = 0.70, which is 70%. Recall is TP / (TP + FN) = 210 / (210 + 140) = 210 / 350 = 0.60, which is 60%. The platform should favour recall. A false negative is a subscriber who leaves without ever being offered anything, and the lifetime value of a lost subscriber is large. A false positive is a subscriber who was staying anyway and receives a retention offer, which costs only the margin on that offer and is not unwelcome. Since the cost of missing a leaver is far higher than the cost of a wasted discount, the threshold should be loosened to catch more of the 140 currently missed, accepting that precision will fall as a result.

In reinforcement learning a system takes actions in an environment and receives a reward or penalty depending on what happens, then adjusts its policy to maximise reward over the long run rather than at each individual step. There is no labelled dataset: the data is generated by acting. The platform could use it to choose which artwork to show for a title, trying variants and learning from whether each viewer went on to watch, or to set streaming bitrate dynamically, where the reward balances picture quality against the penalty of a buffering event. Both suit reinforcement learning because the right answer is not known in advance and the consequences arrive after the decision.

A system trained to maximise engagement learns to recommend whatever is most like the things the viewer already watched to the end, because that is what the reward signal rewards. Each round narrows the range offered, and because the viewer can only engage with what they are shown, the system's own output becomes the evidence for its next recommendation. That feedback loop compounds: a mild early preference becomes the whole of what the platform shows. The effect on the viewer is that the catalogue appears to be much smaller and much more uniform than it is, and they have no way of knowing what was filtered out. On contested subjects the same mechanism narrows what people believe is being said, which is why engagement optimisation is treated as a societal issue and not only a product one.

Free air or liquid cooling rather than refrigeration, which attacks the largest overhead in most facilities. Virtualisation and consolidation so that fewer physical servers run and fewer sit idle. Buying or generating renewable power, and scheduling flexible workloads for when and where the grid is cleanest. Deleting data that serves no purpose and extending the hardware refresh cycle with repair and certified recycling.

PUE is total facility power divided by the power delivered to the IT equipment, so Site A is 2,400 / 1,500 = 1.6. That means 60 per cent again is spent on cooling and distribution losses on top of the computing itself. Site B is 1,980 / 1,800 = 1.1, close to current best practice. Two reasons for the difference. Site B runs at a mean inlet temperature of 27 degrees against Site A's 18, and cooling to the lower figure requires mechanical refrigeration for most of the year, which is where the bulk of Site A's overhead goes, whereas at 27 degrees much of the cooling can be done with outside air. Site B also has 40 idle servers out of 900 against Site A's 310, so Site A is drawing power for more than a third of its fleet doing no work at all, which inflates its IT load and the cooling needed to remove that load's heat. Site B's longer hardware refresh cycle and far higher renewable share do not affect PUE directly but reduce its embodied carbon and emissions substantially.

The rebound effect is the tendency for an efficiency gain to be partly or wholly cancelled out by increased use, because the gain makes the activity cheaper. In computing it is well documented: data centre efficiency has improved markedly over the last fifteen years, with typical PUE falling from near 2.0 towards 1.1, and total data centre energy consumption has nevertheless kept rising, because cheaper computing is used for more things. The same pattern appears inside a product: more efficient video codecs were spent on higher resolutions rather than on lower bandwidth. The practical lesson is that efficiency alone is not a strategy, because it reduces the cost per unit of work rather than the amount of work; reducing the work itself, by deleting what is not needed and not running what need not run, is the part that actually shows up in the total.

The first is its effect on attention and wellbeing. Systems that maximise time spent learn the techniques that do so, autoplay, countdowns, episode-length pacing, and the measurable consequences, particularly for younger viewers, are later bedtimes and shorter sleep; the platform is not trying to produce that outcome, which is precisely the problem, because the optimisation target produces it without anyone choosing it. The second is cultural and economic. At this scale the platform's commissioning decisions shape what gets made: data about what is finished and what is abandoned feeds back into which programmes are funded, which favours formats that perform well on the metrics it collects and disadvantages work that is slow, regional or minority-language. That is a genuine shift in who decides what a culture produces, and it is made by an optimisation process rather than by an editor anyone can name.

The commercial argument is straightforward and genuine. Time spent correlates with retention, retention is what a subscription business is, and a recommendation system that surfaces something a subscriber actually wants saves them the irritation of searching a catalogue of thousands of titles. For most viewers most of the time, better recommendations are a better product, and the board is not wrong to want them. The arguments against are about what happens when that particular measure is pushed hard. Optimising for time spent narrows what is shown, because the safest way to keep someone watching is to offer more of what they already finished, and since they can only engage with what they are offered, the system's own output becomes the evidence for its next decision. That feedback loop compounds quietly: the catalogue appears to shrink, discovery stops, and nobody can see what was filtered out. The wellbeing question is sharper still, because the techniques that raise time spent, autoplay, countdowns, cliffhanger pacing, work by making stopping harder rather than by making watching better, and the measurable effects on sleep, particularly for younger viewers, are well established. There is a cultural cost too, since commissioning follows the metrics, and work that is slow, regional or in a minority language performs poorly on time spent regardless of its value. And there is regulatory exposure: profiling that shapes what children see attracts specific obligations, and a platform that cannot explain why it showed what it showed is in a weak position when asked. Underneath all of these is a measurement problem the board should name explicitly. Time spent is a proxy for value, and it is a poor one: a subscriber who watched four hours and regretted it, and one who watched forty minutes and cancelled nothing, look identical to the metric, and only one of them renews. My recommendation is that the board should not increase the weight on time spent. It should instead change what is optimised: use long-run subscriber value, measured by renewal and by direct satisfaction responses, as the objective, keep a deliberate proportion of recommendations outside the viewer's established pattern to preserve discovery, cap the autoplay mechanisms that work by removing a decision rather than informing one, and apply stricter defaults to profiles registered to under-18s. That answers the commercial case, because renewal is what actually produces revenue, while removing the mechanism that generates the wellbeing, cultural and regulatory risks. Raising the weight on time spent would buy a measurable short-term gain against a metric the board has already acknowledged is not the thing it cares about.