F201 TA2: The Infrastructure Challenges of Big Data

Scaling up means increasing the capacity of a single machine, with more processors, memory or storage. Scaling out means adding more machines that work together as a cluster.

A data lake accepts data in whatever structure it arrives in, including unstructured content such as images, audio and free text that a relational warehouse cannot hold at all. It applies structure when the data is read rather than when it is written, so nothing has to fit a schema at the point of arrival and nothing is rejected or reshaped before anyone knows what questions will be asked of it. That matters for big data because variety is one of its defining characteristics and the useful questions are often not known in advance. The cost is governance: because the lake guarantees nothing about quality, structure or meaning, an undocumented lake becomes a swamp that is expensive to store and that nobody trusts enough to use, so a catalogue, lineage and retention policy are not optional extras.

Four thousand sensors at one reading a second is 4,000 values every second, around 345 million a day, and a limited connection will not carry that continuously, so the design has to reduce the data before it leaves the site. Processing should therefore start at the edge: a local gateway ingests every reading, applies threshold rules immediately so an out of range vibration raises an alert on site within milliseconds, and aggregates the rest into, say, one-minute summaries holding the mean, minimum, maximum and standard deviation per sensor. That cuts the uploaded volume by roughly sixty times while keeping the statistical shape of the signal, and it means the line keeps being monitored when the connection drops, which a cloud-only design would not. Full resolution readings should be buffered locally for a short window and uploaded only around an event, because that is when per-second detail is actually needed for diagnosis. In the cloud the summaries belong in a time series store, which is built for timestamped measurements and compresses them far better than a relational table, with the raw event windows in object storage as a lake for later model training. Alerting should be stream processed rather than batched, because the entire value of the data is in acting before a component fails, and a nightly batch would discover the failure after it had already stopped the line.

The first challenge is that the familiar answer stops working. A single server is improved by scaling up, and big data is by definition the point at which no affordable single machine is large enough, so storage has to become a cluster with the data partitioned across nodes for parallel processing and replicated so that a lost machine loses nothing. That immediately introduces a cost the organisation has not had to think about before: replicas can briefly disagree, so the immediate consistency a relational database guaranteed is replaced in many stores by eventual consistency, and every application has to be examined to see whether it can tolerate that. A balance cannot; a dashboard count can. The second challenge is movement. Data has weight, and at terabyte and petabyte scale the network becomes the binding constraint rather than storage: a migration that would take weeks of continuous transfer may need physical shipment, and cloud egress charges mean that moving data out later is a budget decision rather than a technical one. This is why the governing design principle inverts, from bringing data to the analysis to sending the computation to the machines that already hold the blocks. Third, the processing model changes. Queries become distributed jobs that split, process in parallel and combine, and workloads divide into batch for reporting and retraining and streaming for anything whose value decays within seconds, each needing different tooling and different thinking. Fourth, failure stops being an incident. With hundreds of components something is always broken, so jobs have to be retried, idempotent and checkpointed, and resilience has to be designed rather than hoped for. Finally there is cost and capability: elastic cloud capacity is genuinely cheaper than idle on premise hardware, but a runaway query that scans the whole lake produces a bill nobody approved, so cost monitoring belongs in the architecture, and the platform needs people who have run a distributed system before. Weighing these, the hardest challenge is not any of the technical ones; it is the skills and governance gap. Clusters can be rented in an afternoon, whereas the discipline to catalogue a lake, define ownership, control cost and reason correctly about consistency takes far longer to build, and it is the absence of that discipline, rather than any shortage of capacity, that turns most big data projects into expensive storage.