Data Analytics F200 Paper C: Fundamentals of Data Analytics

Metadata is data about data: information describing a dataset rather than the content of it. The capture time, the sensor that produced a reading and the data type of a column are all metadata.

A query can read only the columns it actually uses rather than every field of every row, and storing like values together compresses far better, so the same analysis moves a fraction of the data.

The temperature of minus 999 for T-01 is a sentinel value standing in for a failed reading, not a measurement, and averaging it would wreck the result. The humidity of 410 per cent for T-03 is outside the possible range for the field and is likely a misplaced decimal point. T-02 appears twice at 08:00:04 with identical values, which is a duplicate that would double count. Beyond those, the status field holds both 'OK' and 'ok', which splits a two-value field into three; T-04 has blank readings with a status of offline; and the second 08:00:05 is missing from the sequence entirely, so a reading has been lost rather than recorded as absent.

An outlier is not the same thing as an error. A reading far from the rest may be a genuine measurement of something unusual, and in a sensor log on a production line that unusual thing is very often the fault the monitoring exists to catch: a bearing starting to fail, a blockage, a sudden temperature spike. Deleting outliers automatically therefore removes exactly the records the analysis was installed to find, and does so silently. The correct order is to investigate first, establish whether the value is impossible, implausible or merely rare, and only then decide whether to exclude it, recording the decision either way.

Observation: sit with operators during a shift and record what they actually do with the software, where they hesitate, what they write down on paper instead and which screens they never open. This finds problems nobody would report, because people adapt to bad software and stop noticing. A structured questionnaire issued to every operator across all three shifts: closed questions so the answers can be counted and compared between shifts, with a small number of open questions to catch anything the designer did not think to ask. The observation explains why; the questionnaire establishes how widespread it is.

An API returns the current state rather than a snapshot that is already stale by the time anyone reads it, which matters on a production line because a maintenance flag raised at 09:00 is useful at 09:05 and worthless the following morning. It also gives control that a file drop cannot: each consumer holds its own key, the requests are attributable and logged, access can be scoped to particular endpoints and revoked in one action, and the query can be filtered server side so only the records needed cross the network rather than the whole table every night.

Technically, the two systems must agree how the data physically moves and in what container: the protocol, the file format, the encoding. If they do not agree here the exchange simply fails, loudly, and somebody fixes it. Syntactically, they must agree the structure: the field names, their order, whether records nest, how dates are written. A failure here produces a file that parses but whose values land in the wrong places, which is usually visible on inspection. Semantically, they must agree what the values mean: whether 'completed' includes cancelled jobs, whether a maintenance hour counts the travel time, whether a shift is dated by its start or its end. This is the dangerous level, because nothing errors. The transfer succeeds, the import succeeds, the report runs, and the numbers are wrong in a way that only shows up when somebody notices two systems disagreeing months later.

There is a clear negative relationship: weeks with a low price show high sales, around 150 to 190 units, and weeks with a high price show low sales, around 20 to 50, with the points forming a fairly tight band running from the upper left to the lower right. The relationship is strong enough to be useful but it is a band rather than a line, so price plainly does not explain everything about a given week. One point sits well away from that band, at a price near 16 pounds with sales of about 176 units, which is roughly what the product sells at its cheapest. That single week is worth more attention than the other twenty nine put together, because it is either an error or the most interesting thing in the dataset: a mis-keyed price, a week when the product was promoted or featured, a competitor out of stock, or a bulk order from one customer. It should be investigated rather than deleted, and if it turns out to be a genuine week the explanation for it may be more commercially valuable than the price relationship itself.

In transit, TLS, so an intercepted request cannot be read. At rest, encryption of the stored volume or file, so a stolen disk or a copied backup is useless without the key.

Pseudonymisation replaces direct identifiers with a reference, with the key linking them held separately. It reduces risk, but it is reversible by design, so the data remains personal data and the whole of the UK GDPR continues to apply to it. Anonymisation removes identifiability permanently, so the data is no longer personal data and the regime no longer applies. The catch is that genuine anonymisation is much harder than deleting the name column, because combinations of ordinary fields identify people: a date of birth, a postcode and a sex pick out most of the UK population, so real anonymisation means generalising, aggregating, suppressing small groups or adding noise, and accepting the loss of precision that comes with it.

Attention to detail, because the analyst who notices that a total is one per cent out and chases it is very often the one who finds the duplicate feed behind it, while the one who rounds it reports a wrong number with confidence. Communication, because every other skill produces a result that exists only inside a notebook: a correct analysis a manager does not understand changes nothing, and a clearly explained partial answer changes behaviour the same afternoon. Scepticism, because the result most likely to be wrong is the one everybody hoped for, and an analyst who checks hardest when the answer is convenient is the one whose figures survive being questioned in a meeting.

A data dictionary lists every field in a dataset with its data type, its format, its permitted values or range, whether it can be null, and crucially what it actually means: not just that 'status' is a three character code but what each code represents and when it changes. It is needed because a dataset is unusable, and worse, misusable, by anybody who did not build it. Without one, two analysts quietly interpret the same column differently and produce two defensible answers to the same question.

The distribution is right skewed: most repairs cluster between one and three hours with a long thin tail of multi-day jobs. The mean is calculated from every value, so those few very long repairs drag it well above what a typical repair actually takes, and quoting it would make the line look slower than it is on an ordinary day. The median is the middle value and is unaffected by how extreme the tail gets, so it describes the typical repair honestly. The manufacturer should report the median as the headline figure, and report the long repairs separately as what they are, a small number of serious events worth managing in their own right, rather than letting them distort the everyday number.

The design should start from the decision rather than the data: the question is which machine to take off line and when, so the output has to be an alert with enough lead time for maintenance to act, and everything upstream is judged against that. Sensor data is the hard part of the collection. A line with hundreds of sensors sampling every second produces hundreds of millions of readings a day, which is a volume and velocity problem rather than a storage one, so processing should begin at the edge: a gateway on site applies threshold rules immediately, so an out of range vibration raises an alert within milliseconds and keeps working when the connection drops, and aggregates the rest into one minute summaries holding the mean, minimum, maximum and spread per sensor. Full resolution readings are buffered locally and uploaded only around an event, which is when per second detail is actually needed. For storage, the summaries belong in a time series database, which is built for timestamped measurements and compresses them far better than a relational table, with the raw event windows in object storage for later model training, while orders and maintenance records sit in the warehouse where they already are. Bringing the three together needs two things that are easy to overlook: a shared machine identifier used consistently by all three systems, and agreement on how timestamps are recorded, because a maintenance record dated by the day of the visit cannot be aligned to sensor data recorded in UTC to the second without an explicit rule. Preparation has to deal with what the sensor log actually contains. Sentinel values such as minus 999 must be converted to explicit nulls rather than averaged, impossible values such as a humidity of 410 rejected by a range check at ingest, duplicate readings at identical timestamps deduplicated, and gaps in the sequence recorded as gaps rather than silently closed up, because a missing reading may itself indicate the fault. Governance is what keeps this usable in two years' time: a named owner for the combined dataset, a data dictionary defining every field including the units and the aggregation applied, recorded lineage so any alert can be traced back to the readings behind it, a retention policy covering how long full resolution data is kept, and role based access with least privilege, since this data reveals production volumes and would be commercially damaging if it leaked. Finally the system has to be evaluated honestly: the manufacturer should record unplanned downtime for a period before deployment and compare it afterwards, because the only measure that matters is whether downtime fell. The hardest part of all this will not be the sensors or the storage; it is aligning the three systems' identifiers and definitions, because that work is unglamorous, invisible when done and produces confidently wrong answers when skipped.