R093 TA2: Research Methods and Sources
Primary and secondary research, quantitative and qualitative data, and how to tell whether a source can be trusted.
Primary and secondary research
Two words, and the paper expects you to use them correctly.
| Primary research | Secondary research | |
|---|---|---|
| Meaning | You collect it yourself, for this project | Somebody else collected it, for their own reasons |
| Examples | Surveys, interviews, focus groups, observation, user testing, your own test shoots | Books, industry reports, census data, articles, competitor products, analytics you did not set up |
| Advantages | Exactly the question you need; current; you control the quality | Fast, cheap or free, often large scale and professionally conducted |
| Drawbacks | Slow, costs money, small samples, easy to do badly | May be out of date, about a different audience, or biased towards whoever paid for it |
Use both
Real projects start with secondary research, because it is free and it shows what is already known, then use primary research to answer the questions the secondary research left open. An answer that recommends only one is weaker than one that explains the order.
A questionnaire is primary if you wrote and ran it. The same questionnaire's published results, read by somebody else a year later, are secondary research to them.
- Primary research is collected first hand: questionnaires, interviews, focus groups and observation.
- Secondary research uses what already exists: websites, books, reports and competitor products.
- Primary is specific but slow and costly. Secondary is quick and cheap but may not fit.
Quantitative and qualitative data
| Quantitative | Qualitative | |
|---|---|---|
| What it is | Numbers and counts | Words, opinions and reasons |
| Collected by | Closed questions, analytics, counts, ratings | Open questions, interviews, focus groups, observation |
| Answers | How many, how often, how long | Why, how it felt, what they expected |
| Strength | Comparable, measurable, can show a trend | Explains the numbers; finds problems nobody thought to ask about |
| Weakness | Says nothing about why | Hard to compare; small samples; interpretation can be subjective |
They are a pair, not a choice
Analytics show that 70% of users leave the booking page. That is quantitative, and it is useless on its own. Watching six people try to book reveals that the date picker does not work with a thumb. That is qualitative, and it is the reason. You need the number to know where to look and the words to know what to fix.
Sampling
- Sample size — small samples swing wildly; a survey of ten people proves almost nothing
- Representative — the sample has to resemble the audience; the wrong fifty people are worse than the right five
- Random — choosing who to ask by convenience builds in bias before the first question
Judging a source
Secondary research is only as good as its source, and the specification expects you to be able to assess one.
| Test | Question to ask |
|---|---|
| Authority | Who wrote it, and what do they know? |
| Currency | How old is it? Platform statistics age in months |
| Purpose | Why was it published? Who paid for it? |
| Evidence | Does it say where its numbers came from? |
| Corroboration | Does an independent source agree? |
| Bias | Does the publisher gain if you believe it? |
Where bias comes from
- Funded research — a study paid for by an industry rarely harms that industry
- Leading questions — the wording suggests the answer
- Self selection — only people who already care fill in the survey
- Cherry picking — quoting the one favourable figure from a mixed report
Ofcom data is extremely reliable, and still tells you nothing if it is about a different age group in a different year. Say why a source fits your audience, not just that it is trustworthy.
Key terms you must be able to define
- Primary research
- Data you collect yourself for this project.
- Secondary research
- Data collected by somebody else for their own purposes.
- Quantitative data
- Numerical data: counts, measurements and ratings.
- Qualitative data
- Non numerical data: opinions, reasons and descriptions.
- Representative sample
- A group whose makeup resembles the audience being studied.
- Bias
- A systematic distortion that pushes results in one direction.
- Leading question
- A question worded so that it suggests its own answer.
Flashcards
Say the definition out loud before you flip the card. Retrieving it from memory is what builds recall, reading it again is not.
Reach the top grade on this topic
The mark is in the justification of the method. Not "I did a survey" but "I surveyed 40 Year 10 students because the product is for that age group, used closed questions so the results could be compared, and then interviewed four of them to find out why the least popular design was disliked."
Mistakes that cost marks here
- Mixing up primary and secondary. It depends on who collected it, not on the type of data.
- Treating quantitative and qualitative as a choice when strong projects use both.
- Quoting a source without checking its date, especially for platform and audience statistics.
- Using a sample that is convenient rather than representative.
- Writing survey questions that lead the respondent to an answer.
A survey question can decide its own answer. 'Should the council waste money on a new website?' and 'Should the council invest in a new website?' describe the same thing and produce opposite results, which is why examiners ask about bias.
Knowledge check
Exam-style questions
Five written questions in the style of the real paper. Write a full answer, then let the marker check it against the mark scheme.
- LA genuine primary method
- LA genuine secondary method
A primary method is running a questionnaire with the target audience. A secondary method is reading a published industry report.
- LIt answers exactly the question needed
- LDevelopment of that point
- LIt is current, or its quality is controlled
- LDevelopment of that point
The first advantage is relevance: the questions are written for this project, so the answers address exactly the decision being made, whereas a published report will have been written to answer somebody else's question about a slightly different audience. The second is currency and control: the data is collected now, and because you designed the method you know the sample size, who was asked and how, so you can judge how much weight the results deserve rather than having to trust a source you cannot check.
- LCheck the date or currency
- LWhy that matters
- LCheck the source, author or funding
- LWhy that matters, or that it should be corroborated
The agency should first check when the figure was published, because platform preference among teenagers changes within months and a figure from three years ago may now be the wrong platform entirely. It should also check who produced the research and who paid for it, because a study published by a platform about its own popularity has an obvious interest in the result; the figure should be corroborated against an independent source such as Ofcom before any decision is based on it.
- LQuantitative data is used to locate the problem
- LA specific quantitative measure is named
- LThe limitation of quantitative data is recognised
- LQualitative research is used to find the reason
- LA specific qualitative method is described in context
- LThe findings are turned into a change
- LRe-testing or measuring after the change
- LA judgement about how the two methods relate
The company should start with the quantitative data it already has, because analytics will show exactly where customers are being lost: the completion rate for each step of the booking funnel, the percentage abandoning on each page, the time spent before leaving and the device being used. That identifies the problem precisely, for instance that 70% of abandonment happens on the date selection step and that nearly all of it is on mobile. What the numbers cannot do is explain the cause, since a drop off figure is identical whether the page is confusing, slow or broken. The qualitative stage answers that: recruit five or six people from the real audience, ask them to make a booking on their own phones while talking through what they are doing, and record every point at which they hesitate, backtrack or guess. That typically produces a single concrete finding, such as the date picker requiring a tap on a target too small for a thumb, which no amount of analytics would have revealed. The change is then made on the basis of that finding, and the quantitative measurement is repeated afterwards to confirm the completion rate actually improved rather than assuming it did. Neither method is sufficient alone: the numbers say where to look and how much the problem is worth, and the observation says what to change, so the sensible order is quantitative first to locate, qualitative second to diagnose, quantitative again to verify.