This excellent article did the rounds in my department at work today, about the methodological rigour (or lack thereof) in ratings, on which I’ve been meaning to write a post for a while.
I refer almost daily to such demographic information – ratings, audience/circulation, readership and particularly advertising value equivalents – as `the department of made-up numbers’ because, basically, that’s what they are. At best, they are a set of figures which, while deeply flawed, are horizontally and vertically consistent, and well-enough understood that their failings can be accounted for (this approximates a definition of any useful long-term demographic data). At worst, they are the patina of officious statistical rigour over a set of numbers tuned to tell people whatever the media outlet, its owners, or its PR company want people to know – and that means they’re designed to fool. Most often, a given dataset lies in between – in the murky liminal zone where it’s impossible to tell whether it’s the former or the latter or something else entirely without access to the raw data and its provenance, which is nearly always impossible to get, and would entail phenomenal amounts of very specialised, expensive, time-consuming work to make sense of even if you could get it.
Despite these dire problems, demographic data, ratings, audience/circulation and advertising value equivalent data are the mainstay of the media and communications industry’s performance measurement infrastructure, for two simple reasons: First, it gives you nice clear figures to prove your department is doing its job; and second, nothing else does, because media demographics is the art of measuring the unmeasurable. So people who are otherwise cautious and crafty and suspicious just accept the numbers at face value and trust them implicitly because the alternative is no data, and with apologies to the Bard, nothing will come of nothing.
This reliance on demographic figures is highly detrimental to the health of the media industry, because the data can’t be verified, and there exists an inflation imperative. I dislike comparisons to communism as a rule, but there’s a parallel in this sort of reporting in the media/PR/comms industry as it presently is to the problems of productivity reporting seen in the 20s in the USSR and the 50s in China. When both producers and their supposedly independent auditors are ranked according to the quantity – not the quality – of the figures they produce, there inevitably emerges a tendency to inflate those figures.
In the USSR and China, wheat and rice yields were inflated in this way, because the producers would be punished if their yields fell, and the municipal authorities didn’t look too closely at the production figures because they would be punished if their municipality’s yields fell. Central government assumed these figures were correct, and based budgets and food allocations and projections and such upon them, planning more than they could realistically achieve because there was in fact less food than they thought in the granaries.
If we substitute `food’ for `ratings, I think the parallel is pretty clear: the media are relying on bad data to demonstrate that their product has value to advertisers first and journalistic merit second and to boost the egos of their stable of opinion leaders third; internal communications departments use it to measure the effectiveness of their campaigns and initiatives; external PR firms use it to prove their worth to client companies; boards of directors rely on it to make decisions about what publicity campaigns to fund, which products to launch, and who to promote. All this is good money thrown after bad – frankly, it’s a miracle it hasn’t all come tumbling down sooner.
L
What are the specific difficulties in measuring it Lew? If you’re talking print media then you already have an exact measure of the numbers printed. Then surely you could over time by survey of outlets and demographic calculations, produce a reliable number of copies sold factored by whether a single copy was read by multiple people and any wastage of unsold product?
I realise the devil is in the detail but this is precisely the kind of issue insurance faces with risk management and they have very well evolved mechanisms to deal with that in reasonably reliable ways.
If you’re talking radio/TV then I presume they use the Nielson approach? Is that reliable and if not, why does the US agencies persist in it.
For PR, I well imagine the difficulties. However again, I would have thought that with intelligent sampling one could produce reasonable results, isn’t this what political polling is all about?
Can’t you take a leaf from the insurance industries and use their methods?
Beautiful Lew – you’ve comprehensively circumscribed the state and nature of our current “political information” “bubble”.
Like our celebrated “credit bubble”, I suspect (and fervently hope), it’s about to burst.
With any luck, Joanna Public will continue the current trend of distrusting and rejecting the paper-tools of our corporate advertisers, and inform and educate herself via the plethora of information available on the wonderful web.
Unfortunately, our paper dinosaurs will plumb any depths to prolong their advertiser-dependent existence, including and especially the blatant and disgusting pro-tory, anti-NZ First “gang-rape” campaign prior to the last election.
Oh fuckit. Greed, money, me, me, me.
Get fucking angry people – beautiful, perfectly-formed (but brown-oh!) babies die every fucking second while these scabrous, moronic, elitist [excised] tell us that they know what’s good for us)
sorry Lew.
reid,
Let me reiterate that I think it is possible in principle to get somemeaning from media demographics, just not as much as people think or would have you believe.
Nielsen ratings are among the best of a very bad bunch. They’re what I had in mind when talking about fatally flawed figures which are consistent enough and well-enough understood to account for the flaws. Circulation is another such measurement – it’s not very good, but it at least measures something concrete (the number of papers distributed, for instance). More on this below.
I’m not overly-familiar with the way insurance demographics work, but from what I gather there are significant differences. First, in insurance there’s a discrete and specific revenue advantage to collecting the best-possible data for each individual case – that is, each case is worth investigating in and of itself. Second, insurance investigates events which actually happen and which can usually be proven to have happened to a reasonable degree of confidence. Neither of these two things hold with media data.
The big methodological problems do share some similarities with market research and political research – authority and availability biases, feedback effects, expectation fullfilment, shame effects or self-identification fallacies, and some of these are even integral to the research – especially as regards advertising. But mostly they’re to do with confidence, and assumptions regarding audience behaviour.
Nielsen TV ratings are a good case in point – remembering that they’re among the best data we have. A sample of households in a target market has a `people-meter’ installed on their TV, which sends data back to Nielsen HQ. Each member of the household has their own indicator of presence in the people-meter, and they are supposed to indicate to the people-meter when they’re watching TV. It’s assumed that whoever is logged into the people-meter is watching their TV with undivided attention, which is an instant methodological fail in any sort of qualitative audience behaviour research – there are literally hundreds of studies which find that attention ranges from virtually nothing at all (making dinner in the other room and popping in now and again) through partial engagement (conversation while watching) to engagement with the programme only (muting the ads) to full engagement – lights dimmed, no conversation, discussion of the programme during ads, etc. – which is actually very rare, especially for long periods of time. Then there’s the fact that people logged in might simply not be there. Little Timmy might log in for Spongebob and The Simpsons, but then when Sally turns over to Home and Away he goes outside, and then when dad turns on the news he does his homework, and then he talks on the phone for a while, then watches the first 10 minutes of CSI, but has to go to bed, and leaves himself logged in because he doesn’t care about the quality of the data anyway, and is logged as watching through until the end of Nightline just because that’s when the TV gets turned off. And then there’s the fact that (in 2006; this may have changed) there were about 400 people-meters for all of NZ – that means each individual TV accounts for about 0.25% of the ratings figures, an abysmally small sample. Houses in which people-meters are installed are demographically weighted but the details of that weighting isn’t made public, so we have no way of knowing if it’s actually any good.
That Nielsen ratings provided to broadcasters take no account of all this stuff is a crystal clear example of the kind of inflation I’m talking about: yes, it’s methodologically very hard to moderate data for peoples’ viewing habits and such, but more importantly it will make the figures look bad.
`Readership’ is another very shaky set of projections, which come from inflating circulation to account for second-hand reads in a doctor’s waiting room, or someone’s coffee table, or whatever; notwithstanding the likelihood that people just read the star section of New Idea, and skip straight to the smutty pages of Cosmopolitan, or whatever.
Advertising value equivalent is even less meaningful – it’s defined as `what this would have cost to run as an advertisement’. Little to no consideration is given to different ad rates, bulk discounts, time or position or prominence, the fact that advertisements are semantically nothing remotely like news copy, that a full-page article might contain one mention of Company X and be counted as being worth a full page ad rate, or any other consideration, such as whether you could even buy that much ad space. It’s a blunt object.
So, in short, it is as bad as all that. But the point of my post was that there seems to be no imperative from within the industry to improve the data, especially on the part of those who use it and pay for it – either directly, or from basing their decisions on poor data.
L
ak,
Easy, there.
You’re right in terming it a `bubble’ – defined by the way people believe information just because other people believe in it, rather than on its strict merit as information, and it is very much like the credit crunch in that regard. But it isn’t to say that the media themselves are at fault, and nor are media or PR agencies – they’re simply fulfilling needs. Your call to arms is right – to improve the system, people need to demand improvements.
L
Yeah Lew, I was saying as you recognise as you continue your 12:35, that the sampling methods need to improve. Statistically reliable data is founded on those disciplines and the insurance actuaries are particularly good in that respect. It’s that leaf which should be taken by the advertising number crunchers.
Very good post and you’re absolutely right, it’s crazy there’s no imperative. I wonder if the fact that a lot of marketing is all about creating a superficial fluffy wrapping around the item of interest, is what’s preventing that from being done. If so, the clients must be fucked in the head not to be demanding a better result. Maybe that’s because the client’s people who are involved in collating the data to the executives, are also grounded in marketing disciplines. A self-perpetuating wheel of vested interest.
\There is nothing more vigorously defended than a vested interest disguised as an intellectual conviction\ is a maxim I see played out all the time, in all sorts of places.
Thanks for the insights.
Easy there
Yeah sorry about that intemperate rant Lew (hard week, painful anniversary and a gin too far…). Here at our wee coalface the dreary nineties re-run of adult tears is making a comeback already, and what’s particularly depressing this time round (in the context of your excellent post) is an upsurge in inter- beneficiary bitterness and suspicion. Hearing the identical talk-back beneficiary-bashing phrases from the victims themselves is a bit hard to take at times… and a too-easy step to anger at the facile and venal scum who promulgate such destructive poison for their own gain. Divide and conquer in the flesh. Dunno if I’m entering a Benjamin-Buttonesque second adolescence or summat, but this time round I feel a strong urge to somehow direct some of these ever-welling blood and tears right back to their comfy, summit-sucking origin. I certainly felt no sympathy for poor wee Micael Laws’ window, and the local electoral office might soon be getting quite a few more referrals…wrinklies of the world unite!