What follows is based upon 18 consecutive puzzles running from the end of May to Thursday 19th June 2008. I had originally intended to accumulate data from 24 puzzles before drawing any conclusions. However, after 6 a pattern started to emerge and subsequent puzzles have served only to reinforce it. Also I shall be away during July and I want to participate in any discussion.
Quality of Data
Every statistician worries most about the quality of the data being used. I have no reason to suppose that these 18 consecutive mainly June puzzles are any different to puzzles in say December and as I haven’t used a sampling technique I’m happy that the puzzles are representative. Where the blogger did a pie chart I used that, subject to my comments below. Where no pie chart was presented, I made my own.
What did concern me from outset was the use of zero, 0.5 and 1.0 scores based upon the blogger’s perception of level of obscurity. This approach introduced a level of subjective personal judgement into the data that I didn’t trust. Take as an example 23,942 of last Tuesday. For me the Masonic Jude the Obscure was just that as was knowledge of Greek vocabulary. For Tom it presented no problem. On the other hand kinetic energy was meat and drink to me but Jack wasn’t sure what it meant.
I therefore decided to use 1.0 for both the 0.5 and the 1.0 scores. Thus my data reflected the fact that some knowledge had been required but not how obscure or otherwise that blogger believed the knowledge to be. Where a topic occurred in a puzzle but was not mentioned by the blogger I took a view, leaning towards inclusion. Some things were far from clear cut. For example in 23,944 we had mention of Gabriel the announcer. Is this worth a point in the Religion Column? I decided yes, others may disagree. Clearly this isn’t wholly satisfactory but such incidents were small in number and do not significantly affect the overall results.
The Results
Based upon the data as defined above the mean (average) puzzle score is 7.3 points with a standard deviation of 2.7 and the median score is 7.0. No less than nine (50%) of the puzzles scored either 6.0 or 7.0 points.
The lowest score was 3.0 and only three puzzles scored less than 6.0
Six individual puzzles scored higher than 7.0 but with a nearly flat line distribution. Two puzzles scored 10.0 and the value 11.00 didn’t occur at all. The highest score was 13.0
The scores are thus skewed and show more kurtosis than a normal distribution.
Turning to individual subjects, there are three clear divisions. Heading the parade neck and neck are Geography and Literature, which between them account for 32% of all the entries. The next tranche contains History, The Natural World, Sports and Games and Other (Sundry). These four between them account for 46% of all the entries. That leaves five topics: Music, Pop Culture, Religion, Science and Visual Arts each scoring between 3% and 5% (making 22% in total).
In summary then for each puzzle you can expect to have to know around seven general knowledge facts, occasionally less than that but a third of the time rather more. How obscure these facts are will be a matter of your own background and knowledge.
Around a third of these facts will come from Geography (16%) and Literature (16%) and approximately a further half from History (12%), The Natural World (12%), Sports and Games (10%) and Other (12%) whilst Music (5%) and Science (4%) together account for less than 10% of all required knowledge. Religion (5%), Visual Arts (3%), and Pop Culture (5%) make up the balance.
In my personal view these are results that The Times should address. In particular I think that Literature is significantly over represented whilst the important subject of Science and Technology is woefully under represented. I believe this shows The Times to be living in the past.
What Next
I see little point in continuing this exercise on a day by day basis. The results from one puzzle are of little value in isolation and the overall pattern is quite clear. I am prepared to repeat the exercise in say six months to see if anything has changed. As of now I suggest we relieve the bloggers of the additional burden of conducting the topic analysis and leave them to concentrate upon clue analysis.
What do you think?
I’m happy to drop the pie charts for the moment and revive them for a while later. Personally I doubt that the charts will achiveve anything in the way of a change of policy unless the xwd ed has some other feedback (letters from “typical” solvers?) which backs it up.
Any new set of charts done later would have to be on the same basis as this current set in terms of deciding what counts. Which seems to mean Jimbo doing or at least helping with the analysis.
I’m personally not convinced that the literary bias is as bad as it seems – there have been times when I’ve felt that non-literary knowledge that didn’t score points was at least as difficult as the lit (or Geog) stuff that did, though Jimbo may have added these afterwards.
Cricket
This is a tricky one. For someone new to the puzzle (especially someone new to cricket) the cricket references can be a barrier. But like muse=Erato, things like on=leg or deliveries=over come up again and again, so aren’t a real barrier for the regular solver.
Tom B.
Another vote of thanks to Jimbo for some outstanding work, even if I understood approximately 15% (+ or – 2) of what you said. I’m going to read it properly (and very slowly) later on today. Statistics is (are?) not my strong suit but quite intriguing nonetheless.
The charts have shown that I probably owe a fair chunk of competitive success to the kind of puzzle where having the right knowledge means that the answer goes straight in and gives you checking letters for the ones where the wordplay is hard or your knowledge is lacking.
On the subject of literary references I’d be the first to admit my literary knowledge is far from strong – but I don’t have an objection to frequent references in puzzles.
Solving the puzzle involves knowledge of language and of, well, general knowledge, but a slight literary leaning is I think understandable. There’s nothing to support the idea that general knnowledge fiends of the pub quiz type are adept at solving cryptic crosswords; such crosswords do, after all, focus the sheer joy of language and can be described almost as a literary form aimed at lovers of language.
Such an audience could largely be expected to have a basic familiarity with a canon of “essential” literary works.
Except me.
Thanks for your efforts on this, Jimbo. Having read your excellent analysis I’m glad I didn’t cause myself embarrassment by offering to have a go at it.
I don’t think a bias towards Literature over S&T demonstrates that The Times is living in the past. But then I’m not that bothered if they are; in fact I’m quite at home with the idea.
On continuing the daily pie-chart, it rather depends how much of a chore it is for other bloggers. Having set up the html script I just fill in the blanks and I doubt it takes me a minute to compile. I do the actual scoring as I write up the blog. I’m happy to continue or not.
If you want to become a better solver, you’ve got to brush up on your Shakespeare (and if you want to do the Listener, you have to read Spenser until your eyes bleed). Is this necessarily a bad thing? A crossword is a literary device originated in a literary medium. The attraction of composing words from anagrams, constructions and puns is going to appeal to people who are more widely-read than others.
Science and technology… I echo Penfold’s comments that science is underrepresented in the world (and it certainly was at the pub last night where I listened to some really awful opinions on energy conservation and alternative fuels – not awful in an evil sense, but awful in that this was a group of people who believed everything they read on blogs and websites about how clean and efficient alternate fuels were). Following on after that terrible parentheses… there’s never going to be a lot of science in crossword puzzles because it is harder to brush up on your Euclid than it is to brush up on your Shakespeare (and unless you’re me, probably less fun).
Geography – again, very easy to brush up on and familiarize yourself with, particularly with that mini atlas gathering dust on the floor beside the computer. We the hardcores who try not to use dictionaries and atlases might complain about overuse of Geography, but it’s going to be easier to verify than hunt and pecking through Chambers.
History – I’m surprised that turned out as high as it did, but it’s usually battle sites that can be worked out from wordplay.
So I don’t think there’s anything unfair that came out from the pie-charts, but if I was to pop a dozen ritalin and hop the pond to challenge the incumbent Biddlecombe, I would only do so after reading all of Austen, Lawrence, Shakespeare, that guy who wrote “She”, and eating my pocket atlas.
To anyone considering that challenge: you do NOT need to read all those books. As suggested by Don Manley a while back, taking a wide interest in the world around you (e.g. by reading most of a newspaper each day) is far more useful. Plus doing some other crosswords including barred-grid ones, though not necessarily the Listener.
As far as I can remember, the number of books I’ve read by some of the important or recently used authors is: Austen: none; Brontes: maybe one or two, a long time ago; Dickens: about six, Haggard (the She man): none; Hardy: maybe one or two (which really means I started about five but can’t remember finishing any); Lawrence: about four, all a long time ago; Shakespeare: I’ve seen maybe eight of the plays in the theatre or on TV, another four or so known via operatic versions and have yet to see Hamlet!; Wells: none
I’ve read a lot of other stuff (understatement! – this house has c. 30 yds of bookshelves, some with double parking), though more fact than fiction these days, and quite often watch TV adaptations, which means that I’ve probably picked up enough Austen, Brontes and unread Dickens.
On the Geography side, I studied it up to A-level at school and have set foot on four of the seven continents including a good chunk of Western Europe and some of N America, which probably cover most of the references. It’s one of my strengths, but ‘old music’ and a pretty weird range of other stuff (Cheeses, card games, London, Athletics (Track & Field)) are up there too.
How useful it might be in the long run, I can’t say, as I’m not necessarily sure there is a problem that needs solving: as I may have said in previous posts, I am aware that my definition of “unfairly obscure knowledge” may just be my own shorthand for “things I don’t happen to know”. The trouble is, of course, that any attempt to define what is obscure or unfair runs up against personal experience and prejudice.
We have, of course, moved on, and the puzzle no longer simply asks you to fill in the missing word in a quotation: I think all we ask is that where specialist knowledge is required, the wordplay is unambiguous enough that an educated guess will suffice.
That’s true up to a point but if a piece of knowledge is something that 90% of people don’t know, then it’s approaching obscure territory whereas something that only 10% of people don’t know is GENERAL knowledge. I think we could all differentiate between a £100 question and a £500,000 question on “Who wants to be a millionaire” or tell a Weakest Link question apart from a Mastermind one.
Most people would know that Shakespeare wrote Hamlet but knowing which Bronte wrote The Tenant of Wildfell Hall, for instance, requires a higher level of literary knowledge.
All of which is my rather clumsy way of saying that I think there’s stuff that few would argue is obscure, and where such a level of knowledge populates both definition and wordplay, or where the wordplay and/or checking letters are “unhelpful”, then I think we have every right to moan.
Incidentally, I’m just reading Simon Winchester’s latest book, about Joseph Needham – a brilliant scientist, but also a communist, explorer, devoted Christian, equally devoted nudist, libertine and terrible poet. Something tells me he was not dull company. There’s plenty of entertainment in science, just as there’s tedium aplenty in literature, a world which has thrown up some spectacular bores (stand up, Henry James, to name but one).
“Some scientific terms are poor material for cryptic crosswords” is not the same as “science is dull”.