Quite a few people have requested a SNITCH version for the Quick Cryptic. I’ve created one here and welcome your feedback on it. In particular, I’d be interested in hearing whether you think the ratings match your experience of the difficulty. I’m also curious about whether I’ve captured the Neutrinos correctly or not.
There’s one bug I know of (one of the graphs is not showing correctly on the Neutrinos page). Please let me know if you see anything else.
And, before you ask, no, I haven’t yet listed the difficulty by setter. I’ll try to get to this.
(By the way, attention for the bloggers: I need to pick up the setter from the title of your blogs, i.e. where you put “by Oink” or another name in the title. Sorry to add to the pressure of blogging, but it would be very helpful if the names are shown consistently, as they almost always are.)
Great job, of course, Starstruck. How did you get the average times? I.e., how far back did you go? (Ironically, I just commented on today’s blog that my target of 6:00 has been too fast for me, and here you give me a 5:45.) Most of the solvers who expressed a wish for this are not listed; can I expect them to be added?
Thanks, Kevin, this is good feedback.
I went back 6 months. But remember that I will only count times that are in the top 100 (since that’s all I’m able to see), so if you’ve had slower times that don’t make the cut, they won’t count towards your average.
Things will improve a bit as I start to pick up data during the day. People only have to be in the top 100 briefly for me to record their time for that day. For yourself, solving early in the day, I’m likely to get nearly all your times, so your average will probably increase.
For the others who asked the same applies. There’s also a basic limit that I need 20 recorded times to start registering people. It’s possible that this number is too high for the Quick SNITCH.
I just looked at the leaderboard for the top 100 for QC 2483, yesterday’s: a quick scan of the 100 turned up two people, Phil Jordan (‘Busman’) and me. That is to say, none of the solvers who contribute to this blog scored high enough to appear on the SNITCH that they expressed a wish for. Today’s QC leaderboard has 64 solvers at the moment, 13 with errors, and virtually nobody among that group (of TfTT commenters).
Great stuff, and thanks for putting in the work. I owe you another coffee. The way the data looks at the moment is that the Snitch is heavily weighted to experienced solvers, not the chatty 10-30 minute solvers that make up most of the posters.
If it were to be a more democratic Snitch, for the posters and members of the SCC maybe you could eliminate scores from solvers who average very fast times.
Anyway. I’m going to make sure I use the club everyday so my times get logged. (48th today and time of post). Maybe many of the QCers use the non-club interface on the iPad (with its handy check and reveal), I always do for the 15×15 and often for the QC as well. Maybe many are also paper solvers.
Starstruck can confirm but I think the SNITCH may be able to pick up your times from your blog comments (see the Blog Solvers section on this page: https://quick.xwdsnitch.link/solvers/neutrinos). This would mean that people wouldn’t need to change where they solve but simply leave a comment with the appropriate formatting.
Thanks, BC, this is an excellent thought.
My initial conclusion is that the Quick SNITCH is doing okay in reflecting the difficulty of the puzzle, but it’s not working well for tracking interested solvers with slightly slower times (and that would include me!).
Johninterred and I are raising the question with the Times folk about whether we can get access to data for more than the top 100 club members. If we success there, it might open up some more possibilities.
In the meantime, getting people to report their times in the TfTT blogs would be a good workaround. I can pick these times up and report them on the leaderboards (as long as that is what the person wants, of course!).
Thanks again for the comment – it’s right on the money.
Thanks, Merlin, that sounds like a good plan. I’ll keep an eye on how your times are being picked up to help develop a solution which is a bit more democratic. (I love the “chatty 10-30 minute” description, by the way.) As BC mentions above, applying the Blog Solvers might be a good answer that would be fairly easy to get going.
Thanks again for the feedback and support.
Looks good – as per Kevin, my average looks better than it is, but probably for the same reason i.e. that my duff times aren’t recorded.
Firstly – mainly thanks for your time and efforts in doing this.
In places it looks right – with the general ups and downs of the SNITCH value looking correct. For example, this week I found Weds and Friday hard and that’s what it shows.
In detail, I’m not so sure I took 11-mins for Monday which is SNITCHing at 89, 25-mins for Tuesday’s Izetti which is about the same at 87. The former is about my fastest possible, the latter somewhere very slightly below average (23mins).
But then, as I have commented elsewhere, I’m not sure what the SNITCH actually reflects. It’s certainly not the difficulty of the puzzle as I have been left with 5 clues unsolved on a 15×15 which snitched at 64, just as I have with one scoring in the 90s. I guess it reflects the difficulty of puzzles for people who are elite at them*.
* I don’t mean this as a criticism of the SNITCH just an observation – ultimately it is better to have some kind of indication based on the wisdom of crowds than to rely on the thoughts of an individual blogger.
Can’t the outliers you mention be explained by the fact the SNITCH represents an average taken from the times of many solvers? There will always be days when you find a particular puzzle harder than most others and days when you find one easier.
Quite possibly. They just seem massively out of kilter to me.
I will look to do some deeper analysis. Copy over the SNITCH values to be alongside my recorded times. And do some twiddling
Edit: a little extra reflection. Certainly slow outliers (e.g. puzzles which take a very long time) can be an individual thing where you get stuck on certain clues and grind through to the solve. But I don’t think you can luck into a quick solve. At the back of my mind, I seem to recall GaryA also did Monday in 11mins then took 40+ on Tuesday’s Izetti and we’re solvers of the same competent level. Two of us with the same outliers? Doesn’t that begin to be a sample …
Hi L-Plates, thanks for your feedback – much appreciated.
Just a quick check – have you looked at the more detailed results for the puzzles (e.g. this link for last Friday’s puzzle)? This shows all the reference solvers individually, so you can see the spread of their results – including which are outliers for the particular puzzle. Do you think that this average over a range of solvers doesn’t give a good score for the difficulty overall? (That’s a genuine question, by the way, as I’m trying to assess whether the way we’ve done it for the original SNITCH doesn’t work well here for the Quick.)
Thanks again for your interest and for taking the time to explore the new site and give feedback.
Thanks for the reply starstruck_au … I hadn’t looked at that individual link and the Quick Snitch links don’t seem to be working at the moment -so can’t access.
I actually began to do more detailed analysis on Saturday – I have a spreadsheet of my times and a friend who is good at these things (avg 7mins, fastest of 3:20ish this year).
But I couldn’t copy all the SNITCH values across. When I copied the grid it simply gave me one long string of data which, in all honesty, I couldn’t be bothered to try and cut up*. I started to type some across but again it was going to be too much.
My spreadsheet has our times listed vertically – one day per row. Date order starting with Jan at top, day-by-day, including Sat&Sun . So ideally I wanted to be able to copy the SNITCH values into a column of their own and then maybe graph the three (my times, friend’s time, Snitch) to see if they visually move up and down together.
* I’m not a completely spreadsheet numpty, could do some transpose or similar. But I’m also not as up to as I was say fifteen years ago when I was into Pivot tables, VLOOKUPs etc.
Hi again, L-Plates. So sorry the Quick SNITCH was not working. It decided to shut itself down last night (my time) – still some teething troubles.
Have a look at the detailed pages and see if this helps your analysis.
Hello again – I had a look through the detailed pages and it’s interesting to note that there is barely anyone taking longer than 6mins on average. And on an easy one, the slowest time I could find was 7:15; but when on the hardest SNITCH (143) I could find there was an 11+ minute’er.
All of this discounts DNFs, non-submissions and quitting, doesn’t it? Which are surely a very important measure of difficulty – of course not sure how you collect that data other than by self-reporting.
Anyway, post comment yesterday I began a thought experiment that I didn’t conclude. Essentially that if you give my 84-year-old mother, who has never done a crossword, an Izetti and a Trelawney she will find them both difficult and not solve a single clue on either.
As a complete beginner they will both be difficult to her. Her judgement is not to be trusted on the difficulty of the grid. At what point does it tip? How much experience do you need?
Is it, as you are doing with the SNITCH, best to look to the experience of the elites? Rationally it makes sense that it is. And yet do they care about SNITCH values? Isn’t the intrinsic value of a SNITCH actually to us lesser-mortals to decide whether to give it a go or not? And therefore is the experience of elites reliable for the experience of the lessers?
It could however be argued an alternative intrinsic value is to reassure good solvers who took a long time that it was due to difficulty, not their own failings.
Then you begin to see diverging reasons for having a SNITCH – whether to have a go or reassurance. And it seems to me how you build that SNITCH value may differ for each reason.
Further thought and philosophising required …
Further to my comment above where I said I think it represents the experience of elite solvers rather than difficulty – I don’t mean this derogotarily . But obvs if you only collect elite solver times than you are missing the data for the numpties like me and therefore not measuring their experience.
Two things I noticed from my nearly two years of solving going from a beginner to competent enough to have solved a QC in under 9mins and three 15x15s.
1) Generally the areas / clues I struggled on at the end of a solve were the same as those of the better solvers. Certainly this begins suggests that the difficulty of grid is determined by how certain words are clued or the familiarity of the answer (e.g. obscure GK)
2) The sticking point for beginners can be a lack of checkers. Decent solvers get many of these on first pass whereas a beginner can be staring at a blank grid and struggling to get going. Often I have found midsolve that just getting one new answer, and the checker it provides, can be enough to bring a bunch of answers tumbling out. For the lesser ability, the grid might be easy IF they can get going – but if they don’t it’s just as difficult.
None of this is said to denigrate your efforts. As I said before, I think having a wisdom of crowds indicator is still good to have. The 15x15s I attempt are those with lower SNITCH values, so a QC version would still be valuable.
Thanks again for your interest and be assured that I take no offence.
I love your curiosity on not just whether the puzzles are harder or easier, but why that might be so. Lack of checkers and more obscure GK must certainly contribute.
I’ve often wondered whether some crosswords suit some groups of us more than others. We see this sometimes with non-UK solvers (including me) who might not know the local GK or slang use of terms. But I wonder whether there might be other causes – perhaps different strengths we have. I’m not very quick at anagrams, whereas I know that others (aphis99, for example) just see them immediately.
Thanks again for sharing your thoughts.
Fully agree with your anagram thoughts – I’m often slow at them too. And, as you say, UK-specific GK is a killer for someone overseas. That raises the question is a clue intrinsically difficult or a lack of ability.
Hey Starstruck_au
I don’t know if you’ll pick this up now but I was thinking more on the QuickSNITCH or, more specifically, on its representativeness.
After yesterday’s difficulties on Teazel (#2491) where many were slowed by a particular last word; it occurred to me that probably the ideal Snitch would sample a range of solvers of all ability levels and not just go on their time. It would actually begin with how many clues they solve correctly. Because while the elites are are able to answer everything or end up with just one wrong.; many of the beginners will get stuck having barely made progress or for the more competent with 3-4 left. They are looking for an indication of how much they will get solved, not just how quickly it gets done.
Of course, how you gather that info is much harder to say. But just my thoughts on what I would see as ideal.
[Edit: this was supposed to be a reply to L-Plates.]
Two outliers are not a significant number when the data is being taken from 100 or more solvers. It is entirely possible (though unlikely) that your solving times for every single puzzle that you have ever solved will not agree with the SNITCH’s level of difficulty for those puzzles. It is also entirely possible (though unlikely) that the same thing could be said for some number of other solvers, however that number is pretty much always going to be smaller than the number of solvers for whom the SNITCH difficulty DOES agree with their times – that’s just how statistical analyses of this nature work. (I say “pretty much always” because there are edge cases, e.g. it would theoretically be possible to have a puzzle where half the solvers found it very easy and the other half found it very hard – the SNITCH would say it was of average difficulty, even though not one solver actually found it of average difficulty).
Part of the information presented in the SNITCH is a graph showing how closely an individual’s times tally with the SNITCH rating. Unfortunately I can’t see the graph for yourself as I can’t find your name in any of the results. Do you solve on paper or perhaps have a different user name on the Crossword Club?
Thanks
How do you select the reference solvers? E.g On Thursday I took 9:26 but I’m not on there.
Thanks for the question. The way it currently works, solvers have to have at least 20 error-free solutions in the top 100 scores in the last six months to be considered. I then check the alignment to the SNITCH (to check that they’re not neutrinos) and, if they’re sufficiently close, I use them as reference solvers.
One problem we have at the moment is that, for historial puzzles, I’ve only got the final top 100. This is full of fast solvers’ times (and some neutrinos). As the SNITCH runs regularly, I pick up more times from people, like yourself, who make it into the top 100 during the day, even if it doesn’t make the final list. Collecting these results means that I get data from more solvers overall and can fill out the list beyond the very fast solvers, so the list of reference and tracked solvers will improve over time.
The other option, as mentioned above, is that we collect time data from the TfTT site (as Blog Solvers). I see that you report your times in your blog comments, which is excellent, as I can pick up some historical and go-forward data from there.
This is brilliant. Thank you!
Thanks 🙂
Great work! Thank you! An incentive to solve in the afternoon / early evening California time so that my times appear – however briefly – in the top 100!
Hey, D, thanks for the comments. A good idea to do the crossword early if you can. And, as mentioned above, make sure you record the time in the TfTT comments as well (as I think you do already).
Taking up Merlin’s point, I am someone who solves fairly regularly, usually in or near the SCC, but using the online Times rather than the club, simply because I skim the news and sport and do the QC and a few other puzzles at the end. I rarely do the 15*15 and I am not remotely competitive so have no personal interest in how my time compares on a table.
I suspect that many others of the less devout solvers use the iPad Times rather the club. The blog shows however that many have the competitive gene that I lack and would be keen to be included if possible. They might well be amenable to using the club to achieve that if necessary.
As I start to attempt a few 15*15s I do find the Snitch v helpful for picking out some old ones that might be more achievable, and I am sure the same would over time be valuable on the QC. All the best in getting it going, I am always impressed by the community spirit that maintains TfTT and makes it so much fun to follow.
Plymouthian’s sentiments pretty well express mine too.
Great comments – thanks. I’m hoping that the current Quick SNITCH version works well for you then, as it shows the difficulty (hopefully well) and you don’t need to have your time recorded.
For the iPad Times solvers who do want to be included, the option of being a Blog Solver will be available. They just need to record their times each day in the TfTT blog (and let me know that they’re happy for these times to be tracked).
Thanks again for the input.
Can you remind us again of the format needed for times to be picked up from the blog. Or, Just post the RegExp command you’re using 😉
You don’t want to see the RexEx – I thought it was very sensible (and funny) when someone years ago told me it was the world’s main “write only” computer language.
Some advice is available on this page: https://quick.xwdsnitch.link/solvers/neutrinos at the Blog Solvers section.
For your convenience, it says:
Notes to Blog Solvers:
“Write only language” 😀😀
I’ve written some monstrosities in my time.
One other thought from looking at the recent QSnitch score. They all seem very similar. Scoring high 80s – low 110s. Whereas with the 15×15 SNITCH there seems to be a much wider range occurring.
Is there a way the scoring range could be broadened?
Would there be any mileage in making the scale 10x larger so if you have two QCs scoring 88 and 89 – granularity would show they’re actually 876 and 894 and therefore further apart than they appear?
Thanks for this thought. What I did on the original was to look at the distribution of scores (using the standard deviation). I’ll try to repeat this for the Quick SNITCH and see what the numbers look like.
Okay.
Incidentally I was looking at the data on Saturday and noticed #2343 is missing. Seems to have been replaced by a random earlier QC.
Sharp eyes – thanks. Now fixed.
Hi Starstruck, let me add my thanks to you for both versions of the SNITCH.
One thing I noticed, the “neutrino” info for the Quick SNITCH seems to carry over some stuff from the other version. For example, I assume the cutoff for reference solvers is 10 minutes rather than 60?
Thanks, branch, for the comment and sorry to take so long to reply. The cutoff is still actually 60 minutes, but because I only get the top 100 solvers, we never get near this. (So, you’re right, my comment on the site is misleading.) I’m hoping we might convince The Times to give us (or maybe everyone) a longer list to work from. In the absence of that, I’m working on getting the Blog Solvers going for the QC, so we can at least track their reported times from there.
Thanks! That also confirms my suspicion that a Saturday SNITCH is too hard. I’d like it for my blogs, but that’s definitely a fringe case! And, explains why my personal average time seemed low.