#1: Brian Bellis (mootpoint) on Feb 25, 2011

I make primarily small puzzles and I think they are rated more harshly than larger puzzles. I've heard plenty of others make the same comment. I have a proactive solution and it comes from the world of teaching.

When a group of teachers are given the task of grading a large number of free response questions such as essays or open ended math problems, we begin by hunting for some "anchors". These are examples of a typical "1" and a typical "2" etc. All the graders are then trained with these anchors including a discussion from the graders on what makes this a 4 and why that is only a 3.

I propose that we hunt for some small (around 20x20) anchor puzzles that we think should have quality ratings of 1,2,3,4,& 5. So that there an no hurt feelings, perhaps we could make the 1's and 2's just for training purposes and not really intended as puzzles.

If we got people to then visit this Rating training forum we might see really great small puzzles getting an average rating or 4.25!

What do you say? Are you up for it?

Great idea! The anchor idea seems very applicable to this and certainly will help us rate puzzles.

I agree it's a good idea, but there are probably far more people solving than checking the forums.

I've made what I think are typical 1,2,and 3 quality puzzles. How about my 11683 Cawabunga! as a 4? It has action and is artistic. It has plenty of white space. The puzzle evokes a feeling. any suggestions?

Once we've chosen a page of anchor puzzles I wonder if we could post a page of solutions as a PDF here that show the anchors side by side.

Ratings are so subjective. I think people are getting way too hung up on them. Personally, I'm just trying to work my way through all the puzzles and typically don't look at them until I've completed the puzzle.

i think its a nice idea but i dont think it will really go anywhere
plus like byrdie said ratings are so subjective
relatively very few people rate anyway and things will likely not change based on a given "template" for rating

there seems to be a few people who:
generally rate high on most everything
generally rate low on most everything

the vast majority of the raters are probably fairly honest and that usually evens things out somewhat

i have seen some fantastic small puzzles that should be rated fairly high but i dont think you can ever convince the masses to ever rate small puzzles as high as large puzzles

Ha. I spend too much time as it is looking at anchors (grading those darned tests takes away from classroom time!) But I'll take a look once you're finished with your project.

Personally speaking, I've tried to be a little more fair to smaller puzzles since I've read posts about that.

My own personal opinion - I don't mind if I get low ratings. I hope for a 3 or better, but if I get lower, I'm okay with that. I enjoy doing small puzzles, that usually get rated a 1 or 2 for quality, and a 1 or 2 for difficulty. On bigger puzzles, yeah, I pay more attention to the rating, but small ones are just fun.

If we carry fourth the analogy of the essay, we grade essays for their quality of thought not the number of words. Why shouldn't an excellent small puzzle get a five for quality?

Nice theory. Maybe it would work if this were a classroom. But it's a realm, and Jan rules. He has a fine philosophy that seems to have worked very well all this time (with one exception).

I agree that many of the ratings don't seem to be made fairly. But, unfortunately, many solvers don't even bother to read the FAQ much less the forum. They are the ones who jump in and start creating puzzles without reading Jan's guidelines, without even honing their solving skills first. Accckkkk!

For those who would like to read up on previous discussions about ratings, read topics #180 and #254 in the regular forum, and #25 and #165 in the archived forum. The last one is MOST interesting. Required reading for all our "students." :-)

I read the threads that Teresa suggested to see if this idea had been thrown out, but didn't see it. I'd like to see some guidelines about rating puzzles. For instance, if someone makes a huge puzzle that is totally line- or color-logic solvable but requires a lot of counting and is very tedious, should that be rated hard to solve? Or should hard to solve be required for puzzles that require something other than simple line or color logic, regardless of size?

Also, how do you rate puzzles for quality? Do you go only by the quality of the finished image, or by whether it's interesting to solve?

Billie, to me quality rating is based on BOTH the image and the solving experience. It's a balancing act: an image filled with colors isn't a challenge to solve, a fun solving puzzle isn't necessarily a good looking puzzle. I also take into consideration the size of the puzzle. My expectations are proportional to the size of the puzzle.

Seeing how many players actually rate puzzles, I am quite disapointed. Maybe if once a puzzle is completed the ratings are set to a default of "3" then players will actually set it to what they really want it to be... What if a puzzle can only be considered complete if it has been rated???

I think since "no rating" is the same as an abstain, and doesn't get averaged at all (rather than counting as a 0 would), I do personally think it's preferable to leave the default at "none" and not require a rating. Defaulting at 3 and requiring a rating would increase the average on the smaller puzzles, sure, but not in any meaningful way. Then we'd just see the glut around 2.5 to 3 instead of 1.5 to 2.5, since most people simply won't bother changing the rating, and will rate EVERYTHING at a 3.

In my opinion, the need of a simple understanding as an experienced user of the site that "smaller puzzles tend to have lower ratings," and knowing that this is for a variety of reasons (more people solve them, many of whom have a less well-defined concept of "good," there simply isn't enough room for the sort of detail you can find in a larger one, etc) is going to continue regardless of how we try to modify the rating system and to "train" people to rate a certain way.

Either the discussion over ratings here is enough to make people rethink how they rate small puzzles (I will admit I have started taking "use of space" into consideration far more frequently than I used to since reading the recent conversations on the topic), or attempting to "train" them is going to go unnoticed as well.

I am not speaking about small puzzles only, all puzzles get low "rating to completion" ratio. How much effort is it to rate a puzzle? What is needed is more participation in the rating process. The greater the size of the rating population, the more useful the rating.

The puzzle is truely complete when the rating is made...

sometimes creating a puzzle is like dipping a girls ponytail in the inkwell... there are teasy puzzles and responses to other puzzles. Most of all this site should be a fun place to hang out.

You are so right Tom,i make puzzles,i solve some,i peak at solutions,i rate some,i make comments on some and i ignore some too.I personally don't pay that much attention to ratings because it is so subjective.If people like my puzzles they will tell me in the comments.I don't make puzzles with hopes of getting high ratings,i just do it for fun.

cool Aldege...i make em cause i HAVE too ... enjoy

I make puzzles because I enjoy solving them. Consequently, I make the kind of puzzles I like to solve. They are small(usually around 20x20), multicolored, not too tough, and usually an interesting perspective or funny result. I'm not fishing for a complement here, lI just hate it when some adorable little puzzle gets a two and a quarter for quality just because it is small. I'm a teacher and think in terms of grades. I ask myself "does this little puzzle really deserve a D?" Where are the As and Bs small puzzles?

I thought I got over this but I guess it still sticks in my craw. Deep cleansing breaths...

Bran...there's a reason for a craw stuck

Not enough bran?

i been calling you bran for a long time now

It's my second favorite misspelling of Brian.

Let me guess, the other is brain.

Of course!

Yeah! I win!

Goto next topic

You must register and log in to be able to participate in this discussion.