Web Paint-by-Number Forum
Topic #68: Rating Puzzles
By Arduinna (arduinna)

#1: Arduinna (arduinna) on Oct 25, 2007

Jan, I have to admit I didn't read your to-do list, so maybe this is on there.

It seems it would be helpful, following your creation of the uniqueness-checker, to update the ratings system. It is arguable that any puzzle with a unique solution may be solved "by logic alone," depending on our definition of "logic." But there is a difference between puzzles solved by following the lines, those solved by edge-logic, and those solved by trial and error. I'm not sure how other people feel, but I think it would be nice to know what "type" of logic is necesary for each puzzle.

I'm sure I'm not being too demanding because I know that you're Superman, right? ;)
#2: Robert Kummerfeldt (rmkummerfeldt) on Oct 25, 2007
We've touched on this debate before. The question marks work good for me to indicate the level of logic you are looking for. No question marks usually means it can be solved with 'normal' logic. The light gray question marks are the ones I look for when I want a challenge - these all have unique solutions, but are not for the faint at heart. Basic row and edge logic usually doesn't suffice. What Jan flags as trial and error, I translate as requiring some sort of elimination logic (trial and error is one method I use to eliminate squares, but not the only one). I don't think you'll find any dark gray question marks - much guessing with 1 solution - at least not once the ratings have accumulated or Jan makes the final ruling. The red question marks mean multiple solutions - light red means the correct solution should be easily decipherable, and dark means enter at your own risk and leave your logic at the door.

Ok - that should be the groundwork for this topic. Now to the real discussion that arduinna brings up - is there a better way to define the ratings system? I like having separate quality and difficulty ratings, but I think the quality rating needs a redefinition (at least in my head). I use it for my overall opinion for the puzzle, combining both artistic and difficulty value, but I don't think this is fair to the beautiful but relatively easy puzzles. I find I never rate a puzzle higher than a quality of 3 unless it makes me go beyond basic logic. I know I have rated the quality of bad looking but challenging puzzles higher than eye pleasing but easy puzzles.

I've been trying to give more weight to the artistic side for the quality rating recently, but I still reserve a 5 for the 'perfect' puzzle.

I know others have a different interpretation of the quality rating, which makes it objective and not as useful as it could be. If the general community agrees that the quality should be artistic only, I can change my thinking and go with the consensus.

After all this, I don't think I responded to arduinna's question very well. I think the combination of question marks and difficulty ratings are a good indication of the level of logic. I use the basic difficulty scale when I rate puzzles:

1 = no or little white space
2 = basic row logic suffices
3 = row logic and some advanced logic (ie: edge logic, color logic)
4 = more advanced logic than row logic, but fairly straightforward
5 = wow - that wasn't easy!!

Maybe just changing the difficulty dropdown list to give a more specific definition to the number of stars would give a better interpretation to difficulty, but those who use the 'intuitive' guessing approach (I like that word) wouldn't be rating the same way.
#3: Jan Wolter (jan) on Oct 25, 2007
Yeah, an awful lot of these difficulty/quality things are under-defined.

But I think coming up with really good definitions is nearly impossible. Robert's are pretty good, but is a big puzzle more difficult than a small puzzle? Is a puzzle where each line has lots of small clue numbers harder than one where most lines have just a few larger puzzles? Is black/blue puzzle harder than a red/green puzzle?

Some people, like Robert, solve strictly by logic. But other people are pretty willing to throw in a guess based on guesses of what the image might be. Their assessment of difficulty might be affected by how obvious the subject of the image is early along.

I tend to rate "quality" based on how much fun I had solving the puzzle. A nice image contributes to the fun. Interesting solving does too. Puzzles that are too easy or almost completely impossible tend not to be a lot of fun for me. But basically it's how much I envy the puzzle and wish I had done that.

I don't think any one definition is ever going to work for everyone. But I think that's OK. It all averages out in the end. The highest rated puzzles on the site aren't necessarily my personal favorites, but they are all recognizably good puzzles.

I do like the idea of a separate "solvability" category for puzzles that are solvable by line logic alone. But before I do that, I'd probably want to upgrade the pbnsolve program so that it has a full line solver. Then it could make that determination automatically.

Note that some puzzles that can be solved by line logic alone are still very hard, because everything depends on finding one very non-obvious pixel. Some of those are much harder than the ones that just need some color logic.

Also I sometimes use edge logic or color logic even in puzzles that don't necessarily need them, because they give me a big leg up. Sometimes I see a puzzle with a 20 in the first column and some 2's in the second column and the first thing I do is edge logic. But having done that, I'll never know if it is solvable without edge logic.

You know, sites like Netflix have people rate movies and average all those ratings together. The result is almost totally meaningless, because (1) everyone who rates has different criteria, and (2) people who hate Sci-Fi tend not to rate sci-fi movies because they tend not to watch them, so in a way it ends up being a rating of how the movie fulfilled the expectations of the people who choose to watch it rather than a rating of raw quality. So the ratings should be garbage. And yet, they are still useful. It's kind of amazing, really, but I think one should resist the temptation to over-analyze these things. Take the ratings with a grain of salt, rate things the way things feel right to you, and don't worry too much.
#4: Nancy Snyder (naneki) on Oct 25, 2007
my rating for quality fall in the "how realistic, how detailed, how pleasent it is to my eye or if you can tell that someone has put alot of effort (heart) into making the puzzle. Sorry to admit that if it's quick & easy I rate average or lower (I would rate a couple of mine in that catagory) I'm tough on myself so I guess I'm tough on others as well.
#5: J.C. Anderson (jc.noserdna) on Oct 25, 2007
My god, what would Mr. Spock say?
#6: Nancy Snyder (naneki) on Oct 25, 2007
is that Dr. Spock or Spock from star trek?
#7: J.C. Anderson (jc.noserdna) on Oct 25, 2007
I'm going to have to let you re-read my question, young lady, and this time read ALL the words -- OH, sorry, just watching a HOUSE rerun, must be channelling. Oops.
#8: Nancy Snyder (naneki) on Oct 26, 2007
JC .. 8-P
#9: Arduinna (arduinna) on Oct 26, 2007
Wow-- tired brain. I couldn't figure out what 8P meant!
#10: Nancy Snyder (naneki) on Oct 26, 2007
lol..I think you need a break
#11: J.C. Anderson (jc.noserdna) on Oct 26, 2007
Whats the difference between roast beef and p soup? Anybody can roast beef.
#12: Gypso (Gypso) on Oct 26, 2007
I remember this question coming up awhile ago. I now rate with puzzles with great ease as Jan suggested then and again here. As it feels right at the time and I don't worry about it. It's suppose to be fun after all... :)
I am a bit more generous with first time posters and even more so with puzzles published by children. A duck may be somebody's mother. *>*
#13: Arduinna (arduinna) on Oct 27, 2007
I agree with the "all evening out" thing, especially on quality. I was really just wondering about the "completely solvable by logic alone." It seems there are more definitions of "logical" than I at first realized!

Mr. Spock would indeed be displeased, JC!
#14: Jan Wolter (jan) on Oct 27, 2007
I think my definition of "logical" is that a person could solve the puzzle in pen, without having to look more than two or three moves ahead in his or her head.

#15: Gypso (Gypso) on Oct 27, 2007
Interesting Jan. I hadn't considered that definition.

You must register and log in to be able to participate in this discussion.