A Search for Consistent Ratings
As each day brings new experiences, each of them are placed along a personal hierarchy of value through little additional effort of our own. A good book you have read, a bad meal you have eaten, a nice show you have watched, and a completely mediocre carpet you have stepped on. Each of these have their own merits and offer to us various types of opinions and conversation starters. Even in fields we aren't experts in, we seek to quantify our enjoyment of various things through the use of ratings. Over the years, I have come up with a rating system that remains largely consistent over my experiences. In this over-systematizing search for consistent ratings, I've made a few observations about the idea of rating things as a whole that I feel are worth thinking about.
What are we rating, really?
Sometimes, it isn't even for enjoyment; oftentimes enjoyment takes the back seat and our understanding of "objective" artistic quality comes in. I think it's a universal experience for something to appear better than something else, while somehow being less enjoyable. That free jazz that fuels your pretension might seem like it's more artistically fulfilling, but is it really more enjoyable than that catchy song you heard once on the radio and can't remember the name of? Maybe, but what if it isn't?
There is legitimate value in looking at artistic merits of a particular experience, even if it doesn't play a large role in your enjoyment of it, but it's easy to care more about artistry in some aspects than in others, and that's especially based on your knowledge and life experiences. Consistent ratings should probably focus entirely on enjoyment, and any artistic merit should probably only be considered if it plays a role in said enjoyment.
Why are books more highly rated than albums?
In the crowd-sourced network of knowledge known as the Internet, ratings play as essential role in indicating value, despite its several problems. A good example of how important ratings are is how YouTube's removal of dislikes has resulted in major wastes of time for many people who are just looking for a fix for some problem [source: me]. Also, when's the last time you trusted (extremely inflated) Google ratings to go to an attraction or restaurant? Even when knowing it's a bad indicator, I still find myself tempted to at least consider it in the decision-making process.
Google's ratings for places are generally pretty inflated, and this inconsistency is found everywhere. Consider the three review sites Goodreads (for books), Letterboxd (for movies), and RateYourMusic (for music), and how a 3.7 would be perceived in each of them. In Goodreads, a 3.7 is well below average, and oftentimes disincentivizes me from reading a book even though a 3.5 is a "good" rating to me. In RateYourMusic, a rating of 3.7 would place an album among the top 30 for year. In Letterboxd, a movie rated 3.7 is nothing much to write home about.
I'm not entirely sure why this difference exists, but I have a guess. Since rating something feels only fair if it's something you've finished, and since bad books are much more likely to be abandoned due to the sheer amount of time it takes to read them, books will generally be rated highly. In contrast, an album or EP is easy to get through in less than a hour (generally), so it is more likely to have negative reviews. Movies are in the middle with their two-hour(-ish) runtime.
However, this doesn't explain why Google ratings or ratings on sites like IMDB are extremely skewed. Perhaps, as we move towards less hobbyist sites, it's more likely for ratings to be distributed in a U-shape, with maybe 0's and many 5's (if we're talking about a five-point scale). After all, what is 74% enjoyment, actually?
74% and the collapse of precision
Can we really break down our understanding of our own enjoyment to such a high level of precision? Can we really quantify any type of emotional response?
While it might be a controversial opinion, I do believe that we can, to a certain extent. Personally, between two things that I would give the same score to, I would usually prefer one of them over the other. Our subjective understanding of quality does vary with a high level of precision. However, beyond a certain point, precision just loses meaning because of the effect of variations with time. If I found Disney's Up to be 68.26% enjoyable at a given time, that will probably change a few seconds later. Even as I'm typing this, I'm remembering how good the introduction was and I feel like bumping my (false) rating up.
To be consistent, ratings need to be only quantized to the point where there is little variation across time. However, this needs to be balanced with having a useful measure of relative quality, or else we'll all just end up with a binary system of classification that has no personal usefulness.
Of course, are ratings really meant to be useful, or are they just meant to be fun? That's another conversation to be had. I'm interested in learning about how people actually handle ratings, and what ratings actually mean to them. Importantly, what kind of scale do you use? How do you benchmark it? Do you care about consistency across different mediums? Does enjoyment or artistry play a greater role? Do you rate several aspects of a particular experience and bundle it together, or rate everything in unison?