# Machina’s Machinations: Numbers That Mean Nothing

To those of you who weren’t eaten by Sharks last week, welcome back to Machina’s Machinations… and for those of you who were, I express my deepest condolences. Last week I spoke about the decision process of game development companies (found here: Beneath The Bottom Line) and this week I’m going to continue where I left off by speaking about something completely unrelated.

It’s not much of an assumption to make that if you consider yourself enough of a gamer to read gaming blogs like eGamer then you must read gaming reviews as well. Whether you read reviews in magazines (like NAG), websites (like GameSpot), stores (like Take2) or on the back of soft drink cans (like Coke), the format tends to be pretty similar. The reviewer types a long story about their experiences with the game and then ends off the review with one thing: a number. And using this number in conjunction with the other numbers that the various game reviews give you, you now need to make the difficult choice of: which games do you play and which games don’t you play? Because simply put, you don’t have the time and the money to play everything – for those of you who do have the time, get a life, for those of you who do have the money, well played – and I, personally, don’t believe that a number is enough to help you make the difficult decision of which games to buy and which games not to buy. So in this article, I’m just going to discuss a few of the reasons why the number review system is a failure.

- You’ll Be Able To Play (Expensive) PS2 Games On Your PS4 Now | 2 months ago
- Jessica Jones Disempowers Its Male Characters And The Effect Is Refreshing | 2 months ago
- Hell Is 30 000 Deathclaws Tearing Through Boston And It’s Glorious | 2 months ago
- Sony Santa Monica Is Teasing Something Truly Strange | 2 months ago

P.S. For those of who complained about the length of my article last week, I’ve remedied it by making this week’s article even longer…

The first problem that I’d like to discuss with regards to the number system is interpretation; and believe me, it’s a lot more abstract than you’d initially think. Now for the rest of the article, I’m going to assume that games are reviewed on a scale of 1 to 10 with scores out of 100 simply being a number from 1-10 with an attached decimal point. It might seem fairly simple to most people that rating something on a scale of 1 to 10 indicates how good or bad the game is but what’s *never* made clear is: what exactly is being quantified here? Does the number from 1 to 10 represent the expected ‘fun’ that will be generated from playing the game? Or how good it is from a technical and objective perspective? Does it represent the likelihood that you’ll enjoy the game? Does it represent how close the game is to perfection? Or does it perhaps represent what quality class the game fits into? Now any single one of these interpretations would be useful as long as it’s clearly stated which of these the number in question is referring to, but, as I stated above, it never usually is.

If this still isn’t clear to you then I’ll illustrate it with an example. Let’s take the number 9 for example. 9 out of 10, 90 of 100, 90%. If you get 90% for a Maths Exam, you display a near perfect mastery of what has been taught to you. If you get 90% of people to vote you in as president, then 90% of the voting population consider you to be the best out of all available candidates. If RottenTomatos tells you that a movie was liked by 90% of critics, then you know that there’s a very high chance that you’ll enjoy the movie. If you know that eating Smarties has a 90% chance to give you AIDS, then your chances of walking away free are close to non-existent. If you own 90% of the shares in a company, then it’s pretty much your company. If the judges rate you 9 out of 10 on idols, it means that the quality of your singing is close to perfect. Now in all of the above scenarios, there’s only one number, the number 9, but in each of them, the meaning is completely different. The same can apply the number ratings for games, in that it’s not perfectly clear what’s being indicated.

Now those of you who aren’t following might say “The number tells you how good the game is, what’s so hard to understand about that?” and I’d respond by saying that it isn’t all that hard to understand. What’s hard to understand is what exactly you mean by ‘good’. The word itself is so broad and vague and has so many interpretations itself that using it in a definition doesn’t really clear anything up. But all the above is simply semantics. The real problem lies in the next three sections, all of which end up coming back to the root problem of interpretation.

The second problem that I’d like to address is called relativity, and it’s a lot more obvious than the first one. I’ll illustrate using a simple example: if every single game ever made was rated 1 out of 10 then a game which was rated 3 out of 10 would be considered to be fantastic even if it’s only rated 3 on the number scale. And this problem is actually a lot more apparent than you think; usually with the numbers 7 and 8. Since it isn’t clearly defined what the numbers represent, the number 7 is more or less supposed to mean decent or playable while the number 8 usually means fairly good. Now the problem with this is quite obvious, the median score for games tends to be 7 rather than 5 and because of this, the number 7 literally does mean ‘average’ when rating games. I, personally, would not pay R700 for what I consider to be average so when websites rate games 7 or lower, I tend to avoid them, which is much more of a problem than you think. You see the problem is that the number 7 fits on the higher spectrum on the 1 to 10 scale. 5 is actually the average of all integers from 1 to 9, while 7 is actually in the upper quartile (read above average). The way games are rated nowadays seems to suggest that *most* games are *above average*, which actually makes no statistical sense when you think about it.

The next problematic number in the whole relativity number is 8. Now on standard 1 – 10 number scales, 8 is supposed to mean really good. It doesn’t matter what foreign country you come from, 8 is on the upper end of the number spectrum and it’s only one step down from the highest reasonable score – the number 9; I’ll discuss the problems with 10 in a moment. It therefore follows logically that if you buy a game rated 8 out of 10, you shouldn’t just be satisfied, you should be pretty damn elated. Regardless of what your number scale represents, 8 is supposed to exist for games that are *better* than the rest. But if you read game reviews then you already know that this isn’t the case. Just to test my theory, I went to GameSpot.com *today *and clicked on latest reviews. Of the 10 latest reviews, there was one game rated 5, three games rated 7, *five *games rated 8 or higher and one game rated 9. Now there’s obviously a problem with the system when *nine* out of *ten* games reviewed fall into the 7-9 category. The problem is that 8 no longer represents truly great games, only games that are above average or are moderately good. And the problem with this is that 8/10 is no longer any kind of guarantee. You can’t be sure if a game that scores 8 is good, nor can you be sure that you’ll even like it.

Another problem with 7 and 8 being so prominent is that the numbers 1-6 have become almost useless in the 1 to 10 scale. Since most people are already skeptical of games with a score of 7, a game that gets 6 or lower is mostly considered to be unplayable. For this reason, the *majority* of the 1 to 10 scale represents essentially the same thing: a bad game with almost no distinction between any of the scores. In the end, there’s probably no difference whether you rate a game 1 or you rate a game 4, you’ve essentially grouped it into one single category.

And that just leaves the numbers 9 and 10. Now, for the most part, the number 9 seems to be done right. 9, in general usually does represent exceptional games that are the of the highest caliber in their respective classes. Now due to the failure of the number 8, there are some games worthy of an 8 that usually end up leaking into 9, but the problem doesn’t really lie with 9 to begin with. No, the next problem actually centers around the number 10, the highest possible number on the 1 to 10 scale. Theoretically a game that scores 10 out 10 *cannot* score higher and thus, we must objectively conclude that there’s nothing that could be done to make the game better. I’ll illustrate it with this example. Let’s say that you can choose between ‘getting your car stolen’ or ‘getting free sandwiches’. Now it doesn’t matter how you look at it, option 2 is *unambigously* better, there is no debating this. So if we rate Option 1 on the number scale at, say, 7 out of 10, then Option 2 must then get a rating of either 8, 9 or 10. But if we rate Option 1 at 10 out of 10, then there’s absolutely no way we can rate Option 2 higher without breaking the number scale and so we have to conclude that they are equal… which makes no sense.

The same logic can be applied to rating video games. When you rate a game 10, you are essentially claiming that it’s so good that it cannot be improved upon. And depending on the interpretation of the number scale you could also be claiming that everyone will like it. Now if you look back at GameSpot’s archive to the games they’ve rated 10 in recent times you’ll see Super Mario Galaxy 2, MGS4, GTAIV. I haven’t played Mario Galaxy 2 but I have played MGS4 and GTAIV. I thought MGS4 was fantastic but this was conditional on me being an MGS fan. And I honestly thought that GTAIV was a bucket of shit that felt more like work than actual fun. But since the game is rated 10 out of 10 and is considered to be perfect, GameSpot is implying that he problem lies with me and not the game in question. I could go on about the number 10 for another 2000 words or so but I’m too lazy so instead I’ll say that there’s a very good reason why games should *not* be rated 10.

Now if you’re still alive after reading that giant wall of text, then I have but one last point to address before I go back to solving Rubik’s Cubes and trying to invent Human-Flavoured Bubblegum and that’s the issue of comparison. Thus far, I’ve mostly talked about the numbers in isolation but the truth is that the number system only becomes more and more unintuitive the longer you use it. One of the biggest reasons *not* to use the number system is the impression it creates when you compare one game to another. Now there are two problems here: first the issue of comparing one score to another based on numbers and secondly the issue of comparing one game to another via the scores.

So let’s start with comparing scores. Now as I’ve already explained above, there’s enough of a problem when you try to distinguish between games rated 7, 8 and 9 when you’re not sure about what the numbers are supposed to represent but the whole thing becomes a mindf@#k of epic proportions when you add in decimal points. You know what I’m talking about, the biggest problem is when game reviewers use their infinite wisdom to rate a game 8.1 instead of 8. When this happens, I just scratch my head in amazement and say ‘what?’. Now I hold a triple PhD in Mathematics, Applied Mathematics and Statistics (ok not really) so it must be quite a big deal for numbers to confuse me. Let’s say, for example, that you ask Mr Genius reviewer to explain the difference between a game that deserves 8.0 and 8.1. Most likely, he can’t. Then ask him to explain the difference between 8.1 and 8.2 and it’s even less likely that he can explain that. When you mark someone’s Accounting Test, the difference between 81% and 82% is a physical tick on a page that corresponds to a question that you got right. When it comes to rating a game, that 1% difference is a non existent subjective anomaly that you suck out of your thumb. This becomes more of a problem when you realize that while there’s no difference between 8.0 and 8.1, no difference between 8.2 and 8.3 and no difference between 8.4 and 8.5, there’s quite a noticeable difference between rating a game 8.0 or 8.5. If this doesn’t seem like a problem to you, then you’re either drunk or are a communist… neither or which is appropriate for readers of this column.

Now the second problem is when you actually take two games and try to rank them against each other using these scores. Now if you take the number system from 1 to 10 and you’re confined to rating playable games 7,8 or 9, you’re going to end up saying that a hell of a lot of games are more or less equal to each other. My theory is that the decimal point system exists to make up for this terrible failure by making the system fail even harder. It’s one thing that you can’t explain the difference between 8.1 and 8.2 but it’s quite another thing when your system implies that a game rated 8.2 is *better* than a game rated 8.1 and you can’t explain it. Because that’s what the number system suggests. The whole point of the 10 point scale is that each grade is supposed to be unambiguously better that the grade that precedes it, because if it isn’t, then the entire point of rating games in the first place falls away. So the problem surfaces when your number system just ends up breeding inconsistencies and shooting itself in the foot repeatedly.

Let’s use IGN – because I love insulting those Wii-loving Nazis. If you read their review of MGS3: Snake Eater they claim that it’s the best Metal Gear Solid title they’ve ever seen and they rated it 9.6. This is actually quite funny when you consider that they’ve rated MGS2: Son of Liberty 9.7. Now you might argue that the score is more relevant for its time and whatnot but it’s an inexcusable joke when you find out that they rated MGS2: Substance 9.1 when it’s *the same game but with more features*. Intuitively the number system is actually saying that the game was 0.6 decimal points *better* when it had *less* content which is not really as simple as it sounds. If I was someone who just looked at the scores before deciding to buy a game I would conclude that Sons of Liberty must be *better* than Substance and then go on to buy the game which has less stuff. It’s like someone offers you the choice between a ticket to a soccer match seated next to a rabid Sabretooth Tiger or the same ticket to a soccer match but seated next to Carmen Electra and you choose option 1.

The sad part is that all of this could have been avoided by rating each of the MGS games 9 out of 10 and then just pointing out that they’re all the same quality class but Substance is an improved version of Sons of Liberty while Snake Eater is the best game thus far.

So in the end, when you try to compare games using the number system there’s always going to be discrepancies. It’s enough work to distinguish which class of quality a game fits into and this problem only becomes that much worse when you introduce decimal points.

So in conclusion, I hope that I’ve at least gotten you a bit more skeptical of the 1-10 rating scale when it comes to summarizing reviews and rating games. You should think harder before you trust a single number to help you decide which games to buy and which games not to buy. eGamer has taken a step in the right direction with their summary box but I would only rate their system an 8 out of 10 because 9 belongs only to exemplary world-class rating systems while rating the system 10 would imply that it cannot be further improved… which would put me in quite the predicament should a better system come along. The truth is that games can be rated on so many different levels that there probably can’t be a perfect universal system for each game… but that doesn’t mean we shouldn’t try to create one.

Till next week, here is a cat… also Portugal sucks…

- The_Lurch
- ipull