Wednesday, September 29, 2010

Reviewing the Reviewers, Part I: Average Score

Opening up the latest issue of Game Informer, I head to the reviews section and am greeted by the following scores:

Transformers: War for Cybertron - 8.5
Toy Story 3 - 8
Singularity - 8.5
NCAA Football 11 - 9
LEGO Harry Potter - 7.25
Limbo - 9
Crackdown 2 - 7.25
Metal Gear Solid: Peace Walker - 8
Dragon Quest IX - 8
Puzzle Quest 2 - 7.75
Persona 3 Portable - 8.25
Disgaea Infinite - 6.5
Snoopy Flying Ace - 8.5
Naughty Bear - 6

The last five games being little one-paragraph blurbs at the end the of the section. Take those away, and the scores range from 7.25 to 9.

What does this mean? Is every single one of those games an above average title worth buying and playing? After all, 5/10 is average, right? (Actually, Game Informer has the courtesy to let us know that 7/10 is considered average to them, while 5/10 is flawed. But with this batch of reviews, the average is closer to 8.)

What is the point of using a 10 point scoring system if you're only going to utilize about half the points on the scale? This isn't just GameStop-owned Game Informer trying to hype up games and generate sales, it is a problem that affects the entire reviewing industry as a whole.

What I'm going to do for now is find out the true average score among all video game reviewers.

The Average Video Game Score
 
Summary: I'm going to gather all the useable scores from the GameRankings database, add up all the points awarded by the reviewers, and then divide by the total number of reviews to find the true average review score from across all the reviews in the database.

GameRankings is a site that gathers review scores from several web and magazine publications. You might have heard of/used Metacritic, which is very similar, but much less useful because it only covers recent consoles, doesn't provide us with the raw scores (instead using its own completely worthless and biased scoring system), and has been reported to take down reviews that the users complain about. GameRankings, while certainly not perfect itself, provides us with a very valuable resource for this experiment: a listing of all review sites in its database, the total number of reviews from each site and the average score that each site gives. You can find this listing here:

http://www.gamerankings.com/sites/index.html

Unfortunately, we can't just use this data: like MetaCritic, GameRankings tries and fails to convert non-standard scoring systems into a universal 1-100 rating. For example, if you click on the first item in the list, 1UP, you'll see that they actually use a letter grade to review games. GameRankings tries to change this into a number grade, where A = 90, B = 80, C = 65, D = 50 and F = 5. How do they come up with this? We don't know, so we can't use such scores. I'm going to go through each individual site and see what kind of scoring method they use, and then throw out the sites that use letter grades.

I don't really feel comfortable using scoring systems that range from a scale of 1 to 5, either. I'll include them in the average for now and might do a separate average later with such sites removed to see if there is a difference.

There is another small problem: some sites change their scoring methods over time. The previously mentioned 1UP formerly used a 1-10 scoring method before switching to a letter grade. I'm not going to look over every reviewer's history to see if they used weird scoring systems in the past. If they're using a standard system now, then all of their reviews are included. If they're using a letter grade now, all of their reviews are removed.

Sample Size: 251 sites, 257,065 total reviews, 18,576,521 total points given when converted to a 1-100 scale.

Here are the results:


SiteLatestReviewsAverageTotal Pts Awarded
2404 - PC Gaming9/13/201025579.56%20287.8
360 Gamer Magazine UK11/1/200714867.23%9950.04
3D Avenue2/14/200753978.78%42462.42
411mania9/28/201044175.52%33304.32
Absolute PlayStation5/21/200474981.40%60968.6
AceGamez9/28/2010322672.84%234981.84
ActionTrip9/27/201082775.90%62769.3
Adventure Classic Gaming9/27/201037071.19%26340.3
Adventure Gamers9/20/201048265.85%31739.7
All Dreamcast6/29/200110477.83%8094.32
AllRPG3/18/200834880.41%27982.68
AntKids.com6/16/200155178.43%43214.93
Archive 642/4/200412575.34%9417.5
Armchair Empire9/27/2010194576.04%147897.8
AtomicGamer9/22/201063082.94%52252.2
BonusStage5/3/200670878.92%55875.36
Boomtown5/17/2010126870.21%89026.28
Cheat Code Central9/27/2010397871.56%284665.68
Cincinnati Enquirer1/25/200643978.52%34470.28
Club Skill10/11/200636376.93%27925.59
Computer & Video Games UK9/29/2010104767.78%70965.66
Computer Games Mag4/1/2007174663.37%110644.02
Computer Games RO9/26/201047880.22%38345.16
Computer Gaming World5/1/2006110962.89%69745.01
Console Gameworld10/2/200851776.80%39705.6
Console Monster9/29/201072873.62%53595.36
Console Obsession9/29/201071173.66%52372.26
Contact Music2/6/200734076.69%26074.6
Cube Europe9/4/200520776.62%15860.34
Cube UK12/1/200543371.90%31132.7
Cubed39/29/201080372.28%58040.84
Cyber Gaming Net12/16/200662873.92%46421.76
D+PAD Magazine9/29/201042869.81%29878.68
Da GameBoyz7/29/201099077.15%76378.5
DailyGame11/28/2009105674.70%78883.2
DarkStation9/8/2010111974.49%83354.31
DarkZero9/19/2010105172.65%76355.15
DC Swirl9/19/200119162.09%11859.19
Deeko8/27/200977275.99%58664.28
Digital Chumps10/1/201056174.35%41710.35
Digital Sports7/28/200545878.71%36049.18
DreamStation.cc7/27/200749074.71%36607.9
DS Central1/29/200526161.61%16080.21
DS-x29/26/201066467.57%44866.48
EAGB Advance11/28/200315872.72%11489.76
Edge Magazine UK10/1/2010146763.76%93535.92
Electric Playground1/10/2007223473.15%163417.1
Electronic Gaming Monthly1/1/2009433267.77%293579.64
ESC Magazine7/3/200642175.05%31596.05
eToychest2/15/200777976.40%59515.6
Eurogamer9/29/2010352366.02%232588.46
Extreme Gamer9/29/201090675.98%68837.88
Final-Level5/4/200536570.11%25590.15
Firing Squad12/1/200732178.49%25195.29
Fragland9/24/2010143975.94%109277.66
Futuregamez9/20/201078578.05%61269.25
G4 - X-Play3/27/2009175263.55%111339.6
Game Axis10/13/200653077.33%40984.9
Game Blitz8/17/200464984.12%54593.88
Game Chronicles9/29/2010298676.06%227115.16
Game Fan7/3/200333477.99%26048.66
Game Helper4/11/200734774.33%25792.51
Game Industry News9/29/2010107980.18%86514.22
Game Informer10/1/2010521373.06%380861.78
Game Over Online9/29/2010255076.38%194769
Game Positive6/15/200915974.83%11897.97
Game Power7/20/200161673.00%44968
Game Power AU1/4/200769577.28%53709.6
Game Raiders3/25/200566579.27%52714.55
Game Rankings7/11/2006114872.72%83482.56
Game Vortex9/29/2010445380.53%358600.09
Game XC1/25/200746379.27%36702.01
GameAlmighty4/29/200833670.89%23819.04
GameBoy Station2/27/200219166.34%12670.94
Gamecell 9/10/2010106672.10%76858.6
Gamecenter2/6/2001108866.34%72177.92
GameCritics9/29/2010133667.48%90153.28
GameDaily11/3/2009223470.36%157184.24
GameFocus9/28/201067376.25%51316.25
Gameguru Mania2/10/200848574.20%35987
GamePen9/4/200250670.16%35500.96
Gameplanet9/24/2010137776.01%104665.77
GamePlasma3/11/201056676.89%43519.74
Gameplayer8/5/200944572.97%32471.65
GamePro9/29/2010440976.70%338170.3
Gamer 2.05/22/2009124076.05%94302
Gamer Limit9/27/201060075.74%45444
Gamer Web6/14/200390072.84%65556
Gamer Within3/18/200858274.02%43079.64
Gamer.tv4/20/200429977.91%23295.09
GamerFeed3/16/200696768.15%65901.05
GamerNode9/28/201086975.00%65175
Gamers Daily News9/29/201038074.07%28146.6
Gamers Europe1/23/200850268.91%34592.82
Gamers Pulse8/8/200482875.94%62878.32
Gamers Wave9/22/200341079.45%32574.5
GamersHell9/13/2010104175.64%78741.24
GamersMark4/15/200991677.93%71383.88
Gamervision9/23/201086973.92%64236.48
Games Are Fun4/27/200966177.40%51161.4
Games First!12/5/2007129470.79%91602.26
Games Master UK10/1/2010388772.42%281496.54
Games Xtreme9/20/201053579.31%42430.85
GameSpot9/28/20101142467.40%769977.6
GameSpot UK (Pre-2003)8/20/2002112773.47%82800.69
GameSpy9/22/2010482969.92%337643.68
gamesTM10/1/2010163565.87%107697.45
Gamestyle9/29/201065065.77%42750.5
GameTap5/27/200827873.31%20380.18
Gametrailers9/29/2010133375.84%101094.72
GameZone9/28/2010654275.13%491500.46
Gaming Horizon10/14/200782675.38%62263.88
Gaming Illustrated7/26/200738981.67%31769.63
Gaming Maxx4/15/200432681.13%26448.38
Gaming Target9/29/2010193877.44%150078.72
GamingExcellence9/28/2010111973.33%82056.27
GamingTrend9/27/2010129179.57%102724.87
GamingWorld X9/16/200450277.75%39030.5
GD Xbox12/18/200321772.33%15695.61
GF3K1/23/200432578.21%25418.25
Giant Bomb9/23/201043070.51%30319.3
Globe Technology2/1/200648771.29%34718.23
GMR Magazine2/1/2005117372.64%85206.72
GotNext7/5/200871672.25%51731
Hardcore Gamer Magazine9/27/201095571.46%68244.3
HonestGamers9/23/2010162068.00%110160
Hooked Gamers9/29/201053076.74%40672.2
Hot Games8/19/200396073.24%70310.4
Hyperactive7/29/200346080.62%37085.2
IC-Games7/6/200867277.49%52073.28
IGN9/29/20101202568.41%822630.25
ImpulseGamer9/29/201037980.88%30653.52
Incite Games6/23/200031869.37%22059.66
Independent Gamer9/22/200330768.36%20986.52
InsidePulse9/13/200654165.03%35181.23
Irish Player2/16/200330977.73%24018.57
JIVEMagazine2/20/200831277.61%24214.32
Just RPG11/28/200743280.30%34689.6
Kikizo9/26/201060876.24%46353.92
Kombo9/22/2010157975.32%118930.28
Lawrence4/5/2010120477.54%93358.16
Loaded Inc6/15/200663273.36%46363.52
Mad Gamers8/22/200338274.09%28302.38
Media & Games Online2/5/200345676.47%34870.32
Mobile GameFaqs UK12/12/200618081.82%14727.6
Modojo12/18/200963264.21%40580.72
MS Xbox World9/29/201087775.67%66362.59
Multi-Player Online Gaming1/17/200232967.15%22092.35
Mygamer9/29/2010125372.29%90579.37
Netjak9/25/2008105071.65%75232.5
N-Europe9/22/201046273.98%34178.76
Next Generation1/1/1999100162.64%62702.64
Next Level Gaming3/25/2007185977.82%144667.38
NGamer UK8/1/2010109660.87%66713.52
NGC Magazine UK6/1/200657167.20%38371.2
N-Insanity1/22/200731576.86%24210.9
Nintelligent Network1/17/200616075.07%12011.2
Nintendo Gamenet9/27/200512186.32%10444.72
Nintendo Insider5/15/200625974.15%19204.85
Nintendo Life9/26/201082365.17%53634.91
Nintendo Power10/1/2010334268.01%227289.42
Nintendo Spin7/6/200823675.61%17843.96
Nintendojo6/8/2010102773.28%75258.56
NintendoWorldReport9/22/2010119369.65%83092.45
N-Revolution Magazine UK12/5/200819670.33%13784.68
NTSC uk5/18/201086266.83%57607.46
NZGamer9/28/2010112577.64%87345
Official Nintendo Magazine UK10/1/201093169.81%64993.11
Official Playstation Magazine7/1/2008218563.67%139118.95
Official Xbox Magazine11/1/2010167770.55%118312.35
Official Xbox Magazine UK10/1/2010137668.53%94297.28
Operation Sports12/18/200721573.26%15750.9
P2 Magazine UK12/1/200566470.72%46958.08
PALGN9/28/2010191069.18%132133.8
PC Accelerator1/1/199918763.64%11900.68
PC Format UK9/1/2010109768.20%74815.4
PC Gameplay10/9/200219573.69%14369.55
PC Gamer11/1/2010181968.16%123983.04
PC Gamer UK10/1/2010181067.18%121595.8
PC Gameworld10/3/200877476.11%58909.14
PC Power Play10/1/201021070.24%14750.4
PC Zone UK10/1/2010239467.68%162025.92
Pelit (Finland)9/1/200878081.54%63601.2
PGNx Media10/6/2009150283.99%126152.98
Planet Xbox 3609/29/201048176.85%36964.85
Play Magazine5/1/2008208276.36%158981.52
Play UK10/1/2010172172.49%124755.29
play.tm9/28/2010102773.87%75864.49
Playstation 2 Mag UK7/1/2008142863.07%90063.96
Playstation 2 Max UK9/27/200456071.54%40062.4
PlayStation LifeStyle9/29/201031072.16%22369.6
Playstation Official Magazine AU8/1/201019872.63%14380.74
Playstation Official Magazine UK10/1/201078265.28%51048.96
Playstation Official Magazine US10/1/201045669.43%31660.08
Playstation Universe9/24/201028576.95%21930.75
Pocket Gamer UK12/18/200969666.08%45991.68
Portable Review4/24/200842273.47%31004.34
PSE Magazine5/1/200681885.91%70274.38
PSM Magazine12/1/2007178770.40%125804.8
PSM3 Magazine UK10/1/2010181968.45%124510.55
PSW Magazine UK6/26/200960969.11%42087.99
PSX Extreme9/27/2010127375.06%95551.38
PSX Nation5/19/2005121771.64%87185.88
Quandary5/3/200843272.54%31337.28
RealGamer9/29/201056971.04%40421.76
RewiredMind4/5/200749267.30%33111.6
RPGamer9/27/201049367.25%33154.25
RPGFan9/29/201072781.37%59155.99
Sega-169/27/201060363.96%38567.88
Silicon Fusion7/29/200736777.05%28277.35
Sports Gaming1/25/200647278.42%37014.24
Talk Xbox12/10/200937278.78%29306.16
TeamXbox9/27/2010141374.90%105833.7
TechTV5/9/200494865.32%61923.36
The Entertainment Depot9/27/201064169.20%44357.2
The Gamers Temple9/29/2010155973.26%114212.34
Thunderbolt9/29/2010176270.23%123745.26
Total PC Gaming5/1/201020368.67%13940.01
Total Video Games9/4/2009110775.72%83822.04
TotallyGN9/27/201041672.02%29960.32
TotalPlaystation9/29/201078275.44%58994.08
UltraNinjas6/7/201039176.32%29841.12
VGChartz9/28/201061676.81%47314.96
VG-Force1/4/200654277.27%41880.34
VGPub11/30/200749977.02%38432.98
Video Game Talk1/11/201085266.73%56853.96
Video Gamers First8/27/200555775.59%42103.63
VideoGamer9/29/2010209369.91%146321.63
VideoGames NZ4/6/200948781.24%39563.88
Videogameslife9/27/200555965.97%36877.23
WonderwallWeb2/23/201073378.54%57569.82
Worth Playing9/29/2010362772.32%262304.64
X360 Magazine UK10/1/201077264.14%49516.08
XBM UK12/1/200545267.79%30641.08
XBN Magazine2/1/200554759.34%32458.98
Xbox 2k7/21/200311277.02%8626.24
Xbox Evolved7/22/201054773.85%40395.95
Xbox Exclusive1/4/200716881.58%13705.44
Xbox Gamer Mag UK4/1/200542664.25%27370.5
Xbox Impact4/8/200413577.25%10428.75
Xbox Maniacs2/2/200417776.80%13593.6
Xbox Rules1/30/200515075.00%11250
Xbox World 360 Magazine UK11/1/201061162.75%38340.25
Xbox World Australia9/29/201038477.68%29829.12
Xbox World Mag UK6/1/200820966.32%13860.88
XboxAddict9/24/201060979.69%48531.21
XboxSolution12/22/200624280.82%19558.44
XboxZone3/5/201017777.39%13698.03
XGP Gaming11/13/200632182.31%26421.51
Zentendo4/28/201023872.02%17140.76
ZTGameDomain9/29/2010142577.68%110694
SUM25706518576521.35
TRUE AVERAGE72.26390738

The average score when throwing out the letter grade sites is a 72.26 out of 100. Including the letter grade sites (using GameRankings' conversions), it is a 72.45.

So the next time somebody tries to tell you, "Hey, it got a 7 out of 10! It's a good game!," you can tell them to sod off.

Then again, it depends on the reviewer in question. Some reviewers give out an average of 8, others a 6. That's probably what I'll go over with the next update.

No comments:

Post a Comment