Sometimes blog posts just don't turn out the way you want them to. Whether due to the content or the implementation, they'll often end up smashing through the central reservation of the Good Intention motorway, hurtle down the Mediocrity slipway and end up in Dumb Idea ditch, back wheels still spinning.

This is what happened with the firebrand titled "Does modern anime suck?", a post from one year prior which attempted to use Science™ to prove that modern anime doesn't suck and the familiar cries that echo every quarter of a year claiming as such were flat wrong.

That was the theory, the practise turned out to be less than convincing.

I am no statistician which presents issues when I deal with anything more vexing than averages, variance and standard deviations which, for the most part, has carried me happily through life so far. Since I wrote the post I'm slightly more genned up and phrases like "bimodal distributions" no longer elicit a vacant stare. This of course doesn't solve the issue of normalising the data that I had obtained for the post.

A story unto itself, I've always been a strong proponent of open data which means when the big three anime data sources: Anime News Network's Encyclopedia, AniDB and MyAnimeList do not make their general data available in a meaningful format, I take umbrage. Sure AniDB and MyAnimeList have APIs but the former is archaic and the latter next to useless for what I wanted it for.

So screenscraping it was. This got several blobs of data but with no deep knowledge of how to meaningfully interpret it, I plonked it unceremoniously into graphs and hoped the result would speak for itself.

It didn't.

I have a strong scientific background so I know the argument that regardless of the outcome, if the process was thorough and structured the results are unquestionable. This was not science though and the results made for nuanced rather than sensationalist reading. And when I say nuanced I of course mean dull.

Despite being entirely content complete, I abandoned the post for the reason that it didn't meet my own quality standards due to the flaky research and lack of interesting angle on the subject. It read much like it the small project had progressed: a disruptive idea let down by lack of knowledge but hey, graphs right?

I present it here as a way to put to sleep a topic which I don't have the resources to do justice to. Perhaps it will inspire someone more learned than I in the dark and arcane arts of statistics to make a more meaningful stab at penetrating the issue. Or perhaps it is a cautionary tale on the hubris of "big ideas". Regardless, the original thrust of the post was to bring a different voice to the dreary and insipid cries which travel along with each new season of anime like an STD. It certainly would have brought a different voice to the caterwauls, it would have been just as screeching and ill-informed though.

--

Last draft of original post: 16th June 2011

Sparked by a conversation on Twitter on Monday about the seasonal calls of "next season looks rubbish!" and "I'm watching twenty shows now, only three next season look any good", I mused as to whether there was a way to canonically prove one way or another whether anime is indeed getting worse as time went on. Or perhaps whether other more mundane psychological reasons are to blame for this repeated inanity like recency bias, nostalgia or plain old attention seeking behaviour.

With little idea as to what the outcome would be, I started parsing data, crunching numbers and generating graphs. So does modern anime suck?

Short version: no. Longer version: eh...

Average score of anime per year from 1970 - 2010

If you want the take-home message without any of the data analysis then the above is it. Average scores for anime have been on an upward trend for close to the last decade, peaking in 2008 then remaining steady for the last two years. Please bear in mind the scale of the graph: the lowest is 6.44, the highest is 7.24 meaning there is only 0.8 between the scores of the "best" years and "worst" years.

Average score of anime per year from 1970 - 2010 broken down by source

Please note: no data was available for 1970 with AnimeNfo

This data comes from four main sources: Anime News Network's Encyclopaedia, MyAnimeList, AniDb and AnimeNfo and is correct as of 13th June 2011. The publicly accessible scores were aggregated along with the start / end dates and the type of show (movie, OVA, TV series etc.). Due to the different weights offered by sources such as ANN and AniDb, the one collated across all of them was the weighted mean. Separating out the sources shows, with the exception of AnimeNfo (more on this later), the scores more or less follow one another in terms of trend, MyAnimeList users are most generous while ANN users are unilaterally harsher.

AnimeNfo is counted in the main graph and the source graph but no further as the site seems to have been all but abandoned. End dates for relatively recent anime such as Fullmetal Alchemist: Brotherhood are missing and the number of reviews is pitiful, even for popular series such as Bleach (182 reviews versus MAL's 105,954). There is also the issue of earlier shows (pre-1985) having wildly varying review scores which is likely the reason for the erratic early years in the main graph.

Average score of anime TV series per year from 1970 - 2010 broken down by source

Narrowing the data set down even further to only include TV series answers the question originally posed: is anime getting worse? Year-on-year is a mixed view, good years and bad years, so 2000 vs. 2001, 2002 vs. 2003 the latter years win in terms of average score. Going by individual source scores rather than the overall one is a mixed picture. MAL and AniDb have a definite trend upwards with MAL not seeing any huge deviations; ANN on the other hand puts the past five years as only just catching up with the 1970's.

Though the scores are different, all sources seem to agree on the good years and bad years though. So 1997 was a poor year (Evangelion hangover?) with a precipitous drop from the year before and one that it only just seems to be recovering from but with a solid peak in 2008 (post Lucky Star and pre K-ON! confusion).

The short version of the answer? If you count only the past ten years: no, anime is getting better. If you count the past twenty: it was pretty bad 1997-2004 but now it's about the same. And the past thirty: no it's pretty good now, good years and bad years.

Summary

Data, by nature doesn't lie, its interpretation can lead to wildly different conclusions though and in this case, the outcome is far from incontravertible. Anime suffers from the same problem as IMDB in that given enough votes, every entry will normalise to around 7. This covers the gamut from absolute hatred (1's and 2's) to the empassioned fans (9's and 10's) and everyone else in between, tending to vote more in favour than against. The end result is percentage points so minor in difference as to be next to worthless.

One can stare at the graphs long and hard to try and scry some meaning from them, search for patterns or sequences or correlations but the reality is more chaotic than that.

Caveats

1. I'm not a statistician or mathematician so while I'm aware of approaches such as weighted means, standard deviations and variances, I did not use these techniques for this analysis.
2. All data used was publicly accessible. I do not have access to any of the listed source's databases (even though I'd really like to)