Dead letter office: Does modern anime suck?

Graph porn

Sometimes blog posts just don't turn out the way you want them to. Whether due to the content or the implementation, they'll often end up smashing through the central reservation of the Good Intention motorway, hurtle down the Mediocrity slipway and end up in Dumb Idea ditch, back wheels still spinning.

the results made for nuanced [...] reading. And when I say nuanced I of course mean dull
My drafts folder is littered with these. Some are just a single word, like the opening to my Final Fantasy XIII review which was simply "bleh"; or they're almost complete and all that prevented pressing the enticing curves of the publish button was a lack of faith.

This is what happened with the firebrand titled "Does modern anime suck?", a post from one year prior which attempted to use Science™ to prove that modern anime doesn't suck and the familiar cries that echo every quarter of a year claiming as such were flat wrong.

That was the theory, the practise turned out to be less than convincing.

I am no statistician which presents issues when I deal with anything more vexing than averages, variance and standard deviations which, for the most part, has carried me happily through life so far. Since I wrote the post I'm slightly more genned up and phrases like "bimodal distributions" no longer elicit a vacant stare. This of course doesn't solve the issue of normalising the data that I had obtained for the post.

A story unto itself, I've always been a strong proponent of open data which means when the big three anime data sources: Anime News Network's Encyclopedia, AniDB and MyAnimeList do not make their general data available in a meaningful format, I take umbrage. Sure AniDB and MyAnimeList have APIs but the former is archaic and the latter next to useless for what I wanted it for.

So screenscraping it was. This got several blobs of data but with no deep knowledge of how to meaningfully interpret it, I plonked it unceremoniously into graphs and hoped the result would speak for itself.

It didn't.

I have a strong scientific background so I know the argument that regardless of the outcome, if the process was thorough and structured the results are unquestionable. This was not science though and the results made for nuanced rather than sensationalist reading. And when I say nuanced I of course mean dull.

Despite being entirely content complete, I abandoned the post for the reason that it didn't meet my own quality standards due to the flaky research and lack of interesting angle on the subject. It read much like it the small project had progressed: a disruptive idea let down by lack of knowledge but hey, graphs right?

I present it here as a way to put to sleep a topic which I don't have the resources to do justice to. Perhaps it will inspire someone more learned than I in the dark and arcane arts of statistics to make a more meaningful stab at penetrating the issue. Or perhaps it is a cautionary tale on the hubris of "big ideas". Regardless, the original thrust of the post was to bring a different voice to the dreary and insipid cries which travel along with each new season of anime like an STD. It certainly would have brought a different voice to the caterwauls, it would have been just as screeching and ill-informed though.

--

Last draft of original post: 16th June 2011

Sparked by a conversation on Twitter on Monday about the seasonal calls of "next season looks rubbish!" and "I'm watching twenty shows now, only three next season look any good", I mused as to whether there was a way to canonically prove one way or another whether anime is indeed getting worse as time went on. Or perhaps whether other more mundane psychological reasons are to blame for this repeated inanity like recency bias, nostalgia or plain old attention seeking behaviour.

With little idea as to what the outcome would be, I started parsing data, crunching numbers and generating graphs. So does modern anime suck?

Short version: no. Longer version: eh...

Average score of anime per year from 1970 - 2010

If you want the take-home message without any of the data analysis then the above is it. Average scores for anime have been on an upward trend for close to the last decade, peaking in 2008 then remaining steady for the last two years. Please bear in mind the scale of the graph: the lowest is 6.44, the highest is 7.24 meaning there is only 0.8 between the scores of the "best" years and "worst" years.

Average score of anime per year from 1970 - 2010 broken down by source

Please note: no data was available for 1970 with AnimeNfo

This data comes from four main sources: Anime News Network's Encyclopaedia, MyAnimeList, AniDb and AnimeNfo and is correct as of 13th June 2011. The publicly accessible scores were aggregated along with the start / end dates and the type of show (movie, OVA, TV series etc.). Due to the different weights offered by sources such as ANN and AniDb, the one collated across all of them was the weighted mean. Separating out the sources shows, with the exception of AnimeNfo (more on this later), the scores more or less follow one another in terms of trend, MyAnimeList users are most generous while ANN users are unilaterally harsher.

AnimeNfo is counted in the main graph and the source graph but no further as the site seems to have been all but abandoned. End dates for relatively recent anime such as Fullmetal Alchemist: Brotherhood are missing and the number of reviews is pitiful, even for popular series such as Bleach (182 reviews versus MAL's 105,954). There is also the issue of earlier shows (pre-1985) having wildly varying review scores which is likely the reason for the erratic early years in the main graph.

Average score of anime TV series per year from 1970 - 2010 broken down by source

Narrowing the data set down even further to only include TV series answers the question originally posed: is anime getting worse? Year-on-year is a mixed view, good years and bad years, so 2000 vs. 2001, 2002 vs. 2003 the latter years win in terms of average score. Going by individual source scores rather than the overall one is a mixed picture. MAL and AniDb have a definite trend upwards with MAL not seeing any huge deviations; ANN on the other hand puts the past five years as only just catching up with the 1970's.

Though the scores are different, all sources seem to agree on the good years and bad years though. So 1997 was a poor year (Evangelion hangover?) with a precipitous drop from the year before and one that it only just seems to be recovering from but with a solid peak in 2008 (post Lucky Star and pre K-ON! confusion).

The short version of the answer? If you count only the past ten years: no, anime is getting better. If you count the past twenty: it was pretty bad 1997-2004 but now it's about the same. And the past thirty: no it's pretty good now, good years and bad years.

Summary

Data, by nature doesn't lie, its interpretation can lead to wildly different conclusions though and in this case, the outcome is far from incontravertible. Anime suffers from the same problem as IMDB in that given enough votes, every entry will normalise to around 7. This covers the gamut from absolute hatred (1's and 2's) to the empassioned fans (9's and 10's) and everyone else in between, tending to vote more in favour than against. The end result is percentage points so minor in difference as to be next to worthless.

One can stare at the graphs long and hard to try and scry some meaning from them, search for patterns or sequences or correlations but the reality is more chaotic than that.

Caveats

  1. I'm not a statistician or mathematician so while I'm aware of approaches such as weighted means, standard deviations and variances, I did not use these techniques for this analysis.
  2. All data used was publicly accessible. I do not have access to any of the listed source's databases (even though I'd really like to)

Responses to “Dead letter office: Does modern anime suck?”

Thanks for sharing. Data may not lie but once it touches human hand, it's ruined!

Actually one thing I wish that would exist, as it would be useful, for this sort of analysis is to also take into account the time when the rating was made. I mean, how else can you take into account recency bias? (I think it's largely imagined as GL phrased it.) Some way to measure deviation from a pre-determined average as to stand in for "test of time"? Do some study on time-to-rating for shows people finished watching immediately or a long time after? Old shows being watched now versus old shows being watched long time ago (study how ratings change over time)?

The great thing about data is how much it tell us how little that we know.
Interesting. I personally believe that modern anime doesn't suck. Looking on the data, its a world wide data varying countries and different ages, right? Isn't the target population too big?

My Personal experience in MAL is that users would just rate anime subjectively. Given that most of the users are young and fan of modern anime, then anime from the 1970s will normally be forgotten. What was the anime back then? Heidi and Blue Blink or something, right? XD

I like this idea though. Its really interesting.
Are you familiar with aquabluesweater? http://aquabluesweater.wordpress.com/

They haven't posted in awhile but it's all juicy statistics. Determining correlation between studio and quality or studio and genre, whether moe really is infesting anime to the extent some believe, and whether the winter/summer seasons really are worse than spring/fall.
It was very brave of you to take on this question, and I applaud the time and effort you put into getting the data together. I'm sure we've all wondered the same thing, but I don't know of anyone else who's tried to use hard numbers to answer it. I suspect that the trend isn't statistically significant given the range (6-7.5 looks a bit narrow) but, like you, I come from a scientific background but not a mathematical one (proof, if it was needed, that biology is "the science for scientists who aren't good with maths"!).

The sheer number of variables and potential interferences simply puts me off drawing any conclusions. Taking a mean value of each title/year usually gives a very 'ironed-out' result, so almost everything occupies a similar range. By which I mean, Goodreads ratings rate most books between 3 and 4 stars, so the difference between, say, a book that the members collectively like and collectively love is the difference between a 3.4 or a 3.7 or something. Really hard to draw conclusions!

Tastes change over time, so "is a 1970s viewer's score of a contemporary show more or less important than how a viewer of today would score that show?" is one question I think. Heck, I score some series and movies differently when I rewatch them just a year or two afterwards.

To summarise: it's interesting and fun to think about all the hard work you've done here, but sadly I suck at statistics so can't draw any definitive conclusions from it either!

Is anime getting better or worse over time? No idea. I am sure of one thing: the current state of "most of it's rubbish to me but I find a minority really enjoyable" was probably always true. Sturgeon's Law and all that.
@omo: I'd love to do an analysis of these questions, even taking into account popularity (how many people watched it) or those who watched a show week to week versus marathon'ing it. I'm sure the data in some form exists, getting access to it is another matter though and while I have a good idea who to ask, I wouldn't know what to ask for.

@TPAB: I'm not sure if the location of a person who voted is stored on any of sites I pilfered the data from. It would be interesting to see though and being able to provide very targeted slices would be phenomenal, like you say, seeing what younger viewers thought of older shows (potentially twice as old as they are). On the subject of sites though, if you discount AnimeNfo then MAL users seem to rate more highly while ANN users are more judicious but by and large the overall trends up and down seem to match.

@Martin: Tastes definitely do change over time both personally and generally but I wonder if other factors would have a greater effect on an older show's score like word of mouth and the formation of common wisdom as to what shows are the cream of the crop. I'd love to have the statistical clout to be able to analyse this data to such a degree but, at the risk of sounding defeatist, I doubt number crunching will ever be my strong suit.

--

Many thanks for reading and for your comments everyone!
I love digging into analyses like this. It's not just reveals information about the anime, but the userbases of these sites as well. The way the major sites parallel each other in movement, but different in yearly averages suggest that the users are distinct, but still overlap in many ways.

The spikes in the data are also interesting. What made 1978, 1985, 1996, and 2008 universally well received, while 72, 81, 97, and to a lesser degree 2010 so unliked? Do a handful of titles skew the averages, or is there a noticeable change in perceived quality across the board?

During the last Spring season I played in MAL's Fantasy Anime League, a game that tracks the MAL statistics of newly airing anime week after week. I didn't follow through on keeping data every week, but the trends over time on weekly viewing numbers, dropped statuses, score average, favorites, and forum discussions lead me to identifying profiles for certain kinds of shows, at least in my head. The FAL group actually kept track of much more data than I, so if you asked them they might surrender their data, which they've had since 2008.