Tuesday, April 11, 2017

Warhammer 40,000 Skill vs.Codex Debate: The Myth Is Busted

"Anecdotal" is what people say when they don't wan’t to have to actually admit someone wins with the unorthodox.   It's a slap in the face.

What no one gets to do is pretend your wins did not happen.  Nor do they get to suggest that you could never in a million years win with a list you just won with.  That’s just spitting in the face of the most relevant data you have:  actual games.

In this game, the general matters more.  Not because you say so, but because the facts support you on that.

Here was a little noodling I did after I was challenged on the efficacy of Dark Eldar, but this study really speaks to a lot of factions.  I wrote this as an answer to someone’s arguments online and have expanded it here.  I hope you find it interesting and if you do, share it.

There were 4 Dark Eldar Generals in the Independent Tournament Circuit (ITC) that scored in the 400 point club.  That refers to those whose points total for the 2016 ITC season in their best five events was 400+.  I am further talking about those who did it within a specific faction.  Obviously these are the most relevant scores, the top representatives of their faction much like a Gold Glove Third Baseman is the best fielder at his position or the team MVP is the most valuable player to his team.  These are the best at what they play and they show how the position can be played.

Only four Dark Eldar generals did this out of 157 who attempted it .  That's only  2.5% of Dark Eldar who could live up to that level of play using that Codex.  Sounds terrible, doesn’t it?  I was wisely told to believe things for good sound reasons instead of reasons that sound good.  So i looked at the rest.  Do you know what I found?  

28 attained it as Eldar; but it took them 714 Generals to do it.  True story!   3.9%, which means it’s not even happening 2% more often!  Not such a jump as you might have been led to believe by the all-knowing Internet is it?  If someone had told you when you were buying your army "hey you’re only going to have a 4% chance of reaching the 400+ club with that army” would you play it?  Would you even buy it, once shown proof that few ever do?  Of course you would have.  Right?

Chaos Space Marines had just 3 Generals in the 400+ Club out of 370 Generals who tried.  0.81%.  yet you cannot ignore that three did it.  The actual frequency is worse, but still we see it happening.  It sounds worse than 3.9% but to be honest...  neither sounds particularly high when faced with the reality of the number either way.

When you look at competitiveness, you have to ask yourself the following question if you want the answer to mean anything:  of those attempting to compete in the ITC, who are demonstrably the best players as proven by their points, using a certain army, can the Codex compete to reach that elite sort of 400+ club standing everyone wants to be in?  Those are generally your local heroes at Warhammer.  They are the ones you expect to win a lot of the time.  They are the bellwethers for the codex, not the rest.  They have somehow grasped that which the codex offered when it was put out there while the others have not (yet).

In that context, it’s quite surprising how far apart those percentages aren’t!  As they are relevant and considered "power armies", here's a few more for you to chew on:

Cult Mechanicus only had four players make it to the 400+ club out of 211, and no one is screaming about how bad War Convocations are.  That’s 1.89% and worse than Dark Eldar!

Adepta Astartes armies, despite their Battle Companies, boasted only twelve out of 1107 Generals making it to the 400 club.  About 1.1%.  Easily the most surprising number of them all.  Still want to argue that the general doesn’t matter?  Still want to argue that Codex’s win?  It would seem that Dark Eldar give you the better chance!

These are objectively true numbers, proven in actual games, with no less than an entire YEAR of results comprising an enormous number of games played at tournament only levels!

How did Dark Angels do, who won some really big events this year?    Just four made the 400+ club, out of 326 who tried.  Just 1.2% could do it, yet people give all kinds of praise to the Codex and none for Dark Eldar who produced the same number of stars in far less attempts.  Why?  Someone good won something with it, that's why.  Not because the Codex was so much better.  It had lots of chances to prove otherwise 

Another shocker:  nine Tau Generals made it...but there were 601 of them trying.  Here again:  1.5%.  Yet we hear claims of their cheddar echoing across the Web.  Those are the facts, folks.  Not my facts.  The facts.

I can go on.  The take away is obvious.  Consider this as well:

Player scores that went much higher were included in this, and those were people willing to travel the country in pursuit of as many major events as they could afford to (and for that matter couldn't truly afford to).  The guy who took first place in the ITC went to 50% more events than the guy who took second, and thus had far more chances to get "best five" results.

Matt Root, the ultimate winner, won mostly with Cult Mechanicus…you know… the force only 1.89% of the players could even reach the 400 club with!  Do the math.

Skill matters most.  If Matt Root committed the same energy to another force, it’s quite likely he’d be right up there again vying for the finals.  Before you say no, I'll give you proof.  Matt Root was in the top 1% in 2015 as well (not a surprise).  Would you like to know what he racked up those points playing?

He was in the 400+ Club that year too, in the top 1%...playing Orks.  Enough said, I think.

23 comments:

  1. That's a really interesting breakdown, though in trying to understand what it's actually showing I'm coming to two significant conclusions.

    1. That the truly top players in the ITC format (it makes such significant changes to method of winning the game that the results cannot be assumed to apply beyond that format) do not need a good codex to be successful, they are good enough at picking out the most reliable method of winning games and the units from the codices that support that method that they don't need the support of a good codex to achieve it. At that sort of rarified level then general's skill is what matters, because they will have worked out how to counter all the toughest elements of the 'power' codices.

    2. That the comment 'everything counts in large amounts' applies not only to guns within the game but also to the games themselves. It would be interesting to see if the results differed in terms of average results for those players instead of the best five - ie you mentioned that the guy who took first place played at least 50% more games than anyone else - are his best five results in tournaments that his other competitors didn't take part in, or is he consistently scoring that high in all his events.

    Personally I'm a big believer that general skill most certainly does matter, I see it weekly at my club where a couple of guys consistently win because they are fundamentally better at making the decisive choices in their games than their opponents. At lesser skill levels however I think codex power does tend to have more influence, partly perhaps because of the psychology of the game too - an average player with an ork list facing an Eldar army is going to be disheartened even before the games starts and might even accept that he's going to lose from turn 1. I had a game last week where my opponent's aim was to survive to turn 3, and lo and behold at the end of turn 2 he had 3 guardsmen and a chimera left. A more positive and aggressive attitude might very well have helped him survive beyond that, as would thinking more about his units positioning.

    ReplyDelete
    Replies
    1. The average results of the players is something you can determine, as well as what, say Matt Root played in. The average might not be AS helpful because a ppayers worst game doesn't represent his Codex, it represents his bad day, argument with the wife, mortgage payment coming due, oh and probably three rounds of bad luck on the dice. =) I went to the Bay Area ope nand injurde my back aftr game three and i was sweating in that 100 degree room barely able to move or think. I fought through it because I had come so far and spent so much to be there. I promise you my first three games were far more representative of what I could do than the last three as i just wanted to die the rest of the tournament. Lol. Outlyers don't truly inform us. In other words there's a reason they don't use the average for this. It would discourage some people from playing any more. Imagine if I won my first five tournaments and said "well I don't want to mess up my score and if I go to anymore it will". Not a great scoring system if you went with averages. not a great barometer of the Codex to count the outlyers. I can say this: Winning any tournament requires your A game and a lot to go right. theres only one winner in every tournament. So for someone like Matt oot to have won as many as he did, let alone the number he placed highly in is quite significant.

      Delete
  2. I feel there's an uncontrolled variable here. I completely accept the argument that the biggest arbiter of victory is skill, however, presenting the raw percentages as some measure of codex quality assumes the skill profile of the players of each faction is consistent.

    There are f'rex a *lot* of marine players. I would venture that a disproportionate number of new-ish players pick marines as their first army, thus the low win rate isn't all that surprising even if the codex were "strong".

    Similarly, it may be the comparatively better players that have the skills to play a "weak" codex and not get discouraged and move to something else.

    My point is, causation is a tricky thing to make concrete statements about. I think the way to tease this out would be to compare the win-rates of the same players playing with different armies. If you had enough data you could derive an averaged set of army-army comparisons

    ReplyDelete
    Replies
    1. Well what you said in the first paragraph isnt 100% the point I made. I made the point that those who know how to use their codex exhibit the TRUTH of whats possible with a codex as compared to the myth of whats possible. It is acknowledged here that certain codex's allowed a guy a little better chance of getting to the 400 club, but as Matt Root shows, it certainly did not doom many worthy generals FROM that club. that's maybe a more accurate way for you to look at it.

      A lot of new players play a lot of things. I played 5 different armies i nthe ITC this year and was in the 400 club. Not optimal for your chances to bounde around like that but i did . Though its not that common, some of these newer players you mention even will go to tournaments in that state of newbishness. Maybe. But if they do, they are not going to know or exhibit the true strength of their Codex for a while and that really is the point. They aren't representatives of the Codex. How could they be? Over the years they will see a very steady rise up the ITC and i suspect any new guy could make it to 400 if he's a bright lad whose well taught by his mentors, eats his Wheaties and reads my blog (sorry couldnt help it, shameless plug).

      But in all seriousness, to evaluate the truth of the myth, you have to look to those who did what the Codex allows, and see how many could. Those numbers represent more than 50,000 recorded games. Think about that number. its more games than any ten players combined has probably played in their life. that body of evidence is substantial and trustworthy.

      Delete
    2. I wanted to follow up on this thought because you hit on something I wanted to ask more about momerathe. I'm assuming you mean "Does the win ratio of the same general go up when playing one army vs. another"? Legit.

      I like this idea. My trouble in making that number stick is that you are kind of applying the unskilled guys results across multiple armies with the guys like Matt Root who were 400+ for two straight years with two very different armies. You'd be weighing the player results equally, yes? I mean if you weigh the 400 guys results more, then we are already back to looking at those Generals, as we have here and for the same reasons!
      The other thing we'd have to overcome is that a less skilled player will simply find adjusting to different armies difficult. I myself played five in competition last year. Five has to be a high number right? Most people only attend 12 tournaments a year if you look at the numbers, if that. My experience is that it affects me when I switch. You can miss obvious steps when running the army (giving orders or some other such silly lapse). I'm not sure we could find reliable numbers because of it. Finding guinea pigs who are adaptable enough not to corrupt that data (man does that sound pretentious or what?) would be a challenge. Tough one. That brings you back to the "consistent winners" and following how THEY trend. They end up being the most reliable weather balloons so to speak to tell you which way the wind is blowing. I can tell you from my perspective that my overall winning percentage is 75% and that is consistent across my armies. I used to track every game and against which armies I got the wins or losses. But my Tau empire army was closer to 80%% and that was true long before 6th Edition. I quit tracking it late in 5th Edition because it was just too much hassle for too little return and the numbers didn't seem to be moving much. My personal take is that the difference between them came partly because I just am in love with the Tau and second, they fight in the way I think best. The Codex itself was considered "terrible" for a very long time but I maintained a really nice win record both long before and long after 6E came and went. So I'd say it would be tough to figure the question of "Does the win ratio of the same general go up when playing one army vs. another"? I'm not sure I could even diagnose that in myself, given how I've seen the Tau pendulum swing, which is the only outlyer of my many armies in results. What I can point to is actualized games played and whether the top players enjoyed much more in the way of success given their differing codex majorities. Still, its a fair question.

      Delete
  3. That's interesting for sure. And you are correct in 1 regard: the best generals are going to float to the top no matter what the circumstances. I have no doubt Matt Root could break the 400 mark with Harlequins if he wanted to. However, there is a fundamental flaw in your numbers: you are take the elite players (3%) and making assertions of the rest of the field (97% of the players). Those type of sweeping assumptions almost never work. A analogy that would be close would be to examine the students that got into an Ivy League school and then make sweeping generalizations of the rest of the students that didn't get into Ivy League schools.

    What would be more edifying is seeing the percentage of average players that had winning records with their codex. When win-loss ratios of codexes is compared across the board without regard to player placement you find that some codexes have far higher winning ratios than do others. Further complicating the analysis is that "pure" codex players are few and far between. Consistently winning with Tyranids or Dark Eldar is a far greater accomplishment than it is with Eldar or Tau. The best players will still be the best players, but if you want to compare codex power level, look at how the whole population of players do with them, not just the elite players.

    ReplyDelete
    Replies
    1. I dont find that edifying, Matt, because the ITC scores show you what the best five results were for any of those generals listed. Their best efforts. The question is not whether average or poor players are going to struggle because...we already know that. The codex strength cannot be accurately reflected there because the skill wasn't there to use it. How do we KNOW the Codex had little to do with it? Well we looked at what was POSSIBLE if you use the codex to its potential. That's how we know. We know that when you really are using the Codex to its potential, the variance isn't great.
      The point to be made here is that a Codex doesn't determine your wins and losses very much. You do. The numbers show that. I know some people don't like hearing "get better' as advice after a game but really, that's what they need to do, right? There aren't just a certain number of "slots" available in the 400 club. Anyone can make that club and you don't have to win a single tournament to do it (though you likely will). Even someone who is consistently third place (always a bridesmaid, never a bride) can make it. So when you really think about it, the average and poor players are only proof of one thing: they are average or poor players. That WILL change if they have the capacity to change, keep reading about strategies and tactics, keep watching games and seeing how people time their stuff and so on. If I studied the low end numbers all I would find is a list of people who had the opportunity to compete but essentially didn't. they didnt go to enough events, probably dont play as often and aren't truly competing and so they are getting victimized by people playing Eldar or whatever and it gets in their head. The internet reinforces it by magnifying the impact of any certain unit by a thousand (no one should be arguing that Scatterbikes are just silly and contribute to that 3.9%) and suddenly people start treating it like conventional wisdom instead of challenging that wisdom as I have here with raw numbers.

      Delete
    2. Oops I meant: "no one should be arguing that Scatterbikes aren't just silly and do contribute to that 3.9%"

      Delete
  4. Looked at from one perspective, it's "only" a 2% difference in DE vs Craftworlds, but looked at from another Craftworlds players are showing up nearly twice as often- that would seem to indicate that your codex _does_ make a signficant difference.

    Codex and player skill (as well as luck, matchup, terrain, and other factors) all contribute to performance in games. A codex is just a tool, and the tool is not the artist- but an artist is still limited by their tools. You can buy a $5000 tablet and editing suite, but that doesn't mean you'll be able to produce great animations; but for someone who _does_ have the skills to make use of them, a powerful set of tools like that can significantly increase their productivity and performance.

    (It should also be remembered that ITC scores are tracked only by primary faction and do not count allies- virtually none of those armies posting scores were "pure" Dark Eldar, and in fact the majority of them likely had very large Craftworld or Corsair components, much as the Top 8-scoring Dark Eldar armies at LVO have been. The fact that the Corpsethief Claw, by itself, is enough to qualify you for DE primary is clearly tilting the numbers more than a bit, since as often as not that is the only DE part of an army like that.)

    ReplyDelete
    Replies
    1. A Corpsethief Claw clocks in around 650 points most of the time so its not enough on its own but sure. It is true that a Dark Eldar player can have other elements in his list from other codex's. No argument there. As can...every single person in every single game of Warhammer played around the world, nearly. So this isn't really saying anything. The "pure" codex list died when 6E came. If you're still bemoaning it, I understand. I am too. Its not relevant to the argument. If someone finds that their list is "best" when Dark Eldar dominate its contents... then that is a reflection on the Codex. If they choose not to...so is that. I can tell you for a solid FACT that Alex, a very good player, is only technically playing Dark Eldar in the ITC when he registers but then...that is the same for the guy playing his Eldar with a couple Grotesquerie or a Tau Empire player who brings Coteaz and some friends, or a Necron player who brings a Chaos Cabal. I've seen it all (not literally but I do play a lot...)

      You yourself are a very good and highly ranked ITC player, Abusepuppy so this isn't new information to you. You've beaten me in every matchup I can remember (though I suppose it's possible I've forgotten one) As i recall you made frequent use of allies. Skathach WraithKnights and similarly friendly things. We didn't call you less a Tau player in those instances. That is now a given in any discussion.

      Delete
  5. I feel that you are missing the point. Let's try it from a different angle. I've ridden a motorcycle all my life (ok, 38 out of 43 years, but still). I'm a pretty competent rider. My main bike is a 1200cc naked sport bike. It's pretty quick. There isn't more than a handful of riders in the world that could outrun it on a track with a 250cc sport bike. That being said, I've seen many superbikes get owned on the track when they were ridden by an average rider and you had a pro on the 250. That being said, no one would argue that my bike has less ability than does an entry level bike. It's rider ability that makes all the difference. Just because Valentino Rossi could beat average riders with the 250 against my bike doesn't mean my bike is less powerful than the 250. That is essentially the same argument you are making when you assert that Root/Nayden/Grant etc. winning with "X" codex proves it's potential. We aren't disagreeing at all that best players make the most difference, it's just that the one argument doesn't really inform the apparent stance of your blog that there really isn't much difference between power level of codexes.

    ReplyDelete
  6. But I do agree that many players just need to get better and stop whining that they are losing because of the lack of power with their codex.

    ReplyDelete
  7. Your final argument is your most compelling. Your usage of statistics with vs dark eldar is poor. A 3.9% placement rate is 56% higher than 2.5%, not less than 2%, and its about what i would expect from a massive codex advantage. A 50% higher victory rate.

    Generals matter, but so do scatbikes. And it matters that some codices have nothing remotely as efficient. No amount of practice changes that.

    Maybe 8th makes this a moot point. One thing's for certain: i won't miss codex eldar.

    ReplyDelete
    Replies
    1. I meant eldar vs dark eldar.

      But as i said, the ork overall victory is much more impressive.

      Delete
    2. Also, the eldar rate is a massive 300% higher than csm, not the trivial amount you represent it as. I actually expected csm to be much better.

      Delete
    3. This comment has been removed by the author.

      Delete
    4. UNfortunately Martel732 you're abusing those percentages. The relativity here is to a standard. How many REAL people made it to 400, using primarily this codex. The reality was that this represents the codex being played to its potential. The CODEX is being graded ONLY by those who could actually see what the Codex can do...and did it. THOSE Codex exemplars didn't do it noticeably more often than any other codex exemplars. One codex did it 3.9% and another 2.5% which is not actually that much difference in incidence. INCIDENCE is what we are measuring. If all you get from one codex to another is a 1% better incidence of reaching 400, then the codex clearly makes little enough difference. People who cannot or sometimes WILL NOT use their codex to maximum advantage are not exemplars of the Codex and do not show us what the Codex can do. There is a sea of complaining Chaos Marine players who "don't get" how those who got to 400+ got there but what they all have in common is: that's not really the Codex's problem. It's theirs. People can and did primarily use Chaos units and profit'd. So the question has been answered: If you master your codex, as you should be EXPECTED to do, can you reach 400. You can. And if you choose the Chaos Marines, you will have a lesser chance, but not really as significantly as people constantly and unremittingly make it out to be. YOU, in the end, are really the issue.

      Some people pontificate about whether This codex or that codex won this or that tournament but if it were the Codex, Matt Root wouldn't be champion according to these numbers. So clearly... Not the case either. Not only must a lot go right for you to win 5-6 games in a weekend, but YOU have to be good because no codex can win it for you. If they could, how ever could Matt Root do what he did two years in a row NOT using the Codex voted "most likely to succeed". You know it's true.

      Delete
  8. It's a 56% increase in incidence. I'm not abusing anything. Most generals are slightly below average skill, average skill, or slightly above. The eldar codex seems to be helping them quite a bit. 56% and 300% increases atest to this.

    ReplyDelete
    Replies
    1. You are. if two codexes compete, and out of 100 players, one produces 2 400+ clubbers and another produces 1 400+ clubber, you are going to suggest this is a massive difference? Lol. It isn't.

      Delete
    2. That's not what happened. But there's no reason to keep discussing it. I don't get the same conclusions out of the data that you do. That's all that really matters. The most compelling argument in your post is anecdotal, ironically.

      Delete
  9. What's true for matt root is likely not true in the general case, hence the increases in top 400 representation. The tau case does perplex me, though. I'm thinking i just don't see or play their counter lists.

    ReplyDelete
    Replies
    1. there are numerous cases where the numbers do not match. For example Dark Angels and BAO. You must accept the evidence in front of you. Its very simple. The Codex isn't the decising factor and the numbers bear it out. It helps a couple more people get to 400, but yet does not decide the matter as Matt Root and NUMEROUS others I saved the space on that I could ALSO mention showed. Numerous others follow this same pattern. It's clear that a codex used to its potential will give only the slightest of edge in making it to the 400+ club and also will NOT decide the championships, the BAO nor much of anything else. Players matter more. It isnt that Codex's DON'T its that players matter more and showed it repeatedly. They will do so again.

      Delete
  10. As long as you keep conflating codexes and generals you cannot have a meaningful discussion. By your very numbers you are only talking the top 3% of players. You can't take the 97th percentile and then apply it to the whole pool of players. The top 3% win more games no matter what codex they play. You could do the reverse and show how the bottom 3% of players always have a losing record at tournaments. Using the same data analysis you could make the claim of which codexes are the worst by the ones that never got a win. I played an ITC GT recently where a Militarum Tempestus player didn't win a single game. He was in last place of players that finished, would you say that this shows MT can't ever win a game?

    ReplyDelete