IDPA Old vs. New Scoring data analysis

[FTDMFR]

Given all the weeping and gnashing of teeth regarding the new IDPA scoring system, I decided to run some actual numbers.

The numbers used here are from four random matches from my local club.

Match results are analyzed here in two different forms:

• Match Score Percentage: The winning score is 100%, and the scores are scaled off of that score (e.g. if the winner's match score is 80s, a guy who shot a 160s would be 200%). Lower is better.
• Match Rank: To compensate for different matches having different numbers of competitors, rank has been normalized to a scale of 1-100.

By Match Score Percentage

First, here's a scatterplot of match percentage. X-axis is match percentage under the old system, and Y-axis is the match percentage under the new system. A point below the line means that shooter's score is better under the new system, and a point above the line means that shooter's score is worse under the new system. Again, 100% is the match winner.

For the top shooters (near 100%), the points are clumped pretty close to the line, meaning that the new scoring system won't affect them much. As the shooters get worse and worse, the points start deviating further and further from the line, meaning that the worse the shooter you are, the bigger the effect the new scoring system will have.

Here's the same data, but viewed a little differently. The X-axis is still the percentage under the old system, but the Y-axis is now the difference between the new percentage and the old percentage. Again, below the line means better score under the new system, and above the line means worse score under the new system.

So, in terms of match score, the worse the shooter is, the more the new scoring system will affect them. Which is pretty obvious, I guess.

By Rank

Here are the same two plots as above, but for Rank rather than Match Score.

Unlike Match Percentage, where the difference made by the new scoring system becomes uniformly greater the worse the shooter you are, for Rank, difference is the smallest for both the best shooters AND the worst shooters, and the greatest difference is with the shooters in the middle.

So, the top shooters will still be the top shooters (because they're both fast and accurate), and bottom shooters will still be the bottom shooters (because they are neither fast nor accurate), and the shooters in the middle (say MM and SS) will be the ones most affected.

Conclusions

Based on this data set, the new scoring system isn't going to affect the rankings for the the guys near the very top OR near the very bottom. However, the vast majority in the middle should start aiming.

(Nothing that hasn't been said a million times already. But graphs are fun.)

------------

When I have more time, I'll repeat this using the data from the 2015 Worlds and see how things look when the pros are included. I'll also see if I can quantify exactly how the optimum balance between speed and accuracy will change for the middle-of-the-pack shooters.

Edited October 1, 2015 by FTDMFR

[MilkMyDuds]

Great graphs, but my first observations is that so few shooters are NOT affected by this change, e.g. the dots on the lines. My second observation is that the top 10% shooters (e.g. the top 20 ranks) have huge spread in terms of rank changes due to the new scoring system. Plus or minus 5-8 ranks is pretty big change for the top dogs.

[motosapiens]

my first observation is that the shooters that aren't complete idiots are going to adjust their shooting style to take advantage of the new dumb scoring, just like folks currently adjust their shooting style when they switch between minor and major in uspsa. or switch between uspsa and idpa.

[FTDMFR]

Agreed. But I think this shows that, even at the club level, the higher level shooters aren't winning by just spraying and praying.

Edited October 1, 2015 by FTDMFR

[v1911]

Agreed. But I think this shows that, even at the club level, the higher level shooters aren't winning by just spraying and praying.

Agreed.

If you aren't practicing the fundamentals and relying on just speed to improve your standings, you're going to feel the hurt.

[MilkMyDuds]

I think everyone agrees that the top 2% will stay on top 2% with the new or old rules, but they only show up once a year to shoot nationals.

The bread and butter of the sport is the mid 20%-80% spectrum, who will be significantly impacted by the change.

If we define "high level shooters" as top 20%, you can see from your graphs how their ranks will fluctuate widely with the new rules. 5-8 positions fluctuation is huge if you are in the top 10-20 rankings.

The bottom line is, the new rule is likely to do nothing to affect the very top DMs and MAs (top 10 in national/worldshoot). However, the lower MA and EX/SS/MM shooters will require major adjustments to stay afloat.

[FTDMFR]

Here's the data from the IDPA 2015 World Championship.

I'll start with the difference in a shooter's division rank between the new system and the old system. Here, above the line means worse rank with the new system, below the line means better rank with the new system. Rank here is actual division rank, not normalized to the number of shooters. I've also broken it out by classification. I'll start with SSP:

I'll add the other divisions as time permits.

NOTE: This data includes the infamous Stage 11. I might rerun the data with Stage 11 removed.

Edited October 1, 2015 by FTDMFR

[Chills1994]

Let's say you calculated the mean (or median) final score for all the shooters in a division under the old scoring system. And calculated a standard deviation.

Then you adjusted everybody's final score based on the new scoring system. Again, calculate a mean and a stand dev. (Or possibly the median, then compare that to the mean, because I think the curves on the histogram will be skewed).

Then compare the two means and SD's against each other.

[johniac7078]

Are you planning to assign significance? Without, it is not possible to tell much.

[johniac7078]

I think everyone agrees that the top 2% will stay on top 2% with the new or old rules, but they only show up once a year to shoot nationals.

Agrees based on what?

[johniac7078]

There is also an issue with your conclusion. Yes, the worse you are as a shooter, the more your score will increase. But that has little to do with overall placement. My data suggests that the top 20% percent of shooters are most likely to have there finish rank altered.

[FTDMFR]

I wanted to put the charts up first to get a qualitative feel for the difference. I'm planning on doing ANOVA as well.

Edited October 1, 2015 by FTDMFR

[FTDMFR]

There is also an issue with your conclusion. Yes, the worse you are as a shooter, the more your score will increase. But that has little to do with overall placement. My data suggests that the top 20% percent of shooters are most likely to have there finish rank altered.

I don't have time to post more data from the Worlds, but from what I can tell so far, it looks like pretty much everyone's ranking changes. It's just a matter of by how much. In SSP, it looks like the biggest changes are with the shooters between 20-60% range (where 0% is best).

Edited October 1, 2015 by FTDMFR

[johniac7078]

I wanted to put the charts up first to get a qualitative feel for the difference. I'm planning on doing ANOVA as well.

SWEET! I am not surprised that a lot of ranks change, it's just how much. At the moment, I am not seeing the kind of changes that really reflect a need to revise scoring. Basically everyone goes up, and a relatively small number of the top dogs undergo the greatest change in terms of where they place. Seems like a lot of trouble for not much return. All the scoring software will ahv to be changed, the classifier....oh well. Back to work now. If you want to look at stats together, PM me.

Thanks man