Classifier system: a modest proposal

[ATLDave]

Since USPSA has opened the box of tinkering with classifier HHF's (a system that, at least for established divisions, was reasonably effective at dividing the shooting population into roughly appropriate boxes), I'd like to suggest that we re-think the way classifications are calculated.  

 

First, we need to agree on what the purpose of a classifier system is.  I would posit the purposes are as follows:

• To test, under uniform conditions, core shooting skills (as distinct from stage planning or movement skills);
• To sort competitors into groups of similar shooting skill (i.e., to function as a handicap system); and
• To allow shooters to gauge their overall progress and standing within the population in terms of skill.

 

If those are the goals, then a system based on percentiles, rather than percentages, seems more appropriate.  (If you do not know the difference between percentile and percentage,  the rest of this post won't make any sense.)  It would also be easier to administer (automatic, really), remove subjectivity, make it possible to include new classifiers easily, make it easy (automatic) to develop appropriate classifications for any new or changed divisions, make it possible for every classifier shot to have an equal likelihood of being used in the system (avoiding the do-no-work classifiers that currently tend only generate "Below" scores in the system),  avoid the problem of artificially low/easy HHF's on a few classifiers resulting in easy grandbagging, and help to more meaningfully separate the truly world-class shooters from the "mere" local GM's.

 

How would it work?  As now, shooters would shoot classifiers as stages in matches.  There would be no change to how the classifiers are scored or how they play into the match.  No changes at the match/MD/local level would be required.  The only difference is what happens when the scores are plugged into "the system" on Tuesday night (or whenever it rolls).  When a shooter's score is uploaded, it will be compared to all the uploaded scores on record of all other shooters in the division.  Instead of a percentage being calculated against a particular HHF (however that is chosen/calculated at present), the percentile of the score is used.  I.e., if 50% of the scores are above a particular shooter's HF on a classifier, and 50% are below, then that is a 50% for classification purposes... regardless of what percentage it is of the highest HF on record or other arbitrary HHF.  One's classification would simply be an averaging of some selection of these percentile scores (we could keep the 6-of-8-minus-exclusions methods if desired).

 

Why would it be better?  For a whole bunch of reasons.  Because the percentiles would reflect the actual performance of shooters, there would be no risk of a poorly-chosen HHF.  Because it would update continuously, it would prevent HHF's from being outdated - if the population got better, then the percentile rank of a given HF would move down, and vice versa.  It would automatically and swiftly set the proper levels for new divisions, or divisions after gear/rule changes.  It would allow competitors to accurately know how they rank against the population of USPSA shooters, not just an imaginary HHF shooter.

 

At a deeper level, it would do a better job of capturing the real differences in performance among shooters at all skill levels.  Let's make up a pair of hypothetical stages.  One is similar to El Prez, but shot at 10 yards with relatively close spacing between the targets, all of which are open.  The other has exactly the same targets and the same distances and spacing, but has no shoots in between the targets eliminating the sides of the D-zones.  For a high-level shooter on a good day (maybe what HHF is supposed to capture?), the times and hits on those two stages will be very similar... they'll be able to rip all A's quickly... they wouldn't be likely to hit a D, so having no-shoots covering some D-zone makes no difference to their score.  On the other hand, for a lower-skilled shooter, they may have to slow down by a measurable amount in order to be assured of staying penalty free.   Or they will have a much higher chance of getting one or more penalties.  A HHF selected to reflect what is appropriate for a strong shooter will either mean lots of "below" coded scores for less-skilled shooters on the harder stage or lots of outlier/grandbagging scores on the easier stage.  Alternatively, a HHF selected to reflect this change in difficulty/set appropriately for lower-skilled shooters (which would require a lower HHF on the harder stage) would result in high-skill shooters smashing the HHF and posting scores well over 100%.  Similar dynamics manifest with lots of the long-distance "standards" type classifiers, where lower-skilled shooters simply cannot make the shots reliably enough to get scores that even count in the current system.  

 

It would also reflect the full range of approaches to a classifier stage.  Both the "match mode" runs and the "hero or zero" (regardless of how they turn out) runs would factor into the score distributions.  

 

This system would also do a much better job of measuring the differences between highly-skilled shooters.  As skill level/performance level increases, real gains in skill/performance come in smaller and smaller increments.  Each successive tenth of time shaved off a draw or reload is harder than the previous one.  At the upper reaches of the sport, a repeatable 3% difference in HF on a set of classifiers may represent a far larger gap in skill - a gap that then shows up as bigger differences in overall match performance.

 

 There are other reasons this would be a superior system, but these seem like more than enough to me.  As long as USPSA is tinkering, let's go to a rational system.  People and organizations whose livelihoods depend on predicting outcomes based on tests almost universally use percentiles, rather than percentages, for their systems.  It's simply a better approach.  

 

Edited July 9, 2018 by ATLDave

[rowdyb]

I'd buy that.

[HoMiE]

I see nothing wrong with the current classification system as is, it was just poorly executed and the HHF had not been adjusted for some time. If the HHF are just maintained regularly and stages constantly refreshed then it works.

 

The drawback with percentile is you don’t have even population of shooter classes. 

[ATLDave]

You can adjust HHF all you want, but certain problems are inevitable -see the example stages discussed in my post.  This shows up in the fact that there are many classifiers in the system that do relatively little work.  Plus, it requires conscious/deliberative "updating." 

 

I agree the current system "works" reasonably well.  But if HQ is going to mess with it at all, why not go to a better system?  

Edited July 9, 2018 by ATLDave

[phoenixsomd]

Since USPSA has opened the box of tinkering with classifier HHF's (a system that, at least for established divisions, was reasonably effective at dividing the shooting population into roughly appropriate boxes), I'd like to suggest that we re-think the way classifications are calculated.  
 
First, we need to agree on what the purpose of a classifier system is.  I would posit the purposes are as follows:

• To test, under uniform conditions, core shooting skills (as distinct from stage planning or movement skills);
• To sort competitors into groups of similar shooting skill (i.e., to function as a handicap system); and
• To allow shooters to gauge their overall progress and standing within the population in terms of skill.

 
If those are the goals, then a system based on percentiles, rather than percentages, seems more appropriate.  (If you do not know the difference between percentile and percentage,  the rest of this post won't make any sense.)  It would also be easier to administer (automatic, really), remove subjectivity, make it possible to include new classifiers easily, make it easy (automatic) to develop appropriate classifications for any new or changed divisions, make it possible for every classifier shot to have an equal likelihood of being used in the system (avoiding the do-no-work classifiers that currently tend only generate "Below" scores in the system),  avoid the problem of artificially low/easy HHF's on a few classifiers resulting in easy grandbagging, and help to more meaningfully separate the truly world-class shooters from the "mere" local GM's.
 
How would it work?  As now, shooters would shoot classifiers as stages in matches.  There would be no change to how the classifiers are scored or how they play into the match.  No changes at the match/MD/local level would be required.  The only difference is what happens when the scores are plugged into "the system" on Tuesday night (or whenever it rolls).  When a shooter's score is uploaded, it will be compared to all the uploaded scores on record of all other shooters in the division.  Instead of a percentage being calculated against a particular HHF (however that is chosen/calculated at present), the percentile of the score is used.  I.e., if 50% of the scores are above a particular shooter's HF on a classifier, and 50% are below, then that is a 50% for classification purposes... regardless of what percentage it is of the highest HF on record or other arbitrary HHF.  One's classification would simply be an averaging of some selection of these percentile scores (we could keep the 6-of-8-minus-exclusions methods if desired).
 
Why would it be better?  For a whole bunch of reasons.  Because the percentiles would reflect the actual performance of shooters, there would be no risk of a poorly-chosen HHF.  Because it would update continuously, it would prevent HHF's from being outdated - if the population got better, then the percentile rank of a given HF would move down, and vice versa.  It would automatically and swiftly set the proper levels for new divisions, or divisions after gear/rule changes.  It would allow competitors to accurately know how they rank against the population of USPSA shooters, not just an imaginary HHF shooter.
 
At a deeper level, it would do a better job of capturing the real differences in performance among shooters at all skill levels.  Let's make up a pair of hypothetical stages.  One is similar to El Prez, but shot at 10 yards with relatively close spacing between the targets, all of which are open.  The other has exactly the same targets and the same distances and spacing, but has no shoots in between the targets eliminating the sides of the D-zones.  For a high-level shooter on a good day (maybe what HHF is supposed to capture?), the times and hits on those two stages will be very similar... they'll be able to rip all A's quickly... they wouldn't be likely to hit a D, so having no-shoots covering some D-zone makes no difference to their score.  On the other hand, for a lower-skilled shooter, they may have to slow down by a measurable amount in order to be assured of staying penalty free.   Or they will have a much higher chance of getting one or more penalties.  A HHF selected to reflect what is appropriate for a strong shooter will either mean lots of "below" coded scores for less-skilled shooters on the harder stage or lots of outlier/grandbagging scores on the easier stage.  Alternatively, a HHF selected to reflect this change in difficulty/set appropriately for lower-skilled shooters (which would require a lower HHF on the harder stage) would result in high-skill shooters smashing the HHF and posting scores well over 100%.  Similar dynamics manifest with lots of the long-distance "standards" type classifiers, where lower-skilled shooters simply cannot make the shots reliably enough to get scores that even count in the current system.  
 
It would also reflect the full range of approaches to a classifier stage.  Both the "match mode" runs and the "hero or zero" (regardless of how they turn out) runs would factor into the score distributions.  
 
This system would also do a much better job of measuring the differences between highly-skilled shooters.  As skill level/performance level increases, real gains in skill/performance come in smaller and smaller increments.  Each successive tenth of time shaved off a draw or reload is harder than the previous one.  At the upper reaches of the sport, a repeatable 3% difference in HF on a set of classifiers may represent a far larger gap in skill - a gap that then shows up as bigger differences in overall match performance.
 
 There are other reasons this would be a superior system, but these seem like more than enough to me.  As long as USPSA is tinkering, let's go to a rational system.  People and organizations whose livelihoods depend on predicting outcomes based on tests almost universally use percentiles, rather than percentages, for their systems.  It's simply a better approach.  
 

Well stated

Sent from my SM-G935V using Tapatalk

[dkamps]

So basically; score classifiers like its a constant organization wide match stage?

[HoMiE]

1 minute ago, dkamps said:

So basically; score classifiers like its a constant organization wide match stage?

And what happens when lots of top shooters are at IPSC world shoot or some other major match and your shooting scores no longer have a large enough population to do percentiles against?

[ATLDave]

6 minutes ago, dkamps said:

So basically; score classifiers like its a constant organization wide match stage?

No, not at all like that.  Percentile versus percentage.  As I wrote, if you don't know the difference, the rest of the post won't make any sense.  

Edited July 9, 2018 by ATLDave

[ATLDave]

3 minutes ago, HoMiE said:

And what happens when lots of top shooters are at IPSC world shoot or some other major match and your shooting scores no longer have a large enough population to do percentiles against?

Irrelevant, since the measured-against population is all scores on record.  

[HoMiE]

Why not just have a more current catalog of stages that get updated every quarter and more like a postal match? Instead of having 100s of classifiers where a few get shot all the time like El Prez and some obscure that get shot barely. 

Edited July 9, 2018 by HoMiE
Speeling

[motosapiens]

this is a really good suggestion imho, and  does away with the problem of needing to adjust HHF's. Done as a percentile, the virtual HHF will adjust itself over time.

[ATLDave]

Why would that be better?  That would likely be worse.  In no small part because you'd end up with mis-set HHF's even more frequently.  

 

Your observation that some "obscure" classifiers "get shot barely" is apt.  The question is why?  Well, a few are obscure because they have stupid/rare/specialized props or bay dimensional requirements that are rare.  HQ just killed most of those.  But there are others that are rare specifically because shooters and MD's have learned that the HHFs are so high that few shooters will post scores that actually count in the current system (as being "below"  - i.e., more than 5% lower than the lower boundary of their current classification).  A percentile system fixes that, while a percentage system will always struggle with it.   

[dkamps]

11 minutes ago, ATLDave said:

No, not at all like that.  Percentile versus percentage.  As I wrote, if you don't know the difference, the rest of the post won't make any sense.  

 

Ok, I was just making sure. I understand the difference, so you are looking for a standard deviation based on long term statistics to put people into percentiles.

 

Id actually be very interested to see a significant sample size taken from scores and placed under this model to see how it would adjust classifications within a bell curve.  Might help hero or zero runs from effecting HHF on classifiers if outliers were tossed out of the statistics too.

 

new shooters would also skew the curve to the left, and it would need a total number of runs based on the shooting populous to maintain and update the curve; throwing out the old scores.

[ATLDave]

Just now, dkamps said:

new shooters would also skew the curve to the left, and it would need a total number of runs based on the shooting populous to maintain and update the curve; throwing out the old scores.

Reasonable people could disagree about whether that's necessary, but I'd be ok with that.

[dkamps]

1 minute ago, ATLDave said:

Reasonable people could disagree about whether that's necessary, but I'd be ok with that.

 

I'm just mimicking the systems I've seen on revolving participant percentile based grading formats that have been accepted in other fields.  Mostly pulling from aviation based grading systems.  Honestly, I kind of like the idea. 

[IronicTwitch]

ATLDave - thanks for putting some thought into your suggestion.  I've struggled putting into words how Hit Factor scoring being a J-curve of points separation leads to questionable results.  It's disconcerting to see the top finishers at bigger matches have such a large points and % gap.  

 

Percentages tell you “how many of the local population/whole” and are an excellent way to present ratios.

 

Percentiles tell you where the local population falls in the distribution of the wider area population. Use percentiles when you want to analyze based on the distribution.  This is seems more in line with a classification system.

[stick]

It makes sense.  

My only problem with the HHF is that almost all of the stages don't have movement.  You're taking out the variables that make some of the best shooters world class shooters (Fleet of foot)  How many GM's do you know that can't run or move quickly?

[ATLDave]

29 minutes ago, stick said:

It makes sense.  

My only problem with the HHF is that almost all of the stages don't have movement.  You're taking out the variables that make some of the best shooters world class shooters (Fleet of foot)  How many GM's do you know that can't run or move quickly?

This goes to what you think the purpose is.  If you look at my first two stated/assumed purposes, they involve shooting skill, as distinct from running.  Or, for that matter, prop manipulation, stage planning, focus over the course of a long day, or any of a number of other factors that do have some influence on match results.  

 

If you think the classification system needs to test, in a representative way, the whole gamut of skills and qualities needed to win matches, then simply looking at weighted match results (the ELO model) is worthwhile.  I think there's some real value in parsing out the core shooting and gun-handling skills - which are, by far, the largest part of overall match performance - and measuring those.  But that's a philosophical question.  No right or wrong answers there.

 

Instead, my post assumes that we want the classification system to basically do what it does today, but do it in a better way.  

[ATLDave]

36 minutes ago, IronicTwitch said:

Percentiles tell you where the local population falls in the distribution of the wider area population. Use percentiles when you want to analyze based on the distribution.  This is seems more in line with a classification system.

 

Seems that way to me, too. OK, so I'm a 79% shooter in LTD.  What does that mean?  Am I 79% as good as Shane Coley?  No, I don't think that's accurate.  Am I better than 79% of the LTD shooters in USPSA?  No, but that's what percentiles would tell us (or disprove!).  

 

I'm probably at least a little more skilled than someone who is sub-70, and probably less skilled than someone who is over 85.  That's about all I can say from the current system.  

[GeneBray]

I see nothing wrong with the current classification system as is, it was just poorly executed and the HHF had not been adjusted for some time. If the HHF are just maintained regularly and stages constantly refreshed then it works.
 
The drawback with percentile is you don’t have even population of shooter classes. 

I don’t think high hit factors requires an even population of shooter classes. HHF is what it says high hit factors. I don’t know the process but it would seem to me to be the average or median of the HHF reported for a classifier. In the process there has to be a floor for cut off of what constitutes a HHF for a classifier. How that is determined I don’t know.

As part of the change I am fairly certain PCC HHFs have been set as opposed to using the Open HHF which I believe was the basis for HHF for PCC until now. And HHF for CO are also probably based on results from CO classifier as opposed to based on another division.


Sent from my iPhone using Tapatalk

[wtturn]

Interesting idea.

[motosapiens]

17 hours ago, stick said:

How many GM's do you know that can't run or move quickly?

 

most gm's are not particularly athletic, unless you are comparing them to sedentary couch potatoes that watch the walking dead.

 

many of the true national-class guys do appear to at least work out a couple times a week, probably as much as a new mother trying to lose the weight she gained during pregnancy.

[reverse_edge]

Not a bad idea, honestly.

 

[Hi-Power Jack]

20 minutes ago, motosapiens said:

 

most gm's are not athletic, unless comparing them to couch potatoes 

 

Hey, I resemble that remark    

[mstamper]

21 hours ago, ATLDave said:

Since USPSA has opened the box of tinkering with classifier HHF's (a system that, at least for established divisions, was reasonably effective at dividing the shooting population into roughly appropriate boxes), I'd like to suggest that we re-think the way classifications are calculated.  

 

First, we need to agree on what the purpose of a classifier system is.  I would posit the purposes are as follows:

• To test, under uniform conditions, core shooting skills (as distinct from stage planning or movement skills);
• To sort competitors into groups of similar shooting skill (i.e., to function as a handicap system); and
• To allow shooters to gauge their overall progress and standing within the population in terms of skill.

 

If those are the goals, then a system based on percentiles, rather than percentages, seems more appropriate.  (If you do not know the difference between percentile and percentage,  the rest of this post won't make any sense.)  It would also be easier to administer (automatic, really), remove subjectivity, make it possible to include new classifiers easily, make it easy (automatic) to develop appropriate classifications for any new or changed divisions, make it possible for every classifier shot to have an equal likelihood of being used in the system (avoiding the do-no-work classifiers that currently tend only generate "Below" scores in the system),  avoid the problem of artificially low/easy HHF's on a few classifiers resulting in easy grandbagging, and help to more meaningfully separate the truly world-class shooters from the "mere" local GM's.

 

How would it work?  As now, shooters would shoot classifiers as stages in matches.  There would be no change to how the classifiers are scored or how they play into the match.  No changes at the match/MD/local level would be required.  The only difference is what happens when the scores are plugged into "the system" on Tuesday night (or whenever it rolls).  When a shooter's score is uploaded, it will be compared to all the uploaded scores on record of all other shooters in the division.  Instead of a percentage being calculated against a particular HHF (however that is chosen/calculated at present), the percentile of the score is used.  I.e., if 50% of the scores are above a particular shooter's HF on a classifier, and 50% are below, then that is a 50% for classification purposes... regardless of what percentage it is of the highest HF on record or other arbitrary HHF.  One's classification would simply be an averaging of some selection of these percentile scores (we could keep the 6-of-8-minus-exclusions methods if desired).

 

Why would it be better?  For a whole bunch of reasons.  Because the percentiles would reflect the actual performance of shooters, there would be no risk of a poorly-chosen HHF.  Because it would update continuously, it would prevent HHF's from being outdated - if the population got better, then the percentile rank of a given HF would move down, and vice versa.  It would automatically and swiftly set the proper levels for new divisions, or divisions after gear/rule changes.  It would allow competitors to accurately know how they rank against the population of USPSA shooters, not just an imaginary HHF shooter.

 

At a deeper level, it would do a better job of capturing the real differences in performance among shooters at all skill levels.  Let's make up a pair of hypothetical stages.  One is similar to El Prez, but shot at 10 yards with relatively close spacing between the targets, all of which are open.  The other has exactly the same targets and the same distances and spacing, but has no shoots in between the targets eliminating the sides of the D-zones.  For a high-level shooter on a good day (maybe what HHF is supposed to capture?), the times and hits on those two stages will be very similar... they'll be able to rip all A's quickly... they wouldn't be likely to hit a D, so having no-shoots covering some D-zone makes no difference to their score.  On the other hand, for a lower-skilled shooter, they may have to slow down by a measurable amount in order to be assured of staying penalty free.   Or they will have a much higher chance of getting one or more penalties.  A HHF selected to reflect what is appropriate for a strong shooter will either mean lots of "below" coded scores for less-skilled shooters on the harder stage or lots of outlier/grandbagging scores on the easier stage.  Alternatively, a HHF selected to reflect this change in difficulty/set appropriately for lower-skilled shooters (which would require a lower HHF on the harder stage) would result in high-skill shooters smashing the HHF and posting scores well over 100%.  Similar dynamics manifest with lots of the long-distance "standards" type classifiers, where lower-skilled shooters simply cannot make the shots reliably enough to get scores that even count in the current system.  

 

It would also reflect the full range of approaches to a classifier stage.  Both the "match mode" runs and the "hero or zero" (regardless of how they turn out) runs would factor into the score distributions.  

 

This system would also do a much better job of measuring the differences between highly-skilled shooters.  As skill level/performance level increases, real gains in skill/performance come in smaller and smaller increments.  Each successive tenth of time shaved off a draw or reload is harder than the previous one.  At the upper reaches of the sport, a repeatable 3% difference in HF on a set of classifiers may represent a far larger gap in skill - a gap that then shows up as bigger differences in overall match performance.

 

 There are other reasons this would be a superior system, but these seem like more than enough to me.  As long as USPSA is tinkering, let's go to a rational system.  People and organizations whose livelihoods depend on predicting outcomes based on tests almost universally use percentiles, rather than percentages, for their systems.  It's simply a better approach.  

 

I am all for re-evaluating the process. Aren't we already using a percentile methodology now with stage point awarding? If so, then making a move to using the same for classifiers should not be such a difficult reach.