We all know it, but does it matter?

[regor]

Last year my club ran a classifier match at the beginning of the year and then didn't run classifiers in our monthly match for several months. We started sending out post-match surveys at some point and one of the questions we asked was whether or not folks (a) preferred a classifier at every match, (b) preferred no classifier, or (c) were indifferent. The break down was about 25-25-50, respectively, but the distribution of the responders' classifications were insightful. The folks who wanted a classifier every match were heavily skewed to lower classifications (plus the upper level A folks who were chasing an M card) and the folks who didn't want one were skewed towards the upper classifications. 

 

The current classification system gives the vast majority of the member body an obtainable goal to chase. Most members don't go to area matches or nationals, and if they do they aren't competitive, so the classification system is their North Star. If you made it harder you would lose engagement, which would cost the org a lot of revenue. I personally wouldn't mind a system where you could get to M via a classification system but GM had to be earned by major match performance, but at this point the cat is out of the bag and switching to that without resetting every GM down to M would create resentment from members who thought they had a shot under the old system but think the new way is unobtainable but, "X got grandfathered in and I beat them every match". 

 

The best USPSA could realistically do is copy chess and have an informal or formal "Super GM" title for folks that have made the super squad or won an area match, but do the folks who would get that title even care? 

[Nathanb]

The answer is in the data.  Pull the high hit factors. See who is setting those. Correlate that to shooters who have set multiple high hit factors. What does that say?  Do we have shooters who have consistently generated and pushed the bar?  Do we have shooters who consistently set the bar at level 2 and 3 matches?

christian sailer is a great example of owning larger matches and pushing the bar higher than what may be indicative of the overall population and would be a use case for creation of a new class. It would be interesting to see the true average of his classifier scores. 

[Fishbreath]

16 minutes ago, Nathanb said:

It would be interesting to see the true average of his classifier scores. 

 

Not too hard to find.

 

His last 6 classifiers shot at local matches average 88.5% (and you have to go back to 2019 for the 6th one). His last 6 classifiers including stage scores on pending classifiers at Nationals average 98.8%.

 

His actual current classification percentage is 100%: two 100% stage scores on pending classifiers at Nationals, and four major match wins.

 

The two 100% stage scores are 105.8% and 104.3% against the HHFs decided post-Nationals, so his adjusted average including those is 100.48%.

Edited September 13, 2022 by Fishbreath

[Bigzona]

1 hour ago, Fishbreath said:

 

By and large, they do. "Paper GMs" are mostly a myth, in that the vast majority of highly-classified people are capable of beating the vast majority of lower-classified people most of the time. Using national champions as the benchmark is unreasonable, because skills and results aren't linearly distributed. A 95% classifier score excludes substantially more than 95% of attempts (chart and statistics courtesy competition.shooting.analytics on Instagram) already.

I saw the post on IG as well and found it quite interesting! Thanks for bringing it in here.

[Fishbreath]

1 minute ago, Bigzona said:

I saw the post on IG as well and found it quite interesting! Thanks for bringing it in here.

 

Sure thing—it would be great if USPSA were to either do some of the analytics work in house, or share data so that we enthusiastic amateurs can do it for free, but that would admittedly go against form, and we're muddling along alright without official access. The competition.shooting.analytics guy's doing great work on classification and its distribution. I think he'll be doing a few posts on the proposed 2022 classifiers, and some mathematically-grounded ideas for the HHFs.

[Bigzona]

1 hour ago, motosapiens said:

why? what difference does it make?

 

in general, If you look at any match results, you'll see gm's at/near the top (along with a few improving m's and A's whose classifications haven't caught up yet), and you'll see lower-classified shooters further down. However, you shouldn't expect to see shooters clustered in the same percentages as the classification percentages. It's statistically unlikely that someone who reached a 95% classification average (when able to throw out all the bad runs) will be able to reach 95% in a major match when they have to keep all their bad runs.

 

We don't disagree. The comment you're highlighting was meant in the context of Major matches and Nationals, in that context, I'd say the best of the best are ranking appropriately at these matches with a few anomalies. 

 

My last paragraph in the post answers the question about the difference it makes.

[Nathanb]

1 hour ago, Fishbreath said:

 

Not too hard to find.

 

His last 6 classifiers shot at local matches average 88.5% (and you have to go back to 2019 for the 6th one). His last 6 classifiers including stage scores on pending classifiers at Nationals average 98.8%.

 

His actual current classification percentage is 100%: two 100% stage scores on pending classifiers at Nationals, and four major match wins.

 

The two 100% stage scores are 105.8% and 104.3% against the HHFs decided post-Nationals, so his adjusted average including those is 100.48%.

This is great. Definitely insightful. It doesn’t seem like Christian was the best example of this since he’s not actually shooting classifiers but just majors 

[Chills1994]

5 hours ago, Fishbreath said:

 

By and large, they do. "Paper GMs" are mostly a myth, in that the vast majority of highly-classified people are capable of beating the vast majority of lower-classified people most of the time. Using national champions as the benchmark is unreasonable, because skills and results aren't linearly distributed. A 95% classifier score excludes substantially more than 95% of attempts (chart and statistics courtesy competition.shooting.analytics on Instagram) already.

 

 

1,414 attempts
12.1593 HF was the highest
5.545 HF was the average
0.14% of shooters shot 100%
0.64% of shooters scored GM
2.4% of shooters scored M or better
8.2% of shooters scored A or better
28.29% of shooters scored B or better
69.52% of shooters scored C or better

 

That tracks with his statistics on classification: 1-2% of shooters are GMs in most handgun divisions. (PCC is about 3.5%, because of the Open HHFs initially used; Revolver is about 5%, because some of the old classifiers still have 6-round HHFs, and some of the 18s were soft up until the most recent adjustments.)

 

If you use 95%+ at Nationals as the GM threshold, there are 10 Carry Optics grandmasters in total:

 

Dazhi Zhang
John Vlieger
Max Michel
Mason Lane
Nils Jonasson
JJ Racaza
KC Eusebio
Hwansik Kim
Phil Strader
Shane Coley

 

A Distinguished Grandmaster title for 95%+ Nationals finishers would make sense, but adding everyone in CO except for those ten guys back into M is obviously wrong, to me.

 

I've been experimenting with Elo rating (also on Instagram), and had it spit out average ratings and standard deviations for each class, against the dataset of all L2+ matches since 2020:

 

GM: 1603 +- 175
 M: 1357 +- 149
 A: 1169 +- 151
 B:  961 +- 147
 C:  744 +- 150
 D:  574 +- 144

 

Each class is about 1.3 standard deviations better than the class below it, and GM is the widest, which makes sense, given that it includes the entire right tail of the skill graph. That seems pretty reasonable to me.

where did that high hit factor information or data come from?

 

can it actually be tracked back to an actual real live person shooting it at some local match?

[BritinUSA]

Back when paper was used for scoring, USPSA would manually tweak the HHF used for the classifiers.

 

My understanding is that it works this way;

1. The classifier is introduced at Nationals and all the measurements are verified by NROI.
2. The stage is part of Nationals so there is a title on the line so every stage score counts.
3. Rather than take the HHF recorded for each stage in each division, they take the average of the top ten. This has the effect of lowering the HHF.
4. Once all the Nationals are done then the classifiers are released ‘into the wild’.
5. L1 clubs run the classifiers, there is no verification of stage setup by USPSA, they just assume that it is set correctly.
6. Scores more than 5% below classification are not included.
7. Competitors can reshoot the stage as many times as they want, and probably practice it before the match starts.
8. There is no National title on the line, and no consequence to a bad run.
9. The top ten HF’s from L1 matches are then used to adjust the HHF up.
10. The net result is that the HHF ends up significantly higher than a top competitor could actually achieve with a Nationals title on the line.
11. USPSA considers the HHF from Nationals to have the same statistical significance as a HHF set at an L1 match, which may not be setup correctly, may have been practiced many times over, etc..

This is why I think the system is broken.

 

[Chills1994]

Back when Single Stack became a legit division the HHF’s were the exact same as L-10.

 

Also at the time, let’s say the HHF for whatever classifier was set at say 10.0 in Open.  Then in Limited, the HHF was set at 9.0 for that same classifier.  And say for Production the HHF was 8.0.  Lastly for the revolver guys, the HHF was set at 7.0 .  So there were clearly these 10% step downs in all the HHF’s as you went from the gamey-est division to the least gamey division.

 

This would have been around say the 2010 timeframe.

 

so it makes me curious where the HHF and graph came from posted on the first page???

[Thomas H]

7 hours ago, Bigzona said:

It should matter. Our top classified shooters should be able to demonstrate match performance reflective of their ranking. Its indicative of the validity of our classification system, and reflective of my own personal experience.

 

And they do, so it works out.

 

Your classification percentage has never meant that you should get that classification percentage at a match.  People who think so are welcome to cite some source that supports that idea.  (But they won't find one.)

 

Classification ranks give a strong indicator of your relative placement in the match---relative to the other shooter's classifications.  Multiple analysis of these sorts of things have happened, and unsurprisingly, have always founds that the vast majority of GMs beat Ms who beat As who beat Bs and so on.  There are outliers (many of which exist because they don't shoot classifiers and are classified lower than they should be, and some of whom exist because they talked their local MD into letting them hero-or-zero classifiers multiple times until they get the score they want and thus are classified higher than they can support at actual matches) but the vast majority of people finish right within their classification group, which finishes ahead of lower classification groups.

 

Every time someone brings up this sort of topic, there are always some people who loudly declaim that "the classification system is broken!!1!" and yet never actually come up with any factual, statistical basis for said declamation. 

 

The classification system works remarkably well.

 

The people who think that your classification percentage SHOULD match your major match finish REALLY don't seem to understand how those numbers work.  After all, in a major match, your score on a stage is based on the person who did the absolute best exactly once on that stage (that in general, no one has ever shot before).  And that stage score is added to other stage scores that again, are based off the person who did the absolute best on that stage (out of a large pool of people, in a single moment in time).  Expecting those match finish numbers to resemble a classification percentage makes very little sense. 

[Chills1994]

Yeah….as a revenue stream for USPSA HQ.

 

For those that interested, USPSA is a non-profit.  So the IRS Form 990’s are published at Guidestar.

 

 

[motosapiens]

1 hour ago, BritinUSA said:

Back when paper was used for scoring, USPSA would manually tweak the HHF used for the classifiers.

 

My understanding is that it works this way;

1. The classifier is introduced at Nationals and all the measurements are verified by NROI.
2. The stage is part of Nationals so there is a title on the line so every stage score counts.
3. Rather than take the HHF recorded for each stage in each division, they take the average of the top ten. This has the effect of lowering the HHF.
4. Once all the Nationals are done then the classifiers are released ‘into the wild’.
5. L1 clubs run the classifiers, there is no verification of stage setup by USPSA, they just assume that it is set correctly.
6. Scores more than 5% below classification are not included.
7. Competitors can reshoot the stage as many times as they want, and probably practice it before the match starts.
8. There is no National title on the line, and no consequence to a bad run.
9. The top ten HF’s from L1 matches are then used to adjust the HHF up.
10. The net result is that the HHF ends up significantly higher than a top competitor could actually achieve with a Nationals title on the line.
11. USPSA considers the HHF from Nationals to have the same statistical significance as a HHF set at an L1 match, which may not be setup correctly, may have been practiced many times over, etc..

This is why I think the system is broken.

 

i rarely agree with you on controversial topics but i think this post is a pretty good analysis. where we differ is that i don’t think this means the system is “broken”, i just think the system is of limited usefulness and kinda dumb, but also meaningless, so i don’t care. 

 

i spent the weekend observing in real time how the best shooters in the sport can’t shoot any better than the luckiest A or B on a fixed time stage. however most of the top guys shoot at a high level on almost EVERY stage.

[Bakerjd]

49 minutes ago, Thomas H said:

 

And they do, so it works out.

 

Your classification percentage has never meant that you should get that classification percentage at a match.  People who think so are welcome to cite some source that supports that idea.  (But they won't find one.)

 

Classification ranks give a strong indicator of your relative placement in the match---relative to the other shooter's classifications.  Multiple analysis of these sorts of things have happened, and unsurprisingly, have always founds that the vast majority of GMs beat Ms who beat As who beat Bs and so on.  There are outliers (many of which exist because they don't shoot classifiers and are classified lower than they should be, and some of whom exist because they talked their local MD into letting them hero-or-zero classifiers multiple times until they get the score they want and thus are classified higher than they can support at actual matches) but the vast majority of people finish right within their classification group, which finishes ahead of lower classification groups.

 

Every time someone brings up this sort of topic, there are always some people who loudly declaim that "the classification system is broken!!1!" and yet never actually come up with any factual, statistical basis for said declamation. 

 

The classification system works remarkably well.

 

The people who think that your classification percentage SHOULD match your major match finish REALLY don't seem to understand how those numbers work.  After all, in a major match, your score on a stage is based on the person who did the absolute best exactly once on that stage (that in general, no one has ever shot before).  And that stage score is added to other stage scores that again, are based off the person who did the absolute best on that stage (out of a large pool of people, in a single moment in time).  Expecting those match finish numbers to resemble a classification percentage makes very little sense. 

Finally someone who actually understands match scoring is on a bell curve. And it's the only way to score stages when you have hits/time=score (hit factor). 

 

The classification system is there to give shooters something to look at as a benchmark, and goal to achieve. 

 

Everyone knows if you want to get higher classification as fast as possible you shoot the newest classifiers. They haven't been shot and had the HHF ramped up to where it's almost impossible to attain a high scoring run. Does this mean that the old classifiers should be thrown out because they have become used to much, or that new classifiers should count until they have reached a certian number of runs to sufficiently create the bell curve? What would that number be?

 

My own thought on the classification system and my placement. I'm a low b class shooter that happens to shoot a lot of majors most years, at least 6 or 7 each year since 2018. When I look back at my overall scores for my division I'm usually middle of the pack or slightly above middle of the pack for my classification. Last year I happen to win c class limited at A5. This year I've been b class in CO all season. My percentage of the division winner has been between 52% and 68%. A5 I was 56% of the winner, Nils. At CO nats this past weekend I was 58% the winner, Nils again. At this point right now if I look at my classifiers that count 2 of the 6 are actual match scores. So for me personally I think the classification system is working well. 

[motosapiens]

1 minute ago, Bakerjd said:

At CO nats this past weekend I was 58% the winner, Nils again. 

u shoulda said hello. mrs moto and i were running stage 9.

[Bakerjd]

4 minutes ago, motosapiens said:

u shoulda said hello. mrs moto and i were running stage 9.

Ahh man sorry. I'm terrible with names/faces. Lol. 

 

Stage 9 was the fixed time stage. It was one of my two really bad stages. I was the guy who was in your first squad of day 2. Got a single shot off and mag jammed up. I was probably not in the greatest of moods after that run and I know I just walked away. I was so disappointed in myself after it. I had a great first day and after that run pretty much knew I had no chance of making my 60% final score goal. 

[motosapiens]

6 minutes ago, Bakerjd said:

Ahh man sorry. I'm terrible with names/faces. Lol. 

 

Stage 9 was the fixed time stage. It was one of my two really bad stages. I was the guy who was in your first squad of day 2. Got a single shot off and mag jammed up. I was probably not in the greatest of moods after that run and I know I just walked away. I was so disappointed in myself after it. I had a great first day and after that run pretty much knew I had no chance of making my 60% final score goal. 

oh, man, I remember that. Sorry it went that way. The bad news is that fixed time is very unforgiving of very small problems. The good news is that even a huge problem is not that big a deal on a 60 pt stage. It was my second stage of the match and went pretty poorly for me too. I spent 3 straight days watching everyone beat my score. Good thing there was beer.

 

Mrs Moto was actually tied for first on that stage after the staff match, with 46 pts. Then the very first shooter in the main match got 50+. Still a solid stage for her, and better than 1/3 of the supersquad.

Edited September 14, 2022 by motosapiens

[Bakerjd]

@motosapiens yeah. I shoot at a place in mi where the MD likes to put a fixed time stage in every so often for a local. I like them a lot but one little bobble and you are done. Both this one and A5 bit me. Other then this stage my other bad stage was 13. I just never got a solid stage plan and forgot two targets. I should have spent more time on that stage and not some other smaller stages in c zone. Either way it was a great match, and I met some very cool people. 

 

Oddly enough having the dates and locations for next year come out sort of has me in a pickle. I like shooting CO but hate loading 9 major and 9 minor. The girlfriend shoots open so I was going to switch to open for next year since I have a super nice open gun sitting at the house. But now with it being 3.5hrs from home for CO nats..... I'm like uggg I want to shoot it. 

[CHA-LEE]

Q - Is the USPSA Classification System a perfect "Measuring Stick" for determining overall Practical Shooting Skill performance?

A - NO

 

Q - Is the USPSA Classification System a fairly decent way of measuring a narrow range of Practical Shooting Skills?

A - YES

 

Q - Is the USPSA Classification System a reliable tool competitors can use to compare their skill performance against others on the national level?

A - YES

 

Q - Is the USPSA Classification System a good method for setting skill performance goal achievement?

A - YES

 

Q - Has USPSA HQ effectively or accurately maintained the Classification System as new Classifiers or Divisions have been released?

A - NO

 

Q - Could USPSA HQ do a much better job of making the Classification System more accurate and relevant to overall Practical Shooting Skill Performance?

A - YES

 

Q - Have the Classifier High Hit Factors been modified inaccurately in the past or present making them either too easy or too difficult to achieve vs reality?

A - YES

 

Q - Does achieving any Classification level guarantee the same "Overall Percentage of the Winner" alignment in Major Matches vs the Classification percentage?

A - NO

 

Q - Do Sandbaggers/Grandbaggers exploit the flaws in the Classification System to achieve their desired classification rating goal?

A - YES

 

 

[llamasabound]

56 minutes ago, motosapiens said:

oh, man, I remember that. Sorry it went that way. The bad news is that fixed time is very unforgiving of very small problems. The good news is that even a huge problem is not that big a deal on a 60 pt stage. It was my second stage of the match and went pretty poorly for me too. I spent 3 straight days watching everyone beat my score. Good thing there was beer.

 

Mrs Moto was actually tied for first on that stage after the staff match, with 46 pts. Then the very first shooter in the main match got 50+. Still a solid stage for her, and better than 1/3 of the supersquad.

 
Haha, you all ran that double bay well. That particular stage flattered me: would that it were reflective of overall match performance!

[rowdyb]

If we have a ranking system it should only be from match results and that's it. All level of matches. Leverage electronic scoring and the rising number of people competing. And do it with a different formula than is currently used!

 

Want a higher classification? Do better at matches.

[wchangose]

[Joe4d]

19 hours ago, zzt said:

 

Bingo.  I'll never get out of C Class.  The raft of weak hand shooting Classifiers the local clubs like to put out guarantees that.  My left hand doesn't work as intended and I can barely control the gun with it.   I shoot way better than C in the match.

Hah,, I shot A on those,,, but cant run with midpack B shooters...  so umm I did really good at those boring classifier stages !..    I will admit though the field courses, activators, texas stars, is what makes the game fun... Never see any smiling laughing joking  and jaw jacking at a classifier stage

[Climbhard]

Classification should be based on percentile.  Your HF earns a percentile against everyone else that shot that classifier.  Average the best 6 of 8 percentile scores.  Then your average goes into the database and if you're in the "x" percentile you classifiaction is "Y".  And yes your classification goes up and down as your percentile changes.  Maybe GM is only the top 1 or 2 percentile??  At least its not based on x% of some arbitrary numbner made up years ago and increased three times by anoymous people.

[RJH]

After thinking about this some more and reading some other people's thoughts, I'm beginning to think the classifier system works pretty perfect. 

 

I don't know if it was designed with this in mind or if it just ended up being this way, but I think the classifier system's best attribute is that it gets noobs interested in improving. Once you get that B,A,M,GM card and are settled into wherever you're going to be, it starts meaning less and less so people naturally figure out ways to "make it better."  But in reality once you settle into whatever classification you're going to end up, basically the rest of your life the classifiers don't really matter anymore and people should just shoot them like the normal match stages and not worry.

 

So in reality I don't think there's any way to improve the classification system overall, and I think the classification system is doing, while maybe not what it was originally designed to do, the best thing it's ever going to do.