How Well Calibrated Are USPSA Classifiers?

[CutePibble]

14 minutes ago, waktasz said:

Just from a quick glance that data is obviously wrong because I have hundos on at least one classifiers they have listed as having 0% GM scores. 

Let’s review it. What is your USPSA number?

 

There are few issues that might be at play here. Are these hundos listed as Legacy scores on your USPSA profile? These don’t have HF in the records so I removed them from output for now. 
 

also you might have something with 100 percent historical, but curPercent much lower due to HHF updates, even if it’s not a legacy score
 

 

[MoNsTeR]

I don't have any commentary on whether classifiers are well calibrated (they're obviously not), but just wanted to point out for any other data/software nerds that may have missed it, the data powering these charts and graphs is available here https://github.com/CodeHowlerMonkey/hitfactorlol/tree/main/data

It's big JSON blobs/arrays, but very easily importable into a database, pandas, R, whatever you like. (Try DuckDB!)

[waktasz]

1 hour ago, CutePibble said:

Let’s review it. What is your USPSA number?

 

There are few issues that might be at play here. Are these hundos listed as Legacy scores on your USPSA profile? These don’t have HF in the records so I removed them from output for now. 
 

also you might have something with 100 percent historical, but curPercent much lower due to HHF updates, even if it’s not a legacy score
 

 

 

Sorry I meant to delete that part of my post. At first I didn't realize I was looking at Open division scores and just scrolling down, saw Melody Line which had 0 hundos on file and knew that couldn't be right. User error. 

 

[broadside72]

If you find your own overall match performance both locally and at majors is generally aligning with your classification (and you aren't sandbagging) then the classification system is working fine, assuming that the distributions of classes in the match is normal

 

If you find your match perf is lower than your class then you are shooting only the classifiers that don't reflect typical match/stage shooting challenges and the system is flawed.

 

If you find your match perf is higher than your class then you are a sandbagger and the system is flawed.

 

 

[Maximis228]

I'm less concerned with the HHF being wacky as I am with the complete lack of consistency from club to club. Many of the USPSA classifiers have very bad diagrams. I've also learned overtime that people don't know how to use a tape measure. I've shot the same classifiers at many different clubs and they often can look different. Most notably no-shoot placement on targets. 

 

This fact alone brings every single classifier into question. 

 

Also... Classifications are an indication of skill level. That doesn't mean you perform at that level every match. There is a reason we use an average of your best 6 of the most recent 8 classifiers. 

 

[motosapiens]

1 hour ago, broadside72 said:

 

 

If you find your match perf is higher than your class then you are a sandbagger and the system is flawed.

 

 

or maybe you just shoot classifiers like any other stage instead of swinging for the fences. the hero/zero. mentality combined with foleys flawed logic in adjusting the hhf’s has made it difficult to get high scores without unacceptable (to me) risk, so i just stopped paying very much attention and worry about my match scores.

 

 

[broadside72]

2 hours ago, motosapiens said:

or maybe you just shoot classifiers like any other stage instead of swinging for the fences. the hero/zero. mentality combined with foleys flawed logic in adjusting the hhf’s has made it difficult to get high scores without unacceptable (to me) risk, so i just stopped paying very much attention and worry about my match scores.

 

 

My post was mainly to stir things up for discussion simply because those the three basic comments I hear from others when this topic comes up.

 

Personally, while I care that I do consistently well with general improvements in classy results over time, I am not obsessed with them. In fact, a hero or zero run I did for fun kind of burned me, bumping me with two lucky shots I called as misses, when I can't perform like consistently at a match. 

[Stepan]

Wow this is awesome!
 

As a chronic grandbagger I quickly learned that there are easy classifiers and hard classifiers. Some are even non-GM-able IMHO. but this really brings the issue into a light. 
 

Personally, I learned how to game the system and it worked ok for me, but it’s really frustrating to see talented and hard working shooters stay in A/M class simply because they weren’t lucky. I have a few friends like that and they constantly place above their classification at majors and get called sandbaggers. But it’s not their fault, in fact they were *trying* to class up!
 

In the grand scheme of things and with enough time/classifers sure it becomes a wash(unless your club puts hard clsssifers on purpose), but I’m sure a lot of this “I don’t care for classifiers” resentment comes from these HHF calibration issues. 
 

I really need to look deeper into this… 

[CutePibble]

3 hours ago, Stepan said:

it’s really frustrating to see talented and hard working shooters stay in A/M class simply because they weren’t lucky. 

That’s one of the reasons this project was started. We don’t want to create more or less GMs with it. Just simply calibrate classifiers and membership stats against each other, making classification process smoother and closer to “real percentage”. 
 

One of the ideas that can achieve that (that was originally posted by competition.shooting.analytics on IG IIRC) is to use percentile of real A-class shooters to make a calibration point AND count the lower bound of that percentile in each classifier as 75% for the division. 
 

this approach of course requires more testing, but so far it looks really promising. It’s already moving hard classifiers into easier HHF and easier ones into higher HHFs. 
 

quick demo on hard and easy classifier IN CO (red vertical lines — current HHF, yellow - proposed A-class calibration, green/blue - similar calibrations against M and G class):

 

 

 

if something like this is implemented — it should NOT change things much, 

more just smooth them over and make so called sandbaggers classify where they should be. 
 

It could also be used for fully dynamic classification range, that can go both up and down, but that’s separate topic, how to calculate classification based on classifier results and their average curPercent/highPercent. 

 

Edited February 27 by CutePibble
Not

[shred]

Using A class may be tricky.  IME shooters often don't stay there long and some are speedy kids already shooting M & GM scores that haven't been caught up to by the lag in the system (admittedly that's less now with weekly runs).  Validating against M is probably a good idea.  B is probably too noisy.

 

 

[Joe4d]

18 hours ago, broadside72 said:

If you find your match perf is lower than your class then you are shooting only the classifiers that don't reflect typical match/stage shooting challenges and the system is flawed.

 

 

 

point I have been making for decade, survivor bias...  
Clubs pick shooting challenge, low round count classifiers,, (or multi string)  then build track meet hose fest stages.. with easy shooting but much more movement.  Its what it is.. Not knocking the stages, obviously it is catering to folks that enjoy it..
If you are going to score people by class at matches , then those classes should be set by matches or simulated match conditions.. Unless things have drastically changed, that simply isnt the case. 
Do they still use major performance as classifier ? dont recall, does classification keep your highest scores or most recent for use leveling up ?
 

[Maximis228]

4 minutes ago, Joe4d said:

Do they still use major performance as classifier ? dont recall, does classification keep your highest scores or most recent for use leveling up ?
 

 

3+ GMs need to shoot above a certain % to count as a classifier for your % of match finish. 

[RJH]

11 minutes ago, Joe4d said:

point I have been making for decade, survivor bias...  
Clubs pick shooting challenge, low round count classifiers,, (or multi string)  then build track meet hose fest stages.. with easy shooting but much more movement.  Its what it is.. Not knocking the stages, obviously it is catering to folks that enjoy it..
If you are going to score people by class at matches , then those classes should be set by matches or simulated match conditions.. Unless things have drastically changed, that simply isnt the case. 
Do they still use major performance as classifier ? dont recall, does classification keep your highest scores or most recent for use leveling up ?
 

 

 

The problem with your "survivor bias" theory is you're wanting to adjust classification based on people who don't stay in the sport.

 

Clubs pick small classifiers because they're easier to set up and less of a headache. Having a classifier that requires several positions that are not boxes and has several walls to set up is never going to be effective because it's never going to be set up consistently from club to club

 

And you still tend to miss the point of when you go to a major the results end up falling in the way you would expect which means the classification system isn't very far off. I know you're going to think survivor bias, but the people sitting at home on the couch really don't matter and adjusting any thing we do based on trying to get them involved in the sport is a fool's errand. I know this because of how many people I've tried to get to shoot over the years and how few actually do. Many new guys show up to one match or two and quit because it's not their thing, not because of the classification system screwed them

[Joe4d]

3 minutes ago, RJH said:

 

 

The problem with your "survivor bias" theory is you're wanting to adjust classification based on people who don't stay in the sport.

 

Clubs pick small classifiers because they're easier to set up and less of a headache. Having a classifier that requires several positions that are not boxes and has several walls to set up is never going to be effective because it's never going to be set up consistently from club to club

 

And you still tend to miss the point of when you go to a major the results end up falling in the way you would expect which means the classification system isn't very far off. I know you're going to think survivor bias, but the people sitting at home on the couch really don't matter and adjusting any thing we do based on trying to get them involved in the sport is a fool's errand. I know this because of how many people I've tried to get to shoot over the years and how few actually do. Many new guys show up to one match or two and quit because it's not their thing, not because of the classification system screwed them

not really, vast majority of members dont shoot majors,, again, just more survivor bias... Guys who can shoot, but not run, simply dont go to majors. Might still enjoy and be active in the sport.. But even then, probably still wont go to majors.  
I mean if people are happy who cares,, but if the class system is to be accurate, then matches and whatever system you use should be similar.
I mean,, class is a goal in itself for alot of people,, kinda a 1 man match to measure yourself against.. In that case its calibration isnt all that important.. but lets not kid ourselves its representative of match performance.

Hvae to look and see how match bumps work, been awhile.. but even then most people dont shoot majors.. Also,,, I think alot of people just look at overall anyways at the club level..  Why if I shoot USPSA, I'll just shoot my iron sight XDM in a Open,, just to enjoy peoples heads explode.

[RJH]

9 minutes ago, Joe4d said:

not really, vast majority of members dont shoot majors,, again, just more survivor bias... Guys who can shoot, but not run, simply dont go to majors. Might still enjoy and be active in the sport.. But even then, probably still wont go to majors.  
I mean if people are happy who cares,, but if the class system is to be accurate, then matches and whatever system you use should be similar.
I mean,, class is a goal in itself for alot of people,, kinda a 1 man match to measure yourself against.. In that case its calibration isnt all that important.. but lets not kid ourselves its representative of match performance.

Hvae to look and see how match bumps work, been awhile.. but even then most people dont shoot majors.. Also,,, I think alot of people just look at overall anyways at the club level..  Why if I shoot USPSA, I'll just shoot my iron sight XDM in a Open,, just to enjoy peoples heads explode.

 

 

I agree most members don't shoot majors, but even at locals the classification system still holds up. The local GM generally beats the local master who beats the A-Class who beats the B class, etc.

 

The only time that changes is when you have somebody who really starts practicing moves up quick and happens to be an old Master class shooter who can't keep up anymore.

 

But overall it still works. I knew it worked that way at my local club, I checked some majors and it still worked that way, I even went so far as to check local matches I've never been to or seen anybody at and sure enough the results tend to fall in order gm, m, a, B, etc

 

So if people keep finishing matches where they're supposed to how badly broke can the system be? I'm going to go with it's not too bad. But as I've said many times in this thread if something is really wonky getting it corrected should be done.

 

But somebody being mad because they can't make a class because they didn't practice hard enough is not a good reason to change the system. Just like if somebody shoots into a class but can't complete against upper people in that same class, that's not a good enough reason to change the system. They just need to practice more

Edited February 27 by RJH

[-JCN-]

You can be an excellent match finisher in a high cap division and just not have a well rounded weak hand and reload game. 
 

Similarly, you can be an excellent static shooter but have orthopedic limitations that affect field course performance. 
 

IMO they test different things and are only a rough estimate of core skill competence. Nothing to get too bent out of shape on either way. Again JMO. 

[motosapiens]

16 minutes ago, -JCN- said:

You can be an excellent match finisher in a high cap division and just not have a well rounded weak hand and reload game. 
 

 

this is known as “not practicing enough”.

 

you know almost all majors (and many local matches) will require SH/WH shooting and reloads, so if u want to do well you must practice those things at least a little.

[-JCN-]

Just now, motosapiens said:

this is known as “not practicing enough”.

 

you know almost all majors (and many local matches) will require SH/WH shooting and reloads, so if u want to do well you must practice those things at least a little.

Enough for what, though?

 

If they do well enough at matches and are happy with that, then great. 
 

We have a local M that just enjoys field courses and is a young, fast guy. 
 

He finishes top 20 CO routinely at Nationals. 
 

He practices a lot, but doesn’t prioritize those things. 
 

Is he a better shooter than a lot of GMs? Sure. He’s happy with M because of classifier skill weighting. 
 

That happened a lot with PCC shooters that came from actual rifle and 2 gun backgrounds. They shot great but reloads weren’t a strong part of their game. 
 

It’s disproportionately represented in classification.  
 

I don’t expect stand and shoot core skills to necessarily translate directly into match performance. There’s just so much more to the sport. 
 

As long as classification is loosely proportional, it doesn’t bother me. It can’t be all things to all people. 

 

[Maximis228]

SHO and WHO makes up less than 5% of overall points available at a match. Usually its less than 2.5-3%. 

 

This is why no one cares to practice it. Its barely even a part of the game to begin with as is. 

[waktasz]

That last few major matches I've shot it made up 0%

[MoNsTeR]

"the classification system as a whole basically works"

 

"individual classifier stages are well-calibrated"

 

These are not the same statement, and one being true or false implies nothing about the other. Some folks seem confused about that.

[VeilAndrew]

Could just look at Lightning and Thunder to see a messed up classifier. I think most C class dudes could realistically pull A/M runs out of that without trying too hard. It's the highest GM% classifier in nearly all divisions.

[motosapiens]

2 hours ago, Maximis228 said:

SHO and WHO makes up less than 5% of overall points available at a match. Usually its less than 2.5-3%. 

 

This is why no one cares to practice it. Its barely even a part of the game to begin with as is. 

its been part of every national match i’ve done, and majority of l2 and l3 matches as well. im not good enough to give those points away, but people who don’t care about their scores at major matches are encouraged to follow their hearts.

[-JCN-]

1 hour ago, motosapiens said:

its been part of every national match i’ve done, and majority of l2 and l3 matches as well. im not good enough to give those points away, but people who don’t care about their scores at major matches are encouraged to follow their hearts.

You’re conflating. 
 

They’re not throwing the points away. They’re spending time getting better at other things that are more representative of a match. 
 

If they’re B class competent weak handed and reloading, it’s not going to cost them much if they spent the time practicing other things like movement instead of spending the time pounding classifier skills that don’t factor predominantly in every stage. 
 

Like I said, I’m talking about a national Top 20 CO guy who prioritizes field course skills over classifier practice and it works out well for him.

 

People have different priorities from each other and there’s not just one way to do it.

Edited February 27 by -JCN-

[llamasabound]

6 hours ago, -JCN- said:

Enough for what, though?

 

If they do well enough at matches and are happy with that, then great. 
 

We have a local M that just enjoys field courses and is a young, fast guy. 
 

He finishes top 20 CO routinely at Nationals. 
 

He practices a lot, but doesn’t prioritize those things. 
 

Is he a better shooter than a lot of GMs? Sure. He’s happy with M because of classifier skill weighting. 
 

That happened a lot with PCC shooters that came from actual rifle and 2 gun backgrounds. They shot great but reloads weren’t a strong part of their game. 
 

It’s disproportionately represented in classification.  
 

I don’t expect stand and shoot core skills to necessarily translate directly into match performance. There’s just so much more to the sport. 
 

As long as classification is loosely proportional, it doesn’t bother me. It can’t be all things to all people. 

 

I’m sympathetic to your example, but the local shooter you described doesn’t exist.

 

There were three Ms in the top 20 of CO last year, two of whom are national champions in other divisions.

 

There was one M in the top 20 at CO nationals in 2022 (who was not in the top 20 last year).

 

There is as one M in the top 20 at CO nationals in 2021 (who is now a G and was 59th last year).

 

If you go back to 2020 and years earlier, the field is so scattered and weak as to be non comparable. Top 20 in 2019 was 80%, ~90th last year.