Changes To The USPSA Classfication System

[Chills1994]

You really are that dense.

 

There is a whole bunch of science strictly dedicated to randomly selecting subjects and collecting data from them.

 

Not one single nightly newscast goes by where there is not at least one mention of "Researchers discovered that people who ___________ lowered their chances of ____________.

 

For any experiment or survey to be legit it needs to be valid and reliable.

 

Valid means it is measuring what it is actually measuring.

 

Reliable means that the experiment or survey is repeatable with the same results.

 

So by drawing some conclusion from just major match results....those match results are NOT a random sample, and therefore, not valid.  You are NOT measuring what you really think you're measuring.

 

If you cannot understand this very simple academic discussion, you cannot be helped.

 

Good day to you too, sir!

 

P.S. this is just anecdotal, but the lowest placed Master at the 2016 finished 263. So obviously proof that 100% the classification system works.  (that was sarcasm) 

Edited August 14, 2017 by Chills1994
homonyms

[Sarge]

IBTL

[teros135]

You guys are really funny 

[Blueridge]

21 hours ago, Chills1994 said:

You really are that dense.

 

There is a whole bunch of science strictly dedicated to randomly selecting subjects and collecting data from them.

 

Not one single nightly newscast goes by where there is not at least one mention of "Researchers discovered that people who ___________ lowered their chances of ____________.

 

For any experiment or survey to be legit it needs to be valid and reliable.

 

Valid means it is measuring what it is actually measuring.

 

Reliable means that the experiment or survey is repeatable with the same results.

 

So by drawing some conclusion from just major match results....those match results are NOT a random sample, and therefore, not valid.  You are NOT measuring what you really think you're measuring.

 

If you cannot understand this very simple academic discussion, you cannot be helped.

 

Good day to you too, sir!

 

P.S. this is just anecdotal, but the lowest placed Master at the 2016 finished 263. So obviously proof that 100% the classification system works.  (that was sarcasm) 

Reliable means that the experiment or survey is repeatable  with the same parameters set and the results of the experiment fall within an acceptable margin of error.

 

 

[CHA-LEE]

The classification system is a primary performance improvement motivator for a lot of USPSA members regardless of what level matches they attend. I think this is a good thing as it gives shooters something to strive for and is a fairly accurate performance measurement to rate ones fundamental shooting and gun handling skills.

 

The Classification system isn't new as its just as old as USPSA its self. Its also a significant revenue generator for USPSA from clubs setting up classifier stages all across the nation at least 3 - 4 times a week as clubs host Weekend and Weekday matches.

 

What baffles me about the Classification system is how USPSA seems to put very little effort into actually validating that existing High Hit Factors are accurate. Its even more perplexing how High Hit Factors are initially set when new classifiers are released or new divisions are deployed. I know for a fact that many of the 09 and 13 based classifiers had significantly lower than expected High Hit Factors when they were initially released. Then the High Hit Factors were updated later to reflect reality. But to me, this is the lazy mans method of defining the High Hit Factor numbers. USPSA could just at easily go into data collection mode for 3 - 6 months on new classifiers or divisions in order to set some kind of baseline performance level to set the new High Hit factor accurately. Once they have enough data points to make an accurate judgement of what the High Hit Factor should be THEN they should deploy the classifier as a valid one that counts against competitors classification record.

 

Vetting new classifiers or new divisions on existing classifiers can be done effectively in many different ways. But I am yet to see USPSA leverage any method consistently. What I have seen is the new High Hit Factors get picked for some unknown reason and then it allows for invalid classification results to be recorded on members results until the high hit factor is adjusted to reality. To me, that is the biggest injustice of this whole process. USPSA claims to provide a product that accurately rates your performance, but any time a new classifier or division is deployed they simply "Wing It" and allow inaccurate classifications to happen.

 

The most recent case in point would be PCC. Many of the PCC high hit factors are based on the Open division high hit factor which for a lot of stand and blast classifiers with no reloads is a joke to compare one to the other. We have well established "Lifer" B and A class Pistol shooters who can magically generate a GM classification in PCC because the high hit factors are jacked up. This level of product inconsistency can only happen when the people behind the scenes (USPSA Brass) are asleep at the wheel.

 

I also don't understand how USPSA can award classifications on divisions that are still in a "Provisional" status and the gun/gear rules are still in flux. (The same could be said for having National Championships for provisional divisions) At a minimum I would expect that all classifier results for "Provisional" Divisions would be suspended until the division exits provisional status and becomes a normal division. If they did that it would give them enough classifier data during the provisional period to at lest come up with a fairly accurate High Hit Factor once the division exits the "Provisional" status.

 

Would shooters have to wait longer until they became classified in these new divisions? Absolutely. But once the classification system was turned on for the new division at least the high hit factor would be valid and based on existing results.

[motosapiens]

6 minutes ago, CHA-LEE said:

What baffles me about the Classification system is how USPSA seems to put very little effort into actually validating that existing High Hit Factors are accurate. Its even more perplexing how High Hit Factors are initially set when new classifiers are released or new divisions are deployed. I know for a fact that many of the 09 and 13 based classifiers had significantly lower than expected High Hit Factors when they were initially released. Then the High Hit Factors were updated later to reflect reality.

 

I recall that some of the 13-series classifiers were run at nationals or other major matches, and the division hhf's were (for a time) the scores of the folks who won that stage at nationals. I even looked up some of those stage scores from the match while waiting for the monthly updating of the cuneiform classification tablets. Of course they were lower than normal, because people were shooting for match score instead of hero or zero to impress people on the internet.

[CHA-LEE]

19 minutes ago, motosapiens said:

 

I recall that some of the 13-series classifiers were run at nationals or other major matches, and the division hhf's were (for a time) the scores of the folks who won that stage at nationals. I even looked up some of those stage scores from the match while waiting for the monthly updating of the cuneiform classification tablets. Of course they were lower than normal, because people were shooting for match score instead of hero or zero to impress people on the internet.

 

I understand that there was an "Effort" made by USPSA to acquire data for setting the High Hit Factors. But it is suspect when that "Effort" produces invalid data compared to normal classifier results in club matches. The fact of the matter is that classifiers will always be shot to max effort in club matches verses majors. If our high hit factors are based on the "max effort" club match results then that is where the data mining should be coming from. This further enforces my comment that USPSA isn't putting much thought or effort into this stuff before they deploy it for prime time. 

Edited August 14, 2017 by CHA-LEE

[Bkreutz]

1 hour ago, CHA-LEE said:

I also don't understand how USPSA can award classifications on divisions that are still in a "Provisional" status and the gun/gear rules are still in flux. (The same could be said for having National Championships for provisional divisions) At a minimum I would expect that all classifier results for "Provisional" Divisions would be suspended until the division exits provisional status and becomes a normal division. If they did that it would give them enough classifier data during the provisional period to at lest come up with a fairly accurate High Hit Factor once the division exits the "Provisional" status.

 

Would shooters have to wait longer until they became classified in these new divisions? Absolutely. But once the classification system was turned on for the new division at least the high hit factor would be valid and based on existing results.

 

Especially since high classification in one division reflects on all other divisions. So much for the word "Provisional" when dealing with a "boutique" division.

[Thomas H]

On 8/13/2017 at 6:14 PM, Chills1994 said:

You really are that dense.

 

There is a whole bunch of science strictly dedicated to randomly selecting subjects and collecting data from them.

 

Not one single nightly newscast goes by where there is not at least one mention of "Researchers discovered that people who ___________ lowered their chances of ____________.

 

For any experiment or survey to be legit it needs to be valid and reliable.

 

Valid means it is measuring what it is actually measuring.

 

Reliable means that the experiment or survey is repeatable with the same results.

 

So by drawing some conclusion from just major match results....those match results are NOT a random sample, and therefore, not valid.  You are NOT measuring what you really think you're measuring.

 

If you cannot understand this very simple academic discussion, you cannot be helped.

 

Good day to you too, sir!

 

P.S. this is just anecdotal, but the lowest placed Master at the 2016 finished 263. So obviously proof that 100% the classification system works.  (that was sarcasm) 

 

You might stop demonstrating that you are continuing to miss the point.

 

"However, since that isn't the question at hand, it does show rather clearly that you don't seem to understand the problem, and also makes clear why you keep saying "sampling bias" when it doesn't apply.  It also shows that you ignored what I said about how the total population for this question is not the total population of everyone who shoots USPSA.

 

Since you don't seem to actually understand the statistical analysis you are attempting to discuss, we can stop here.  Good day."

 

 

Back to the actual point that other people are discussing:

 

I agree that the PCC HHF are....just wrong.  I wish they would have simply not had ANY HFF (and kept everyone unclassified) rather than chose an already-existing set that doesn't fit well at all.  Sure, some of them are close to realistic with no obvious differences, but others are obviously extremely wrong.

 

Cha-lee, I'm betting that the HHF haven't been updated or analyzed because 1) prior HQ "leaders" didn't feel it was a priority and didn't really care, and 2) current HQ leaders think it is important but not a high enough priority with other fires to put out first.  (Whether that belief is correct or not is a different issue.)  In addition, I'm thinking that analyzing HF scores and attempting to reset the HHF will cause LOTS of whining, complaining, and angry yelling.  (Do we use the 10 highest-ever scores achieved by people who heroed it once by pure luck?  Do we set a "C" by the center of a statistical distribution and then scale up?  What if the distribution isn't symmetrical (which it won't be)?    Do we pick current GMs and average their times in local matches?  Or in majors, where they might shoot it differently, and where it shows (more likely) their "real" ability?  What do we do about the fact that the highest-ever HHFs on any particular classifier have probably increased over time for the most part?)

 

I think that having realistic (and reasonable) HHF to gauge skill would be nice.  At the same time, I think that having at least some sort of external comparison to use as a relative gauge seems to be working---people work to get better, people (in major matches, let's not make certain people mad again) tend to mostly finish according to their class, and so on.

 

Between the potential argument about how to reset HHFs, and the fact that the current system is (mostly) reasonable at major matches, I'm betting it is just a low priority for USPSA.

 

 

[CHA-LEE]

Thomas H> I totally understand where you are coming from. And I am sure that a big reason why a complete rework of the classification system hasn't happened simply because it works fairly accurately for the established divisions. 

 

My primary beef is the total lack of effort or forethought put into it for New Divisons or Classifiers. Deploying a new division requires a proactive effort to get the 100% HHF's accurate. What we have seen today with both Carry Optics and PCC is a complete joke. 

 

Classifier fees are a HUGE revenue generator for USPSA. Why they wouldn't want that revenue stream running as accurate and solid as it possibly could is beyond my comprehension. 

 

USPSA isn't run by volunteers. There are paid employees on staff to make sure this stuff is done right. If I was tasked with that effort at my day job and made a monkey show of it like USPSA has I would be fired. Businesses can't neglect revenue streams like that and expect to be successful.

 

USPSA always has the "Grow the Sport" marching orders for its membership and sanctioned clubs. It's a little strange for them to make that type of mandate when the business it's self isn't even looking after their fundamental products designed to service the membership. I say, get your house in order then worry about others jumping through hoops to grow the sport.

[Chills1994]

or #3.  the money from the classifiers scores that were submitted was still rolling in.

 

Don't kill the cash cow or the goose that lays the golden eggs.

[CHA-LEE]

The other thing I want to point out is that screwing up the HHF's in a way that makes it too easy to make a higher classification automatically dilutes the classification achievement others have made in the established divisions. 

 

For example, a B class pistol shooter can cherry pick the easy 100% classifiers in PCC. Then generate a GM classification in PCC. This bumps their Limited classification to Master automatically. 

 

This situation dilutes the value of the pistol classifications. 

[Chills1994]

14 minutes ago, CHA-LEE said:

Thomas H> I totally understand where you are coming from. And I am sure that a big reason why a complete rework of the classification system hasn't happened simply because it works fairly accurately for the established divisions. 

 

My primary beef is the total lack of effort or forethought put into it for New Divisons or Classifiers. Deploying a new division requires a proactive effort to get the 100% HHF's accurate. What we have seen today with both Carry Optics and PCC is a complete joke. 

 

Classifier fees are a HUGE revenue generator for USPSA. Why they wouldn't want that revenue stream running as accurate and solid as it possibly could is beyond my comprehension. 

 

USPSA isn't run by volunteers. There are paid employees on staff to make sure this stuff is done right. If I was tasked with that effort at my day job and made a monkey show of it like USPSA has I would be fired. Businesses can't neglect revenue streams like that and expect to be successful.

 

USPSA always has the "Grow the Sport" marching orders for its membership and sanctioned clubs. It's a little strange for them to make that type of mandate when the business it's self isn't even looking after their fundamental products designed to service the membership. I say, get your house in order then worry about others jumping through hoops to grow the sport.

thread derail in progress....

 

IMO, USPSA HQ does a very piss poor of helping their "franchise" clubs.

 

(sung to the tune of Janet Jackson's "what have you done for me lately")

 

How many classifier stages require a full size pepper popper?

 

Has USPSA HQ partnered up with any AR500 target manufacturers to provide reduced pricing for newly affiliated/chartered clubs?

 

Maybe have a "new club starter pack" for steel targets???

 

back on topic...

 

back when single stack became a legit division...what? like 2007 or 2008? the SS high hit factors were the exact same as L10.  It was pointed out on this very forum.  (and the dude who signaled the alarm might have received a time out for it too)  Soooo... this charlie foxtrot with HHFs for CO and PCC should not really be a surprise.

 

I am sure I have harped on it in this thread and in other places on this forum, but all the more reason to go with a records page of the top 5 or top 10 highest hit factors shot  on each classifier per division.

 

There are enough nosey busy bodies in USPSA who would be stalking the records pages to email somebody at USPSA HQ if they spotted some bullshit.

 

 

 

 

 

 

 

 

 

[motosapiens]

On 8/18/2017 at 8:42 PM, Chills1994 said:

back when single stack became a legit division...what? like 2007 or 2008? the SS high hit factors were the exact same as L10.  

 

 

not much has changed. there are very few classifiers where the HHF for SS is not the same as Limited and L10. Having just switched from SS to Limited, I can assure you that a real limited gun (heavy, full length dust cover) is easier to shoot than SS....... but not really any more fun, so whatever.

[pjb45]

Charlie is right on!

 

I have over 12 hours of statistical and research methodology behind me plus more than a few years in fundamental (physics) and applied (social) research.

 

There are various methods of eliminating outliers.  None are absolute.  Remember statistics uses confidence interval  and a =/- an error factor.

 

HHF are not absolute.  Take the El Prez, so many people practiced it, the HHF just increased more each year.  It is truly representative of overall performance or just of that classifier?

 

USPSA probably needs to have a real statistician review they data and make some recommendations.

 

I know that hospital labs usually require a sample of at least 35 before they will even begin to be comfortable with results.  Even then, the sample size is quite small.

[busdriver02]

The MD at my local club is on the top 20 GM list. My current classification is within 1% of my average match score compared to him.

If the point of the classification system is to roughly predict match scores at a national level, it does a fairly good job.

[rowdyb]

I have to disagree with the above, at least for Production. Very few people shoot their classification at Nationals. Only 6 of the 41 GM shot their classification percentage or higher.

 

Of the M shooters only 6 of 63 master prod shooters shot their classification percentage. And in A only 5 of 72 shooters did.

 

14% of GM,  9.5% of M and 7% of A shooters shot their classification percentage at nationals. That seems hardly indicative of an accurate representation.

 

Let's look at well attended Area match, A 1. Again in Prod.

 

There were 8 GM, 15 M and 16 A shooters. Of the GM only 2 shot their classification. In M the number is 2. For A class it is 3.

 

25% of GM, 13% in M and 19% in A. Once again, a large majority of shooters can't shoot their classification in national or area matches.

 

Classification is a poor indicator of predicted match performance/placement. And if classification can't do a good job or predicting that, what is it good for?

Edited August 23, 2017 by rowdyb

[AWLAZS]

On 8/11/2017 at 6:13 PM, Chills1994 said:

It's still like a buck fifty for the activity fee and then like another buck fifty to get the classifier score submitted, right?

 

Just scrap the classifier system and then make the activity fee $3 .

 

USPSA HQ still gets their "per capita" dues paid in from each of the "franchises" to the same total amount.

 

 

1

Some folks shoot for the classification card. Without the benefit of classification, they have no reason to be a member of uspsa.   

[motosapiens]

1 hour ago, rowdyb said:

I have to disagree with the above, at least for Production. Very few people shoot their classification at Nationals. Only 6 of the 41 GM shot their classification percentage or higher.

 

 

might be a better idea from a math standpoint to take an average of the top 5 or 10 at nationals, and see what the percentage was of that, rather than percentage of the winner.

 

I think when someone points out that the classification system is a good predictor of performance at matches, they mean that in general, most of the gm's are ahead of most of the m's who are ahead of most of the a's.

Edited August 23, 2017 by motosapiens

[rowdyb]

1 hour ago, motosapiens said:

 

might be a better idea from a math standpoint to take an average of the top 5 or 10 at nationals, and see what the percentage was of that, rather than percentage of the winner.

 

I think when someone points out that the classification system is a good predictor of performance at matches, they mean that in general, most of the gm's are ahead of most of the m's who are ahead of most of the a's.

it's fun to see we've never agreed on a single thing.

[ziebart]

Anecdotal, but the last couple lvl 2 and higher matches I've shot hae come out almost exactly at my classifier percentage. The last was against Max at the NM sectional. 

 

Classifiers are tough since they, like the rest of the sport are graded on a curve. You don't know what the 100% will be till everyone shoots it. You can jave a good idea after a couple of the best shoot it, but it is always possible someone beats it. Compared to trap, skeet, bullseye where you know what you need to score to get 100%. Yes someone can make a perfect score including full x counts but once enough people do that then bullseye makes the target smaller. Fixed time stages are the closest we have to that, yet we don't shorten the time.

[BritinUSA]

I think the difference between classification and real match scores are more a result of stage design than ability. Most of the classifiers require little to no movement and apart from a few multiple string stages, the round count is generally small. Large matches have stages with higher round counts and multiple shooting positions, its the movement around the stage that really seems to separate the top tier competitors from the rest. Those differences become more apparent with larger stages.

 

I think that the only true classification measurement is the competitor's score at a major match with all the variety of stages and round counts, and no opportunity to practice those stages beforehand.

 

If USPSA abolished the classification system then there would be a financial impact; It is worth noting that with all the match data that Practiscore has they could probably create their own classification metric based off all match results and bypass the USPSA system all together.

[pjb45]

Busdriver:  A single data point is not statistically significant.  Exceptions do not prove the rule.

[Gooldylocks]

On 8/23/2017 at 7:55 AM, motosapiens said:

I think when someone points out that the classification system is a good predictor of performance at matches, they mean that in general, most of the gm's are ahead of most of the m's who are ahead of most of the a's.

Correct

[GJM]

Comparing PCC to Open, since they share the same HHF, classifiers shot by PCC with a reload are generally harder than Open, and those without a reload are generally easier for PCC. Since the change from a PCC low ready start to butt on belt, the PCC start advantage has shrunk by .30 or .40.

 

With Carry Optics, the HHF are very high, and this is confirmed by the top 20 list, where there are only 3 GMs, and two of those are now 89 and 90 percent. 

 

I do do not think a PCC classification should cause a class bump in other handgun divisions.