Why don’t classifications go down?

[IVC]

On 12/5/2019 at 12:46 PM, B_RAD said:

Looks like half the advancements. Not gonna lie, in some ways that actually might be good. 

Why?

 

If the classification is driven not only by the top shooters, but by the top shooters shooting something they can both practice and go for hero/zero, all you're getting is a system that doesn't really address shooting ability and the bands of ability (classes) as a concept becomes meaningless. That's what's happening already. 

 

If the IDPA Master class is too broad because there is a lot of difference between entry-level Masters and the top shooters, USPSA is now doing the same to the mid-level classes by preventing top shooters in those classes from advancing. USPSA will end up with a few very narrow bands at the top and everyone else will be a B class, so, much like IDPA master class, there will be a huge difference between entry-level B class and the top of the B class.

 

Going for broke on a classifier to move up will be the only way to advance in classes and this is something that only those who care about classes will do. At local matches you can already see a large fraction of G/M/A shooters rarely get their own class when a classifier is just another stage. It's partly because some of them might practice a bit less at different times of their lives, partly because they don't want to lose a match on a classifier. If they didn't have their class in the bag already, many of them would be a class or two below their current classification. 

 

I don't really care either way, but having a meaningful classification system where it can be used to track progress of a shooter and provide a (semi) meaningful way to assess shooters all over the world is a good thing. Making it into "very narrow + very broad" two-class system defeats that goal. 

Edited December 7, 2019 by IVC

[oddjob]

Many years ago when I played golf we used to punch in our scores along with the slope reading (course difficulty) into a computer at the course.  You then would get your handicap.  Your handicap did fluctuate constantly.  I wonder if USPSA could do something similar and therefore your classification (like a golf handicap) would and/or could change.   If you wanted to stay at a certain classification level you would have to keep up through the classifiers or major matches.  This would take into account factors (like age) that make shooters shoot less then their current classifications. 

 

The people I played golf  with always were gunning for a lower handicap... sorta for bragging rights at the 19th hole (bar).  Far as sandbaggers.....golf at the time had a lot!

 

 

[lstange]

On 12/7/2019 at 8:53 PM, IVC said:

having a meaningful classification system where it can be used to track progress of a shooter and provide a (semi) meaningful way to assess shooters all over the world is a good thing. Making it into "very narrow + very broad" two-class system defeats that goal. 

We already have that, it's called classification percent. No matter how you slice it into six bands, you inevitably lose information.

 

A percentile would be even better. Without knowing the shape of the distribution it's hard to say whether 42 is a good classification percent, but "42nd percentile" has an intuitive explanation - in a match with 100 other shooters selected at random you would expect to beat 42 and be beaten by 58. And it's not hard for USPSA HQ to do, just one SQL query.

On 12/7/2019 at 8:53 PM, IVC said:

you can already see a large fraction of G/M/A shooters rarely get their own class

That's by design. Results 5% below the bottom of your current classification band are not used in the average, two worst results out of six are discarded, if classifier is re-shot then the better result counts, all those things bias classification percent up. As long as everyone knows it and is treated the same, it's not a big deal. No matter how hard a B class shooter tries to hero-or-zero, he's still not making a GM. Ask me how I know

[IVC]

17 minutes ago, lstange said:

We already have that, it's called classification percent. No matter how you slice it into six bands, you inevitably lose information.

 

A percentile would be even better. Without knowing the shape of the distribution...

Percent is relative to the HHF so it’s not as informative as you might think.
 

For example, consider a robocop sets a HHF at twice the current value. The GMs are now at 48%+, M is at 43%, A is at 38%. Three bands that are currently the top, all crammed at the D/C boundary - the small percentage difference doesn’t capture the difference in skill because intuitively it’s all “within a few percent.”

[lstange]

6 minutes ago, IVC said:

Percent is relative to the HHF so it’s not as informative as you might think.

Nah, it's good enough. Existing classification percent predicts major match placement with mean absolute error of 8.6%. The best I could do (using trimmed mean classification percentile) was 7.5%. This juice is not worth the squeeze.

 

There are a few classifiers that are unusually hard, e.g., 03-11, 03-12, 09-03 in Production. But they are not nearly as lopsided as in your example. Apparently USPSA throws out robocop runs when setting HHFs.

[IVC]

2 hours ago, lstange said:

Existing classification percent predicts major match placement with mean absolute error of 8.6%.

I would assume you mean the score at major matches, not necessarily the placement, correct?

[lstange]

8 hours ago, IVC said:

I would assume you mean the score at major matches, not necessarily the placement, correct?

The dependent variable was match percent at major matches, but it determines match place so the story is the same.

[IVC]

2 hours ago, lstange said:

The dependent variable was match percent at major matches, but it determines match place so the story is the same.

That's what I thought and thanks for the actual numbers.

 

I am surprised that it's that close - I looked recently at Hi-Cap Nationals and in Limited only the top 5 guys shot GM percentage of the top shooter. There were 36 GM-s total, so only 14% of GM-s shot at 95%+. No Master reached 85% and only 11 of 45 M-s even shot an A class. Similarly, only the top 3 of 43 A class shot 75%+. It's a bit more consistent at the bottom, with top 18 of 68 B and 21 of 32 C shooters shooting their class. 

 

So, of the top shooters (GM/M/A) only (5+0+3)/(36+45+43) = 8/124 = 6% shot their class. I'd call that a "problem" with the current system and it's only going to get worse. 

[lstange]

2 minutes ago, IVC said:

So, of the top shooters (GM/M/A) only (5+0+3)/(36+45+43) = 8/124 = 6% shot their class. I'd call that a "problem" with the current system and it's only going to get worse. 

The slope of the regression line is less than one. C class shooters get about 50% match percent at the majors, but for B class and above match percent is lower than classification percent. One would expect to see such bias knowing the way classification is calculated. There are no re-shoots at major matches, and all stages count.

 

But rank ordering is roughly preserved. if you order, say, shooters at the Nationals by their classification percent and take 42nd from the top, chances are he'll finish around 42nd place.

 

[B_RAD]

29 minutes ago, IVC said:

That's what I thought and thanks for the actual numbers.

 

I am surprised that it's that close - I looked recently at Hi-Cap Nationals and in Limited only the top 5 guys shot GM percentage of the top shooter. There were 36 GM-s total, so only 14% of GM-s shot at 95%+. No Master reached 85% and only 11 of 45 M-s even shot an A class. Similarly, only the top 3 of 43 A class shot 75%+. It's a bit more consistent at the bottom, with top 18 of 68 B and 21 of 32 C shooters shooting their class. 

 

So, of the top shooters (GM/M/A) only (5+0+3)/(36+45+43) = 8/124 = 6% shot their class. I'd call that a "problem" with the current system and it's only going to get worse. 

In my opinion you have to accept that the top 3-5 competitors or a level above legit GM’s. Max, JJ, Ben, Nils, Christian(all won Nats this year). Take them out (and a few others) or add them to a “Pro” class, and then most everyone finishes within their percentage of classification.  I’m not saying that’s what should be done or that there won’t always be a few outliers but it would put the percentage issue at Nats back in line. 

 

so, if there is a problem, is it people not finishing in their  classification percentage? Or is it that the shooters that are in contention to win, actually need to be in a higher class?

Edited December 10, 2019 by B_RAD

[motosapiens]

53 minutes ago, IVC said:

So, of the top shooters (GM/M/A) only (5+0+3)/(36+45+43) = 8/124 = 6% shot their class. I'd call that a "problem" with the current system and it's only going to get worse. 

 

why is that a problem?

[nasty618]

57 minutes ago, B_RAD said:

add them to a “Pro” class, and then most everyone finishes within their percentage of classification

You think IDPA may have been onto something with their Distinguished Master class? 

 

Quote

The IDPA Classifier is a match that classifies shooters. Distinguished Master, however, is only attained by winning the Division Champion title or scoring within 3% of the Division Champion score at the IDPA U.S. Nationals, IDPA U.S. Indoor Nationals, or the IDPA World Championship.

 

[B_RAD]

7 minutes ago, nasty618 said:

You think IDPA may have been onto something with their Distinguished Master class? 

 

 

Maybe in theory. I honestly don’t know what IDPA did it?   
 

 

[nasty618]

I think recognizing the top National level shooters with a separate class is warranted. A class that you cannot get on a classifier, but have to earn at a handful of matches.  Perhaps same matches that are used for the World Shoot Team selection process. 

 

I think this will address a lot of "classification is broken" concerns, highlighting the purpose of the system that was designed to classify a large number of participants from a common database.  I think it might address the "paper GM" BS  as well, but who knows.

 

[IVC]

3 hours ago, motosapiens said:

why is that a problem?

It's a problem because the whole classification system loses relevance. If you use it to track personal progress, like majority of people do, neither your class nor your percentage reflect your ability as a shooter.

 

You can already just ignore classification completely and simply look at where you and up in the Nationals, much like you can completely ignore the handicap in golf and look where you end up in the majors. Of course, in both cases you have the same problem - most people never make it to the top competition, most people don't even care to try, they just want to enjoy the sport and have an idea where they stand. 

 

In the Sig Hi-Cap Nationals I quoted above, there were less than 250 shooters in Limited. That's just a few percent (I believe) of USPSA members with classification in Limited. Not quite a good way to measure progress. 

[IVC]

4 hours ago, lstange said:

But rank ordering is roughly preserved. if you order, say, shooters at the Nationals by their classification percent and take 42nd from the top, chances are he'll finish around 42nd place.

Sure, but that's just saying that the shooting ability and classifier percentage are positively correlated without saying much about how meaningful or useful the current classification system is. Assuming you picked number 42 for its deep philosophical meaning, finding this correlation might be the answer you're looking for, but the question remains undetermined.

 

Also, we will see a change in this correlation now that the HHF-s have changed. Whether it's above the noise level is something to keep an eye on. 

[motosapiens]

13 minutes ago, IVC said:

It's a problem because the whole classification system loses relevance. If you use it to track personal progress, like majority of people do, neither your class nor your percentage reflect your ability as a shooter.

 

i think your class and your match percentage both are relevant, and both reflect your ability as a shooter. They don't have to be identical to be relevant. They are different ways of measuring the same thing. One of them artificially ignores some extremely good scores and most bad scores. The other one doesn't. Forgetting a target on a classifier has no effect on my classification, but forgetting 2 targets at nationals (  ) had a pretty significant effect on my placement there.

 

Quote

In the Sig Hi-Cap Nationals I quoted above, there were less than 250 shooters in Limited. That's just a few percent (I believe) of USPSA members with classification in Limited. Not quite a good way to measure progress. 

I disagree. The *best* limited shooters were there, and that's all you really need to know where you stand.

[IVC]

4 hours ago, B_RAD said:

so, if there is a problem, is it people not finishing in their  classification percentage? Or is it that the shooters that are in contention to win, actually need to be in a higher class?

Neither of the two is really a problem. The problem is that the system that is highly skewed towards the top loses meaning to people in the middle because it creates way too narrow bands at the top and way to broad bands in the middle. It doesn't hurt anyone (except egos, I guess), but it's also not all that useful.

 

It's already getting to the point where people simply look at their times instead of percentages of the HHF on standardized drills such as El Presidente or Bill Drill to measure progress and have an idea about how well/poorly they are doing overall. If Bill Drill at 2 seconds was considered a "master level time" at some time, anyone below it knows they are not all that bad. The fact that most B and some C shooters can shoot at that level tells the story...

[IVC]

1 minute ago, motosapiens said:

I disagree. The *best* limited shooters were there, and that's all you really need to know where you stand.

Heh, probably at your level (I mean this in respectful way). Once you get down past the G/M/A, it's a matter of who showed up... 

[motosapiens]

1 minute ago, IVC said:

Heh, probably at your level (I mean this in respectful way). Once you get down past the G/M/A, it's a matter of who showed up... 

 

If you're talking about overall placement.... yes, to some extent that is always the case. But my experience over the last 6-7 years has been that nationals has a pretty consistent talent pool, and if you finish 125th out of 250 at one, you'll probably finish 100th out of 200 at another, assuming you executed at the same level. I personally pay more attention to percentage of the winner, although that also can be a moving target. The harder the shots are in a match, the greater the separation seems to be between the top guys and everyone else.

[B_RAD]

35 minutes ago, IVC said:

Neither of the two is really a problem. The problem is that the system that is highly skewed towards the top loses meaning to people in the middle because it creates way too narrow bands at the top and way to broad bands in the middle. It doesn't hurt anyone (except egos, I guess), but it's also not all that useful.

 

It's already getting to the point where people simply look at their times instead of percentages of the HHF on standardized drills such as El Presidente or Bill Drill to measure progress and have an idea about how well/poorly they are doing overall. If Bill Drill at 2 seconds was considered a "master level time" at some time, anyone below it knows they are not all that bad. The fact that most B and some C shooters can shoot at that level tells the story...

You said it’s a problem that people aren’t finishing within their classification %?
 

I’m asking is that the issue? Or is the issue, if there is one, the measuring stick used to set the 100%?  Comparing someone’s match finish, in percentage to the winner, is going to yeld a different percentage when a solid GM wins HOA Vs win Max wins. An A class shooter will likely fall within the A class range when the Div is one by a regular GM. 
 

 

The classification system isn’t perfect but the GM’s normally finish at the top. Then the M’s and so on. So, it works about as good as anything probably can. Should there be another class?  I think that makes more sense compared to lowering the classification ranges to fall within the percentages of nationals. Though, I’m not sure it’ll solve anything or even get implemented well. 
 

@motosapiens had a good point. There are kinda two different measuring sticks to asses where you’re at. The classification system and you’re finish at Nats/majors.  For most, those do not line up. 

Edited December 10, 2019 by B_RAD

[lstange]

54 minutes ago, IVC said:

If you use it [classification system] to track personal progress, like majority of people do, neither your class nor your percentage reflect your ability as a shooter.

Only if you try to game it. And there is no point in grandbagging, you'll be only fooling yourself. I certainly don't want to be like that now famous GM who finished 67th of 118 with 62% of the winner in area match.

 

If you study math, you become better at math and your standardized test scores go up. If you study to pass standardized tests, your test scores can go up even faster, but you are not better at math as a result.

Edited December 10, 2019 by lstange
a word

[Chills1994]

On 11/10/2019 at 3:51 PM, StealthyBlagga said:

Simple question: If the classification system is supposed to pit shooters of like ability together, why does classification get locked at the highest class a shooter ever attains?

 

We all get older and less sharp over time,  and HHFs get higher, so shouldn't  the system recognize that reality? Yes, I know there is a mechanism for asking to be downgraded, but my guess is few do this because of the prestige and self-esteem associated with a higher ranking. An automatic downgrade according to current performance would ensure folks were placed in a class reflective of their true ability regardless of ego. An agreeable further benefit would be reduction in the “paper GM” problem, and perhaps a reduced incentive for some folks to work the system to get a GM card by what we might call nefarious means.

 

I get that some folks might sandbag to get into a lower class, but I am confident a suitable algorithm could minimize any gaming of the process. 
 

Thoughts?

Way back when.... say from about 2005 to 2011 ... when I was more active on this forum.. . I used to piss and moan about classification  systems in both USPSA and IDPA... even moreso with IDPA and their easily sandbagg-able 90 round classifier "stage"  Back in late 2007, when USPSA bought the Steel Challenge, I knew it was just a matter of time  before SC would get classifications too.

 

Soooo...back then, my griping about the classification system put me on the admin/moderators's radar here.  I can't remember if I ever got a time out about it though.

 

I remember having done key word searches for both "sandbagg" and "grandbagg".  I want to say "sandbagg" yielded 17 pages of results.  

 

My solution back then that still stands to this day is....

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. Wait for it.

.

.

..

.

....

.

.

.

.

.

. 

Get rid of classifications entirely .

 

Shoot head's up against each other within a division.

 

For a major match, the top finisher in each division gets a trip to the prize table. . Division with the most competitors goes first., then next most populated division, and so on and so forth.  Make some sort of cutoff where like at least 5 or 10 competitors have to be in a division before the division winner gets a trip to the prize table.

 

Then whatever else is left on the prize table gets random'ed off.... basically door prizes.

 

Other than getting rid of the classification system entirely, then my only next most possible solution is to incorporate 3 or more classifier stages at a major match.  What your classifiers there net you score-wise becomes your new classification.  You can't sandbagg those too much or else you'll risk tanking your entire match.

[IVC]

1 hour ago, lstange said:

Only if you try to game it. And there is no point in grandbagging, you'll be only fooling yourself.

We are not talking about the same thing - gaming a system means that you're playing the classification game to mislabel yourself up (grandbagging) or down (sandbagging). Having a classification system that is off and doesn't differentiate well between shooters (on paper) or how an individual progresses, by definition makes that system not particularly usable for tracking progress. 

 

This is similar to IDPA and complaints that making Master is too easy. If we simply go off of "well, you know the time you shot on classifiers (or whatever counterpart)," then there shouldn't be a problem with everyone being an IDPA Master, as long as the rank is preserved by the numeric measure, right? Yet, it's the most common complaint I hear about IDPA classes from the USPSA shooters...

[IVC]

6 minutes ago, Chills1994 said:

Get rid of classifications entirely .

Seems like people already talk more about their time on standard exercises (Draw, reload, El Presidente, Bill Drill, etc.) rather than HF. After all, what is the meaning of classes if the goal posts are constantly moving and percentages need to be tagged with the time/date when it was achieved such that one can figure out the HHF at the time?