New Classifier Percentages

[waktasz]

I have hundo'd Can You Count multiple times so that can't be correct.
Eye of the Tiger is a hard one in Production because you have to go slow and get A's to score well shooting Minor. 

 

Edited September 19, 2018 by waktasz

[IHAVEGAS]

2 hours ago, MemphisMechanic said:

 

Want to see you % go up?

 

Really don't care what my number is, but would like to see the system be designed like most handicap systems, the only goals being that people of similar skill levels are competing against each other and the groups are sort of balanced in size. Not sure what the goal is with the existing system. 

 

It is not that big of a deal I suppose, but I think it is fun to go to a big match and feel like you can do well in your  class if you have your "A" game that day. 

[malobukov]

4 hours ago, ATLDave said:

I kinda have the feeling that they set the HHF at some level of SD above mean.  Maybe 3 standard deviations?

New HHFs are around 99.5th percentile. In a normal distribution this would be 2.6 standard deviations above the mean, but here the distribution is not quite normal.

[malobukov]

4 hours ago, waktasz said:

I have hundo'd Can You Count multiple times so that can't be correct.

This year, in production?

[waktasz]

Not since they have updated the hit factors, no. I was replying to the post that said it was shot 600+ times with no 100s on file

[malobukov]

35 minutes ago, waktasz said:

Not since they have updated the hit factors, no. I was replying to the post that said it was shot 600+ times with no 100s on file

That was a sample of results from 2018 (before the HHF update). If you shot it before then (or after), your result wouldn't be in that sample. Can You Count is quite popular, so can be shot 600+ times in just a few months.

[ATLDave]

11 hours ago, malobukov said:

New HHFs are around 99.5th percentile. In a normal distribution this would be 2.6 standard deviations above the mean, but here the distribution is not quite normal.

Is that just based on the sample you have seen?

 

I agree completely that the distribution is not quite normal (even leaving aside the no-negative-HF's issue of the left end of the curve being cut off).  And I strongly suspect the distribution is perhaps materially different from one classifier to another.  

 

I just wish HQ would be open about their methodology and goals.  Do they have a particular percentage of the active membership that they think should be in each category?  Are they trying to get GM's to represent the top 1%, top 0.5%, top 0.1%?  Someone who is 3 standard deviations better than the mean?  Which mean - means on particular classifiers, or means of overall classification percentage after taking into account all the "throw-out" aspects of the system?  

 

There's obviously no clear "right" answer on this stuff, but it would make for a much more interesting discussion and a more informed membership if they would just lay out their thinking and approach.  

[motosapiens]

2 hours ago, ATLDave said:

I just wish HQ would be open about their methodology and goals.  Do they have a particular percentage of the active membership that they think should be in each category?  Are they trying to get GM's to represent the top 1%, top 0.5%, top 0.1%?  Someone who is 3 standard deviations better than the mean?  Which mean - means on particular classifiers, or means of overall classification percentage after taking into account all the "throw-out" aspects of the system?  

 

 

The problem here is HQ doesn't understand their methodology, or how it is related to their goals. Instead of having someone conversant with math and statistics involved, they just made stuff up.

[ATLDave]

I'm not certain that's true this time.  It could be, but I think they tried to apply some kind of methodology.  I'm not saying it was a sophisticated approach, just that it kind of seems that maybe they were trying, in some way, to be "data driven."  

 

But since, to my knowledge, they haven't disclosed what/how/why they did what they did, we're suck speculating.  

[malobukov]

5 hours ago, ATLDave said:

Is that just based on the sample you have seen?

Correct, just a random sample. It takes a while to download and aggregate the data from individual web pages, so not even a large sample. But still better than guessing.

[IHAVEGAS]

3 hours ago, motosapiens said:

 

The problem here is HQ doesn't understand their methodology, or how it is related to their goals. Instead of having someone conversant with math and statistics involved, they just made stuff up.

 

Goals, as far as I can see, are never defined, was hoping that there would be an explanation for why it is what it is with the classifier explanations on the first pages of the rulebook. 

 

 I think the math part is easy, after you know what it is that you are trying to accomplish. We don't need a rocket surgeon so much as a common sense guy. 

[motosapiens]

3 minutes ago, IHAVEGAS said:

 

Goals, as far as I can see, are never defined, was hoping that there would be an explanation for why it is what it is with the classifier explanations on the first pages of the rulebook. 

 

 I think the math part is easy, after you know what it is that you are trying to accomplish. We don't need a rocket surgeon so much as a common sense guy. 

 

i think you do need someone who understands statistics, which has clearly never been the case from the start. The current method of going by a fixed percentage of raw score is dumb, and will vary widely between different types of stages and different types of guns (particularly when you get rifles involved).  OTOH, using percentiles so that the top 1-2% of scores are gm,  top 5% or whatever are M, etc.... Just decide how big you want those groups to be and set the percentiles accordingly. then they adjust themselves.

[ATLDave]

16 minutes ago, motosapiens said:

 

i think you do need someone who understands statistics, which has clearly never been the case from the start. The current method of going by a fixed percentage of raw score is dumb, and will vary widely between different types of stages and different types of guns (particularly when you get rifles involved).  OTOH, using percentiles so that the top 1-2% of scores are gm,  top 5% or whatever are M, etc.... Just decide how big you want those groups to be and set the percentiles accordingly. then they adjust themselves.

Amen my brother.  Amen.  

[rowdyb]

Amen indeed! Get someone who's got a BS degree where they've at least taken two stats courses ( I only took one) and they would come up with a better metric than we use now. Percentile would be a great start! The practiscore project ELO ranking, like is used in the chess, word and strategy games even on my phone is even better.

[ATLDave]

Percentile of classifier would be great if the goal were measuring basic shooting skills compared to the USPSA population.  I happen to think that's the most sensible goal, so that's the system I like.  

 

ELO would be measuring overall match performance, which would also be interesting, but would of course roll in a bunch of factors beyond just shooting and gun handling.  Sort of like WAR in advanced baseball metrics rolling in defense and baserunning along with batting stats.  A classifier/percentile system would give you something more like OPS+ or some other overall batting stat that is adjusted to the league.  ELO would be what to use if you were trying to predict who would likely win in a head-to-head match.  Also a valid goal.  

 

What we have now is like... counting the difference in hits between you and Freddie Freeman.  It's not useless, but it's not very sophisticated at all.  

[malobukov]

Elo rating system does not seem to be optimal for USPSA. In chess or go only two people are playing and nobody counts by how many points or stones one of them wins. A win is a win.

 

A USPSA match not only ranks competitors, but also gives match percent. You can say that you won against everyone who scored lower than you in a match, but that would discard the magnitude of the difference in match percent and keep only the sign.

[rowdyb]

I guess I wasn't good at writing what I was thinking. If we went to percentile I would like that, I can handle that for "grading" or classifications. But I also would be happy with the ELO replacing classifiers completely. Not as an additional measurement along with classifiers. And I do agree it is suboptimal because of the number of variables in what we do.

 

So maybe that is part of the answer, you need a rating or ranking based off of more than one factor. Classifiers. Match performance relative to your division winner, adjusted for their classification and yours. Percentage of points shot. Time. A to C to M ratios.  Just like how a american football quarterback's passer rating (a single number) is made of up of attempts, completions, yards, touchdowns, interceptions and so on.

 

What data do we get? Total time it took to shoot a stage or classifier and a match. (and I'd weight match data more than classifier data in an overall ranking. and Level 2 or 3 matches more than 1's.) And then you get the type and number of areas on the target hit or not hit. So we could have data about who is just fast. Who has the best A to C ratio. Who shoots the most NS and so on.

 

But that only tells me about me, rather than a comparative analysis or to determine competitive equity. And that's why I'd minimize stand and shoot tests and put more import into match data. No more classifier stages needed at a match. No more thinking a stage has a different meaning than the others. Every stage counts, every stage is used to see how you're doing. Whether you're a club that shoots with 12 people in a gravel pit once a month or a club with 100+ shooters every Saturday and 12 GM's show up. Whether you only shoot locals or you shoot every Area match and Nationals. No more hearing "Well I don't shoot classifiers well" and hopefully less purposeful classifier "management" in either direction.

 

Just playing around, what 4 data points would you use to create a shooter rating? Something that as a stand alone valued your accuracy only. Your number of errors. And then a measure of your competitive performance per stage. Not just match finish. Just quickly thinking about it I'd want something that showed

-what percentage of times do I shoot an A hit (an individual thing) then also compared to others in my division (a group thing) I wouldn't want it to be about A to C ratios or anything like that. Or even have a time element involved. Simply an Accuracy rating.

-the relative difference between my division winner and myself of hitfactor, per stage, as a function of how may people are in the division. so that being the best of 1 or 2 isn't nearly as good of a rating as being towards the top of 30 or 50. (a group thing)

-something that looks at penalties, M, NS, P, NPM (an individual thing)

Edited September 21, 2018 by rowdyb

[ATLDave]

rowdy', if the point is not to provide a "handicap"/weight-class system (i.e., a way to identify a peer group of people of approximately similar skill against whom you can meaningfully compete while GM's are trouncing all of you), then I would say it is not terribly important to try to reduce everything to a single number.  Nor, I think, do we currently have the knowledge or analytical basis to combine them in a sensible, non-arbitrary manner.  And QBR, which is actually a pretty stupid stat, is a good example of what you get when you kind of SWAG your way to a combination that "seems" right.

 

I very much like your discrete proposed stats, though.  I would just track and publish them separately.... it's much more interesting, to me, to understand the different profiles of different QB's than it is to try to reduce their career or year to a single number. If two shooters are doing about as well in matches, but are doing it very different ways (speed vs. high A%; lots of points versus avoidance of penalties; etc.), then being able to see that in numbers would be really interesting.

 

Unfortunately, we'll probably never get to the point of being able to do the kind of stat that would make the most sense for this sport - the HF equivalent of golf's "shots gained" stat.  That stat breaks down the game of golf into discrete areas of performance (driving, shots into the green, close shots from near the green where the prior shot missed, and putting) and measures the performance of competitors relative to the field on each of those dimensions.  And it can be subdivided by specific distances or other conditions.  You can see, for instance, that most of Rory McIlroy's success comes from being better at driving the ball, while Tiger Woods has most frequently excelled in hitting irons (particularly long irons) into the greens.  It gives you a very good way to get a sense of the "shape" of someone's game - not just their overall excellence (or lack thereof), but exactly what it is that they do better than the other top-level guys and what it is that limits them.  Something like that would really be ideal for USPSA, but it requires collecting discrete data on every single shot.  We're not gonna do that.

 

If, on the other hand, we are trying to do something that will very accurately predict match outcomes, then we should collect a bunch of discrete data over some time, then figure out which data fields and in what combination best predicts outcomes.  This is how several of the genuinely advanced stats in other sports came to be... people figured out what was a good predictor, and which combinations of good predictors correlated most closely with overall outcomes.  That's basically the story of sabermetrics in a nutshell.  

 

But back to the topic at hand: simply replacing percentages with a percentile-based system would result in an immediate improvement to the classifier system, by giving clarity to what is being measured and by automatically and continuously adjusting HF's to the correct values.  The only big value judgement would be deciding what portion of the shooting population belongs in each classification.  After that, the system would basically run itself.  

 

Edited September 21, 2018 by ATLDave

[rowdyb]

54 minutes ago, ATLDave said:

But back to the topic at hand: simply replacing percentages with a percentile-based system would result in an immediate improvement to the classifier system, by giving clarity to what is being measured and by automatically and continuously adjusting HF's to the correct values.  The only big value judgement would be deciding what portion of the shooting population belongs in each classification.  After that, the system would basically run itself. 

Totally agree and would totally back that change.

 

I guess I was reaching into other areas in my post..... hahaha. But I'd like to see more done with the data collected from my matches as it relates to a handicap, or predicted match performance or whatever. But to your third paragraph and getting states to define a person's game we could do that if the timer picked up EVERY shot on the stage and the timer wirelessly transmitted the data. There would have to be an advance in the timers we use and interfacing them into data about the cof. Collecting the data in real time, at the range.

[MemphisMechanic]

I agree that a percentile system is ideal, if we’re to maintain the principle behind the current system.

 

And I also know that we’ll never get it. It’s a simple and logical upgrade... to what was an incorrect and complicated attempt at a fix.

 

Edited September 21, 2018 by MemphisMechanic

[HoMiE]

Percentile system only works when you have normal distribution. There are far too many lower classed shooters that skew bell curve to the left. Reduce the number of classifiers so you can get more data points to set the top 10 bar for each one and make sure you maintain it. That’s where uspsa failed in maintaining the HHFs.

 

[malobukov]

30 minutes ago, HoMiE said:

Percentile system only works when you have normal distribution.

Percentile system works with any distribution. More data points is better, but even with currently available data it's possible to set HHFs reasonably well by looking at, say, 50th and 90th observed percentiles and extrapolating from that to 99.5th percentile that roughly corresponds to HHF now. The right tail is not that different from normal.

[ATLDave]

1 hour ago, rowdyb said:

Totally agree and would totally back that change.

 

I guess I was reaching into other areas in my post..... hahaha. But I'd like to see more done with the data collected from my matches as it relates to a handicap, or predicted match performance or whatever. But to your third paragraph and getting states to define a person's game we could do that if the timer picked up EVERY shot on the stage and the timer wirelessly transmitted the data. There would have to be an advance in the timers we use and interfacing them into data about the cof. Collecting the data in real time, at the range.

 

But you'd also need to transmit the shooter's location, the target location, results of the shot, and (probably) the shooter's velocity (i.e., whether the shooter was static, moving slowly, moving at a dead run, etc.).  All this stuff is technologically possible, but would require vastly more effort and resources than we're ever likely to get.  

[ATLDave]

39 minutes ago, HoMiE said:

Percentile system only works when you have normal distribution.

That is wrong, wrong, wrong.  That is the opposite of the truth.  

[ATLDave]

5 minutes ago, malobukov said:

Percentile system works with any distribution. More data points is better, but even with currently available data it's possible to set HHFs reasonably well by looking at, say, 50th and 90th observed percentiles and extrapolating from that to 99.5th percentile that roughly corresponds to HHF now. The right tail is not that different from normal.

 

Yep.  The weirdness is in the left half, where things are collapsed by the no-negative-HF function.

 

But there's a world of difference between two classifiers where the top 10 shoot the same HF but where the fraction of the shooting population that zeroes the stage is significantly different.  It is not hard to imagine two stages where world-class shooters shoot pretty much the same HF, but lower-level shooters have a much harder time with one than the other.  

 

Imagine an El Pres with no-shoots on the outside of each target obscuring the D zone.  At normal El Prez distances, the D-zone is simply irrelevant to a very high level shooter.  He's likely going to shoot a time and hits very close to what he would shoot if those NS's weren't there.  But the score curve for the whole population will not be the same.  Lower-level shooters will start plugging those NS's.  Mid-level shooters will back off the pace to make sure they don't.  A percentile-based system will pick up this difference.  A HHF percentage system is blind to it.

 

Similar dynamics occur on some of the standards-type stages where the absolute shot difficulty means that misses are common for all but very strong shooters.  On one of those stages, a 1.5HF run might be 20% of the HHF, but better than 70% of the scores posted.  Our current classifier system is basically going to treat that as a garbage result... but it may indicate a material level of skill above that of the literally-average USPSA shooter.  A percentile system would capture that.