Jump to content
Brian Enos's Forums... Maku mozo!
PocketPistol

Classifier Data Analysis

Recommended Posts

I am a PCC shooter B class shooter and decided I wanted to do some dry fire practice and compare my dry fire performance with hit factor cut offs. Not so easy as it turns out.

 

Unfortunately http://www.classifiercalc.com/ and https://azshooters.org/classifier does not have PCC data and it is unclear if data represents the recent changes in high hit factors. There is a calculator on the USPSA website but if you try to query more than 15 stages in tells you that you have queried too many, and shuts you down for an hour. Very obnoxious. Fortunately, you can calculate the high hit factors yourself if you have a pair of percentages and hit factors. 

 

Well, I wrote a python script to scrape the USPSA member classifications levels and results from A100000 (mid 2016)-A105000 (mid 2017) (1912 users) users and all the life users L135-L4537 (3889 users). I ignored expired users whose data shows up with an 'x'. This is only about 20% of the USPSA membership, but I think it gives me some interesting none the less. I could scrape more, but haven' taken the time to figure out where the start and stopping numbers are for each of these membership categories (A, B, CA, CAL, CL, F, FL, FY, FYF, HCL, L, RD, S, TY, TYF, etc.). If someone wants to do that, I am happy to scrape more data...

 

The first thing I tried to answer is which division does the average shooter do the best in? Is PCC really easier than limited? Again this is only based on 20% of the membership.

 

  Open Limited Limited10 Production Revolver Single_Stack Carry_Optics PCC
Max 93.2122  98.1042 91.1341 95.0249 76.6005 85.8647 91.4708 99.9841
Average 52.5649 46.2836 41.955412 47.383663 48.41808 42.88808503 48.51231235 57.4404

 

The results seem to show that the average PCC shooter is scoring higher percentages than all the other classes compared to their own high hit factors. Since PCC is introduced so recently, maybe results are being skewed by experienced people moving into PCC vs. other divisions that have new people starting there? Well, here is the data just looking at users who joined mid 2016 through mid 2017...

 

  Open Limited Limited10 Production Revolver Single_Stack Carry_Optics PCC
Max 88.4506 90.2799 78.0854 95.0249 76.6005 77.9958 91.4708 99.9841
Average 51.1567 45.8634 40.186086 47.2301201 49.17671 42.48816879 48.5311964 57.5547

 

So not only are PCC shooters shooting 12% higher than limited shooters from the same time period, not a single annual member shooter who signed up between mid-2016 to mid-2017 has attained GM besides in PCC and Production.  This is probably why the PCC hit factors were changed, right? Well I don't think so...

 

I also scraped all the recent results for the above member accounts (only kept the results that were not "legacy" as legacy results don't have HF's). This got me about 52,000 scores. So for instance, in my data I got 74 scores of Carry Optics shooting 03-02 which fall into 2 groups, those hit factors before 6/28 (7.15), and those hit factors after (7.07). By doing some simple excel magic, I was able to pull out all the older HHF's, and newer HHF's for the 81 stages * 8 divisions minus a couple particular scores that seems like have not been shot (e.g. could not find a single revolver shooter shooting 06-07 among these ~6000 shooters). 

 

Anyway, after doing this, I compared the new HHF's with the old HHF's, then calculated the percent change for each class. The average percent changes are below...

 

Open Limited Limited10 Production Revolver Single_Stack Carry_Optics PCC
3.1% 3.1% 3.1% 2.3% -3.1% -3.5% -4.8% -3.8%

 

So Open, Limited, Limited 10, Production got net harder, but Revolver, SS, CO, and PCC got net easier, on average. Now, it is true there are stages that got much harder -- e.g. 06-05 got 27% harder for PCC shooters, but where 09-13 got 23% easier for PCC shooters... That is a big change...

 

So what is next? Ideas - 

 

My thought is gather more data, and do some data analytics on the data to see what the distributions look like for each stage? For this I could use some help with what ranges of member numbers I need to pull in (e.g. what is the first A number, last A number, is the first member number formatted as A1, or A0001, etc.). The USPSA member sticky was useful, but outdated.

 

I'd really like to understand the algorithm about how the HHF is determined specifically (e.g. is it the average of the 10 highest shooters, etc.)? This may become evident if I had all the data. 

 

For everyone who wonders what the point of this is, well, I got to teach myself python and learn more about USPSA HHF's than I really wanted to know...

 

Anyway, something to chew on...

 

Share this post


Link to post
Share on other sites

Interesting data and analysis.  I would like to see the data from some of the more seasoned shooters that have been around for a few years.

Share this post


Link to post
Share on other sites

Nice!!  I get the sense that you have already put more effort into this than the "statisticians" at the USPSA

Share this post


Link to post
Share on other sites
1 hour ago, stick said:

Interesting data and analysis.  I would like to see the data from some of the more seasoned shooters that have been around for a few years.

 

Easy answer is to look at the life shooters whose data I already have. There is probably a better metric for what defines a seasoned shooters than life, but it is all the data I have at the moment. 

 

  Open Limited Limited10 Production Revolver Single_Stack Carry_Optics PCC
Max 99.9833 100 97.9607 100 99.8677 99.8923 97.3264 100
Average 62.39926 58.57738 55.30444 56.35329 49.5888 53.98289 50.93657 61.95755

 

I have been running my analysis on "Pct" field, not the "High Pct" field for the max/average comparisons. Am I correct that Pct is your average of your best 6 of your last 8 and high percent is your best 6 of any consecutive 8 in history? If so, my data is really a latest snapshot in time for all shooters whose data I have. 

 

Might be interesting to compare the difference between Pct and High Pct? Not sure what that would tell you. The larger this number, maybe the more you have to spray and pray? Don't know.

 

I wonder if the lower average scores reflect when someone reaches a certain point in a division they stop and move to a different division? E.g., I got my A in revolver, so now I am moving to limited. I reached the point in limited I want, so I am moving to open?

 

Share this post


Link to post
Share on other sites
5 minutes ago, PocketPistol said:

 

Easy answer is to look at the life shooters whose data I already have. There is probably a better metric for what defines a seasoned shooters than life, but it is all the data I have at the moment. 

 

  Open Limited Limited10 Production Revolver Single_Stack Carry_Optics PCC
Max 99.9833 100 97.9607 100 99.8677 99.8923 97.3264 100
Average 62.39926 58.57738 55.30444 56.35329 49.5888 53.98289 50.93657 61.95755

 

I have been running my analysis on "Pct" field, not the "High Pct" field for the max/average comparisons. Am I correct that Pct is your average of your best 6 of your last 8 and high percent is your best 6 of any consecutive 8 in history? If so, my data is really a latest snapshot in time for all shooters whose data I have. 

 

Might be interesting to compare the difference between Pct and High Pct? Not sure what that would tell you. The larger this number, maybe the more you have to spray and pray? Don't know.

 

I wonder if the lower average scores reflect when someone reaches a certain point in a division they stop and move to a different division? E.g., I got my A in revolver, so now I am moving to limited. I reached the point in limited I want, so I am moving to open?

 

Great analysis.  Interesting question you pose about different divisions and shooters ability.  An interesting analysis would be seeing if you could see the differences of percentages from shooters with multiple classifications. 

Share this post


Link to post
Share on other sites
11 hours ago, PocketPistol said:

I'd really like to understand the algorithm about how the HHF is determined specifically (e.g. is it the average of the 10 highest shooters, etc.)? This may become evident if I had all the data. 

 

 

i lol'd at that. over time there appear to be different methods used at random, most recently it appeared to be a person with little/no statistical background looking at numbers and making a wild-ass guess.

Share this post


Link to post
Share on other sites
13 hours ago, stick said:

Great analysis.  Interesting question you pose about different divisions and shooters ability.  An interesting analysis would be seeing if you could see the differences of percentages from shooters with multiple classifications. 

 

Here is the data for all shooters I have scraped so far (again, only 20% of members) that are classified in 8 divisions.. 

Capture.PNG.86c9ccbd2c54e881e6b549b4a62b06a1.PNG

 

 

Share this post


Link to post
Share on other sites
6 hours ago, PocketPistol said:

 

Here is the data for all shooters I have scraped so far (again, only 20% of members) that are classified in 8 divisions.. 

Capture.PNG.86c9ccbd2c54e881e6b549b4a62b06a1.PNG

 

 

Interesting numbers and great work.  I'm not sure that shooters who shoot 8 divisions are necessarily the norm.  Can you scrape shooters with 2 or 3 divisions?  I'm feeling that this would be a better representation of the general population of USPSA shooters who should bring the numbers more inline with your initial findings.

Share this post


Link to post
Share on other sites
3 hours ago, stick said:

Interesting numbers and great work.  I'm not sure that shooters who shoot 8 divisions are necessarily the norm.  Can you scrape shooters with 2 or 3 divisions?  I'm feeling that this would be a better representation of the general population of USPSA shooters who should bring the numbers more inline with your initial findings.

 

I have that data, but the problem is there are 56 combinations of ways people can shoot 3 of 8 divisions, and 28 ways people can shoot 2 of 8 divisions... I am not really sure the best way to present that data in a meaningful way? Plus, by the time I take that data, and divide it up into 84 bins, I am not sure if it is statistically significant?

Share this post


Link to post
Share on other sites

Dear lord... actual data!  PocketPistol, if you happen to find yourself in Atlanta, a beer or coffee is on me.  Thanks for your effort and contribution!

Share this post


Link to post
Share on other sites
2 hours ago, PocketPistol said:

 

I have that data, but the problem is there are 56 combinations of ways people can shoot 3 of 8 divisions, and 28 ways people can shoot 2 of 8 divisions... I am not really sure the best way to present that data in a meaningful way? Plus, by the time I take that data, and divide it up into 84 bins, I am not sure if it is statistically significant?

i would think for that data to be meaningful it would need to take into account when the classifiers were shot, and how many in a division.  My L10 and CO percentages look pretty low compared to Lim and SS, but I haven't shot L10 since the 2nd year after started, and I've hardly shot any CO classifiers, so no real conclusions could be drawn from that.

Share this post


Link to post
Share on other sites

Awesome post!  Very interesting data and I look forward to seeing what else shows up.

Share this post


Link to post
Share on other sites
6 hours ago, Steve RA said:

L1740 seems to be doing very well !

 

Based on the dates of his recent classifiers, he has also swung over to the darkside.  ?

Share this post


Link to post
Share on other sites

Per forum guidance, I won't be posting HHF's even though I have them and they are now published on the cell phone app. However, here is the percent change from the HHF update...Missing cells I don't have data for post HHF update so the cell is undefined. Green indicates the stage got easier, red means it got harder. The last row has the average change. 

 

image.thumb.png.b963e8b4b284e7c1a15e6447f39998ee.png

 

 

 

Share this post


Link to post
Share on other sites

Here is a chart of popularity of stages and divisions (green popular, red unpopular) over roughly the last 3 years... I just have a sampling of data but trends should hold...

image.thumb.png.ea7667db84c21a7563768e05c9d66cfc.png

Share this post


Link to post
Share on other sites

I had a request for distribution of shooter classifications. I have data for about 15,000 members (I hear there are about 30,000 members). Not currently too concerned about getting all the data, I think this is enough to draw some conclusions. I am also plotting "percent" and not "high percent" so someone who was a master but is currently shooting A will show up in my data as an A.

image.png

Share this post


Link to post
Share on other sites
12 minutes ago, PocketPistol said:

I had a request for distribution of shooter classifications. I have data for about 15,000 members (I hear there are about 30,000 members). Not currently too concerned about getting all the data, I think this is enough to draw some conclusions. I am also plotting "percent" and not "high percent" so someone who was a master but is currently shooting A will show up in my data as an A.

image.png

Very cool. Based on your data it appears carry optics hasnt pulled away much from production when it comes to HHF and is ripe for more GMs.

Share this post


Link to post
Share on other sites

This is super cool. 

 

One thing that confuses me when trying to interpret the data: why is it that PCC has the highest average classifier yet Open has a higher percentage of A/B classified shooters 

 

Edit: Where did you scrape this from exactly?

Edited by Seanziegler

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...