Totally agree. 90% of the reason I'm in it is because I enjoy it. It's what draws me to USPSA rather than 3 day tactical courses.
The martial arts world in general is full of what-iffers with overactive imaginations. The concealed carry world can be an extension of that. Martial artists do the same thing - "if you're not a jiu jitsu blackbelt, you're unqualified to teach self defense." Meanwhile they've never been in a real fight, can't shoot their way out of a paper bag, and haven't ever stopped to think about situational awareness.
I liken it to a football player with a 500lb squat. Among the strength elite, that number is mediocre at best. But for the football player, the amount of specialization and effort it would take to get his squat to 600 would likely make him worse at stated goal of being good at football, because he'd have to drop his efforts in other areas.