Ok, your issue is just what I thought get the tools out and grind of the right hand side of the safety so you are not pressing it up with your strong hand. You can thin out the whole thing or just where you strong hand is applying pressure to it. Take a look at the left side and do the same thing. I also add a little tension to the detent by altering the hole on the safety. I use a larger bit so the detent locks in better.
I run the Brazos shield to keep my thumb off the slide.