Loose Pins on Frame from Removing Them Too Much?

[Gun1]

If I were to remove the pins on guns too many times will they eventually become too loose and start walking oit by themselves? Also, would this only affect polymer frames, or can metal frames also be affected such as 1911s?

[Rnlinebacker]

The thumb safety usually traps then from coming out. You could certainly have some play that would cause r issues if the holes aren't cut precisely parallel