I think it's that. But then again, it might be more simple. When you clear you holster you hold your gun a certain way. Maybe the first shot "alters" that position in a very small way (drives the grip into the meat of your hand harder). From the second shot on, the grip stays the same. During recoil, you notice if your grip changes or not. This might result in a slower first split.
You could say that it isn't the draw, but the first shot that completes your grip.