Have you tested your visual reaction time in the same way? If your visual reaction time is indeed faster, you're an unusual case.
I did a little reading on reaction time a while back, and research has shown that visual cues produce longer reaction times, possibly because it takes measurably longer for visual signals to reach the brain.
IIRC, visual reaction times average about 20-40 milliseconds longer than auditory times.
If you add any kind of interpretation to the stimulus (e.g., react only if you see "A", not "B"), the reaction time increases significantly, and continues to increase with the complexity of the decision required.
Jay