When I first got my X5, I did not think it was very accurate either, although over time (1,000 rounds maybe?), it developed into an excellent shooting gun over time. My X5 Legion seems to be pretty accurate out of the box.
I have 2 loads that I shoot:
RMR 147 FN FMJ under 3.6 gr. Sport Pistol
RMR 124 JHP under 4.3 gr. Sport Pistol (I have an old load of this bullet under 4.1 gr. of Titegroup that worked well too)
Both of these shoot great in both guns. I don't see enough difference accuracy-wise between them to notice the difference.
I did put a Bar-Sto barrel in the standard X5, but it shoots about the same as the stock barrel. As I mentioned above, when I first got the X5, I was not really happy with the accuracy, so I spent the money on the Bar-Sto. By the time it got here and I had it fitted, the gun was shooting great. I compared both barrels from a rest, and there was really no difference that I could see. Before you do anything drastic, I would shoot yours for a while. As nice as the Bar-Sto barrel is in my X5, I am not sure it was money well-spent, and I wish that I had been more patient.