This is one of a few cases where I want a loose fit
Really though, a loose fit ensures you can slap whatever upper on that lower or vice versa. Accuracy will not be affected enough to matter. Maybe if you are doing benchrest comp for ultra tight groups you might see a MINOR accuracy difference but not enough for it to matter in real world use. At that point you could bed the upper and lower mating surfaces & lugs to each other, i guess......
Even better is a loose fitting upper lower combo with a lower that has a set screw under the rear lug to tighten it up to your desired tightness.