It sounds like to us that its an issue with the interface of the barrel to receiver to extension. There are a couple things that may be the issue.
What I can comment on is our extensions. We test each one using hardened steel gauges to ensure perpendicularity on the front and rear of the flange. Only barrels that pass are allowed to make it to the field. While this does not absolve us, my reasonable assumption is the issue is outside the barrel.
1. We understand that the upper previously had another barrel installed on it. The receiver and barrels (especially if there was high round rounds) can "mate" to one another through repeated compression (especially on the aluminum side). We would encourage you to look at the receiver and place on a very flat surface. See if it is anything but perfectly straight up and down or if there are light gaps. That would indicate an out of spec upper.
*On a side note, I am inclined to lean towards this as when you increased compression on the barrel, it tightened up.
2. Likewise, the barrel nut can also have the same issue but it is very difficult to test. Considering its material (either steel or titanium), it is unlikely it is an issue if your previously barrel worked well.
Now, if the issue does turn out to be the barrel, I will gladly replace it.