I agree with Hawg. The wedge should be tapped in, not finger tight. If the barrel is loose at all, that means it's flopping around out there on the front of the gun. Accuracy will be terrible.
Glad that the arbor is tight. That's a pain to correct. Try tapping that wedge in with a mallet. You don't have to hit hard, just tap it in good and tight. I bet your accuracy improves.