Hello Woodsrunner79,
I've put together a little diagram which might help explain,
Let's say the Black rectangle is the hole in the Barrel looking from above,
and the Red rectangle is the hole in the Frame,
and the Blue rectangle is the Wedge.
There should be a gap in front of the Wedge on the Barrel, and also behind the Wedge on the Frame.
The Wedge is pulling the Barrel (from the rear side) towards the Frame (against the front side).
If you do have a gap on each side after it's in tight, I believe you are correct and your wedge is not wide enough.
I hope this helps (if I didn't mess it up), AntiqueSledMan.