Ah I see, looks like I may have to go back to the drawing board. My original design in the picture below uses a single set screw pushing onto a 4mm D shaft on the gear head. It is designed to support a heavy vertical load but I never really thought about how easy they would come off. Heres what Originally done:
Basically a hub rides inside a brass bushing which is inside an aluminium axle end cap bolted to the axle case (not shown) to support vertical load and a single M3 set screw on the hub presses against the D-shaft.
Thanks for the pics! I'll start working on new design, mabye something build right into the rims
-Chris