So here is my method of turning down the bit:
After the bit has been turned down to fit into the servo horn, I place a small amount of JB Weld in the servo horn.
Then, I place the bit in the servo horn. I then power the servo to adjust the bit to get rid of any wobble and to get is to spin as straight as I can. Let the JB Weld set for about an hour. This should hold the bit firm but still allow to adjust the bit if needed before placing more JB Weld around the whole base where the bit enters the servo horn.