Feynman shared your hesitancy for very similar reasons. He states in Surely You're Joking that he was offered, but never took them.