Do you really not lose your followers if you change user name?

If I change my user name, I don't want to lose all my followers. I have read varying accounts of what happens when you change your user name. Some people said they lost all their followers and had to start over, but some people insist this won't happen. Any one know which it really is?