No, the follower you have gained are yours! Remember – they are following you because they chose to do so, they are interested in your account and what you have to offer.