It's not so much that it bothers me, but I have one person as follower who has withdrawn from site. Which means they don't want to be here anymore. It seems a bit strange that even though they've withdrawn, whomever they've followed are now stuck with them.
As stated -- Blocking a member who has followed you should keep them from appearing on your About Me page. Currently, there is no way to remove that member having followed you.
Gonna close this thread as I think the question has been answered.