I'm confident that only one or two of my followers are from people that actually give a damn what I say. Some people just follow everyone for little to no reason. As Zimber said, though..It's better than nothing, especially if you care about that sort of stuff.
Also, its not currency or anything that increases your social status or something, so don't let it get to you