Leaders with only positional authority may get things done but if the title were gone, no one would listen to them. However, when you start to develop influence, people voluntarily follow you, not because of title, but because they like you. John Maxwell describes this is as Permission Leadership and it is the second step up in leadership.
"So, who cares what people think about me? I'm not here to make friends."
I believe some leaders say this after getting their fair share of kicks in the butt or simply heard someone else say it and took it as good advice. It's easier to keep it impersonal or as some say, "professional." Unfortunately, we are all emotional beings and sometimes that means people will talk about their feelings about you- what they like about you, what they don't like you, what they would do, and what they wouldn't do if they were you. There's a number of reasons why people say things about you, and sometimes is just as simple as they want what you have. Sometimes there may be truth to what people say about you, and you'll have to make a choice to change or not to change. Sometimes what people say is slander, and in extreme cases you'll need to move those people off your team; but, I really believe if you always keep it "professional," you're not going to get influence and you cannot grow as a leader without the ability to influence.
Let your guards down.
I'm not saying be best friends with everyone you lead because you can actually care so much about someone that you become unable take them where they don't want to go, but need to go for fear of hurting their feelings, but you do need to show those that you lead that you care about them personally and professionally so you have a relationship to build on.
Show you care.
A mentor gave me some great advice once and it was this. List everyone who reports directly to you and list one personal thing about them. Is there anyone you can't list anything for? Get to know them. Get to know what passions they have, what's going on in their life at the moment. Find something (not work) to talk about that you relate to and share your stories as well. It may be awkward at first, but if you are consistent in getting to know someone, they'll start to open up and begin to give you the Permission to lead not because of your title, but because they like you.
The five levels of leadership John Maxwell describes are position, permission, production, people, and personhood (very few people reach this high). One cannot be achieved without the others below it. Read more about these in Developing the Leader Within You.