First of all, if a group of people want to be called a certain word (or not called one), it's important to respect that. But that doesn't change the fact that, today in our culture, people take offense illogically. Let's take two words: retard and invalid.
There is a push to ban the word retard and its accompanying versions. The preferred word of the day is a person with special needs. I won't be surprised when that phrase is deemed offensive in a few short years; I find it extremely condescending. The word retarded essentially mean slow. To call someone slow is surely not a compliment, but, it's not exactly offensive.
Meanwhile, a person who is sickly can be described as an invalid without offense. Invalid, in it's most basic sense means not valid, you just slightly change the pronunciation and that's supposed to make everything ok. Since you're not supposed to use the word in the definition, invalid also means lacking substance or not logical. So the person we describe as an invalid lack substance and their existence is illogical. But a word that means slow is offensive?
People often take offense independent of logic. Instead, offense is taken based on feeling. Invalid is acceptable, as far as I can tell, only because of the slight alteration in pronunciation from the word valid. Perhaps retarded is deemed offensive because of its harsh combination of consonants.
Retarded is like gay. It can be used offensively if spoken as slang to mean something is stupid. That retard is gay. That gay is retarded. Both are offensive. But there's nothing wrong with being gay or retarded. They're offensive only because one of the words in each sentence is slang for stupid. The other word is a descriptor of a certain group of people.
Let's take the word Jew. That man is a Jew. That's not offensive. I will jew that man. That is offensive because it is using a term for a particular group in a derogatory way. That's the difference. We need to need to look at the context before we get offended by certain words.