Limyaael's Rules for Writing Non-Humans.
1. Try making them non-human emotionally as well as perhaps physically. I can't tell you how many fantasy stories I've read with "elves" that are prettier, more connected to nature, and longer-lived than humans, but otherwise exactly like them. It robs the story of something when you realize that the "wise, fair folk" everyone's in awe of are basically humans with pointy ears and facial treatments. It also gives me a low opinion of the author who truly expects me to accept this as something new and alien.
Make the characters have emotions humans don't- or reactions to things that would seem inappropriate to humans. How do they treat death? Birth? Marriage? Do they even have marriage? Sculpting the reaction of a human who suddenly finds out that this strange race doesn't weep at a death can do wonders for the story. Or, if you're going to make them alien physically, think about how that would affect the way the characters express emotion. Imagine a race that can't weep. How would they mourn?
2. Make them alien culturally as well. I have a race in one world that doesn't have funerals, as such, because their bodies burn three to five minutes after death. This is a physical difference from humans, but it plays out in their culture. They sing a song that one of their bards wrote a long, long time ago, and the only thing that varies is the language that it's sung in and the pronouns (depending on the gender and number of the dead). Their beliefs about the afterlife are varied and intricate, but they're not obsessed with the body as such, and don't understand the elf-like race in their world who are. If I had made them concerned with elaborate funerals, yet still had the body burning that quickly, it wouldn't have made sense. Again, part of the problem here is that people tend to lift customs wholesale from human cultures on earth, so that perhaps funerals resemble Christian funerals exactly even though their characters aren't Christian. What's the point? Why not look at your non-human race and have them celebrate, mourn, or ignore death in a way that's appropriate to them, instead of to humans?
3. Consider the disadvantages as well as the advantages of the magic and the powers you give your race. The quickly-burning ones, Elwens, I mentioned above have very powerful magic, a number of physical advantages (such as keener senses and more adrenaline), and very long lives, stretching into millennia. It seems as though they would become wise and serene creatures, the way that elves are sometimes portrayed.
Wrong. I also gave them emotions ten times as strong as a human's. It makes them passionate lovers, but it also makes them strong in hate. They believe in vengeance, so blood fueds can go on until everyone involved is dead. And their world is torn by war after war, most caused not by greed or expansionism, as in ours, but by pride, jealousy, hatred, and inability to forgive something that happened ten thousand years ago but which living Elwens still remember.
Don't make your non-humans perfect, and don't ignore the limitations imposed by their magic. Presenting them as perfect only encourages the reader to see them as flat, shallow stereotypes- one of the reasons why, even though I love the concept of elves as such, I am bored silly by the way that many fantasies portray them.
4. If you present the story of a mortal in love with an immortal, or at least a mortal in love with a member of a longer-lived race, consider writing it from the immortal's point of view. How does it feel to see a life flicker, fade, and die? There are surprisingly few of these stories, even though some stories mention the idea. Most of these love stories are written from human points of view, in order to angst about dying so soon (and almost always to make some point about the human lifespan being better, which I hate; see below). Write from within the non-human race's culture. What would they think about one of their kind who fell in love with a human, or other mortal?
5. Don't present the non-humans as inferior to humans all the time. Of course they should have limitations, as discussed above. But almost all fantasy authors seem to take the point of view that, well, sure those races may have longer lives and fascinating cultures, but they're still not as wise as those humans! I don't understand this. A human might mature at a different rate than an elf, and see more of life if an elf stays in his forest all his life and the human travels, but does that mean that a human will really be "wiser" than an elf of comparable experience? I would say not. The danger here lies in walking the line between presenting the non-humans as always perfect and always better, and diminishing them for the purpose of making your human audience comfortable. It's a hard line to walk, but it can be done.
6. Consider making non-humans the dominant race in your world, or at least on a level of power with humans. Following Tolkien, most fantasy authors declare their version of the "Age of Man," and send the magical races away. The sorrow works in Tolkien; it often doesn't work in the imitative fantasy. And I think a part of this is that we can still catch a glimpse of what Elves once were in Tolkien. They sing beautiful songs, had high civilizations, and have done great things. They have a long, rich, beautiful history. The Tolkien-imitators often send their elves off to their equivalent of Valinor without presenting any of this, and this makes the domination of humans a cliche- and makes the reader feel little if any sorrow at the passing of a race that must seem cardboard to him.
I've reversed this almost completely in Shadeemira, the world where the Elwens live. Humans are not native to Arcadia, the main continent, and so enter its history later. They fight two genocidal wars against the Elwens, neither of which they win, and Elwens act towards them as towards hated and defeated enemies. They cannot advance, possessing no magic and having the sentient, easily bored magic of that world destroy any technology they assemble past a certain point. They are doomed to remain an underclass and ultimately to fade from the world through crossbreeding and despair, and that makes me pity them far more than I would have otherwise.
7. Consider writing your story without any humans at all. This may be the best way to give your non-human creatures room to grow and develop, without the constant presentation of comparisons. And it will force you to adopt a viewpoint from within their culture instead of outside it, which is invaluable in terms of writing experience; far too many fantasies take the easy route of having a hero to whom this is all new, and so explaining it over and over.
More I could add, but I think I will save it for another time.