1) Know what role nature plays in your world. For example: is your magic considered a science, too, just in accordance with a different set of physical (or metaphysical) principles, or is science what’s done with the natural fabric of the universe and magic what’s done with the supernatural? Or do they mingle? What works where? How can your characters tell the difference? (Their knowledge can be limited, too, though not necessarily as limited as that of a reader who picks up the book for the first time. See point 2).
I think it’s necessary to know this because, otherwise, it will not be clear what your characters can depend on at a given moment and what they can’t. If it seems possible for them to use flying magic to escape from danger, but it really isn’t, then you’ll need to show why. Perhaps they’re in a place where purely natural laws hold sway, or the character’s flight magic depends on eating a certain kind of flower he hasn’t eaten in a while, or it depends on some other kind of magic which itself doesn’t work here. Make sure you know, because otherwise you have a special kind of idiot plot, where the characters conveniently forget what they’re able to accomplish so that the author can put them in more danger. It’s akin to a character being able to turn into a small bird at one point in the story, and then pretending that a cell with an unbarred window can hold her.
On an even more basic level, not knowing what works and what doesn’t can mess up the fabric of your world and render it incoherent. It might be possible to juggle a completely inexplicable system of science and a completely inexplicable system of magic, but it’s going to take more skill than simply saying, “Okay, science! Okay, magic!”
2) Multiple sources of knowledge are available. Perhaps the great discovery your characters are going to make is that science does work, when only magic was thought to; perhaps they’ll discover that the “magic” artifacts around them are really ancient technology that they can control once they combine them with, say, a mill-wheel; perhaps they’ll discover that they really are the descendants of an ancient galactic civilization. (That’s the premise behind at least two “fantasy” series I’ve heard of, though I won’t say which ones for fear of spoilers). Or perhaps they’ll discover that the technology they’ve just captured from apparently natural aliens in fact refuses to work for them, depending as it does on “exploded” arts among the humans, like sympathetic magic. Perhaps this is a post-apocalyptic world, and they’re being continually stunned by the emergence of magic in what was an ordinary Earth. Or perhaps magic is small and secret—I’ve noted before that one of the great differences between technology and magic is that technology is usually available to many people, or at least can be used by many people, while magic tends to be considered inborn and rare. So, while there are people who can actually conjure food into being, they live far away from the people who can’t, and the average citizen of the world isn’t aware of them.
What’s the advantage of this? It permits you to broaden the characters’ conception of your world slowly, so that it looks more complex than it appeared on the surface. It can resolve apparent contradictions or inconsistencies by promising further revelations. It can show the reader that your book indeed borrows some of the tricks of fantasy despite looking like science fiction, or the reverse, while the characters themselves might firmly believe they are living in one kind of world or another.
It also presents another solution to the problem I mentioned in point 1, how to keep your characters from becoming all-powerful if they have command of both lasers and fireballs. They can’t use something if they don’t know about it. Zelazny uses this solution in the Chronicles of Amber, the first series, where Corwin knows that he wants to create a certain kind of gun, but can’t simply make it himself, because he has no idea of the finer principles behind its creation. He instead has to seek out people who can make it for him. His use of the guns is based on something else unknown to the majority of his siblings: even though gunpowder doesn’t explode in Amber due to its becoming inert there, a chemical from another world does, which Corwin intends to prime the guns with. Thus, even though most of the people in the series are demigods, Zelazny can set limits on what they’re able to do.
3) Seeing what happens. What happens when you mix the different elements of the genres together—sorcerers and robots and dragons and FTL spaceships and werewolves and string theory? Possibly stew. Possibly a mess. But a great part of the delight is in trying, because the elements together enable the author to accomplish something that neither alone can.
The purest example of this I know is Simon R. Green’s science fantasy Shadows Fall. The town of the title is where fairy tales and old legends come to die. Thus there are Sidhe and talking animals present, but also androids, dead rock stars, Christian fanatics, and a threat that resembles magic but is contagious like a disease. It’s debatable whether this works, given that they’ve apparently been mixed with a lot of blood and emotional button-mashing in a blender, but the result is certainly striking. They were apparently all necessary to tell the tale that Green wanted to tell, so he stuffed them in.
Even if it’s not strictly necessary to tell the core of a limited tale, figures like these can haunt along the edges of the story. And they certainly present a challenge to the author who’s only worked in one genre to control them, justify them, and figure out what they’re doing there.
4) A change of worlds. A book that blends science fiction and fantasy is often post-apocalyptic (although sometimes the apocalypse gets shoved to the back of things and the series is mostly fantasy or science fiction on the surface, as in Terry Brooks’s Shannara world). It can also be a crossover, where one character is traveling to a different world—usually one where magic works—but is still capable of returning to a scientific or futuristic one. Thus, place is probably the easiest way to divide the elements of the hybrid up, if you want them divided and not mixed, so that magic only works in certain places and advanced technology likewise.
This can be done in any manner of ways. Different worlds, alternative universes—why not one where magic is real, at least in the sense of the natural laws taking on subtly different forms than they did in ours, or more responsiveness to a human mind?—a single place in the center from which other realities spread out (Zelazny did this with Amber), a metaphysical catastrophe that gave more prominence to one force on a single continent but didn’t affect the others, a city situated to take advantage of a “natural” upwelling of magic, a road that leads travelers through different countries where hostile magic or natural forces might await but which protects the ones on it, a country so inundated with gates and “invasions” from other worlds that it becomes a stronghold of different species in the midst of other nations that look quite different. And once you know what the place looks like, you can extrapolate metaphysics from it, or characters, or plots, or ways to break the boundaries, or imaginings of what a world would look like where all these things were true.
The next rant is apparently on “happy things about urban fantasy.” I’ll get to thinking of those right away.