Magic is one of the fundamentals of fantasy. From Tolkien’s Middle Earth to Rowling’s Hogwarts, fantasy worlds almost always have some sort of magic.
To make magic convincing to readers, you need to make it consistent. It’s therefore worth thinking about your magic and working out the system behind it before you write fantasy.
Model and Theme
Like anything in a story, magic is at its most powerful when it connects to your theme. If your story is about temptation, you could create a magic system that offers great benefits for people willing to do bad things, tempting them to ever darker acts. If it’s about the clash between change and stability, perhaps you have two types of magic - one preserving and one transforming.
It’s worth looking at models you could use to build your system upon. If you want your fantasy world to be evocative of fairy tales then look at how magic works in those stories and build on that. If you’re exploring questions of faith then perhaps your magic users could organize themselves in a way similar to the ancient church.
The Important Details - Limitations
If the magic in your world doesn’t have limits then it will destroy your story. Magic that can fix any problem removes the challenge for the characters, dissolving conflicts and eradicating plot. If the characters don’t use magic to get out of a problem, readers will be baffled at why they didn’t.
So think about the limits of your magic. What can it do and what can’t it do? Why isn’t everybody using it?
One of the most important and interesting limitations is the cost. What price do people have to pay to use this magic? Do they exchange favors with spirits and so end up indebted to them, as in so many folk tales? Do they have to spill the blood of innocents, with all the consequences that implies, as in Ian Tregillis’s Milkweed Triptych? Does magic use up valuable and limited resources, as in some of Brandon Sanderson’s books?
As you can probably see from these examples, limitations also provide inspiration. Once you work out your limitations, you can use them to create challenges for your characters.
Making it Personal
Think about how magic affects the characters in your story. After all, characters are the reason why readers care.
For those wielding magic, what is the sensation of using it like? Does it hurt them or exhilarate them? Is it satisfying, scary, or both? How does it look, taste, sound, and smell?
How does being a magic user affect their life? Do they have to be part of a coven or guild? Do they spend time hunting down spells and ingredients? Are they admired for their power or ostracised as freaks?
Think about how the existence of magic affects other people as well. Do they get excited to see it in action or frustrated that they can’t use it? Do they encounter it often?
By thinking through these details, you ground the magic in the reality of your story, flesh out the details, and give readers a reason to care about it.
Communicating Your Magic System
Having worked out your magic system, how do you communicate it to readers?
A lot of this will come from seeing the magic in action, showing what it can do and what the cost is. Some of it will be about explanations - either conversations between characters or passages of exposition. Try to break the latter up, spreading the information out so that readers aren’t deluged with information they can’t take in and that interrupts the story.
Whenever possible, connect information about the magic system to something that’s at stake in the world. For example, if spells don’t work in the presence of cold iron then you could create a situation early on when a character needs to cast a spell but there’s iron in the area stopping her. Because this has consequences for your characters, readers will be excited about it and remember this part of the system.
For a magic system to work it should be coherent, have limitations, and be connected to its theme. And critically, it should be something that you can communicate to readers, so that they will understand and be interested in how it works.