The universe operates on some fairly simple rules. Time moves forward, left and right are equivalent, nothing moves faster than the speed of light. You don't need to spell these things out (you don't even need to understand them) because that's the world everyone lives in so we all know the result - it's our world. But if your world contains magic or technology that allows travelling between worlds or time periods, you need to explain how that works, even if the explanation is a non-explanation.
What that means is that you need to give the reader the sense that there is some logic behind what can physically happen in the world. Not because we crave logic, but we need to get a sense of what CAN happen in order to understand what DOES happen. If your hero is setting out on a quest to fulfill an ancient prophecy, we need to know that your average magician can't just wave his hands and make the entire evil empire disappear. We need to know if magic can affect everyone, if it takes studying to learn, if it's powered by concentration or by life force.
The same is true for technologically advanced novels. We need to be told if it's now easy to get into space; if virtual reality is at a level indistinguishable from real reality; if aliens breathe the same air as us. We all know that future technology basically makes past magic real (flying, talking over long distances, visiting the moon). So if we're in your future, we need to know what level of technological achievement we're at.
The level of detail of the rules only needs to match the detail of the story. If you don't have a lot of magic happening, you don't need to spell out that much (Tolkien is a good example). But if the practice of it is essential and your characters are using it, or suffering from it, regularly, you need to give us enough to figure out the consequences (the Star Trek universe). Think of it like explaining baseball to a foreigner who's never seen the game. If you want them to know who won, all you have to say is that teams score runs and the most runs win. If you want them to play right field, you're going to need a lot more details if you want them to do a good job.
Aside from the physical nature of the world, we also need a sense of what kind of place it is. Are people friendly or scared? How many of them are there? Is life brutal for all or only some? What kind of government do we have here? This goes beyond magic and technology but into the basic nature of society. Thus, it's applicable to all stories, not just speculative ones. Once again, it affects how the characters go about solving their problems, how they interact with each other, so it's fundamental to the story.
Once again, if your character is trying to fulfill that prophecy, are there a lot of people who will help her? If she has to travel to the far side of the kingdom, does that take ten days by horse or three months by steam engine? If instead she's repelling an alien horde, how many ships are in their armada? Is the government full of worthless bickering politicians or can the nations of the earth unite for a common cause? Share enough to let us know what options your hero has and why it makes sense for them to do what they do.
Even if your story is a contemporary novel that takes place in the normal world, it still lives in a world of its own. Is it set in a hospital? People act differently in hospitals: visitors speak quietly, workers have their own language, rules structure physical interactions and locations, even the lighting and smells are unique. You want the reader to feel they're part of that world and you have to explain how things work there.
The physics and feelings of a setting are completely up to you, the writer, to design as you wish. But the rules need to be consistent or the reader will feel cheated. Once we've learned the rules to baseball, we'll get upset if the losing team all of a sudden gets double the runs for every score just because their behind in the ninth inning. So your magician shouldn't suddenly perform a spell that would have saved everyone a lot of trouble earlier on. Your computer gadget shouldn't suddenly be able to decrypt the enemy's communication. Rules are about setting limits and you can't adjust them because they limit the writer as much as the characters.
I've found that breaking my own rules often happens on the first draft. You get to a point in the story and you want to do something cool that will solve a problem. That's fine. You just need to go back and evaluate how that capability would affect all the earlier elements of the story. You may want to change the earlier parts, you may want to change the later part, but you NEED to make the rules consistent in both parts.
This principle is probably most important when it comes to characterization. It's not about keeping one character consistent (that's important, but it's not really worldbuilding). It's about keeping the characters in general consistent. If most people in your world are challenges to be overcome, then suddenly having a helpful bystander is inconsistent (it may be plausible and even realistic, but it weakens the story). If everyone in that hospital is a stickler for the rules, don't let them become lax to make your solution possible. If you need that laxness, then make sure you include some earlier, so exception becomes consistent. That's what revising is for.
So how realistic does your world need to be? Not at all. Realistic isn't the measure. Magic is not realistic. Hyperspace is not realistic. The hero never missing a shot while an army of trained soldiers never hits him once - not realistic. But possible? Plausible? Yes, because books have their own rules of plausibility.
Readers want to suspend disbelief. That's a major reason people read - to escape this drab world and lose themselves in a new one that has slightly different rules. They like that giant insects could develop intelligence and don't really care that mass scales differently than strength and a human sized insect would collapse under its own weight. They don't care that time travel is impossible because it violates causality and no amount of logically justification makes it work.
On the other hand, I've found most YA novels to be much looser with their worldbuilding, often to the point of highly improbably and completely implausible future worlds (yes, I'm looking at you Divergent Chicago). I believe many of the readers of YA speculative fiction judge worldbuilding on a easier scale than more traditional fans, focusing more of the other elements of the story. Take that and use it as you will.
There are certain conventions in writing, from using magic to having lucky heroes, that people want to accept and so they do. Read the other books in your genre, the ones creating world at the same level as you, and see what works and what doesn't. Let the readers' reactions be your guide and you'll see that they forgive a lot if you tell them what the rules are and have some consistency. Be thankful but don't abuse it - readers have their limits and punish harshly those who fail them.
All this is well and good, but it's also very important to understand how you present this world you've built. The worst thing you can do is to start a story by explaining your rules, taking the time to lay our all the information that will be necessary at some point in the story. That's called info-dumping and it bores the reader. Yes, they want to know the rules, but only as the need them. They want story first, and only the world as it's necessary for the story.This means that you slip in the worldbuilding as it comes up. You don't need to discuss the limits of magic until someone is using it.
And even then you don't need to stop and explain things. It works much better to show your world than to tell about it. Show the spell-caster losing consciousness with the effort. Have your laser gun auto-aim for the character. Don't point out the security necessary on a contagious disease floor - have the security guard turn away the wandering Alzheimer patient.
This is another time when revising is your friend. I find it works well not to spend time trying to explain things and then let my poor beta readers be a little confused. When the first readers point out what confuses them about your world, then you know you need to work in some explaining information. But if they get it, then you already have enough. The common error is over-explaining, so try working from the other direction and you're more likely to come up with a better balance.
Worldbuilding is a large subject and volumes have been written on how to do it. Some authors hate it and some get lost in it. The truth is that all authors do it, whether they realize it or not. There are lots of ways to get it done, and I think it's perfectly fine if you aren't aware of doing it, but you do need to recognize the result. That's what I tried to focus on here - what makes a world successful. But if you want the nuts and bolts of how to get it done, here a a few sites that I think have good advice:
A whole book on the topic: World-building by Stephen Gillet
Worldbuilding doesn't have to be a encyclopedic exercise. For me it's part of the fun of creating any story. Sometimes it's a completely different world that requires lots of thought and careful tracking to maintain consistency. Sometimes it happens of its own accord as my character drives the story. But it's a part of all writing, and every writer should find the level and method of developing their world that works for them.