You'll find others out there who have studied the art of storycraft and can break things down on an academic level. That's useful, but it's not my goal. I plan to keep this simple and write from my own perspective, without research or references. The hope is not so much to create a prescription for creating a good story but to provide a bag full of ideas to pull from.
The truth is that people change less than they think. But we all want to believe we can change and hopefully improve. Stories are a way to live vicariously, to see things happen that we either don't want to happen to us or that we wish could one day happen to us (but probably won't). When bad things happen, we want to see what the result is and how it can be overcome - it's good to know we could overcome it as well without having to actually suffer through it. When good things happen, we like to imagine we could experience that happiness ourselves - though we know we won't actually get super powers or a second chance at life. In either case, it would change us and we want to be shown exactly how it could change us. We want to see the possibilities.
There are some stories where it's acceptable to end up back where you start. These are circular arcs. Think The Simpsons. Twenty years later and the family is still the same: Homer works at a nuclear power plant, Bart still gets into trouble in school. But every week they change, there is something that disrupts the normalcy and then crazy adventures ensue. Often one character learns and grows. Sometimes they merely get into a mess and have to work themselves out of it. At the end of the day they are back to normal and ready for next week's story. There is an arc, it just wraps back around to reset. That works because we expect another story next week. Circular arcs work for series, but there still needs to be movement and development within the circle. We need to travel along with the characters.
So how do you create this arc? Creating a story is a complicated thing that is best done simply. I like to start with a basic premise and let it play out in my head. What would it be like if your loved one was kidnapped and you had to get them back? That will lead to an arc. What would you do if it turned out you were a werewolf? Big time arc there. What does it feel like to fall in love with someone who is dying? Sad arc, but definitely an arc.
Once I have a premise, I normally start by thinking about where I want to end. It doesn't have to be a definite place, but I need to have a target. Maybe I want to examine how a person can come to grips with great tragedy. I don't need to know how they do it - that will come with the writing - but I need to know that eventually they do do it. That's where my arc finishes. I want to know my resolution exists even if I'm not sure exactly what it will be.
Then I go to the beginning. What is the current state of affairs and how can I contrast that with where I will end up. If it's tragedy, it's good to start in a place of happiness. If it's fantasy, I like to start with reality. Picking something far from your goal forces you to travel along the arc. You may find that the finished book should start somewhere closer to your ending and that you've just been creating backstory, but that's okay. That backstory will help give you depth and make the portion of the arc that you share with the reader more believable. You can often cut out the bits along the path that are less interesting, but by knowing they exist they keep you from jumping randomly along.
Because it's important for an arc to flow smoothly. There needs to be signs early on of how the arc will progress and the progressing needs to be in stages. It's a path, not teleportation. You can throw in some random story elements that shift everything sideways, but the reader needs to be able to make the turn. That means they need preparation or they need to have time to adjust to the new path. The more you know your arc the more you can gently hint at where it's going; the more your character makes incremental changes the more the reader can follow their footsteps.
(While I'm on it, characters themselves also arc. As I said, we want to see people changed because we want to be changed. We want to see people learn so we can learn from them. We want to see people ride an emotional roller coaster without having to risk the crash ourselves. I'll go into more detail on what I think makes for a good character arc and how to create it in a future post.)
Ultimately arc IS the story. Otherwise we have a series of random events that might each be interesting but the reader will never get invested. If you keep your arc in your mind while you craft the story, you will build a path that the reader to follow to the place where you want them to end up. That place is a happy place (even if the ending is not).