greenvoid:
So my question is not how to circumvent the problem, but how to give prepared spellcasters a way to be able to use any spell in an emergency, without making them unbalancingly powerful.
This goal is more or less impossible unless the cost to switch is drastic. You're talking about removing one of the very, very few checks to a caster's control of the narrative, a check that is already largely circumvented by spells powerful enough to force the universe to play your game instead of the other way around (
glitterdust, for example, allows you to say "I don't want to dodge arrows and I don't want to have to watch my back for rogues") and by spells that are inherently versatile (summons, polymorph effects, illusions). Is an hour drastic enough? Eh. I'd probably say "and you have to burn a slot a level higher," which among other things means you could never do it for 9th-level spells, not that they're the kind of thing that typically needs swapping.
But spontaneous casters already break this limitation! Yes, and they're usually a spell level behind and casting from a limited list, which is why prepared casters are traditionally considered more powerful - if you're smart, wizard beats sorcerer every time, because you made the right choices at the start of the day and therefore ignore the theoretical drawback of being locked in. That's why sorcerers are T2 and wizards are T1. Situations where you don't have enough information or you just didn't make the right choice are the only thing that make sorcerers competitive. A wizard who can do what a sorcerer can, or who even edges into that territory, is like how clerics can also be fighters by casting
divine power but if fighters had a class-enabled narrative range beyond "physical damage to accessible physical targets."
Buuut...to be honest if a GM is using full casters at all it's less about "are you breaking the game" and more about "is the ruin you've made of that nebulous paradise called 'balance' still fun for the other players to play in." And that's a question that you can't really answer except in context, so. An ye harm none, do as ye will and all that.