All that said, I can't help but wonder what has compelled them to take this gamble. Have ratings and merch sales been declining deeply enough and for a long enough time that they feel a massive shakeup is in order? Or is it a new guard of showrunners who want to take this series new places brand recognition be damned and were able to convince corporate to give the greenlight?
Probably to shake things up. Back in Best Wishes you can begin to tell the anime team were starting to get a bit tired. It's why XY felt more like a shonen series at times, Sun & Moon was a slice of life (mixed with other things on whatever the writers whims were), and finally Galar just tossing the region idea out and having a world tour. Wouldn't surprise me this was all planned sometime during Sun & Moon, liking not being restricted so decided to make Ash Champion so that he could next gen then compete in a Champion League to become the Top Champion to finally end his story (and it just so happened Gen VIII gave them the perfect story & character(s) to do that with).
Now the anime team can pretty much do anything they want. New characters, no "baggage" from previous gens, just gotta include stuff from the current gen which is no problem as that provides plenty of story potential.
More than anything they really just wrote themselves into a point of no return for Ash.
His charm as a character that made him such an effective main character for the anime is that he was always the "underdog". Early Ash especially, but Ash in the earlier series (particularly the original series) was basically the loser kid that nobody liked, who was seen as incompetent, stupid, and someone who had no hopes of ever reaching greatness, and yet despite all that (yes, he is stupid and reckless), his determination, his genuine love for Pokemon, and just his overall grand ambition made him compelling to a point where you couldn't help but root for the guy all the same.
He worked as such for so long because he was always the underdog. Later series are where Ash started to become a more competent and mature person and Trainer, and soon grew into a role model for others who could inspire others (especially starting with XY, where he really started being put into that position) but even then he could still make a compelling protagonist because he hadn't achieved a position of greatness...at least not yet at the time. But then come SM and Ash becomes the Alola Champion, finally actually winning and earning himself a position of greatness.
But even then Journeys could still keep him and the world tour format allowed him to still serve as an underdog to a degree despite him being a Champion and an experienced Trainer, because the World Coronation Series was a world competition and the top of the brass, the Masters 8, was people who were also Champions, especially more veteran ones like Cynthia, Lance, Diantha, and Steven (and of course, his ultimate opponent, Leon, at the top of them all), and Ash was the "underdog" in that he was the new kid Champion among them, the new rising star, pitted up against more experienced Champions who were practically the strongest competition he could find.
And then Ash came out of that tournament beating the strongest Trainer in the world, Leon, and becoming the World Champion and the top Trainer in the world himself. Now there's no turning back from that. Being a regional Champion is one thing, which is why he could still work with what Journeys did but being the greatest Trainer in the world is basically a grandiose position and now Ash is practically a living legend in-universe. You
cannot make a compelling underdog protagonist for kids to watch and relate to with Ash anymore after this. He's locked out of the old infinite loop he used to be in. He's already the best in a way. Sure, he intends to continue travelling in hopes of becoming a Pokemon Master, as there's even more for him to explore in the world, but there's nothing for us to watch that would make him compelling
as a protagonist. As such, I think they realized this is it for him and there's nowhere they can take him as a main character now, so after this magnitude of an accomplishment, with Ash having become the greatest Trainer in the world, they decided it was finally time for him to step down.
With that said, however... (in regards to Pikachu)
As for the main anime, they could maybe have a recurring Pikachu character appear once an episode or every so often, not being part of the main cast but certainly at least witnessing what's going on (maybe even act as audience surrogate at times).
Who's to say that the recurring Pikachu character in question won't be Ash Ketchum himself?
Yes, his time as the main character is over, but that doesn't mean we'll never see him ever again. Ash may have stepped down as the protagonist, but odds are despite us having new MCs, it's still highly likely that the Gen 9 anime series still takes place in the same universe that the main anime has always took place in, just with focus on new MCs now.
We may not be focusing on what Ash is doing anymore, but Ash is still out there travelling the world, meeting Pokemon, having battles to his heart's content, which means there's always room for our new MCs Riko and Roy to run into him every now and then. We could even get follow up on what Ash is doing every once in a while, when he makes guest appearances as a recurring character, he could even be a role model for our new MCs. Mr. World Champion and living legend Ash Ketchum, being an inspiration for new and aspiring Trainers, including Riko most likely.
He is still an iconic character and a face of the franchise, and if they wanna keep showing the mascot Pokemon, Pikachu, he's always right there to showcase it. After all, Ash and Pikachu are practically synonymous with each other. Our new MCs could always run into the guy every so often, and Ash is a popular and iconic character so the anime writers only benefit from continuing to have him show up as a recurring character. Not super frequently, but every once in a while, he could still show up, him being world famous as a top tier Trainer. Seeing him battling against other Trainers and maybe big spectacles starring him and his great team and beating other strong Trainers.
It would be a solid continuity nod if they decide to do it. Ash is out of the spotlight, but he's still out there travelling the world. The new kids Riko and Roy could always have a chance encounter with him.