Any game after X4 would be a tough act to follow regardless of it's quality. I liked X5 well enough at first, some of the stage music is nice, it's a tougher game, certainly. But the more I play it, the more the small things irked me.
I think my biggest issue is how lazy the whole goddamn thing is. After X4's wicked anime cutscenes (which I had never ever seen before), it was easy to be let down by X5's static images. Sprite animation took a massive tumble back down the stairs, being nearly as lazy as X3 in that regard. Capcom's spritework at one point was damn near unrivaled, so I was a little let down.
I barely touched X6, and haven't ever played X7 or X8, so I don't know (or care, really) how the story develops. It was kind of weird how much focus was on Zero towards X5's conclusion, though, and I hear that only got worse as the series went on.
I still wouldn't say no to a 16-bit X9, done in the same vein as Megaman 9, though.