Being with Blaze Bayley and not Bruce Dickenson killed them there.
I agree that the Blaze era was, on the whole, a failure and it is given very little attention by fans. This may be because of the two albums they did with him, the second one, Virtual XI is an unmitigated disaster, by far the most mediocre album they ever did. Performing live with Blaze was little better, in fact it was considerably worse. The man's vocal range was and remains severely limited... when the time came to sing their back catalogue he slaughtered it so badly, you wonder how is it possible they hired him in the fist place. Did they not ask him to sing at least a few of their hits in an audition? There are plenty videos on YouTube of Blaze singing so off-key, you want to put your fist through the screen to somehow punch him in the mouth. Blaze also suffered from a variety of health problems, often on tour. As a result the X-Factor tour was cut short as was the Virtual XI one, although in that case I think it happened because his relationship with the band had already broken down.
Having said all that, I still think the X-Factor is a small masterpiece. The songs perfectly complement Blaze's voice. The tuning dropped a bit, the bass pushed a bit more forward in the final mix. The whole record is considerably darker. Harris suffered some sort of a depression after Dickinson left the band... that is evident in the X-Factor's lyrics, many of which are hopeless and pessimistic at least four songs have to do with war. The melodies are also considerably darker, nothing like happy-happy tunes like "The Trooper". Songs like "Fortunes of War", "Judgement of Heaven" and especially the stellar "The Edge of Darkness" have dark guitar melodies and ferocious, thought-out, well-planned solos of such quality that I think if a certain generation of musicians (what was once called "New Wave of Swedish Death Metal") was asked, they would admit being influenced by them.
In any case, the band didn't continue in this direction with the next album and nobody really asked them why, because the X-Factor was one of those Maiden albums that is ridiculously hyped when released, only to be forgotten a few years later.