The once friends had turned to foes in spilt second, now it was Gandalf who controlled the city and he had more men. One hundred and forty thousand strong, one hundred and twenty thousandof his men and the addition of twenty thousand who General Haul brought to the city, these men despised Castellan and they chose to stand with Gandalf.
Like they always say "An enemy of my enemy is my friend" and indeed these Prophisian men who still believed Taelins were usurpers were ready to side themselves with anyone who wanted the throne as long as it meant they will not support the Taelins.
Maybe Castellan had ignored the wounds that still lingered inside many hearts of Prophisians for quite some time. Mithandirs were cheated out of what was respectfully theirs, and good folks would never forget it. When Castellan stood before his own gates blooded and overly exhausted, he knew the end had come.