A contingent of so many people trying to come in into the Empire borders would raise all kinds of alarms over the Empire.
If he was the king of this kingdom, the moment that the people from the outside enters the wall, they would send an army to escort
Of course, he would not make the stupid decision to attack since that would make the refuges band together but instead he would use the carrot and stick approach.
But maybe it is not so easy to do that kind of thing in this continent. Since most of the people here very much value their tribal relations.
And if by some kind of miracle some people managed to take one of the states of the Empire, then with all kinds of failsafe in place, that state would be isolated.
The Empire had conquered a lot of land but they also know to how to balances out power.
The strategy of restraining one with the other, overcoming fire with water, is a method the Empire is very fond of using.