Just like Sir Yu had said, they wouldn't have been able to stop Lin Fan if he wanted to go anywhere.
With his cultivation, it could be said that he was one of the more powerful people in the Mu Empire. The only ones that could actually keep up with me were the powerful experts of the Mu Empire.
Those experts were ones that would remain hidden in the Mu Empire since they were treated as high ranking members. They would be loyal to the royal family and would be part of the group that would ensure the safety of the Mu Empire if anything happened.
As such, they wouldn't be free to move around that easily.
They would usually stay in the capital or other important parts of the Mu Empire.
So there wouldn't be anyone that would be able to match Lin Fan if he were to do anything, or at least not on the side of the Mu Empire.