Xiao Wen ultimately stayed in the Prince Mansion and didn't follow the Prince of Yundian back to Yundian Royal Mansion.
However, this was only temporary. Xiao Wen had already agreed with his father to return to Yundian Royal Mansion, he just wanted to wait until his siblings were born before leaving.
The Prince of Yundian couldn't stay in Capital City for long and, after spending less than ten days with Xiao Wen, he hastily left the city.
Thankfully, his years spent searching for Lady Yu and her son gave him enough experience in disguise to avoid detection.
Because a few days earlier, The Fourth Prince had been suddenly assaulted, prompting the Emperor to order a rigorous search throughout the entire city.
Prince Mansion was searched, of course, but such a large estate made it too easy to hide someone.
The departure of the Prince of Yundian from the city was facilitated by his disguise and stealth.