Laz followed the guards as he was lead from the black castle down a path and back into the fortress. While there was plenty of room in the mountain retreat, most of the free space already had a purpose and they couldn't just let the castle sit there. So in the end they had to retreat to an area behind the fortress that was still hidden from view and wind before setting down.
As for the people currently residing in it, while Laz didn't want to kick them out, they didn't have the supplies for them to remain and so they followed him back into the fortress along with the members of Laz's legion. Since they didn't know what to expect, the legion remained armed and ready. They were considered deserters in the end and such a fate never ended well for those types, even if they were forcefully drafted.
Nagi ended up going with them as well, curious about the outside world. When Laz asked her if it was safe, she simply responded,