The Island

After departing from the Selfless Pavilion, Di Tou was once again on the road. As this time he wasn't returning to the Academy, he didn't return to the teleportation formation, and instead could leave directly from the island, which Di Tou later found out after parting from Qing Xin was the 'mortal' branch of the Selfless Pavilion while chatting with a nice uncle who sold fish.

In short, as the Selfless Pavilion had the Stone of Knowledge to pick out seeds suitable for cultivation and miscellaneous tasks around the sect, they didn't go through the bother of recruiting externally and having a talent search every however many years. Instead, they just set up a few small villages from which they drew population, and those who were found suitable to be in touch with the cultivation world doing any miscellaneous tasks, with or without talent for cultivation, would be situated in this island.