An hour on Earth would mean an entire day within the tower. Although Arthur needed to go back to his home, there was a great advantage in training in the tower first before he went back to Earth. It was no wonder that the tower was similar to his legacy, since the two had the same creator.
The scholar guardian used his resources, which he gathered from reviving the challengers and taking parts of their memories, to further strengthen them. While the main reason was his desire to preserve his tower and keep it hidden, he still allowed them to grow stronger.
The tower was an excellent place for Arthur to grow stronger. Inside the tower, the voices were easier to communicate with. Although Arthur assumed that's because his powers within the tower were sealed, the guardian explained the real reason.