Hou Yuxiao had also put himself in Hou Yuduan's shoes—if he had faced such a dramatic upheaval, what transformations would his heart undergo? But, he was not the person involved, so no matter how much he empathized, he could never truly experience it first-hand.
The only thing that bothered him was the final act of manipulation by Ji Yanzhi...
All the events in Tongling were orchestrated by Ji Yanzhi behind the scenes, a simple fact that Hou Yuxiao believed. Given Old Fifth's intelligence, he definitely would be able to see it. So, logically speaking, Old Fifth should have also seen through Ji Yanzhi's final act of manipulation.
But the issue was that recognizing it couldn't cover up the crucial role that Hou Yuxiao had played in these matters.