Agneli managed to get away because his priority was Anja, she was an important figure not only because of the position she held but the information she had regarding how Ragnar operated.
This siege wouldn't have been possible without her, she predicted this outcome because she knew how important Lagertha was to Ragnar.
This included the known tension between Ragnar and Askild, she gave them all the information they would need but once again, Tyr's death worked against them.
Asger too was an unwelcome factor that threw the entire plan into shambles because if they could get Lagertha within their grasp.
If they had Lagertha, they could have influenced Ragnar's decision. There was no way he would act this rashly with his wife in their possession, especially after he had lost his son.
Asger was the problem and right now, nothing was stopping Ragnar from letting loose.