Jimmy had waited most of the day, certain that these people had no intention of leaving, and decided to go. It wasn't that he didn't want to go in and wipe them all out, but he didn't know who else was inside with Scott and what sort of people they were.
Jimmy had indeed killed many, but apart from the criminals he knew for sure, the rest were mostly drug dealers or the like. He had no interest in hurting ordinary people and absolutely didn't want to harm them.
Not to mention, at the very least, the two men in sunglasses and suits who were always with Scott clearly looked like bodyguards. Although they never spotted Jimmy's car following them, such types usually came from various security companies, hired just to make a living, not deliberately involved in gangster business like the dregs he used to deal with.