Daniel gave the address to his penthouse to Philip and instructed him to drive towards there. When Samantha heard this, she threw a wondering look at him and thought that he just wanted them to stay separately. She was not aware that Daniel lived in his penthouse since they divorced.
Feeling uncomfortable, Samantha ordered Philip to take her home to Sapphire Gardens instead which also surprised Daniel. It had been a long time since she went there and knowing that she did not have plans on actually going to the Cliffside Villa, it left a bitter feeling on him.
Arriving back at Samantha's house after their car ride remained very silent, she took Ash inside and did not even bother inviting the other two men in. But that did not mean that Daniel did not let himself inside Samantha's place on his own.