Oliver realizes what his brother must be thinking right now, which makes him pull away from him like he's making him sick. He indeed said hurtful words to him this morning, but he was not in a normal state.
Now that he sees the recoil and disgust on his brother's innocent face, he says to himself that he's the one who made him this way. Because of him, he now realizes that his behavior towards him is a tad suspicious and inappropriate.
He placed his hand over his mouth, realizing his gourd as his eyes widened in dread. "No, no," he said to himself, "not that, please. I didn't mean it that way, and I don't want you to change your habits with me. I was just beside myself and said foolish things with that foul mouth of mine."
"I know you're my brother, and you love me as I love you, but don't let my words sway you, and what's between these boys doesn't concern us. For us, it's different. I know it."