EXPLANATION TIME: i have a love/hate relationship with im3 and I read this awesome headcanon somewhere saying that the movie was intentionally full of one-liners and snark (a bit too jarring imo, esp in more serious scenes) because Tony himself was the one narrating the movie to Bruce (i.e. us, the audience), and it would be so like Tony to talk about his own near-death experiences/horrible events in his life with as much glib and snark as possible.
SO I took the liberty of expanding the hc to think that Tony himself was an unreliable narrator, and left out a few things in the story. Because Tony Stark is a jerk, but he also has a heart.
(Additionally: Harley wasn’t surprised when Tony didn’t fall for his guilt trip; in fact he pretty much expected it, as from previous experiences of abandonment. So when Tony did come back, Harley was amazed and kept his mouth shut the entire drive home)