Are you documenting each and every time he's let or doesn't pick up his child? If not, you need to start doing so. I know that you would like for him to be on time when picking up his child, but he's not going to do it. He may be genuinely late or he may be doing it deliberately to annoy you. Whatever the reason...document. He really is gaslighting you and trying to control the situation.
You said what was on your mind, now you need to step back, document and try not to get into it again w/him. You are dealing w/someone who is emotionally immature and acting irrational about having to be responsible.
You handled the situation the best you could. I just hope he doesn't become physical in the future towards you.
Sit quietly, the answers will reveal themselves when you least expect them to. The past is gone, the present is a gift and you need to focus on today, allow the future to reveal itself when it is ready.