Rachel its his kids too and if he would like to come and see them, he needs to agree with you when that will happen. Say tuesday/thursday eve and every other weekend, whatever works for both of you really. Dont let him make you feel guilty, it is his relationship woth the kids and he needs to maintain it, it is his responsibility.
I think for now you could get away with not letting him take the kids, but long term you might not have a choice and essentially you cant stop him, he is their dad.
So maybe just saying to him that emotional wellbeing of the children is really important and at this stage introducing them to anyone isnt a good idea. General advice is 6-12 months after bd. So you have a long way to go.
But once again depending on hiw he takes it you might not have a choice