I was in a bad place at this time and he was to. I believe this was the cause of the crisis. I never really forgave him for the first A and we were both in a really bad place. I was looking for anything because my trust was completely gone. I think when I began to accuse him of the EA he then realized that he realized he ruined things and no matter what he did he could never conceive me that he was trustworthy. He folded after that! He gave up.
I'll caveat this by saying I've never been through an affair, but I have betrayed and been a betrayer, and I think you did nothing wrong and were in fact smart to begin again with zero trust. Trust is earned. Where you may have gone off the rails is assuming wrongdoing (instead of just verifying) and you say no matter what he did he had no path to ever convincing you he was trustworthy.
If you want to rekindle the relationship you must give them a path to a good place. Now, if you were just verifying, and he took that as assuming he was in the wrong--he may be gaslighting you. This is not uncommon with cheaters and so I enumerate that possibility for consideration. A huge blessing of these forums for me has been seeing reality from a less invested perspective.
There was an XGF I betrayed who I was trying to rekindle a friendship with. When she assumed the worst of me for a second time, some people suggested dropping her. I tried, "If you only see who I was and not who I am, I don’t want to be friends." Since then she's been more vulnerable and our friendship hasn't been particularly close but it has been kind, open, and low-drama.