I think a man doesn't touch you if he doesn't want to. When a person wants done and is in a position to be completely done, like your H is, he sure as heck won't make contact with you AT ALL. This guy is still grabbing you and grabbing you in special spots.
More specifically, your H is living elsewhere. He can go very dark if he wanted to, but doesn't. He has dark spells but always comes popping out. I think it's safe to assume when has a dark spell, he's trapped with OW and wants no drama so goes dark. But I've learned when the OP does that, makes the person go dark, that just makes the H or W A.) resent OP, and B.) makes H or W think of their LBS.....and further more, they wonder why they aren't with their LBS.
I think yes, leave him be. Let him figure it out but try to maybe flutter around in his head....meaning when you see him, be hot smell good. Be nice and supportive, ask how he's doing and really care about the answer. Avoid any more mentions of OW. That's falling apart I think and he's really wondering what the heck he's doing.
As I've been told often lately (and I love it) "You are the prize.....act like it."