That is a very good question... when to raise something and when to try to sort out our own emotions... something I struggle with, as you know
But... it sounds like at this point, there is nothing valid for you to raise an issue about. People at work say stupid things like that without knowing your sitch. But, I would trust your instincts about that woman, and if there does end up being any socializing, increased contact etc. definitely say something.
That's my two cents for what it's worth at this point in my own sitch...
