BUT, if I divorce my wife because she had sex with another man (or women, or animal) then she is already guilty of adultery and I have not made her commit adultery (remarriage is always the assumption). This is why it says to the men: whoever marries a divorced women commits adultery. What we see then, is that when a man divorces a women and she remarries you have made her an adulteress, except in the case of her committing adultery on her own. Read Matthew 5:31-32 like this:
but I say to you that everyone who divorces his wife makes her commit adultery (except for the reason of unchastity, in other words except in cases where she has in fact committed sexual sin) and whoever marries a divorced woman commits adultery.(this is for the men so that they don't get a free pass to marry divorced women)
So in reality the "vaunted exception clause" is about how divorce results in adultery, unless adultery lead to the divorce in the first place.
1 Corinthians 7:8-16