The Loving v. Virgina case was settled on the grounds of equal rights by the 14th amendment. If I have a right everybody has the same right. I have the right to marry a white women, therefore everybody has the right. It does not suffice to suggest that everybody has the right to marry somebody of their own race. Likewise I have the right to marry a women, so everybody has the right to marry a woman. Thusly, even women have the right to marry women. Equal protection under the law legalizes gay marriage without needing to say anything about it being gay.
It is a restriction on my rights to say that my sister can do something that I cannot do. Although, I have no yen to marry men, it's a right and must be equally honored for all people. It isn't gay marriage which is being restricted by prohibitions against gay marriage. It's the rights of all individuals to marry those of the same sex... even though only gay people would care to do so.
When gay marriage is legalized by the supreme court, that'll be the logic behind it.
Adding Video, via Pandagon