I had this problem with my GID ex wife. She became unapproachable and getting her to have sex was like playing "wake the dead", a complete boner killer. For a long time I thought it was me because of my weight but then after the separation, I hooked up with a woman for a weekend. I was relieved to see I had ZERO problems with her even if I didn't find her very attractive. Also had no problems with the ex girlfriend afterwards, was quite attracted to her.
What I learned was that even if age/weight play a factor, things will work out of there's a connection and your mind's in a good place. And yes...even on the off times when it isn't cooperating, a little patience will result in being able to be hard all the way to finishing.
If your GID ex is having problems, its very likely he's in a bad place in his head or not being honest with what he wants. The pressure to perform when he isn't into it just compounds everything. A comedian once said (paraphrasing) "the mind can tell itself stories all day but the D is the best lie detector."
Have you not found a difference with being with a straight man? Being with a straight woman was like night and day compared to my ex. You can just tell they want you and that wall I got used to just isn't there. I see no benefit in trying to be with a person who wants something else than you.
That's my 2 cents on this.