Wow, I don't know where to begin.I am a woman 67, living with a man 65.   I have been in this relationship for 12 years.  I was in a very long term (27 years) relationship with a narcissist/emotional physical abuser. It took me many years in that relationship to see the Red flags and to understand what a narcissist is and what was happening to me.  I left that relationship and met my current partner (male). We bonded over similar trauma, though his former girlfriend and he were only together 4 years. Both our ex's were narcissists. I sold my home and we moved in together in his home.  Things have been good between us, except for sex.  At first, I was the one initiating, then he told me he had ED.  Said he was taking meds for it. We have had sex maybe 5 times in the 12 years all of which occurred early in the relationship.  We have had no sex for 8 years.  We are physical with lots of hugs,cuddling, and occasional kissing. I had some health issues that prevented me from being physically intimate.  Menopause and loss of libido so I am not blameless here. He has always been a drinker but not a drunk, and he has pre-diabeties. So I always thought that was the reason for his ED. But a couple of times during a couple of his all-nighter drinking sessions, he began crying and acting like something was wrong. When he tried to tell me what it was, he just kept shaking his head no and saying it was too embarassing.  Was that my first red flag?  maybe!  Then a month or two ago he said he wanted to "get in shape" he is on Medicare, he went to the local Y , but they would not take his membership as it is not covered by insurance.  When he came home he said he met a guy there that had a home gym and would like him to come and work out! (second red flag?) He went a couple times a week for about a month, then said he didn't like going there because the gym was in the basement and too hot.  Now he has seemingly never lied to me, nor I him, but.....?  Then this week another guy shows up at his business (refered by the home gym guy)  and has him do some work for him.  He does the work and for payment, he has the guy take him to lunch.  The guy is married and in fairness my guy did ask me if I wanted to go with them.  I had other plans and did not go.    I missed so many red flags in my previous relationship, wondering if I have missed more than these two incidents.  To not draw this out, here is a list that makes me wonder.   
He lived with his Widowed mother until he was in his late 40’s,
Never married, no kids ( me either for that matter)
Strict catholic upbringing
Very down on gay men
Moves away or moves my hands if I try to intimately touch him when cuddling in bed.
 
In his defense, he is high functioning Asbergers, has been a self employed businessman most of his life. (building his business solo with not much time for relationships).  We have had almost no conversations about the lack of sex except once when I told him I no longer initiate because I don’t want to make the ED worse. (performance anxiety) and he said that if I want it, to let him know. He never initiates, seriously.   So is this my fault???
I feel so stupid because when I read all this, I think AGAIN I have ignored red flags. My gut tells me something is off. And I am a known deny-er  But other than the lack of actual sex, the relationship has not been a bad one and I feel like he IS the person I love and want to spend my life with.  I honestly just don’t know!   And to add we are both very private about our phones, computers etc.  So no snooping has occurred on my end.