You do you man and don't give 2 fvcks what people think. I was never a big pot head, maybe a little bit in my early 20's but now I turn it down all the time. It's just not for me. It's only akward if you make it akward. If they don't like who you are and who you want to be then find new friends that will. They are out there.
I can barley remember the last time I had sober sex. I mean, I do remember it's just been awhile. It's not easy finding sober ch*cks since they are like sheep to a heard but they are out there. I'll admit it, I'm an functioning alcoholic and battle with it all the time. It's hereditary and runs in my family like a wild fire.
I don't know if it legit but there are sober dating sites. Ha I just found this one: 12StepMatch.com