Help with a relationship in terms of height and self esteem?

Alright, well the first thing to know is that I am a gay guy. Disclaimer: I respect everyone for their opinions and will not start any arguments on who gets what right and so on and so fourth. Everyone has the right to an opinion with their beliefs. Now, I have a partner who is a great friend, confidant, and person. We always have a good time together and all is well. I, by nature, am ually ambiguous. Many wonder if I am gay or straight. He is the same way. However....I am 5'7 and 140 lbs. He is 6'4 and 180 lbs. I have always suffered from self esteem issues in regards to my height. It makes me feel small and like a push-over. I try to respect myself for me, and sometimes it works! Others, mmmm, not so much. I am not feminine and I believe that in gay relationships, there is no one who serves as the "woman". They are both two men! That is why it's gay! :) But with this guy, albeit, I am happy in ways, I feel so small and insignificant with him. I do not want to end it, nor does he, but I don't know what to do. By being the smaller guy with the larger one, I get labeled as the "woman" and being gay is hard enough, but being called a that one is not, well...a new level. I wish I could just get over it, but I seem to have issues. So here we are. Does anyone have any ideas or advice they could share with me on how to deal with this? I really am happy with him, I just don't know how to get over my self-esteem issues regarding my size. Any advice? Thanks, guys.