Perception of colors

I often times find myself really upset when men who practice heterosexuality have more feminine characteristics than male.  One characteristic that I deem as feminine is being able to recognize, distinguish and identify multiple colors without having formally been trained.  I add the training in because if you work for Sherwin Williams or Home Depot, it is your job to be able to distinguish between say "Egg" and "White."  My utter helplessness in this matter has made me think about colors because very few creatures on Earth can see the wide variety of colors we do.  Shockingly, I have found the following questions brought up to me by other people; interestingly, all of them are my second cousins.  When I bring it up with others, they find it interesting yet they never thought of it before. It's possible it's unique to my family but in this public forum I ask if my theory holds. 
      When I look at a color that I identify as "Red" is it the same thing that you see as "Red"?  If, say you learned that what I perceive as blue, you perceive as red, neither of us would ever know because we'd both say "Red" when we saw the color.  Some have speculated to have spectrum of different shades of red and see how far the spectrum goes till someone sees a new color, but that doesn't solve the problem that it is quite possible we could be seeing two completely different colors.  You couldn't even go to brightness because yes Black is darker than White, but what's darker between Red and Blue?  As far as the brightness, it's kind of equal.  Same goes for say Green or orange.  I only recognize 11 colors, (Black, white, green, yellow, orange, brown, red, blue, grey, pink, purple).  From brightness factors I think White, Yellow, pink .... Brown, Black from brightest to darkest.  The other six colors seem interchangeable to me. 
      Now that the basic argument is out of the way, I'll go into why I came up with this.  I've looked at colors that I would swear are one color and most people see a different one.  So I wonder if we're seeing the same thing.  I have never seen anything that I conceive of "Blue" in what people call "Midnight Blue."  I only see black.  The box of 64 crayons is meaningless to me.  I've stared at sheets that everyone kept telling me was brown but all I could see was "black"  I tried very hard to see what others were seeing but I just couldn't do it.  I guess the spectrum theory is the best test for this, but it's not the most conclusive to answer these questions. 
      With all this said, I wouldn't advise you to ask me to decorate anything and I'd beg for your patience if my clothes don't match because to me, it looks like the same color, but it may not be the majority opinion (this is always why I try to wear khaki, white and black since it goes with everything...at least I think it does).