Could someone PLEASE explain why Affirmative Action is fair anymore?

Just saying "because there is still prejudice in the world" isn't cutting it anymore. We have a black president now. This nation (or any nation for that matter) will always unfortunately have people who are ignorant and small minded against other ethnic groups. It will not be a 100% hugfest EVER. Why are people getting jobs, positions, school admissions based SOLELY on the color of their skin when others are MORE qualified? I would much rather have someone who scored higher on a test and who can handle a position better based on intelligence than some subpar person who can't do a good job just because their skin is darker or their culture is different.