Why does sex on the Court matter?

I was disappointed in reading this interview between Ronald Kessler (Newsmax.com) and former chief of staff Andy Card.

President Bush selected Harriet Miers for the Supreme Court after a search for other possible female candidates outside the White House began to lag, former White House Chief of Staff Andrew H. "Andy" Card, Jr. said in his first extensive interview since leaving the White House.
"The president was criticized for not nominating a woman when he nominated John Roberts," Card said, as he petted Sophie, his Wheaton Terrier, in the living room of his home in northern Virginia.
"And then when William Rehnquist left, you know, ‘What do you mean you're not going to nominate a woman? You've had two opportunities, and you haven't nominated a woman.' And so he was looking to nominate a woman — not blindly, not any woman, not just to nominate a woman."
As the search committee kept suggesting female candidates who did not seem right, Bush pushed for more selections.

Why? Why not pick the best and the brightest, regardless of sex? Samuel Alito was a terrific pick, as was John Roberts. I was not a fan of Sandra Day O'Connor, but it had nothing to do with her gender. It's politics.