“We have been sold this meme of Islamophobia, where every criticism of the doctrine of Islam gets conflated with bigotry toward Muslims as people,” and Affleck jumped in for the third time, more or less declaring the mid-show interview over: “Now hold on—are you the person who understands the officially codified doctrine of Islam? You’re the interpreter of that?”
As many have since pointed out, Affleck and Nicholas Kristof then promptly demonstrated my thesis by mistaking everything Maher and I said about Islam for bigotry toward Muslims. Our statements were “gross,” “racist,” “ugly,” “like saying you’re a shifty Jew” (Affleck), and a “caricature” that has “the tinge (a little bit) of how white racists talk about African Americans” (Kristof).
The most controversial thing I said was: “We have to be able to criticize bad ideas, and Islam is the Mother lode of bad ideas.” This statement has been met with countless charges of “bigotry” and “racism” online and in the media. But imagine that the year is 1970, and I said: “Communism is the Mother lode of bad ideas.” How reasonable would it be to attack me as a “racist” or as someone who harbors an irrational hatred of Russians, Ukrainians, Chinese, etc. This is precisely the situation I am in.
My criticism of Islam is a criticism of beliefs and their consequences—but my fellow liberals reflexively view it as an expression of intolerance toward people."
"Kristof made the point that there are brave Muslims who are risking their lives to condemn “extremism” in the Muslim community. Of course there are, and I celebrate these people too. But he seemed completely unaware that he was making my point for me—the point being, of course,
that these people are now risking their lives by advocating for basic human rights in the Muslim world."
When I told Affleck that he didn’t understand my argument, he said, “I don’t understand it? Your argument is ‘You know, black people, we know they shoot each other, they’re blacks!” What did he expect me to say to this—“I stand corrected”?
Although I clearly stated that I wasn’t claiming that all Muslims adhere to the dogmas I was criticizing; distinguished between jihadists, Islamists, conservatives, and the rest of the Muslim community; and explicitly exempted hundreds of millions of Muslims who don’t take the doctrines about blasphemy, apostasy, jihad, and martyrdom seriously, Affleck and Kristof both insisted that I was disparaging all Muslims as a group. Unfortunately, I misspoke slightly at this point, saying that hundreds of millions of Muslims don’t take their “faith” seriously. This led many people to think that I was referring to Muslim atheists (who surely don’t exist in those numbers) and suggesting that the only people who could reform the faith are those who have lost it. I don’t know how many times one must deny that one is referring to an entire group, or cite specific poll results to justify the percentages one is talking about, but no amount of clarification appears sufficient to forestall charges of bigotry and lack of “nuance.”
http://www.samharris.org/blog/item/can-liberalism-be-saved-from-itself
That's a bit of Harris' response to the criticisms of his statements being called "racist", there's much more on his blog of you want to read it