The Chair of the Conservative Party, Baroness Sayeeda Warsi, is to speak out today against prejudice against Muslims, which she claims is the last acceptable form of bigotry in Britain.

She is due to speak later today at the University of Leicester, however some snippets of her speech were publicised today in the Telegraph. In her speech Baroness Warsi will speak out against the "patronising" and "superficial" way in which issues of faith are discussed in the media, but she will also tackle the subject of "Islamophobia".

One does not like to comment before the full speech is given, but if current reports are anything to go by then Baroness Warsi would be best advised to look at why anti-Muslim prejudice has lodged itself in parts of our society.

Prejudice against anyone is never a good thing, but it is worth remembering that stereotypes, no matter how twisted they become sometimes have a seed of truth in them.

Jews for example have long been stereotyped as being at best rich and at worst money-grubbing. This unflattering view came about because for many centuries Jews in this country were not permitted to do any job other than that of moneylender, while the rest of the population was banned by the Church from such work. As a result Jews came to be associated, through no fault of their own, with moneylending.

Prejudice against Muslims however takes a different form. Baroness Warsi is expected to say that it is wrong to categorise Muslims into "moderates" and "extremists" as it will lead to people automatically saying things like "The family next door are Muslim but they're not too bad".

This may be so but it is a sad fact that Muslims themselves feel the need to do this because of the actions of their co-religionists. I remember on one occasion having a general chat about life with a chap from Africa who then told me that he was a Muslim. Before I could say anything else he immediately added, "But don't worry I'm a moderate not a suicide bomber".

Seeing as I was standing right next to him I was very relieved to hear this, but it goes to show that any prejudice felt against Muslims is generally not a result of irrational racial bigotry, but of real concerns about the actions of a considerable number of Muslims both in Britain and across the world.

After all if anti-Muslim prejudice is the result of irrational fear and loathing of the other, then it follows that Britain's Hindus and Sikhs would also be the victims of similar prejudice. However prejudice against either of these groups is not generally found, at least in this country.

It is the violent language and actions of a significant minority of Muslims which have created this prejudice, just as in centuries gone by, violence and the promotion of assassination by the Catholic Church ended up turning Catholics into second class citizens.

With Islamist violence and extremism, perhaps the most shameful aspect of it, in Britain and abroad, is the use of fellow Muslims as human shields.

Hamas is of course accused of doing this in the literal sense, putting mortar squads right next to schools and hospitals so that whenever the Israeli Defence Force passes by it is the killing of Palestinian children that gets in the news.

But even in the West the fellow travellers with Hamas, bin Laden and the Ayatollahs in Iran, like to use innocent Muslims as metaphorical human shields. Whenever extremists are faced with criticism for their despicable and murderous beliefs the first weapon they reach for is the word "Islamophobia". An attack on them is a disgraceful and prejudiced attack against all Muslims they say.

And so it is that the people who suffer most from the effects of Islamic extremism are decent law-abiding and non-violent Muslims who are just trying to get on with their lives. Firstly because they are humiliatingly forced to add the words, "I'm a moderate" to the words, "I'm a Muslim", thanks to the reputation given their religion by the extremists.

More seriously it is of course Muslims who are the victims of the violence and terrorism carried out by their brothers, strangely in the name of Islam.

Only today 45 people were killed by suicide bombers in Iraq at a Shia Muslim festival. Before today however countless Muslim lives have been lost in Iraq, Afghanistan and Pakistan as a result of bombings, shootings and attacks carried out by Islamic extremists.

That is not to say that many lives have not been lost as a result of the Western interventions in these countries, but let us not forget that many of those proudly fighting the invading infidels have done so mainly by killing their fellow Muslims, often while they are at prayers in their mosques.

Baroness Warsi is right that there is a lot of mistrust and even prejudice towards Muslims in this country, leading to formation of ugly groups such as the English Defence League. However the source of that prejudice is not irrational bigotry on the part of non-Muslims, but it is the real concern about the words and actions of many within Islam. Only when that changes will Muslims be accepted in the same way that Hindu's, Sikhs and even Catholics have been.