This week, I passed the four lovely Portland stone pillars near Hyde Park corner in a taxi. These are Memorial Gates dedicated to the South Asian, African and Caribbean soldiers who fought in the two world wars. Both myself and my driver, a young Sikh man, were wearing a poppy. I asked him if he knew about this monument which took its place only in 2002. He didn't.

He also didn't know about the five million of these soldiers, the hundreds of thousands of them who died. So who does he remember with his poppy? 'Oh you know, English and American soldiers who saved us from Fascism. Heroes like'. He'd learnt about the Holocaust and great battles but never about the part played by his people in the war sagas. His own parents had never told him.

Muslims and Sikhs were exceptionally brave. They were even praised by German soldiers. Field Marshall Sir Claude Auchinleck, Commander in Chief accepted that 'Britain could not have come through both wars if they hadn't had the Indian army'. Caribbean, Arab and African soldiers played their part. A good number of soldiers of colour and valour received George and Victoria crosses. Yet many of us who hail from the colonies are often told by native Brits: ' You can never be British. We didn't fight the Germans to have you come and steal our country'. To them the wars were won by men who looked like Michael Redgrave or Henry Fonda.

Much of this past lies under our feet, unknown. Our establishment likes to keep historical narratives simple and uplifting, and white. Britons of colour who've tried to get the opportunity to present a television history series are rebuffed, told such programmes are 'difficult' and 'complicated'. That is about to change as at long last. David Olusoga, mixed race and hugely talented, excavates the buried stories of black and Asian people going right back to the Roman occupation. Africans were, in England, then noblemen and women. The series Black and British: A Forgotten History, begins on the 9th of November, two days before Remembrance Day. There is also an accompanying book.

War
Our whitewashed history ignores the heroic role of people of colour in the war, says Yasmin Alibhai-Brown iStock

Hitherto, the only substantive book on the black presence in Britain was Staying Power – by the late Peter Fryer – published in 1984. Most Britons have never heard of this enlightening and eloquent book. Fryer, an Englishmen, once told me that history was the passport to national belonging. He was so, so right.

Children of all backgrounds need to know about slavery; the truths, good and bad, about the empire; migrants who came from the 16th Century onwards; England's love of the east – the subject of my last book Exotic England. The design of St Pauls, for example, was described by Christopher Wren as Indo-Saracenic. Queen Elizabeth 1 adored Ottoman style and fashions, invited Istanbul's oligarchs to London and gave them a good time. That Koh-i-Noor diamond in the queen's crown was taken from Maharaja Duleep Singh, a boy prince of Punjab, whose land had been annexed by the British. The child was separated from his mother and vigorously Christianised. Samuel Johnson loved his young black servant, Francis Barber, who was sent to school and left an annuity by his great master.

War
Sikh men attend a remembrance service for the five million people from the Indian sub-continent, Africa and the Caribbean who served with Britain in the two world wars in Hyde Park in central London, 10 November 2006 Getty Images

Warren Hastings, the first director general of the East India Company, was shabbily treated by British politicians partly because he went native and fell in love with India and its peoples. He built an Indian manor house, a sanctuary in the Cotswolds. Most Britons have heard of the noble William Wilberforce, but how many know Olaudah Equiano or Ignatius Sancho, freed slaves who inspired the abolitionist movement? Peggy Ashcroft had an affair with the black American singer/actor Paul Robeson and fell a little in love with him. There were three Asian MPs in our parliament in the late Victorian age, all elected by white people. When Gandhi came to London, the white working classes came out in their thousands to show their love and respect.

Young British Muslims perhaps would not feel so alienated if they knew that that hundreds of upper class Englishmen converted to Islam in the 1800s, that Winston Churchill gave Muslims land in Regent's Park so they could build a place of worship in the heart of the city, that the first scientists of the Royal Society learnt deep Arabic so they could read the books written by early Arab intellectuals, or that Muslims helped liberate Jewish people in Europe.

These facts really matter. Those who have whitewashed history are guilty of either terrible deception or terrible ignorance. Britons from elsewhere helped make this country; our roots go deep into the soil. We have inviolable rights. We belong. Brexit makes it more important than ever before to lay down that claim and fight for it.