When I started my grammar school in 1999, after passing the 11-plus, I wore a different uniform to almost everyone else. The standard school blazer, a conspicuous emerald green number which made you a glowing target for the rough wit of local comprehensive school kids, cost around £60.

But, for those like me who were entitled to free school meals, there was a cheaper alternative. It was much darker green, you had to sew on the badge, and it only cost about £20. So there I was, in an emerald sea, the dark green sheep on my first day at school. There was a handful of us and everyone knew we were the poor kids. It is daunting enough to start big school, more so with the stigma of a different colour blazer.

I think the logic of the school was they were helping out by offering us a cheaper alternative. If they had thought about it, they should have just subsidised some of the cost of the proper blazer. But there we are. A few years later, as I entered sixth form, they scrapped the dark green blazer. And that was that.

Despite this and some other challenges, I look back very fondly at my grammar school. The camaraderie, the lifelong friends, the wheezing pipe organ, the remembrance services with the veterans, the competitiveness (including the ludicrously named "Cock House Championship"), the thumping school song, the culture of aspiration and achievement, the sprawling old boys' network. But more than all this weepy nostalgia, it was genuinely a route out of my social class.

I was the first in my family to get GCSEs and A-levels, to go to university. I was helped on by a couple of supportive family members and an excellent state primary school, which despite taking in 1,000 of the town's most deprived children was one of the best in the area. But it was also in large part thanks the to the quality of the education I received at the grammar school. I am the sort of person grammar schools are supposed to help. So I am living proof that the system works. Or am I?

Because it has meant so much to my family and me – my mum ripped the letter saying I'd passed the 11-plus because she was so desperate to open it, and then cried when she found out – going to a grammar is an important part of my identity. So I have a sentimental, emotional attachment not just to my grammar, but the grammar system more broadly. It is human to struggle to detach emotion from our reason, to be objective about the things that are important to us. And it is from this battle in my own mind that I am reluctantly coming around to the idea that I am not proof the system can work, but the exception proving the rule that it doesn't.

Few of us came from low income backgrounds. Many went to primary schools in affluent areas. Their parents paid for piano lessons and extra tuition. They were families who went skiing in the winter and bought branded food from the supermarket. Often these children were privately educated, went to posh prep schools where they are relentlessly trained to pass the 11-plus. Grammar school was a way for affluent parents to save money on educating their children. And it still is. It is largely a preserve of the middle classes.

Chris Cook, who was an education journalist for the Financial Times and is now the policy editor of BBC Newsnight, crunched the data on grammar schools and poor children. His verdict: "If you plot how well children do on average by household deprivation for selective areas and for the rest of the country, you can see that the net effect of grammar schools is to disadvantage poor children and help the rich."

At their peak in 1964, there were 1,298 grammar schools in England, educating 37.8% of all state-funded secondary school pupils. As of 2014, there were just 163, educating 5.1%. Of the 22,679 pupils in selective education, 98.8% achieved five or more A* to C grades at GCSE. This compares to the average for all state funded schools, of which there are 548,421 pupils, of 66.3%.

Source: House of Commons Library

Selection and separation based on academic ability is not unique to grammars. Most schools do the same thing every day. They carve children up into top, middle and bottom sets so teachers can tailor their lessons appropriately. Nobody is left behind, nobody is held back. So perhaps selective schools like grammars can work, if only they are reformed. Stop giving unfettered access to the grammar system to privately educated primary children. Maybe have quotas for low income children.

But this all seems unnecessary. Selection can still take place within the academies and free schools system. And it is much more fluid – a child can easily slip from middle to top set within a school, but changing school entirely is much harder, disruptive and more stressful. Bluntly judging ability at 11-years-old is absurd. Moreover, children benefit from mixing with pupils from a diversity of backgrounds. There is a richer diversity in the mainstream system than there is at grammars.

None of this is to say that the grammars have nothing to teach the rest of the system. They have a lot. A grammar-style culture of can-do and aspiration, that you can achieve anything you want to achieve, has often been said to be seriously lacking in too many schools. A school identity, a sense of heritage, pride in the institution, a strong alumni network for contacts and socialising – these should be universally beneficial, not exclusive to private and grammar schools. All of this helps to embolden children and give them that extra leg up.

Confidence: you will find many grammar or private school children who say they can do something when in reality they can't, and you will meet far too many other children who say they can't when, actually, they could if only they believed in themselves.

I understand why grammars hold mythic status in our society. They are symbolic bastions of aspiration, of achievement, of hard work paying off, of the social mobility that we crave. But at the very least, they need reform, if not abolishing altogether. Given how many parents still want them, the latter seems unlikely.

Those of us who hold dear our grammar school experiences need to ask deeper questions about whether they really are the vehicles of progress we want them to be. We may not like the answer. But I would hope our grammar education can give us the intellectual robustness to accept what the evidence tells us, and to do what is right. My old school's motto comes in handy once again. Forti nihil difficile: To the brave, nothing is difficult.