This week's Tory conference has quickly been defined as a battle between displaced Cameroonian Remoaners and tails-up Brexiting May-ers. As a Cameroon I'm afraid I haven't got that memo. I've already been described by two journalists as the most loyal of sacked ministers. When I left the government, I told the PM I would be a loyal supporter from the backbenches, and that's what I intend to be.

The media wants there to be a war between the Cameroons (and Ken Clarke) and the new government. Every pronouncement, every article (including this one), every drinks order at the bar will be scrutinised for evidence of disloyalty and fomenting civil war.

There is some dissent, but not as great as people think. This is a new government, and it is keen to stake out new territory and tell us why it's different. The chancellor today is announcing the end of austerity, and is ripping up the deficit reduction targets set by the last chancellor. That's being spun as a break with the past. But that's also prudent in a post-Brexit world, when no one really knows what will happen to the economy over the next few years.

The prime minister wants more grammar schools, and it's true that's caused a mild fluttering among the Cameroons – but it's not earth-shattering, wrench in a new direction stuff. Again, it's good politics, and amounts to little more than a mild change of emphasis.

We should recognise that May is absolutely in tune with the Tory activists, in a way that perhaps the Cameroons weren't. She spoke in my constituency last autumn, and went down a storm. The foot soldiers who flog their guts out for the party recognise her as one of their own. They will protect her from too many dissenting MPs.

There is some concern that the PM is too immersed in the detail, but that doesn't mean she won't make good, timely decisions. In my own little world of media policy, she quite rightly overturned a decision to allow the chairman of the abolished BBC Trust to move seamlessly to become the chairman of the new BBC board. A small decision, but the correct one.

In any event, that decision shows the approach the PM intends to take. She wants to be seen to be across policy, and to be taking the right decisions for the right reasons in the right way. It is becoming a bit tedious to see the relentless briefing of the last government as a sofa government. In my experience, all ministers from Cameron down were hard-working, and clear processes were followed. But I guess, if it suits the narrative, go for it.

Anyway, the only thing that really matters is Brexit. I can see how Brexit could become a proxy war between Cameroons and May-ites, between traditionalists and modernisers. There is definitely a feeling among the leading Brexiteers of "we are the masters now", and it suits them to tilt at the windmills of the Remoaners while they work out the detail for Brexit. I hope that sense of triumphalism abates, as the next few years will be hard pounding.

But I would exempt Theresa May from any charge of triumphalism. She has played a blinder so far. Just before conference, a momentum was building for clarity and a sense of direction for Brexit. And that is what she gave the conference and the country on Sunday. We now know, for better or worse, that the process will begin next March. We don't know much about the detail, but that is mainly because this is a negotiation, and it's hard for detail to emerge when the other side, the European Union, won't negotiate.

I remain a confirmed Remainer. I am not going to pretend that everything will be rosy, and I hope the rumblings from Nissan in Sunderland don't become an earthquake. But I accept that a clear verdict has been given, and I will back the government in seeing through this extraordinarily difficult process. Just don't ask me to drink the Brexit Kool-Aid as well.

Of course, there is one thing astonishingly absent from this narrative. The Labour party. Last week they plunged into the abyss. At one of the most testing times for our country, the official opposition has left the field of play. That's not good for the nation, and it's not good for the governing party, which needs to be tested and needs to avoid turning in on itself.

Some might expect Theresa May to call an early election in these circumstances. But again, she has made clear that "2020 means 2020". As political journalists study the tea leaves, and try and work out who is in and who is out, what's being ditched and what is being kept, perhaps they are missing what is under their nose. With Theresa May, what you see is what you get. The country voted for Brexit, and Brexit will be delivered. See you in 2020 for the general election.

Ed Vaizey is Conservative MP for Didcot and Wantage and was Minister for Culture & Digital Economy 2010-16