David Cameron and Jose Manuel Barroso
David Cameron has dismissed Barroso's warnings Reuters

Ed Miliband has used a mini-reshuffle to toughen Labour's stance on Europe just as moderate Tories are starting to fear a panicked David Cameron is leading the UK towards inevitable exit from the EU.

Miliband has appointed former Blairite frontbencher and "hard-headed" pro-European Pat McFadden as shadow Europe minister in what is seen as a clear sign he is not about to execute a U turn and back Cameron's 2017 in-out referendum as many have been predicting.

The appointment is significant as it comes as the prime minister is facing fears from his own loyalist supporters that he has painted himself into a corner over Europe in his desperation to head off the Ukip threat, particularly in the looming Rochester by-election.

And, once again, he faces charges of abandoning long-term strategic thinking in favour of day-to-day headline chasing leading to a hardening of his position on immigration which threatens to dominate the general election campaign.

Ignore the daft ambitions of people who've made interests for just getting out of Europe. Let's look after Britain in the modern world, keep us a modern economy.
- Ken Clarke

The election strategy was supposed to be focussed on the reviving economy although, with borrowing back on the rise and the euro crisis threatening to return, that may now be less powerful than originally hoped, landing Cameron with a double challenge to get back on track.

The EU fightback has been led by veteran former cabinet minister and passionate pro-European, Ken Clarke, who said some of his fellow backbenchers should be members of Ukip and claimed there was a "wave of almost hysteria" about immigration.

And in a direct message to the prime minister, he urged him not to compete with "ignorance and bigotry", adding: "Ignore the daft ambitions of people who've made interests for just getting out of Europe. Let's look after Britain in the modern world, keep us a modern economy."

Those remarks, coming from Clarke, will not surprise Cameron but it is the growing sense of nervousness from other backbenchers with less passionate views about the EU that should worry him.

They are the larger group of loyalists who are eager to see sensible reform of the relationship between the EU and the UK but are determined to stay within the Union.

And they are becoming concerned that option is being closed off by Cameron's increasingly divisive, Eurosceptic language, and the "obsessive" focus on immigration.

Those fears were fanned by the prime minister's response to comments made by outgoing EU Commission president Jose Manuel Barroso that Cameron's plans to limit immigration from existing EU states was illegal.

He also compared Cameron unfavourable to Margaret Thatcher, writing in the Daily Telegraph: "It would be impossible to hear a Ukip speech coming from Margaret Thatcher, completely impossible."

Cameron quickly sent a "you are not my boss" message to Barroso - the man he urged the Scots to listen to during the independence referendum - while Tory party chairman Grant Shapps branded him an unelected bureaucrat and who would be proved wrong.

Pat McFadden
Pat McFadden is a 'hard-headed' pro European Reuters

That sort of language, with its echoes of Nigel Farage's rhetoric, delights hard-line Tory Eurosceptics but deeply worries others who believe it is born out of short termism at a time when the UK needs to be building alliances inside the EU if it is to have any chance of securing the sort of renegotiation Cameron has promised.

And that is the second problem they perceive; that the prime minister has abandoned his previous, they say sensible decision not to slap down red lines before those negotiations and is issuing ultimatums on a regular basis under pressure from his rebels.

They repeatedly warn Cameron that attempts to "out-Ukip Ukip" are bound to fail because he Nigel Farage can simply up the anti-EU, anti-immigration rhetoric.

And even recent events that would harm other parties just slide off Farage's back.

The latest are claims of racism over a pro-Ukip Calypso sung by a former DJ in a fake Jamaican accent, and the fact Ukip have welcomed a far-right, Hitler apologist to its group in the Brussels parliament to ensure it keeps receiving funding.

But so long as Ukip is seen as a means of kicking the establishment, none of that is likely to stop disillusioned voters supporting the party.

The problem comes if that trend fades either in Rochester, which the Tories should still hold despite the expectation management, or at the general election where any protest vote should be squeezed out of the system.

It would then be impossible for Cameron to start rowing back from some of his more extravagant promises and, in any case, the damage done to UK relations with the rest of Europe may by then be irreparable.

And that could all lead in only one direction – the "out" door.