I try to avoid the wittering of TV chef Jamie Oliver like the plague, but a few years ago I was forcefully struck by something he said. I mean that in the pejorative sense: what he said was even more mindless and silly than usual. Yet as so often with Jamie Oliver, liberal England sagely nodded along as the nonsense inexorably flowed from his mouth like sour milk poured from a jug.

Young British workers, Oliver said, were "wet behind the ears". Sounding every bit the putative Mike Ashley of the culinary world, Oliver claimed that young workers today needed to be able to "knock out seven 18-hour days in a row", adding that 48 hours was "half a week's work for me".

As has become almost obligatory, he contrasted torpid British youngsters with industrious European immigrants, without whom, he said, all of his restaurants "would close tomorrow". They were willing to put in the shifts that British youth were not, you see; "a 48-hour-week! Are you having a laugh?".

The level of debate surrounding immigration is often so low and sordid in Britain that as long as you strike an uplifting note you call slip all manner of nonsense under the radar like a low-flying stealth bomber. At one time being made to bash out "seven 18-hour days in a row" by your employer would have been enough to send anyone with a social democratic bone in their body into violent paroxysms of rage. Today, however, sounding off like Ebenezer Scrooge elicits a nod of affirmation from both the "captains of industry" and the Labour front bench – so long as you sound sufficiently bien pensant about immigration. After all, where would Britain be without those Romanian builders who are willing toil away on building sites for no more than a box of chicken? (if you don't believe that this happens, read This is London by Ben Judah).

And yet this state of affairs isn't all that "pukka" – as Jamie Oliver might put it – at all, is it?

Before I try to explain why, a pre-emptive throat clearing is in order. It has become so ubiquitous to bash migrants in the name of the "white working class" that I want to make it clear that this is not in any sense where I am coming from. I think immigration is on balance good for Britain, and despite everything I am about to write I am still not convinced by the argument which says that historical progress is furthered by more – rather than fewer – restrictions on where people can go to live and work. [End of disclaimer]

The Conservative 'row' over Philip Hammond is based on a false assumption about Brexit

My reminiscences regarding Jamie Oliver were prompted by the very Jamie Oliver-like noises that have been coming in recent weeks from the mouths of those who I am sure would not hesitate to call themselves "progressives". Their logic runs something like this: because the Brexiteers want, for lamentable and short-sighted reasons, to put an end to the free movement of people, every consequence of large-scale economic migration to Britain must be defended as stubbornly as the Russian army dug in at Stalingrad. To do otherwise would, after all, be gifting "ammunition to reaction", in the common left-wing formulation. Better to stay quiet or tell a lie if it is for the greater good. You can't make a borscht without cutting up beetroots, or something like that.

Twitter reacts to Home Secretary Amber Rudd's proposals for firms to reveal foreign workers IBTimes UK

This weekend liberals were proudly sharing a doom-laden Sky News article which warned us that "British vegetables will disappear from supermarket shelves if post-Brexit immigration controls prevent thousands of Eastern Europeans from working in the UK". The words were those of Guy Poskitt, whose company Poskitt turns over £35m ($42m) a year and employs agencies which pay workers the minimum wage to do back-breaking work in the fields of Yorkshire. Poskitt claimed that he "can't recruit local workers".

This is a ubiquitous bellyache. I heard it recently in Rugeley, a small town in Staffordshire, when I enquired as to why most of the staff who worked at the vast Amazon warehouse there were from Eastern Europe. "The locals don't want to do the work," I was casually told by a rep from one of the employment agencies used by the company.

Yet speaking to a few of the much-maligned locals, a different story emerged: many locals did want to do the work when the company first arrived in 2011; yet they were put off when they encountered the Victorian-style conditions they were expected to endure. A former miner in the working men's club – hardly the sort to shy away from arduous graft – was very clear with me that he "wouldn't do that work". "I wouldn't do it because I'd fall out with them [the managers] over how they treat people," he told me.

Defending the rights of migrants to come and work in the UK is one thing. Echoing the rhetoric of a hard-hearted and remorseless permutation of British capitalism is quite another.

Travelling around the country I heard the same thing over and over again: young British workers who had the temerity to expect a degree of dignity at work were dismissed as "soft" and "unwilling to graft" by both employers and otherwise progressive-minded people. Eastern European migrants were deployed as a rhetorical device in the same way that the previous government evoked "strivers" in order to bash "shirkers".

Budget 2016 sugar levy
Jamie Oliver once said that without immigrants from Eastern Europe his restaurants would close tomorrow Getty Images

Yet judging by the 100s of conversations I had, very often young British workers were simply unwilling to destroy their family lives by working 80-hour weeks to fill the pockets of celebrity chefs like Jamie Oliver. At companies such as Amazon, they refused to accept a situation in which they could be disciplined for taking a single day off sick or even for going to the toilet.

This unwillingness to roll over and take it ought to be a mark of progress. The British labour force has standards as to what it will and will not tolerate. Meanwhile, migrants from poor former Eastern bloc countries who have no idea what to expect and who are afraid to make demands in an unfamiliar country will in many cases put up with nearly anything. Bosses like Guy Poskitt say they cannot find British staff; yet were they unable to call on a vast reserve army of labour from poorer countries you suspect they would have to try a bit harder to attract them.

So, yes, good luck to those who come to the UK to graft. But let's not delude ourselves as to what is actually going on. Nowhere is it more in evidence that the right has trounced the left in the economic war of the past 30 years than in the adoption by progressives of the miserly narrative that there is something virtuous in being worked ragged by an employer. Defending the rights of migrants to come and work in the UK is one thing. Echoing the rhetoric of a hard-hearted and remorseless permutation of British capitalism is quite another.

James Bloodworth is former editor of Left Foot Forward, one of the UK's top political blogs, and the author of The Myth of Meritocracy.