Gary Barlow performs at a fund-raising concert in front of royalty at the Royal Albert Hall in London
Gary Barlowhas been urged to pay back avoided taxes Reuters

On one thing at least the political leaders are united, Gary Barlow should pay back the millions he kept out of the taxman's grasp, but there's no need for him to hand back the gong he received from the Queen.

The Take That pop singer is the latest wealthy celebrity to be found by the courts to have used schemes deliberately designed to avoid paying tax. He has not been accused of tax evasion, which is a crime.

David Cameron, Boris Johnson and shadow chancellor Ed Balls have now all agreed what he did was wrong and that he should pay back the money. But only Labour's Margaret Hodge, chair of the Commons public accounts committee, has called on him to hand back his OBE.

Cameron explained that the celebrity had done lots of good work for charity and music, for which he had been rewarded with the honour.

"Tax avoidance is, in these cases, these very aggressive tax avoidance schemes. They are wrong and we should really persuade people not to do them," he said.

"Gary Barlow has done a huge amount for the country, he's raised money for charity, he's done very well for Children in Need. The OBE was in respect of that work and what he's done," he said.

Comedian Jimmy Carr was previously found to have used the same scheme and some critics have pointed to the difference in language used by Cameron in relation to him compared to Tory supporter Barlow.

Questioned about Carr at the time, Cameron said: "Think of all those people who work hard, who pay their taxes, and out of their post-tax income save up to see Jimmy Carr.

"He's taking that money and stuffing it into something where he doesn't have to pay tax on it. That is not fair. That is not right. It isn't morally right to do that."

On this occasion, the prime minister has stopped short of accusing Barlow of "immoral" behavior although he has been as robust about the need for him not to use the schemes.

The problem for politicians, of course, is that they design the tax laws and, either by design or default, create the loopholes which give professional advisers the tools to help their clients avoid paying tax.

The fact it is invariably the wealthy who use these schemes only adds to the sense of public anger at such behaviour and, as Carr discovered, it is something the public deeply disapprove of.

Successive governments have promised to come down hard on such schemes and there has been some progress in recent years.

But, while avoidance remains legal, it appears the greatest pressure will come from ordinary members of the public using their muscle to express their distaste at the behaviour.