Lord Fink
The Tory peer had previously threatened to sue the Labour leader over his allegations BBC News

A Tory peer has admitted to "vanilla" tax avoidance measures after he threatened to sue Labour leader Ed Miliband for "untrue and defamatory" remarks.

Lord Fink, a former Treasurer for the Conservative Party, revealed that he had transferred shares into family trusts while in Switzerland.

"The expression tax avoidance is so wide that everyone does tax avoidance at some level," he said.

His latest comments come after the peer urged Miliband to re-state his tax avoidance allegations against Fink outside of the House of Commons, where speakers are protected from libel under parliamentary privilege.

But the former hedge fund boss, believed to be worth £200m, has now admitted he did not object to the Labour leader's use of the word "tax avoidance".

"Because you are right: tax avoidance, everyone does it," he told the Evening Standard.

"I don't even want to sue Miliband," he said. "In my life I have been libelled a few dozen times and I have never sued anybody, even for some comments that were quite outrageous.

"If he simply uses the words 'Lord Fink did ordinary tax avoidance' then no, I couldn't sue him.

"But if he made the statement 'dodgy' about my bank account, that was potentially libellous. That was the issue I took exception to."

He added: "I also took exception to him saying I had questions to answer. In fact, whenever anyone has put questions to me I have answered them."

"I chose the mildest end of the spectrum that I was advised on," he said. "What I did ... was at the vanilla, bland, end of the spectrum."

'Dodgy prime minister'

David Cameron and Ed Miliband
Cameron and Miliband clashed in the Commons Reuters

The revelation follows a clash in the Commons between David Cameron and Miliband over the HSBC tax avoidance scandal.

Miliband claimed seven Tory donors revealed in the leaked files have given nearly £5m ($7.6m, €6.7m) to the Conservative Party.

The Labour leader also alleged there was a "revolving door" between Tory headquarters and the Swiss branch of HSBC.

But Cameron hit back at Miliband and claimed no government had been tougher than his on tax evasion.

"Well, I saw this list just before coming to PMQs and one of the people named is the Labour donor Lord Paul, who funded Gordon Brown's election campaign," he said.

"I'll be very clear: people should pay their taxes in this country and no government has been tougher than this one in chasing down tax evasion and tax avoidance."

However, Miliband said the difference between him and Cameron was none of the donors had "given a penny" to Labour on his watch.

"None of these people have given a penny on my watch and he's up to his neck in this. Let's take Stanley Think, who gave £3m to the Conservative Party," he said.

"He actually appointed him as Treasurer of the Tory Party and gave him a peerage for good measure. So now can he explain what steps he's going to take to find out about the tax avoidance activities of Lord Fink?"

Cameron replied: "I will tell him, Mr Speaker, about the differences between him and me. When people donate to the Conservative Party, they don't pick the candidates, they don't choose the policies and they don't elect the leader.

"When the trade unions fund the Labour Party they pay for the candidates, they pay for the policies and the only reason he's sitting there is because a bunch of trade union leaders decided that he was more left-wing than his brother."