Ukip leader Nigel Farage went on the offensive after senior Tories derided the party for the views of some of its supporters.
Farage insisted the Conservatives had embarked upon a smear campaign because they are scared of how Ukip's new-found popularity will impact at the English council election on Thursday (2 May).
Government minister Kenneth Clarke restated a claim by Prime Minister David Cameron that Ukip was full of "fruitcakes and closet racists".
He said: "Some of the people who have assured me they are going to vote Ukip I would put in that category."
Ukip was shaken last week by allegations of allegedly anti-Semitic comments against one of its candidates. Council candidate Anna-Marie Crampton was suspended over comments posted online alleging Zionist conspiracies.
But Farage vowed he would carry on talking about levels of immigration in Britain.
The Ukip leader and European parliament member incurred the wrath of Twitter users for warning about "criminal" Romanians and Bulgarians on BBC's Question Time.
Farage told This Morning: "These are tough subjects to talk about. All we are doing is talking the truth. There is nothing in Ukip that is racist in any way at all and Ken Clarke knows it. If he throws this abuse out, we will start having a proper debate.
"They know that the British public are genuinely concerned about opening up the doors to Bulgaria and Romania next year.
"They are concerned because we have a million youngsters unemployed, we have wages being driven down and I am afraid a crimewave in London being caused by Romanians already."
Tory London mayor Boris Johnson broke ranks with fellow senior Conservatives party by calling Farage a "rather engaging geezer" who "sticks up for Britain" in his Telegraph column.
The anti-EU party is fielding a record number of more than 1,700 council candidates this week. Ukip has drawn support from large numbers of disgruntled Tory voters.
Deputy leader Paul Nuttal told IBTimes UK that the party's long-term plan had been revised to capitalise on surging support.
Lashing out at Clarke, Farage said of the Conservative party: "They have lost the argument so Ken Clarke and others have decided that, rather than going for the ball, they are going to attack the player. That is exactly what this is about."
He said Ukip was the only party that had a blanket ban on former members of the far-right British National Party joining.
"The only party in British politics that does not allow former members of the BNP to even join us as a member, let alone be a candidate," said Farage. "Labour has sitting councillors who are former BNP members, the Tories have BNP members in their party. We forbid it."