The Ukip candidate for North East Hampshire Robert Blay was suspended with immediate effect, according to the party.
A video published by the Daily Mirror shows Blay threatening to shoot Ranil Jayawardena, his Conservative rival for the seat.
The footage was secretly recorded by a Mirror reporter at a party event in Kent on Saturday.
Blay told a Mirror reporter that Jayawardena had been tipped to potentially be the country's first British Asian Prime Minister.
The former member of the Conservative Party said: "If he is I will personally put a bullet between his eyes.
"If this lad turns up to be our Prime Minister I will personally put a bullet in him. That's how strong I feel about it.
"I won't have this f***** as our Prime Minister. I absolutely loathe him."
He added: "His [Jayawardena's] family have only been here since the Seventies. You are not British enough to be in our parliament.
"I've got 400 years of ancestry where I live. He hasn't got that.
"I said to his dad about two months ago, 'when did you come to Britain?' He said, 'in the 70s'. I said, 'why did you come?' He said, 'things weren't very good politically in Sri Lanka and I came here and I could train as an accountant'.
"So he's come here and ponced off us, hasn't he, like all the east Europeans are? That's what is happening. Continually.
"His name's Jayawardena and I'm told that name is a Tamil name. Well the Tamils were Indians which went to Sri Lanka to take it over and they got their asses kicked. So, he comes here, ponces off us and then his son's in our political system."
A UKIP spokesman said: "Mr Blay has been suspended with immediate effect."
He added: "Any comments of this sort have absolutely no place in British politics or public life, and the party would like to take this opportunity to apologise to Mr Jayawardena for any distress caused."
Blay's comments were filmed outside a meeting where Ukip leader Nigel Farage was delivering a speech to voters in Ramsgate part of the Thanet South seat the UKIP leader is trying to win.
North East Hampshire is one of Britain's safest Tory seats and Jayawardena, who is Conservative Deputy Leader of Basingstoke and Deane Borough Council, is tipped as the favourite to become its next MP.