Settlement talks between one of the largest investors in Bernard Madoff's Ponzi scheme and the trustee appointed by a court to recover money for swindled victims have broken down.

Hedge fund Kingate Management and Irving Picard, the court-appointed trustee who is liquidating the bankrupt Bernard L. Madoff Investment Securities LLC, were in discussions to settle for $800m (£493m, €593m).

The settlement talks fell apart after the Justice Department announced its decision to exclude Kingate from payouts out of the agency's $2.35bn Madoff Victim Fund, which is overseen by former SEC chairman Richard Breeden.

Breeden's $2.35bn victim fund will be made available to thousands of indirect investors and third party victims of Bernard Madoff's fraud; Picard's office, which has been reimbursing direct Madoff investors, had expected to secure $800m for Kingate from Breeden's fund, according to reports in the Wall Street Journal.

Picard has so far recovered $9.51bn for Madoff investors through a separate compensation fund.

British Virgin Islands based hedge fund Kingate invested $1.73bn in Madoff's firm from 1994 to 2008, according to court filings.

US Attorney Preet Bharara said the $2.35bn comprising Breeden's Madoff Victim Fund represents money obtained by federal prosecutors through criminal and civil forfeiture actions, after Madoff's $65bn (£40bn, €48bn) Ponzi scheme collapsed in December 2008 and led to investors losing around $19bn in total.

Breeden's fund will allow Madoff victims who invested through Kingate Management to collect payments directly, rather than awaiting a settlement for the hedge fund.

Breeden defended the move and said he prioritised returning cash to Madoff's victims over Picard's negotiations with various feeder funds.

"You can't say the distribution of a couple billion dollars to the victims of crime has to structure itself around commercial litigation settlements. It's the other way around," he said.

So far Picard has denied 10,921 "third party" claims, according to his website, though some claimants continue to challenge his methods in court.