Royal Bank of Scotland (RBS) and Nomura Holdings were ordered to pay a collective fine of $806m (£511m, €707m) over mis-selling of mortgage-backed securities to US-state owned mortgage firms Fannie Mae and Freddie Mac ahead of the 2008 financial crisis.
US District Judge Denise Cote in Manhattan ruled the banks must pay the full amount requested by the Federal Housing Finance Agency (FHFA), after having found them liable in a lawsuit brought by the agency.
"The offering documents did not correctly describe the mortgage loans," Cote wrote earlier. "The magnitude of falsity, conservatively measured, is enormous."
Cote ruled Fannie Mae must receive $26.6m, while Freddie Mac will be paid $779.4m by the two banks. The mortgage bonds, worth an estimated $434m-$479m as of 26 March, will be returned to the banks.
Japan-based Nomura said it will appeal the decision, adding it "takes this situation very seriously and strongly disagrees with the outcome of the case".
"Nomura will continue to vigorously defend its interest in this matter and looks forward to taking its case to the US Court of Appeals," the company said.
Nomura sponsored $2bn of securities sold to Fannie and Freddie, while RBS underwrote four of the deals.
There were 18 banks that were sued in the US for selling subprime loans that led to the US subprime mortgage crisis, which was one of the seeds of the worldwide financial crisis of 2008.
The FHFA collected nearly $17.9bn in settlements from banks including Bank of America, JPMorgan Chase and Deutsche Bank.