AIJ Investment Advisors president Kazuhiko Asakawa and three others are arrested on fraud charges amounting to $1.4bn (£890mn) on Tuesday.
Asakawa was accused of defrauding the country's two corporate pension funds, one in the country's capital Tokyo and the other in the northern Japanese city of Nagano.
The police also arrested Shigeko Takahashi, an AIJ executive and Hideaki Nishimura, an employee of AIJ's sales arm, ITM Securities, as well as another person.
An investigation into the scandal was going for quite some time with Japan's Securities and Exchange Surveillance Commission raiding the company headquarters in March.
Company officials admitted to the authorities that it could not account for most of the pension money it managed.
The scandal broke out in February and Asakawa apologised to the clients saying that he was confident of recovering the money.
AIJ is accused of hiding the losses for nearly a decade and Asakawa earlier testified before Japanese lawmakers of the cover -up.
AIJ who handled funds from over 100 companies was stripped of its registration in March.
The losses mainly occurred at the derivatives trading in futures and options in shares tied to the Nikkei Stock Exchange and in government bonds, the Wall Street Journal reported.
If convicted, Asakawa and the executives could face prison term for the financial irregularities.
AIJ is the latest incident of fraud that hit the Japanese corporate sector. Earlier, Camera maker Olympus had hidden $1.7bn losses from market regulators for nearly 20 years which came to light only in late 2011.
Pension funds in Japan are under pressure to provide high returns despite the subdued growth of the Japanese economy which grew only one percent over the past decade.
The Bank of Japan has kept deposit rates to nil since 1995, but pension funds are required to provide 5.5 percent return.