The Passport Office will be brought into the Home Office after the agency's "summer of chaos".
The Home Secretary Theresa May announced the move, which will mean the organisation is directly accountable to ministers, after June saw a backlog of more than 500,000 passport applications.
But the Home Office said the number of outstanding applications have plummeted to around 80,000.
"As the events of the summer showed, it is essential that the Passport Office is run as efficiently as possible and is as accountable as possible," May said.
"I also know that its hard-working staff are committed to delivering a high-quality service to the public.
"I believe these changes will put them in a stronger position to do so."
The Home Office said the agency's current chief executive, Paul Pugh, will remain in his post until his successor has been appointed.
The news comes after the cross-party Home Affairs Select Committee called on May to revoke the Passport Office's agency status.
The group of MPs also said the organisation should be brought back under direct control of ministers, but should still retain a director general.
The report, which investigated processing delays of passport applications, also said the government's contingency measures to respond to the backlog of cases were "too little, too late".
"This has been a summer of chaos at the Passport Office," said Keith Vaz MP, chairman of the group.
"At its peak, the backlog of passport applications reached half a million, with British citizens unable to go on holiday and sick children unable to return to Britain.
"The state should not be exploiting its own citizens by making a profit on what is a basic right.
"A British passport, an essential document for travel by British citizens, had become the subject of emergency statements and crisis management."