The Nigerian Army killed approximately 600 people after a Boko Haram attack on a military barracks, according to Amnesty International.
The military raid in the city of Maiduguri saw escaped detainees captured and killed, eyewitnesses told Amnesty.
Boko Haram militants allegedly freed hundreds of Islamists before many were extrajudicially killed by Nigerian soldiers.
"I saw the soldiers asking the people to lie on the ground," a witness told Amnesty.
"There was a small argument between the soldiers and the civilian JTF [a self-defence group]. The soldiers made some calls and a few minutes later they started shooting the people on the ground. I counted 198 people killed at that checkpoint."
The human rights group said that half of the 1,500 people killed in the conflict between the Nigerian military and Islamist insurgents this year have been civilians.
"More than 1,500 deaths in three months indicate an alarming deterioration in the situation. The international community cannot continue to look the other way in the face of extrajudicial executions, attacks on civilians and other crimes under international law being committed on a mass scale," Amnesty's briefing said.
"Civilians are paying a heavy price as the cycle of violations and reprisals gather momentum," it added.
The insurgents want to turn Nigeria into a strict sharia state and their campaign of terror has been waged mainly in the northeast of Nigeria. Three hundred people have been killed in February alone.
The state of Yobe has been placed under emergency rule along with two other predominantly Muslim northeastern states - Borno and Adamawa.
A military campaign to combat the militants has failed.
Boko Haram, which means "Western education is forbidden" is designated a terrorist organisation by the United States and has targeted many schools in the past year.