Seven political prisoners were freed by the South Sudanese government on Wednesday.
The release has partially met the rebels' demand to free 11 detainees who are accused of an attempted coup d'état against President Salva Kiir.
"We don't feel bitter, we don't feel the president is our enemy," one of the freed men, South Sudan's former justice minister John Luk Jok, told Reuters in Kenya's capital Nairobi.
The four remaining detainees could be released after unspecified "legal clarifications," the officials said.
Former prime minister and rebel leader Riek Machar said that the accusations against him and the rebels are "baseless", the BBC reported.
"I personally have not planned a coup and my colleagues who are under detention have not planned a coup with me so I see no reason why we would face such charges," Machar said.
Machar added that he hoped mediators in the crisis would ensure the release of his remaining four allies.
"The government should differentiate between the alleged coup and the current rebellion," Machar continued.
The conflict in South Sudan started in December 2013 after President Salva Kiir, who is an ethnic Dinka, accused his former vice-president, Riek Machar, an ethnic Nuer, of an attempted coup.
Machar denied the allegations and in turn accused the president of planning a violent purge.
Peace talks started in the Ethiopian capital of Addis Ababa four weeks ago. The warring sides agreed on a ceasefire but they have both warned that this is only a temporary measure; further talks will be necessary to achieve peace.
The talks plunged into deadlock as the government refused to accommodate the rebels' requests.
Besides the release of the 11 political prisoners, the rebels urged Ugandan troops to stop supporting the government.
Over 1,000 people have been killed and around 500,000 have been displaced in the conflict.
The UN said that government soldiers and rebels had committed atrocities. NGO Human Rights Watch said that crimes against humanity had been committed by both the rebels and the government. The crimes include mass killings, extrajudicial killings, arbitrary detention, sexual violence, looting of property and use of children in conflict.
The fighting has also severely harmed the future revenues of the country's oil industry.