Violent skirmishes at a prison in the town of Taiz led to the escape of over 1,000 prisoners.
This is the biggest incident so far in a number of prison breaks that have freed Yemeni militants in recent years in a country divided by civil war.
"Groups of al Qaeda supporters... today attacked the central prison in the city of Taiz and more than 1,200 of the dangerous prisoners escaped," state news agency Saba quoted a security official as saying.
Another local official told Reuters some of the escapees were "suspected of belonging to al Qaeda" who then scattered, leaving the area after clashes between warring factions in the city.
The security official said army forces linked to Yemen's former president Ali Abdullah Saleh allowed the prisoners to escape as the militiamen, dubbed "popular committees" by their supporters, pressed forward.
"Heavy fighting took place near the central prison and the popular committees approached and seized control of the area, but Saleh's forces opened the prison doors," the official said.
Shi'ite Muslim Houthi fighters advanced on Taiz in March, in a push from their base in the capital Sanaa that drew military intervention from a Saudi Arabian-led coalition.
However, three months of air strikes have had little effect on the group and units in Yemen's army that are loyal to Saleh, the country's ex-president.
Another group of al Qaeda militants escaped from a prison in the eastern city of Mukalla in April after army forces suddenly quit the city.
The conflict in Yemen has killed over 1,000 civilians and displaced more than a million since March. Houthi rebels forced President Abdrabbuh Mansour Hadi to flee to Aden and then to Saudi Arabia.
UN Under-Secretary-General for Humanitarian Affairs Stephen O'Brien warned of a "looming humanitarian catastrophe," with at least 11.7 million people affected by the conflict. He requested $1.6bn in emergency aid, according to an AP report.
"People across the country are struggling to feed their families. Basic services are collapsing in all regions. Millions of families no longer have access to clean water, proper sanitation or basic health care. Deadly diseases such as dengue and malaria have broken out, and supplies for acute trauma care are running dangerously low," he said in Geneva.