Sports Direct shareholders have dealt a blow to billionaire sports retailer Mike Ashley by rejecting the re-election of the company's chairman.

A stormy annual meeting Meeting saw 53% of independent investors vote against Keith Hellawell amid major criticism over the way the company is run.

But the chairman's position was saved after he was backed by Ashley, who founded the business and still owns 55% of the company.

City investors have tried for three years to remove Hellawell in favour of a more experienced head.

Hellawell said: "I take this clear message from our independent shareholders seriously, and I will do my best to address their concerns and earn their confidence over the next year."

"I have confirmed today that should I not receive the support of a majority of our independent shareholders at next year's annual meeting, I will step down at that time with immediate effect."

At Wednesday's meeting Standard Life head of stewardship and governance Euan Stirling said he would like to see Ashley in a new role "supported, as well as challenged, by a group of talented and experienced executives".

The investor said it would vote against all of the firm's non-executive directors and its director's pay report.

Stirling was speaking at the meeting held in the firm's Shirebrook warehouse in Derbyshire, in the latest salvo highlighting the growing frustrations between shareholders in the scandal-hit firm and Ashley.

A growing number of institutional investors want a clear-out of the existing board, and the introduction of new members with enough experience to challenge Ashley. Shareholders want to oust chairman Keith Hellawell, but the board has stood by him despite his offer to resign over the weekend.

Sports Direct
Keith Hellawell chairman of sportwear retailer Sports Direct arrives at the company's AGM after his offer to resign was rejected by the board, at the company's headquarters in Shirebrook Darren Staples/ Reuters

L&G Investments director of corporate governance, Sacha Sadan said: "At absolute minimum, we believe the current chairman should step down immediately and an external, independent appointment made to oversee management and protect the interests of all stakeholders – including employees, suppliers, and shareholders."

Royal London Asset Management corporate governance manager Ashley Hamilton Claxton said he was "astonished at the lack of improvement in corporate governance matters at Sports Direct".

Hamilton Claxton added: "We agree with other investors that the chairman should resign, with a new chairman to be appointed within the next six months. We are voting against the non-executive directors yet again, to express our frustration that governance has not improved.

Shares slumped as much as 11% today, and have fallen by almost 60% over the past year. Shareholders and other observers at the meeting said changes outlined by the business did not go far enough to overhaul working practices and corporate governance.

Sports Direct
Employees of sportswear retailer Sports Direct walk past the company's headquarters on the day of their AGM in Shirebrook Darren Staples/ Reuters

Among a range of changes the retailer said it plans to put a workers' representative on its board, cut back on zero hours contracts and end the "culture of fear" at its Derbyshire which employs 5,000 staff.

But when a Unite union member said the firm's pegging back of zero hours contracts was lacklustre, Ashley accused him of "showboating."

He added it was partly "Unite's fault that we are in this situation because we can't talk to you in an open manner".

At one point Ashley offered to leave the business himself. He said: "In the end if I keep failing and failing and keep failing, it'll be best for everybody if I go. But give me some time to see if I can fix the problems first."

The retailer agreed to carry out changes after a report by law firm PRC found serious shortcomings at its key warehouse after MPs said conditions there were like a Victorian workhouse this summer. PRC has long-standing links with Sports Direct, and shareholders say this means in cannot give independent advice.

Investigations by the Guardian newspaper last year found that staff at its Derbyshire warehouse were harangued for not working fast enough, while long staff searches after work sometimes meantworkers earned below the minimum wage. There were also allegations that female staff were promised permanent contracts in exchange for sexual favours.

The Financial Times also reported in August that a company owned by Ashley's brother is involved in distributing products sold by Sports Direct, while in January his daughter's 26-year boyfriend was put in charge of the group's vast property portfolio.

An eye-catching moment in the day came in a tour of the warehouse after the shareholder meeting. Ashley had to empty his pockets to go through airport-style security entrance and pulled out a large wad of £50 notes, joking he had "just been to the casino".