Four million people in England and Wales will be in need of affordable housing even if the UK hits full employment, according to a group of local councils.

The Local Government Association (LGA), which represents over 370 councils, claimed 3.98 million people of working age will still need affordable — meaning sub-market — housing options by 2024. That is 39% higher than in 2011.

If full employment is not reached, and those currently unskilled are not helped into higher paying work through training, as many as 5.4 million people will need affordable housing, said the LGA. The claims are based on an analysis of official census data from 2011 by the Learning and Work Institute, a thinktank, for the LGA.

"Bold new action is needed to solve our housing crisis following the vote to leave the European Union," said Cllr Peter Box, the LGA's housing spokesman. "A renaissance in house building by councils must be at the heart of this. The private sector clearly has an important role to play but the reality is that it cannot build the homes we need on its own, and will likely be further restricted by uncertainties in the months and years ahead."

Councils are calling for the government to lift the cap on their borrowing so they can raise money to invest in building more homes; to keep all of the revenue raised through the sale of council-owned homes under the Right to Buy scheme; and to use these and other funds to pay for the construction of more housing by private developers, commissioned by local authorities.

Currently, Right to Buy receipts will go to a central pot at the Treasury and used to fund homeownership policies, such as Help to Buy. Councils are also restricted on how much they can borrow to keep a lid on public debt and prevent the Treasury having to bail out any authorities which get into financial trouble.

"More council housing has been built since 2010 than the previous 13 years – but as part of making Britain a country that works for everyone we need more housebuilding," said a spokeswoman for the Department for Communities and Local Government (DCLG).

"Local authorities have £2.5bn in housing reserves and borrowing headroom of £3.4bn, so they should be able to build the homes their communities need. Furthermore, we've set out the most ambitious vision for housing in a generation, including investing £8bn to build 400,000 more affordable homes."

Government figures show that there were 142,390 completions in England and Wales in 2015, a 20% annual rise, but well below the estimated 250,000 new homes a year needed just to meet demand. Ministers are targeting a million new homes by 2020.

There are concerns that the vote to leave the EU in the 23 June referendum will curtail housebuilding by weakening demand for property. Private developers may choose to cut back on work or invest less in the future, hurting efforts to solve the housing crisis.

A report commissioned by the National Housing Federation (NHF), which represents housing associations, identified £7bn of government funding that could be redirected to them to boost housebuilding. Public money is available to support the construction of specific types of home. For example, there is £2.3bn dedicated to "starter homes", discounted properties for first-time buyers.

But the NHF argues that by using the money as a general housebuilding fund instead, and allowing housing associations to decide exactly what needs to be built and where, housebuilding levels can be maintained even as private construction firms cut-back on work amid the Brexit uncertainty.