Families in Britain risk being separated as young people face "exile" from where they grew up because of the UK's high house prices, the communities secretary will warn.

Greg Clark, who is expected to speak at a Local Government Association (LGA) conference on 2 July, will urge councils to tackle the affordable housing problem by taking power from Westminster.

"For centuries, to be exiled – to be sent away – was considered to be an extreme penalty, reserved for the most serious of offences against the community," the top Tory will say, according to the Press Association.

"Yet in many parts of our country, it has become normal for young people to leave, though not out of choice. This might be to find work, but more and more, it is to find a home that they can afford."

Clark will add: "If we want to maintain the chain of community – and a place for the next generation – then we must make sure we have the homes to welcome them to. The responsibility lies with us – national and local leaders alike."

Harriet Harman, Labour's acting leader, will also address the conference and argue the government's plan to extend the right-to-buy scheme to housing association residents will only exasperate the UK's housing shortage.

"Ultimately, we'll see what the government brings forward, but the test for any housing policy must be whether it eases rather than deepens the housing crisis," she will say.

"Proposals which don't address the key problem – the chronic shortage of homes – will see the dream of home ownership drift further and further out of reach."

Harman will issue the warning after new figures suggest that UK house prices fell in June, in an unexpected move.

Building society Nationwide said prices fell by 0.2% in June to an average of £195,055 (€274,893, $304,325), with an annual increase of 3.3%, down from 4.6%.

The firm explained the drop means yearly price growth has fallen to its lowest level in two years. However, the data showed the rise is well above the latest Consumer Price Index (CPI) inflation rate of 0.1%, recorded in May.

Robert Gardner, Nationwide's chief economist, said: "House price growth continues to outpace earnings, but the gap is closing, helped by a pick-up in annual wage growth."