Department of Health (DoH) staff take twice as many sick days as the national average, according to new figures.
More than 50,000 working days have been lost to sickness over the last three years at the Westminster department led by health secretary Jeremy Hunt.
Civil servants at the DoH took an average eight sick days each year between 2014 and 2016. The national average was four, according to the Office for National Statistics.
Insiders at the London office put the lost work days down to stress and staff cuts, reported the Mirror, who lodged a Freedom of Information request.
The news comes after Hunt was forced to apologise to patients in England who were told earlier this month that around 50,000 non-urgent National Health Service operations would be suspended.
Critics say the underfunding in the NHS means the current winter crisis threatens to stretch the UK's health service to breaking point.
Liberal Democrat health spokeswoman Judith Jolly told the paper: "It's little wonder the NHS is in crisis when the Department of Health can't even look after its own staff."
In total DoH staff called in sick 50,037 times over the three-year period, which the Mirror said cost £107.85 a time, putting the Department's bill at £5.4m (€6.1m).
The department also revealed the total number of staff at the Westminster office had been cut from around 2,000 to 1,650.
A DoH source told the newspaper: "Many staff felt they were bullied, or at least strongly encouraged, to accept voluntary severance. The alternative was to apply for one of the remaining posts, but many staff were informally told that they would not be successful and so should accept early severance. This was very unsettling with management behaving very aggressively."
But the DoH said days lost to staff sickness have been coming down since 2014.
A department spokesman added: "We take the health of staff very seriously and are committed to fostering a culture where they can take time off if needed."