Cloudflare, the web optimisation and security platform used by over five million websites, has confirmed a massive coding error resulted in the leaking of sensitive data of its customers for months – including passwords, internet cookies and private messages.
The bug impacted big-name brands including 1Password, Uber and dating site OKCupid, and has been blamed on a "memory leakage." Cloudflare said in a technical summary of the flaw that around 1 in every 3,300,000 HTTP requests sent through its system were at risk.
What to do if your Cloudflare was affected
For any individual, or organisation, concerned about the bug there are some simple steps that can now be taken to stay protected. According to security expert Ryan Lackey, who published helpful advice after news of the bug broke, this includes enforced password changes.
"From an individual perspective, this is straightforward – the most effective mitigation is to change your passwords," he said. "While this might not be necessary (it is unlikely your passwords were exposed in this incident), it will absolutely improve your security."
For any websites using Cloudflare's services he said: "While it may be disruptive, you should seriously consider forcing a password change for all of your users. Be careful not to "scare" them, but it may be prudent to push such a change." He added: "I believe this is a serious security issue and should be evaluated by any major service using Cloudflare."
Alongside the noted brands, there is now a running list of potentially impacted domains being published to GitHub. So far, they include patreon.com, uber.com, tfl.gov.uk, medium.com and many more. The count is currently sitting at over four million (4,287,625), however these are not yet confirmed victims.
According to John Graham-Cumming, Cloudflare's chief technology officer (CTO), the number of impacted websites is closer to 3,400. However, a true figure remains unclear at this time. "We identified 3,438 unique domains," Graham-Cumming said.
The most vulnerable period for users was between 13 February and 18 February this year, however admitted the issues may have been around for up to five months, since September 2016.
The wide-ranging issue was uncovered by Google Project Zero researcher Tavis Ormandy, a well-known software bug hunter in security circles. In a detailed run-down of his discovery, he exposed the shocking scope of the coding snafu, reluctantly labelling it "CloudBleed".
What was leaked by the Cloudflare bug?
"I didn't realise how much of the internet was sitting behind [Cloudflare] until this incident. The examples we're finding are so bad," Ormandy wrote on 19 February.
"I'm finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings," he said. "We're talking full HTTPS requests, client IP addresses, full responses, cookies, passwords, keys, data, everything."
Graham-Cumming said his firm, which worked overtime to release a fix this week, has not discovered "any evidence of malicious exploits" as a result of the bug, however confirmed hit impacted Http cookies, authentication tokens and unspecified "other sensitive data."
"One obvious piece of information that had leaked was a private key used to secure connections between Cloudflare machines," he said, in relation to a secretive crypto key that secured the connection between the firm's own internal network.
What's worse, a large amount of the leaked data was reportedly cached – or saved – by public search engines including Google, Bing and Yahoo. Ormandy said his organisations is trying to clean this up, but described any efforts to do so as "a bandaid."
"Cloudflare customers are going to need to decide if they need to rotate secrets and notify their users based on the facts we know," he noted. The researcher also said he did not know if the bug had been exploited but that it's likely other web crawlers were able to collect the data.
In his notes, Ormandy appeared overall pleased with Cloudflare's response to this disclosure, but said at one point the firm "pointed out" its bug bounty programme as one method of informing them of vital issues. The problem with this, he said, is that the top prize is a t-shirt.
He also described a number of delays on Cloudflare's side, however this is likely explained by the sheer scope of the problem the service was facing. When presented with a copy of the draft notification, he said it "severely downplays the risk to customers."
In his blog post, Graham-Cumming responded: "Our natural inclination was to get news of the bug out as quickly as possible, but we felt we had a duty of care to ensure that search engine caches were scrubbed before a public announcement."
The websites hit by the bug were not notified prior to Ormandy's disclosure which, as expected, has caused many to face instant complaints from concerned customers." Now, some internet-based organisations are being forced to address the issues head-on.
Jeffrey Goldberg, a security expert with 1Password, issued a statement on the incident. He wrote: "No 1Password data is put at any risk through the bug reported about CloudFlare. The security of your 1Password data remains safe and solid."
Meanwhile, on Twitter, encrypted email provider, ProtonMail, said: "We do NOT use Cloudflare, #Cloudbleed does not impact your ProtonMail credentials." Others suggested such bugs are likely to have been exploited, especially by well-funded hackers at the NSA.