A background on the vulnerabilities: an easy-to-read description of what they are and the immediate and longer-term effects that Meltdown and Spectre will have on the services and software that we rely on. And why having a really secure platform for the 1st time ever (i.e. PoW Blockchain) is such a big deal. — TJACK
Meltdownattack.com has a full list of vendor advisories.
The academic paper on Meltdown is here (PDF).
The paper for Spectre can be found at this link (PDF).
Additionally, Google has published a highly technical analysis of both attacks.
Cyberus Technology has their own blog post about the threats.
Google’s Project Zero (GPZ) is a think tank of leading edge security researchers who have established a track record of ground breaking research. Yesterday they announced a set of flaws in CPU architectures that create two kinds of vulnerabilities.
It is early in the year, but this may be the most important and impactful security vulnerability in 2018. This affects any software running on Intel chips, no matter the operating system or vendor. This affects every Intel processor since 1995 that implements out-of-order execution, except Itanium, and the Atom before 2013.
The vulnerabilities were discovered by collaborating researchers at University of Pennsylvania, University of Maryland, Graz University of Technology, Cyberus Technology, Rambus Cryptography Research Division, University of Adelaide and Data61 along with researchers at GPZ.
The flaws were first reported confidentially by researchers to CPU makers Intel, AMD and ARM on June 1st, 2017. Disclosure was under embargo until next week, but public speculation on kernel patches that fix this issue lead to early disclosure starting on January 1st, 2018. Most information was finally disclosed by the researchers involved yesterday, January 3rd. Research associated with the security flaws was published on the Google Project Zero blog.
They have named the flaws Spectre and Meltdown. You can find the academic paper on Spectre on this page (PDF) and the paper on Meltdown on this page (also PDF). I am providing mirrored copies of both PDF papers on our site because at the time of writing, both source websites were down, probably due to excess traffic. Spectre Mirror and Meltdown Mirror.
Both of these vulnerabilities stem from performance optimizations in CPUs. The security fixes may have a performance impact. Some news sources are claiming up to 30% performance impact, while more authoritative sources indicate this number is exaggerated. Intel’s official statement says “Contrary to some reports, any performance impacts are workload-dependent, and, for the average computer user, should not be significant and will be mitigated over time.”
Intel has been accused of downplaying the seriousness of the vulnerability, both in terms of how badly Intel CPUs are affected and the negative effects of these vulnerabilities.
The Meltdown Vulnerability
Meltdown is the first of the two vulnerabilities that GPZ disclosed. It exploits a CPU performance optimization known as ‘out-of-order execution’ to read arbitrary kernel memory locations. The attack is independent of operating system and does not rely on any software vulnerabilities. In other words, it is a vulnerability in chip hardware that is exploitable on any system, no matter what operating system it is running, no matter whether the software on the system has a vulnerability or not.
Meltdown allows an attacker to read memory that they should not have access to in other processes, other virtual machines on the same system and across various other permission boundaries. This affects a huge number of cloud service providers and personal computer and device users.
There is a mechanism that operating system developers can use to protect against this attack. You will be seeing a large number of operating system patches released and deployed during the coming days to secure systems against ‘Meltdown’.
Spectre is a vulnerability that exploits another performance enhancement in modern CPUs, known as ‘speculative execution’. Hence the name, Spectre.
Modern processors use speculative execution to improve performance. The mechanism allows processors to guess which code will execute and to go ahead and execute that code while waiting for a memory location to be read. Once the read operation is complete, if the processor guessed right, it keeps the results of the computation. If it guessed wrong, it discards the computation results. This improves performance.
Specter attacks will get a victim processor to perform operations that would not occur during correct program execution. These operations leak confidential information.
This attack violates many security models including process separation, containerization and others.
Of particular concern to those of us in the website security community is the following passage from the research paper:
According to the research, makeshift processor-specific countermeasures for Spectre are possible, but a long term fix will require a fundamental improvement to CPU architectures.
Fixing Meltdown and Spectre and Their Impact
Both of these vulnerabilities are hardware level vulnerabilities that exist because of a flaw in CPU architecture. They are very serious vulnerabilities because they are operating system and software independent. The long term fix for both of these issues will require that CPU makers change the way their chips work, which means redesigning and releasing new chips.
That is not feasible for existing chips and in order to fix this issue for existing CPUs, operating system vendors are going to have to release fixes. That means that you will see security fixes for the following OS’s released in the coming days: Windows, OS X, Linux and probably Android. When you see a fix available for your PC or device, apply it as soon as is practical because it will probably contain a fix for these issues.
Because the vulnerabilities are in algorithms in CPUs that improve performance, the fixes may have a performance impact. Chip vendors like intel are playing down the impact, while some news media is playing it up. I would suggest taking a wait-and-see approach, because ultimately, benchmarks of the new operating system patches are the only accurate way to reliably determine if there will be any performance impact and if so, of what magnitude.
If you are a hosting provider that uses cloud services for your customers, expect your cloud provider to reboot systems during the coming days and have your operations team on standby to ensure that everything cycles back normally. And of course, keep your customers apprised of the situation.
If you use hosting services, like WordPress hosting, you should be aware that your hosting or cloud provider may need to reboot systems over the coming days as they apply patches for Meltdown and Spectre. Unless you have a 100% fully managed WordPress site, it may be up to you to check that certain services for your site came back up after the reboot. Keep a close eye on bulletins from your host over the coming hours and days and ensure you check your site and systems as soon as they come back up after any reboot or down time.
So far we are seeing notifications of maintenance or reboots for the following hosts and cloud providers:
- Amazon is reporting that they have patched most of the underlying operating systems for AWS and will complete the rest soon. They are saying that customers are responsible for updating the operating systems of their instances and have provided information to do that.
- Linode are saying that they will need to do a “fleet-wide reboot” to protect against these issues. Keep an eye on their blog for updates.
- DigitalOcean are reporting that they also may need to reboot droplets and are monitoring the situation.
- Vultr are reporting a reboot may be needed.
If your cloud provider is not listed above, keep an eye on their blog and Twitter account for updates.
At this time we are not seeing updates from major hosting providers to their customers. The operational impact of these updates will probably flow upwards in architectural terms. In other words, CPU vendors were first notified and responded, then operating system vendors, then cloud providers like AWS and Linode and next we will see service providers respond.
These would include hosting companies, DNS service providers, storage providers, backup providers and other providers of services and applications. In many cases, for service providers, there may be no operational impact if they have built redundancy into their application and are able to perform partial fleet reboots without disrupting service.
Chrome and Firefox Affected
“Our internal experiments confirm that it is possible to use similar techniques from Web content to read private information between different origins. The full extent of this class of attack is still under investigation and we are working with security researchers and other browser vendors to fully understand the threat and fixes.”
They have already implemented and released fixes to mitigate the issue but as the above quote indicates, more fixes are probably forthcoming. Firefox users should update to Firefox 57.
Google Chrome is also affected, and according to Google, Chrome will receive a fix in Chrome 64 which will be released on January 23rd. Chrome also provides options for users to enable that will help reduce the effectiveness of these attacks:
Performance and Business Impact
Systems that receive these security updates may experience a performance impact though it is currently difficult to say to what degree. If you are in an operational role, it is important that you evaluate system performance once you have applied OS patches to determine if it will impact your customers.
At an executive level, consider that in a worst case scenario, system performance may degrade 30% across the board. If you are running your systems at 90% capacity and your financial margins are thin, you may find yourself in a crisis situation which results in raising prices or making other changes to adapt to CPUs no longer delivering the performance to which your business model has become accustomed.
As a customer or end-user, I would reserve judgement on any performance impact until benchmarks are released. If someone tells me that sunspot activity is slowing down my workstation, I tend to notice slowness on my workstation. It is difficult to quantify performance changes until someone does the work to produce accurate and precise benchmarks.
Impact On Hardware Design
Meltdown and Spectre are a new class of vulnerability, both in their sophistication and impact. They use timing attacks to exploit flaws in the underlying hardware we use for a majority of our applications today, both in the cloud and on desktops and devices.
A complete fix for Meltdown and Spectre is going to require a CPU replacement. As CERT says, the solution is to “Replace CPU Hardware”.
It is inevitable that other hardware vulnerabilities like these with wide impact that require hardware changes will emerge in the coming years. We can’t buy new hardware every time this happens. So a long term fix may require that we invent a way to dynamically patch the hardware that our software relies on.
This Was Disclosed Early
These vulnerabilities were under embargo until next week. On January 1st, speculation started on a blog titled Python Sweetness, about a major vulnerability that was hardware based and involved memory manipulation. On January 2nd, The Register published a story with some details.
Yesterday on January 3rd, GPZ published full details on their blog, resulting in a huge amount of press and official statements emerging.
An extract from Intel’s official statement makes it clear the vulnerabilities were disclosed early:
“Intel is committed to the industry best practice of responsible disclosure of potential security issues, which is why Intel and other vendors had planned to disclose this issue next week when more software and firmware updates will be available. However, Intel is making this statement today because of the current inaccurate media reports.”
This story is now major news with plenty of coverage and commentary. The authoritative sources for this story are the GPZ blog, the research papers, statements from chip makers Intel, AMD and ARM and the blog posts from cloud providers like AWS and Linode. Check your vendor blogs and vendor Twitter accounts for updates on security and service interruptions.
If you have any additional reliable and accurate resources, research or commentary related to this, I would appreciate if you would leave them in the comments.
Mark Maunder – Defiant Founder/CEO.
- Google Project Zero Announcement
- Spectre Paper
- Meltdown Paper
- Intel Official Statement
- ARM Official Statement
- AMD Official Statement
- Amazon AWS Response
- Linode Response
- DigitalOcean Response
- Early speculation on Monday which lead to early announcement
- The Register coverage on Tuesday breaking the story
- Mozilla commentary on Firefox being vulnerable
- Chrome fix release date
- Chrome advice to mitigate these attacks
- CERT official vulnerability note
- Sophos technical analysis and commentary
- BleepingComputer coverage on the vulnerability of Chrome and Firefox
By Mark Maunder, CEO, Defiant
[04 JAN 2018]
Cybersecurity agency’s suggested fix for Intel bug comes with a hefty price tag
The tech world continues to come to grips with Wednesday’s revelation of very serious vulnerabilities associated with central processing units (CPUs) that affect, well, just about everyone with a computer.
Two flaws have been identified by researchers: Meltdown, which is believed to affect most Intel CPUs produced over the last 20-plus years, and Spectre, which affects processors produced by multiple companies, including Intel, AMD, and ARM.
Now, the Computer Emergency Response Team (CERT), the government-sponsored cybersecurity organization based out of Carnegie Mellon University, has issued its report on how to fix computers affected by the widespread bugs… and it’s not cheap.
The underlying vulnerability is primarily caused by CPU implementation optimization choices. Fully removing the vulnerability requires replacing vulnerable CPU hardware.
TL;DR, if you’re really worried about it, it’s probably safest to just buy a new machine that doesn’t have one of the vulnerable processors tucked away inside.
As our Jack Morse pointed out yesterday, the Meltdown vulnerability could put you at risk:
Not only is CERT’s suggestion an expensive proposition, it suggests that if you really, really want to make sure you’re secure, the patches companies are rolling out to users won’t fully fix everything. They’re not worthless, but they’re also not comprehensive and are even likely to slow down computers with older processors.
For the most part, most private users should be okay with the patches and adjusting browser settings. There are vulnerabilities, but it’s businesses and government organizations that are already most susceptible to hacking attempts that will want to make sure everything’s on lockdown.
[04 JAN 2018]
Two New Hardware Bugs Affect Most Devices, Private Keys Vulnerable
Researchers have published a report on two hardware bugs that allow programs to steal sensitive data on affected devices, which is “most” devices worldwide.
For crypto users, these bugs are a direct threat to the security of their private keys, making the need for secure hardware storage of crypto funds even more pressing.
How do the bugs work?
The two bugs, known as Meltdown and Spectre, exploit security vulnerabilities in Intel, AMD, and ARM processors in any device, including PCs, laptops, tablets, and smartphones.
Meltdown affects all devices with Intel chips, which are estimated to be in 90% of all computers (desktop and laptop combined), the BBC reported.
Spectre potentially has an even wider reach, affecting Intel, ARM, and AMD chips in any kind of device. Meltdown and Spectre also work in the cloud.
The BBC also reported that the tech industry kept the threat a secret for up to six months via non-disclosure agreements, but now fears are mounting that public awareness could lead to real-life exploits.
Protecting your funds
Bitcoin core developer Jonas Schnelli referred to the newly reported security flaws in terms of how they affect Bitcoin users, laying out three steps to secure cryptocurrency holdings:
“Using a (hardware) wallet is now more important than ever.”
In October 2017, the Ledger Nano S hardware wallet was number eight in the list of top ten best selling items on Amazon’s Computers and Accessories section. Today, Jan.4, the Ledger Nano S is number one.
[05 JAN 2018]
Scary Chip Flaws Raise Spectre of Meltdown
Apple, Google, Microsoft and other tech giants have released updates for a pair of serious security flaws present in most modern computers, smartphones, tablets and mobile devices. Here’s a brief rundown on the threat and what you can do to protect your devices.
At issue are two different vulnerabilities, dubbed “Meltdown” and “Spectre,” that were independently discovered and reported by security researchers at Cyberus Technology, Google, and the Graz University of Technology. The details behind these bugs are extraordinarily technical, but a Web site established to help explain the vulnerabilities sums them up well enough:
“These hardware bugs allow programs to steal data which is currently processed on the computer. While programs are typically not permitted to read data from other programs, a malicious program can exploit Meltdown and Spectre to get hold of secrets stored in the memory of other running programs. This might include your passwords stored in a password manager or browser, your personal photos, emails, instant messages and even business-critical documents.”
“Meltdown and Spectre work on personal computers, mobile devices, and in the cloud. Depending on the cloud provider’s infrastructure, it might be possible to steal data from other customers.”
The Meltdown bug affects every Intel processor shipped since 1995 (with the exception of Intel Itanium and Intel Atom before 2013), although researchers said the flaw could impact other chip makers. Spectre is a far more wide-ranging and troublesome flaw, impacting desktops, laptops, cloud servers and smartphones from a variety of vendors. However, according to Google researchers, Spectre also is considerably more difficult to exploit.
Microsoft this week released emergency updates to address Meltdown and Spectre in its various Windows operating systems. But the software giant reports that the updates aren’t playing nice with many antivirus products; the fix apparently is causing the dreaded “blue screen of death” (BSOD) for some antivirus users. In response, Microsoft has asked antivirus vendors who have updated their products to avoid the BSOD crash issue to install a special key in the Windows registry. That way, Windows Update can tell whether it’s safe to download and install the patch.
But not all antivirus products have been able to do this yet, which means many Windows users likely will not be able to download this patch immediately. If you run Windows Update and it does not list a patch made available on Jan 3, 2018, it’s likely your antivirus software is not yet compatible with this patch.
Google has issued updates to address the vulnerabilities on devices powered by its Android operating system.
Apple has said that all iOS and Mac systems are vulnerable to Meltdown and Spectre, and that it has already released “mitigations” in iOS 11.2, macOS 10.13.2, and tvOS 11.2 to help defend against Meltdown. The Apple Watch is not impacted.
Patches to address this flaw in Linux systems were released last month.
Many readers appear concerned about the potential performance impact that applying these fixes may have on their devices, but my sense is that most of these concerns are probably overblown for regular end users. Forgoing security fixes over possible performance concerns doesn’t seem like a great idea considering the seriousness of these bugs. What’s more, the good folks at benchmarking site Tom’s Hardware say their preliminary tests indicate that there is “little to no performance regression in most desktop workloads” as a result of applying available fixes.
Meltdownattack.com has a full list of vendor advisories.
The academic paper on Meltdown is here (PDF); the paper for Spectre can be found at this link (PDF). Additionally, Google has published a highly technical analysis of both attacks. Cyberus Technology has their own blog post about the threats.
[06 JAN 2018]
[08 JAN 2018]
Meltdown Patch Performance Loss – Cloud Service Provider
Attention Fortnite community,
We wanted to provide a bit more context for the most recent login issues and service instability. All of our cloud services are affected by updates required to mitigate the Meltdown vulnerability. We heavily rely on cloud services to run our back-end and we may experience further service issues due to ongoing updates.
Here is a link to an article which describes the issue in depth.
The following chart shows the significant impact on CPU usage of one of our back-end services after a host was patched to address the Meltdown vulnerability:
Unexpected issues may occur with our services over the next week as the cloud services we use are updated. We are working with our cloud service providers to prevent further issues and will do everything we can to mitigate and resolve any issues that arise as quickly as possible. Thank you all for understanding. Follow our twitter @FortniteGame for any future updates regarding this issue.
[18 JAN 2018]
The Hidden Toll of Fixing Meltdown and Spectre
In the early days of 2018, the engineering team at the mobile services company Branch noticed slowdowns and errors with its Amazon Web Services cloud servers. An unexpected round of AWS server reboots in December had already struck Ian Chan, Branch’s director of engineering, as odd. But the server slowdowns a few weeks later presented a more pressing concern.
“We had six engineers crammed in a small war room all staring at charts, deploy logs, revision histories, and latency graphs looking for the cause,” Chan says. “We spent a few days eliminating possibilities one after another, but were unable to find a root cause. We were seemingly chasing a non-existent bug in our system.”
The team kept Branch’s services operational by reworking some of their architecture, and purchasing more server capacity from AWS to stabilize the workloads. “At some point someone floated the hypothesis that it was an underlying performance issue due to the Spectre and Meltdown patches being applied by AWS,” Chan says. “The mystery reboots from just a few weeks earlier suddenly made sense.”
Branch’s struggles turn out not to be unique. Last week’s public revelation that most mainstream computing processors could be manipulated to leak data between programs led to a frenzy of patches and confusion. Even before Meltdown and Spectre were officially revealed, there had been hints that the fix could significantly degrade performance. And while system administrators, internet infrastructure providers, and cybersecurity managers now largely agree they’ve dodged the early worst-case scenarios, they’ve taken a tangible toll.
Taking Your Medicine
The Meltdown and Spectre vulnerabilities exist because for years chipmakers have taken steps to prioritize performance and speed that, as a side effect, turned out to impact security. By reining in some of these data fast tracks, the fixes slow down certain types of operations, particularly for programs that require a lot of requests to the kernel, an operating system’s most fundamental and secretive inner sanctum.
‘I remember first looking at it and thinking ‘oh, shit,’.’ — John Michener, Casaba Security
Early testing and benchmarking of the Meltdown and Spectre fixes indicated that their impact could be severe. Even just the complexity of applying and managing the patches—particularly for Spectre, which is more a class of vulnerability than a specific bug—has created a real strain on the industry. Lots of vulnerabilities require large-scale patches. But Meltdown and Spectre are unique in that they involve overhauls of both standard operating system software, and more rare updates to the firmware and microcode that coordinate and control hardware.
“I remember first looking at it and thinking ‘oh, shit,'” says John Michener, the chief scientist at the security consulting firm Casaba Security, which has helped retail vendors with Meltdown and Spectre remediation. “We’ll see Spectre-related bugs for the next five years. But in general this type of thing has happened before. We may see a marginal impact and take a bit of a hit, but the newer processors don’t have a huge loss. Older processors have more of an impact.”
Dampening the potentially crippling performance issues has required a massive, coordinated effort behind the scenes. Some companies, including the open source enterprise IT services group Red Hat, had advanced notice about Meltdown and Spectre before the public disclosure, getting a head start on the patching process.
“There certainly is a performance impact, but what we had to do is kind of use the big hammer initially to mitigate, and then we can go back to iterate and refine,” says Red Hat chief ARM architect Jon Masters. “There’s potential for improving these fixes.”
That’s not to say everything’s fine and rosy. While Intel and other processor manufacturers initially worked to downplay potential performance problems from the patches, the industry immediately started feeling ripple effects.
In a Tuesday update, for example, Microsoft said that consumer devices with processors from 2015 or earlier running Windows 7, 8, and 10 would be more likely to exhibit slowdowns. The company added that, “Windows Server on any silicon, especially in any IO-intensive application, shows a more significant performance impact when you enable the mitigations.”
This means that millions of Windows PCs and servers around the world, even those that are just a few of years old, could get noticeably more sluggish—as much as 20 percent slower in some cases. Intel also published benchmark and user data on Wednesday, which similarly shows deeper losses for older generations of silicon.
Millions of Windows PCs and servers around the world, even those that are just a few of years old, could get noticeably more sluggish
Those losses will hit consumers hard. Large-scale organizations have minimized problems by testing patches in advance, and adding other efficiencies to offset losses, but individuals are pretty much stuck with the solutions tech companies provide. On Tuesday, for example, Microsoft paused distribution of its Meltdown and Spectre patches for certain AMD processors after the update bricked some machines. Microsoft claims that its patches were flawed because of inaccuracies in AMD’s chip documentation. On Thursday, Intel also admitted that its Meltdown and Spectre patches for older Broadwell and Haswell processors are causing more random reboots than usual. The chipmaker may push another patch to deal with the glitch.
And that’s before you even get to performance dips that stem from third-party service providers, like cloud platforms.
The video game maker Epic Games, for example, recently detailed patch-related performance declines in the popular battle royale game Fortnite. “All of our cloud services are affected by updates required to mitigate the Meltdown vulnerability,” Epic Games wrote last week. “We heavily rely on cloud services to run our back-end and we may experience further service issues due to ongoing updates.”
Fortnite players have experienced problems with log-ins, slowdowns, and downtime—not ideal for a competitive gaming environment. The problems have persisted since Fortnite initially outlined them last week. The company tells WIRED that it is still working with its cloud providers on a total resolution.
Industrial control systems and critical infrastructure have so far avoided Meltdown and Spectre slowdowns by not yet deploying fixes. That’s typical of these sectors, given the importance of understanding how patches will impact systems before they’re deployed. If something went wrong it could go really wrong.
“We definitely don’t see anyone in critical infrastructure patching on the fly,” says Jonathan Pollet, the founder of Red Tiger Security, which consults on cybersecurity issues for heavy industrial clients like power plants and natural gas utilities.
In working with the Meltdown and Spectre patches so far, Pollet notes that industrial systems generally have low processing and bandwidth requirements anyway, meaning less potential for performance degradation. The bigger complication will be identifying all of vulnerable devices, and making sure patches reach them eventually.
“When there’s a vulnerability at the chip level our customers are struggling with figuring out which of their components out in the field or in plants and factories actually have this particular bug, because they’re not really tracking their supply chain and inventory down to the chip level,” Pollet says. “So it took a few days for some of our clients to figure out where they actually had infrastructure that required an update.”
That type of time investment applies to internet infrastructure as well, one sector where lack of protection against data exposure vulnerabilities like Meltdown and Spectre could pose a real and large-scale security risk long-term.
“The thing that’s unusual about this bug is the scope of it,” says John Graham Cumming, chief technology officer of the content management and internet infrastructure company Cloudflare. “It affects pretty much all computers, it’s a very high percentage, and the problem is that people really find ways to exploit these security problems over time. So you’ve got to patch, there’s no way to get away from that, you’ve got to roll it out everywhere.”
‘You’re suddenly in an emergency situation where there’s kind of a fog of war.’ — John Graham Cumming, Cloudflare
Google has been refining a mitigation approach called Retpoline, which the company released last week to help manage performance issues in cloud platforms and other massive enterprise systems. And Amazon Web Services told WIRED in a statement Thursday that, “There have been isolated cases where a specific workload needed attention after patching. Our engineers have helped customers optimize their applications and in almost every case, prevent significant changes to their costs.”
For its part, Cloudflare, which claims to manage almost 10 percent of internet requests worldwide, says that in the end it managed the performance issues with the Meltdown and Spectre patches by putting extensive resources into testing the fixes before pushing them out. “You’re suddenly in an emergency situation where there’s kind of a fog of war,” Cumming says. “We sell performance, so if it was going to slow us down that would have a very big impact on our business.”
And though installing the Meltdown and Spectre patches has been an enormous effort and caused real grief, many in the industry remain upbeat about the challenge. Even after all of its struggles and the money it had to spend to handle the problem, Branch says it sympathizes with AWS, and everyone working to deploy the patches. In fact, AWS pushed out yet another refinement on Friday to improve performance right as this story went live.
“We’re still investigating the longer term impact on our system,” Branch’s Chan says. “Despite the performance impact, AWS was protecting its customers. They did the right thing.”
- You’re going to want to read the inside story of how four security research teams independently found Meltdown and Spectre within a few months of each other
- Meltdown and Spectre are as devastating as they are complicated. Here’s how they work, and why they’re such a menace
- Fortunately, some important steps have already been taken to fix the problem—but a full solution is still years away