A Tale of Two Supply Chains

This is the first of a multi-part series where we will share some of our methodology for supply chain verification in situations where there is very limited information. This content was previously shared by Sophia d’Antoine at Square’s r00ted¹ Conference on November 14th, 2018 in NYC.

We have previously shared our thoughts on the importance of supply chain validation with regard to hardware attacks, but this blog series will delve deeper into the specifics related to case alleged in Bloomberg Businessweek’s “The Big Hack” article.

On October 4th, Bloomberg Businessweek published the now infamous article, “The Big Hack”². In the article, Bloomberg claimed that Chinese spies placed small “rice-sized” components on motherboards made by the US-firm, Super Micro. One German researcher jokingly tweeted out some “hardware backdoors” he found:

So perhaps a better name for this blogpost could be: Of Rice and Men.

Those components are “nation-state-level hardware implants” allegedly placed by Chinese spies on Super Micro motherboards during their manufacturing prior to shipment to US companies such as Apple and Amazon.

Apple, Amazon, Super Micro, and the US and British governments (through DHS and GCHQ respectively) have all vehemently denied the report and are calling for Bloomberg to retract the article³.

The article had a severe impact on the Super Micro stock price – which lost nearly half its value the day of the report.

Several U.S. Senators are starting investigations. Senator John Thune, a senior member of the Senate Commerce Committee, in his letters⁴ to Apple, Amazon, and Super Micro, requested a briefing on the hack allegations. Senator Marco Rubio and Senator Richard Blumenthal also sent letters to Super Micro requesting information on how the company responded to the allegations. The Director of National Intelligence, Dan Coats, publicly stated⁵ that he’s “seen no evidence of Chinese actors tampering with motherboards made by Super Micro Computer.”

To the best of our knowledge, no one has verified the claims made in the Bloomberg Businessweek articles.

Due to the seriousness of the claims made in “The Big Hack”, we conducted a review of Super Micro motherboards in an effort to verify the claims.

The Chinese press also vehemently denies these claims.

Additionally, Joe Fitzpatrick, one of the two named sources used in the original article, expressed doubts⁶ about the article. He said he found failings in substantiation and felt his quotes were taken out of context. In light of all of the backlash Bloomberg has said it is devoting more editorial resources to prove its story.

This type of supply chain intervention attack is troubling to both Super Micro but also to companies who rely on Super Micro products in their own products and services. Claims like these can deter clients as well as investors from trusting and investing in these companies.

But where do supply chain threats come into play? Consider a standard technology stack.

A tech product usually contains some or most of these components. From a security standpoint, a company can produce and secure the software aspects of its products. For example, a company like Apple can develop a secure boot method which is cryptographically secure, or user can verify an application by taking a hash of the file. Additionally, software, including firmware, can be loaded onto the device at various stages. This can be done outside of the supply chain’s potentially untrusted environment using a variety of methods including UEFI updates, flashing the memory, and others.

Although not always utilized, the industry has semi-automated solutions to verifying a system’s software components – why has hardware verification fallen by the wayside?

Companies have not invested in hardware verification as it’s often outside a company’s control and it’s not an easy problem to solve.

A supply chain has several security critical stages. Each has unique security requirements.

1. Design Time First, a specification is developed – this can be checked for backdoors or weaknesses prior to manufacture. Subverting this stage to introduce a backdoor provides the greatest access.

2. Hardware Manufacturing Manufacturing is often subcontracted to a third party and is not easy to check. Manufacturers frequently substitute parts due to availability and cost constraints. Small malicious changes are possible at this stage. The ease of doing so depends on the device and format of the plans the attacker can access and modify.

3. Third Party Hardware & Firmware Integration Manufacturers frequently act as integrators and subcontract manufacture of subcomponents. This third-party integration leaves room a malicious actor to introduce backdoors or exploitable flaws into the system.

4. Supply Distribution Time By the time the manufactured device reaches distribution, the company and consumers have little ability to verify the device matches the specification as originally designed. Devices can be replaced wholesale with counterfeit devices or modified to include additional malicious components.

5. Post Deployment In the final stage, defense depends largely on the end customer’s physical security and processes.

In the “Big Hack”, the attack Bloomberg described would take place at stages 2 and 3.

Through investigating Super Micro’s public disclosures⁷, we find that Super Micro performs the majority of R&D in-house. They are a huge supplier for US companies: 4,950 SKUs (including power supplies, etc.), 1,200 “server systems”, 600 “serverboards” as of June 30, 2016. Super Micro is a business-to-business company reporting about 800 customers, mainly distributors, in June 2016.

However, Super Micro contracts manufacturing and supply chain logistics. They use Ablecom Technology: a company which manufactures and provides warehousing before international shipments (US, EU, Asia). Ablecom is a private Taiwanese company run by the brother of Super Micro’s CEO, largely owned by the CEO’s family and is critical to Super Micro’s business processes. They rely on them to accurately forecast and warehouse parts from various contract manufacturers to be able to create their products.

If we attempt to simplify the above supply chain, Super Micro’s public disclosures would suggest that the steps applied to their case is, at a high level, as follows:

1. Super Micro designs product
2. Ablecom coordinates manufacturing with contract manufacturers; contract manufacturer produces and ships to Ablecom for warehousing
3. Ablecom ships to Super Micro facility (San Jose, Netherlands, or Taiwan) for final assembly
4. Super Micro ships to distributor, OEM, or customer
5. Customer utilizes product

So how are hardware threats remediated and proactively responded to? Through a manual analysis and a deep understanding of the underlying hardware and its functionality – as well as the functionality of its subsystems.

Technical Approach

Hardware backdoors are – by their very nature and purpose – extremely hard to detect and remediate. How do experts verify the integrity of their hardware? To verify the claims were false, manual analysis was required on each layer of the PCB as well as the IC connections to look for anything extra, unexpected, or unusual to investigate further.

In the case of investigating the Bloomberg article’s claims, no schematic was available from Super Micro to check against – which makes each step slightly harder. All connections (traces), components, and other subsystems had to be analyzed for their purposes and likely uses. Where applicable, net lists (the connections of the different components on the board) had to be manually reconstructed as well.

An example of something suspicious would be a component that connects several seemingly independent components, such as an external serial bus to the storage controller – something you wouldn’t normally see. We also looked closely at privileged areas such as those that implement management planes, DMA controllers, and external BUSes.

This process is time intensive – if it’s so hard for experts, how can multitasking IT specialists even hope to protect their hardware? Especially at scale.

Some valid abnormalities may be detected as false positives: on server hardware the part count is high and sometimes varies across devices. There are often also unpopulated or “missing” components on a PCB, which in our experience happens usually either due to the vendor wanting to maintain flexibility to substitute parts depending on const and availability, or if one circuit board design is being used for various SKUs where some have additional features included.

When we were investigating this situation, we performed a full disassembly of the devices, including removal of both the external cases as well as internal shields on the PCBs. Any documentation for chips where data sheets, application notes, or other relevant information could be obtained were analyzed.

The traces on the circuit board were manually examined to determine likely affiliations between chips and the sub-systems on the board. This was done with a combination of high-definition scans and electrical probes.

In some cases, X-ray imaging was used to introspect inside the board’s layers to either gain additional understanding of the circuit traces, or to validate that there were not ‘buried’ components (which one Bloomberg article² alleged).

Further analysis could include decapping specific chips of interest to inspect the dies under scanning electron microscopes, however this level of analysis wasn’t a great fit for a scenario where no suspicious chips were found to inspect. Further, no available information pointed to chips which warranted closer inspection.

Although the focus of this discussion has been on hardware attacks, supply chain vulnerabilities are often be used to enable software attacks. While we often are engaged to hunt for previously-unknown firmware vulnerabilities in customer’s products, this type of analysis was not the objective of validating the claims in Bloomberg’s article. For this task, we analyzed the firmware obtained from historical archives available to us and compared them against the contents of memory which we extracted from the device. The compared chips partially contained software instructions, along with configuration and other data.

Our team developed customized automated tooling to verify the memory did not contain additional or modified code – and to quantify other changes – such as configuration data. To narrow down the search space for these abnormalities our tooling used byte comparison as well as entropy analysis to define regions. As an example, the below images show the differences in entropy between two similar firmware images taken from the Super Micro website.

Typically instead of comparing two firmwares such as for this example, we instead compare the distributed firmware from Super Micro with the memory we dumped from the motherboard. As in the above example, they often differ. However, this isn’t necessarily wrong and doesn’t in-itself indicate an issue – certain properties of flash needed to be accounted for in these comparisons. For example, the default state of flash memory is logical 1 (0xFF’s) as it is less expensive to set this memory when needed. If it were logical 0x00 it would require an entire block to first be erased then written with the intended value, as the minimum block size is rather large, it is better to have flash memory default to 0xFF which we see reflected in the memory dumps analyzed.

This blog post covers only a portion of the analysis we performed. Some other techniques we often use for projects include low-level radio-frequency (RF) analysis, detailed network analysis, and PCB delayering – but such techniques are outside the scope of this blog post.

So what are the takeaways?

Even if one board is verified perfectly, this assurance cannot be transferred to the other ten thousand devices coming off of the manufacturing line. For high assurance, each individual device would have to be separately assessed – or ideally, created from scratch in a trusted supply chain.

Hardware security verification is still a complex and nascent field. A thorough verification effort includes chip decapping and PCB delayering – destroying the board. As such, overall it is time consuming, expensive, and requires a team of experts for each step.

While “The Big Hack” may turn out to be fantasy, we should heed its warning. This is not the first report of supply chain intervention by Chinese fabricators. The threat is real and potentially devastating.

A user on Hacker News shared a story about his employer – a credit card company – that attempted to automate verification by weighing every single device coming in and it still wasn’t enough.

As an industry, we should be thinking about how can we automate this and how can we apply a solution at scale. Until such capabilities are developed, tested, and proven – it will be a team of experts called in to provide verification for a small set of the unknown.

Thanks to the entire hardware team at River Loop Security for helping to develop a comprehensive set of analysis which can be put into practice to verify the claims. Also, while we believe there is still a lot to learn about these attacks, we believe that improvements will come as a from the fact that Bloomberg Businessweek brought the topic of hardware verification into the mainstream discussion.

If you have questions about your supply chain security in general or in response to specific risks, consider contacting our team of experienced hardware security experts to learn more about what you can do.

In our next post we will describe various general types of hardware “backdoors” and more.

NOTE: For this blog post we used stock images which are in no way intended to indicate any suspicion of compromise in anything depicted in these images.