Sometime in March 2020, on what was assumed to be a normal, regularly scheduled software update, malware was introduced on SolarWinds’ Orion software platform. SolarWinds is a long-established international company that provides network and security management software. Out of its 300,000 customers, fewer than 18,000 were on the Orion software update list. However, a number of these Orion customers are U.S. federal agencies, critical infrastructure systems, other national governments, and large multinational corporations. The malware was engineered to be extremely stealthy, meaning its creators designed it to be very difficult to find.

The hack began with a compromise of someone or some system at SolarWinds. In the SolarWinds system, the attackers had to have spent a considerable amount of time poking around to find the right attack vector. The hackers were able to package the software so that it would be extremely small and could piggyback on a normal software update. The routine software update provided access into the host’s system and security management system software.

Once inside the system, the malware sat there for about two weeks before it opened. Like an alien in some science fiction story, it waited, tested the atmosphere, and then unraveled itself. It began to slowly communicate with the command-and-control center. The communications were infrequent and hopped from one address to another, as the malware never established a common or repetitive communications link; it was random.

These communications were masked by the system’s normal communications traffic. Most intrusion protection systems look for outlier activity, something that is uncommon, something with a pattern. Here, the malware was masked by randomness and the noise of regular communications and internal system activity.

For about nine months, the malware moved within these systems creating and deleting files. It was able to capture user passwords and authentication credentials, and execute “jobs” where it downloaded files and established its own access rights. It was able to disable functions and system services. The malware ensured its ability to control these systems over the long term by granting itself access, credentials, and administrative privileges.

Sometime in the middle of December, FireEye, a SolarWinds customer, discovered the malware. FireEye exercised the cyber-incident response process, and SolarWinds began their incident response by notifying their customers, and began a containment-and-mitigation response. Currently, a host of federal agencies and the IT industry are involved in identifying the scope of the breach and developing the means to remove the virus.

Big questions remain unanswered. How did the malware get into SolarWinds, and what can be done in the future to ensure the integrity of these automatic software updates that the world relies on, how can intrusion detection systems improve, and others? Over time, the entry point will be discovered, and it may be a number of things. It is the impact or the long-term persistent threat that remains the real question. There is no doubt that this was a nation-state actor; this was a multi-year investment in resources that no individual or criminal network could support.

The remaining issue is the integrity of the systems compromised; how do you know that the virus has been truly eradicated? It is integrity, the most neglected of the holy cyber trinity — confidentiality, integrity, availability — that is the focus. After being compromised by a highly sophisticated attack, one in which the beginning and end are still obscure, the integrity of the systems is now suspect. The goal of the attacker is dangerously subtle; it was not to take over or shut down systems but to collect information and nudge decision-making. After all, operating on information that you know to be true, but instead has been compromised and is false can only lead to disaster.

The next issue is how to avoid this. The honest answer is you cannot. In this case, armies of professionals were compromised. No small business could have withstood this attack. At some point in the supply chain, you have to trust the software developer. There’s an expression in system security: It is not a question of if you will be attacked, but when. Then it is all of the things you do beforehand, in terms of the system security principles that will allow you to survive.

In light of this, the security recommendations remain the same. These make it hard to get into your system: Use complex passphrases, multi-factor authentication, encryption, role-based access control, system monitoring, and the like. These are the mundane actions that a company can take to make it difficult for the hacker. If your system has been compromised, these difficulties often force the hacker to expose themselves.

Then there is recovery. In light of this massive breach, we are changing our backup and archive process, meaning we will use two external air-gapped drives and start an annual backup or archive at six-month intervals. One drive will archive the system at month one, and again archive 12 months later. The second will archive the system at month six, and then again in 12 months. In this example, alternating the annual archives will allow for a recovery in a worst-case scenario.

However, as with many things, the real challenge is implementing and sustaining all of these good ideas in real life.

With 30 years of experience in information technology, Mike Olivier brings his expertise to small-business System Security Planning with San Diego-based 171Comply. As a small business owner working in the federal space both as a prime contractor and as a subcontractor, he understands the realities of running a small business. Contact Mike at mikeo@171comply.com.

Shares: