4 Lessons Learned From the Log4j Vulnerability Response

By Augusto Barros, Cybersecurity Evangelist, Securonix

What is Log4j vulnerability?

Log4shell is a major flaw in the widely used logging programme Log4j, which is used by millions of machines worldwide to run internet services. It is expected to have an impact on a wide range of people, including organisations, governments, and individuals. If the problem is not fixed, attackers can break into systems, steal passwords and logins, extract data, and infect networks with malicious software.

​​After the rollercoaster ride that was Log4j a few weeks ago, it’s time to shift away from the firefighting to extract some lessons from the incident.

Lesson 1: SBOMs Need to Become Standard Security Practice for Software Developers

One of the most interesting points of this incident was that it wasn’t related to a “missing patch”. Many recent large scale security events were related to some worldwide attack or worm exploiting an easy to patch vulnerability. That would automatically drive the conclusions and advice to “you just need to patch better, faster”. The Log4j vulnerability, CVE-2021-44228, the attack vectors and all implications from it, however, were different.

The first simple conclusion is that the time for the Software Bill of Materials, or simply SBOM, has come. Software development today is closer to LEGO building than actually writing code, so it’s critical to know what LEGO pieces are part of your product. Log4j could be described as one of those very common 4×2 LEGO pieces; it’s everywhere, and it is part of many software products out there. But developers in general have been very lax about tracking what they use in their software. When an event like this requires us to identify whether some library or component is used by our code, that lack of traceability becomes a major pain point. It turns a simple exercise of checking inventories and SBOMs into a complex scanning process, with many opportunities for false positives and false negatives. If we ever needed a wake up call for this need, we’ve got a big one with Log4j.

Lesson 2: Good Security Posture Is More Important Than Ever

The attack method to exploit this vulnerability has also reminded us of the importance of a good security posture. In order to successfully exploit the Log4j vulnerability, the attacker must be able to instruct the vulnerable host to download Java classes from a host they control. That means they need to be able to establish an LDAP connection from the victim to the system where the classes are hosted, usually on the Internet.

That’s where good old network security practices come in. Why would you allow a server in your network, for example, to open an outbound LDAP connection to the Internet? There may be cases where that is indeed required, but usually you know in advance where that connection is going to. Instead of allowing any outbound LDAP connection, your firewall, or whatever other network security resource you use (security groups for cloud hosted systems, for example), should only allow the connection to the specific, authorized endpoint required by that application. If you do that in your network, chances are high that even with systems carrying the vulnerable library you wouldn’t be at risk from exploitation.

Lesson 3: Understand Attack Paths and Threat Models

A less evident observation from all the response activity around Log4j was the general lack of understanding of how the vulnerability is exploited. As the first news started circulating, some proof of concept attacks and automated scans also surfaced. We saw, for example, many cases where the exploit string was added to the “User-Agent” header of HTTP requests. Those were mostly “copy-cat” attacks derived from the initial Proof of Concept exploit. But that’s not the only way the exploit can be delivered.

The most intriguing thing about the Log4j vulnerability is that the ways it can be exploited are only limited by the imagination and expertise of the attacker. These examples using common HTTP headers were used because it’s very common for applications to write the content of those headers in logs (writing the received content in logs is what triggers the vulnerability). But virtually any application input could trigger the exploit. For the attacker it can be hard to even foresee which application or system will be exploited. They can deliver the exploit string through an input field of an application in one of your business partners, for example, and when that application calls a web service from your environment to get some extra data, the exploit could end up being executed in your system. The challenges in threat modeling from this vulnerability are tough, and I expect we’ll still see exploitation cases where the attack scenario was completely unexpected by the victims.

Lesson 4: Detection Must Go Beyond the First Exploits

Finally, blue teams and security vendors were quick to respond, with signatures, policies and rules to detect and block the first observed attacks. However, what we saw was a series of quick actions to create content designed to detect those specific attacks only. Although that content would certainly help you to see those attempts against your environment, they were focused on static indicators of compromise. Attacks using slightly different, but also efficient approaches, would be missed.

It’s understandable to see a first wave of detection content targeting highly visible exploitation attempts, but it cannot stop there. Alarmingly, I haven’t seen much content being published or developed to address the exploitation attempts in a more general, encompassing manner. This challenge is partially caused by the difficulty in threat modelling that I mentioned before, but it is also caused by limitations on blue teams in developing good threat detection content. We are seeing many organizations pushing content to look for exploit code on HTTP headers, but what about those outbound LDAP requests from your systems I mentioned before? If something like that happens, the chances are high that one of your systems is being exploited; looking for those connection attempts is far more effective than looking for specific exploit delivery methods, which are numerous, can be done through encrypted channels and using multiple obfuscation techniques.

In summary, the Log4j vulnerability event can help us extract valuable lessons for multiple security practices: secure development, security hygiene, threat modeling and detection engineering are just some of them. That points us to the final, and maybe most important lesson to be learned from this holiday season: keep learning, keep evolving and maturing. That’s the only way a security program can keep up with the ever growing threat landscape.