Following the Log4Shell vulnerability fallout, OWASP Gothenburg hosted a "Log4Shell Special" virtual meetup on 2021-12-16 with Jonas Lejon from Triop, Fredrik "Stök" Alexandersson from TrueSec and our very own Assured consultant and Säkerhetspodcasten host Peter Magnusson. Hosting and moderating the event was Assured CEO and OWASP Gothenburg board leader Jonas Magazinius. Attending were 80+ people within DevOps, penetration testing and similar technical areas. This blog post is a short summary of what was talked about during the 1 hour session.

What is the Log4Shell vulnerability?

The session started up with the speakers casually explaining what the Log4Shell vulnerability is, how it came to be and what it implies. If you are reading this you probably already have enough knowledge on what the vulnerability is. If you don't there are some good writeups in Portswigger: 'Log4Shell' vulnerability poses critical threat to applications using 'ubiquitous' Java logging package Apache Log4j and Naked Security: "Log4Shell" Java vulnerability – how to safeguard your servers.

An interesting aspect of how the vulnerability was introduced was discussed: It seems that initially as the vulnerable feature was requested, it was not obvious that it would imply a security issue. The feature itself was probably not used extensively until very recently as the internet has now exploded with active exploitation, security researchers testing every potential method of triggering the vulnerabilty, and general making fun of the issue in social media.

An obvious question could be asked, and was asked by the moderator: "Why would it make sense to allow a message parser functionality to execute arbitrary (remote!) code?". Historically, Java web servers were often decoupled from Enterprise Java Beans (EJB) servers to separate business logic from web applications. Between the different servers was an almost full trust as it were, and JNDI was therefore not a thing to really worry about. Come 2021 and we find ourselves in a world where microservices pop up all around with less-than-perfect network rules and segmentation.

To allow for (as an example) dynamic log routing the JNDI API would be utilized to contact a friendly server to download logic for sending the logs to the correct place without having to alter configuration etc. Little would one have guessed that since log messages are parsed by the JNDI feature whether they contain user supplied (untrusted) data or not, the user could tell the server to lookup a remote server of their choosing to collect the code, and ultimately running it.

So with the original assumption that there is trust between web server and the remote server, any server could be contacted. The last week has shown us that even a simple DNS lookup could be bad enough as the attacker could craft a FQDN containing secrets fetched from JNDI lookup strings (such as environment variables).

What is affected by the vulnerability?

At the moment, if your Java application depends on log4j version <= 2.15 you are probably affected. It also depends on what you are actually logging and if you've configured the bad default behavior away. The 2.15 fix was deemed incomplete, so if you patched early and took a few days vacation you may have missed the last 2.16 update. If you're currently thinking "I'm on log4j version 1, good I didn't upgrade", think again. Since you're running an end-of-life dependency it probably means you have additional technical debt, such as old and vulnerable struts, spring and similar components.

What about the timeline from disclosure to exploitation?

The audience was eager to fill in and ask further questions, such as highlighting the rumour/notion that the vulnerability was being exploited before the public disclosure on December 9. The speakers discussed their understanding that the original disclosure was made to Apache on November 24, and that a patch was being developed on or around November 30. The next day, there were indications of active exploitation.

What would it mean that servers were exploited about 9 days before public disclosure? Did someone accidentally leak information around the issue, or was the development branch available before disclosure so that anyone could deduce the vulnerability from the patches being made? The timeline is still muddled, it seems, and we should put paranoia aside for now.

About mitigation and prevention

Could this vulnerability have been found sooner? The speakers discussed what we as a security community could improve to find and handle similar issues going forward. One thing could be to improve fuzz testing (Google's OSS-Fuzz project recently introduced JNDI detection), another to better detect bad and unexpected behavior (should Minecraft server really pop the calculator?). On the mitigation side, we should also become better at isolating systems and limiting servers' DNS and internet access.

Some good mitigation strategies presented during the meetup were:

• (An obvious one) Patch the log4j (2.17 or newer) dependency, or get the latest patch for your application depending on log4j.
• You can configure away the dangerous default (in older versions) behavior: log4j2.formatMsgNoLookups=true
• Don't allow JNDI lookup to connect to untrusted, remote servers
• Avoid logging unnecessary/untrusted data without proper filtering (caveat: this may sound easier than it is)
• DNS: only allow cluster/internal dns, don't resolve on the internet
• Scan and test your code for vulnerable dependencies, JNDI sinks and similar.
• Scan locally on all your systems, using JNDI payloads.
• Implement EDR (Endpoint Detection and Response) to detect bad behavior

Predictions on what's in the near future

Other open source Java frameworks will probably be the target of extensive future research to see if there is any reference to JNDI lookups in them. Expect the issue with JNDI message parsing not to stay at log4j implementations alone. Expect also the bug bounty community to thrive on this type of vulnerabilities in the future as the issue becomes eligible for bounties.

Conclusions

We haven't got the full picture yet, but many eyes are on the issue at the moment. Expect more to come in the near future: better detection, new ways to trigger the bug, new insight. Make sure to patch your dependencies, scan your code and your systems. Oh, and if the patch was applied without your knowledge, assume a compromise and investigate!

I would like to thank OWASP Gothenburg for hosting the meetup. Join their mailing list to make sure you don't miss future events.

If there are any questions, don't hesitate to contact us.

Update 2021-12-20: A Denial of Service vulnerability in log4j was fixed in the 2.17 version, updated patch recommendation.