Interview with Holden Oullette: Elixir Security, the Sobelow Project, and Scaling Developer Education

Michael: Hi Holden, what are you currently working on? How is it related to Elixir?

Holden: Hey there! Currently I operate as the Security Architect at Podium where I help our various security teams build out the future of security for a rapidly growing SaaS business which caters to local businesses around the globe.

Podium Engineering is a pure Elixir shop (I believe one of the largest in terms of Elixir specific headcount) and as such; all of our microservices are built on the Elixir Phoenix framework. We consider it a lighthouse technology!

A good portion of my role is thinking about how best to secure our codebase on a continual basis and instill a “Shift Left” mentality in our Engineers when it comes to securing the Software Development Lifecycle.

Michael: What is Sobelow? Could you talk about the history of the project, and your current involvement in the project?

Holden: Sobelow is a static code analysis tool primarily focused on detecting potential vulnerabilities in Elixir Phoenix applications - but through its use of Abstract Syntax Trees under the hood, it makes a pretty decent Static Application Security Testing (SAST) tool for normal Elixir applications too!

Sobelow was first created by Griffin Byatt during his tenure at NCC Group - he even spoke about it at ElixirConf 2017 when it launched! In my opinion, it was released at the perfect inflection point for the community. Elixir as a language had been around since 2012; but by the time 2017 rolled around it was picking up tons of steam - for instance you had Discord (who had already been using it for a few years) very publicly endorsing the language.

Even with all this traction though, there wasn’t all that much released about Elixir Security! The vulnerabilities were out there, so why weren’t there more resources about building secure software in Elixir? On top of that, one of Elixir’s core strengths is its robust support for metaprogramming - which makes it a goldmine for the approach that Griffin took when building out Sobelow.

With it filling such a crucial void in the Elixir Ecosystem, it was only natural that Sobelow took off. Almost every paid solution I’ve seen in the space for Elixir SAST now uses Sobelow under the hood (e.g. GitLab, Paraxial.io) and the default recommendation I see when Googling “How to secure Elixir apps” is simply “Run Sobelow”.

After witnessing such community support and adoption of the tool, it really shocked me to see development stall out over time. This isn’t a complaint against Griffin at all - he’s done enough work for the community by building the tool in such a robust way that it hasn’t needed that much work / maintenance! But regardless, Elixir the language is much different than it was 5 years ago and no releases enhancing detection capabilities have come out since mid-2020 - not to mention outdated dependencies come for us all at some point haha.

I had always been a long time fan of Sobelow - having integrated it into Podium’s CI/CD pipeline as one of my first projects upon starting. Not to mention I like to keep tabs on notable folks/projects coming out of NCC Group (ever since I had a summer internship there in college). When I saw discussion brewing of people wanting to fork the project since no one had permissions to merge things in, I realized it would be an absolute travesty to fracture the community like that if a split were to occur; so I decided to reach out to Griffin on Twitter.

We discussed the project a bit via email and I got his blessing to use my best judgment to keep development going. So that’s where my involvement on the project comes from; I’m the new volunteer custodian for the repository and hopefully will start to feel comfortable enough in the codebase to push out some new feature releases soon (beyond just the little fixes).

Podium has a vested interest in the continued success of Sobelow and as such I am making it a priority that there will always be someone from the Podium Security team involved in the project - even if I weren’t at Podium any longer. My hope is to avoid another potential community lockout so progress on developing Sobelow can always continue.

Michael: You mentioned “new feature releases” for Sobelow, anything you can speak to right now?

Holden: Well I started off my maintenance duty by tidying up the repository a bit (which is still a WIP) and merged in some “low hanging fruit” issues. So once Griffin and I finalize setting up the shared credentials for pushing to Hex.pm, I’ll probably roll up the fixes, minor features and dependency bumps into a minor version release.

But after that I really want to deep dive into kickstarting the community around development and defining a clear picture of the roadmap / how to contribute to the project. The community is really what made Sobelow so successful and I’m far from the best Elixir programmer out there, so lowering the barrier of entry to contributing as much as possible is a focus of mine.

In terms of actual features, I believe that there’s room to grow the tool for supporting more of the Elixir Ecosystem. Nerves, Scenic, Livebook, LiveviewNative and more are all projects that either have come out or have come into their own in the past few years that Sobelow just wasn’t taking into account when it was initially built. Like I mentioned before, Sobelow still does a pretty dang good job at analyzing Elixir applications in general outside of Phoenix - but there’s always room for improvement and I’d like to comfortably drop the association with Sobelow only being for Phoenix projects.

Additionally I’d like to take my learning experience and problems I’ve run into securing a massive Elixir codebase to build out more robust testing in Sobelow. Things like Authorization / Authentication issues seem ripe for ideation and even higher confidence findings through cross file associations (similar to r2c’s DeepSemgrep functionality) are an area I’d like to target.

Lastly I would like to iterate on how developers can interface with Sobelow - for instance I’ve already started a draft PR for LiveDashboard to surface findings there instead of relying on the CLI. I could also see the development of IDE integrations to give feedback even quicker in the development cycle. When developer time is at a premium, a fluid UI/UX can cut down the time spent on the feedback loop.

Michael: If a business is using Elixir, should they be using Sobelow?

Holden: Without a doubt - yes. Will it catch every insecurity in your Elixir codebase? No. If it did I’m pretty sure I’d be out of a job haha. But I do think that Sobelow should become a critical part of every Product Security team’s SAST strategy for Elixir codebases.

One of the strengths of Sobelow is its flexibility in how you integrate it into your Secure SDLC:

1. It can easily be run in CI/CD pipelines

2. It can be run locally

3. Many third party tools integrate it (GitLab, Paraxial.io)

4. Potentially more in the future!

Plus as I outlined before, renewed development on Sobelow is just beginning and Podium’s commitment to supporting the tool should hopefully reinstill confidence in any company looking to integrate Sobelow into their workflow that support hasn’t gone away.

Michael: You seem to work on a well funded and mature security team. For companies that are getting started with Elixir security, what should they do?

Holden: That’s definitely the (multi-)million dollar question haha. There’s a relevant quote that I’m quite fond of from Gene Spafford that reads, “The only truly secure system is one that is powered off, cast in a block of concrete and sealed in a lead-lined room with armed guards - and even then I have my doubts”.

When I stepped into my role at Podium, it was the first time I was really forced to understand what Elixir was and the state of the language - I’ve come to truly believe in Elixir. It almost forces wonderfully written code, provides the foundation for the next generation of applications worldwide, and has “Secure By Default” best practices baked into its core - but I do think we’re just waiting for the other shoe to drop when it comes to Elixir Security and/or one of the underlying mechanisms that powers the Elixir Ecosystem to be compromised. Even though it inherits much protection/functionality from its predecessor Erlang, it’s a young ecosystem that is changing by the day and more complexity to any data system increases the chances of compromises just that much more.

The C programming language had been around for 24 years before Peiter “Mudge” Zatko released “How to Write Buffer Overflows” and sent it to Elias “Aleph” Levy who subsequently released his seminal work “Smashing The Stack For Fun And Profit” in 1996 - redefining how crackers for generations to come would approach reverse engineering programs. That’s not to mention the plethora of various exploits released that had simply been undetected for many years waiting to be discovered and abused (e.g. Heartbleed was introduced in 2012 and remained undetected for 2 years).

Simply put, it’s not a matter of if Elixir based applications can become more easily compromisable, it’s when and how. So my advice to companies looking to get started in Elixir Security would be this: by using Elixir, you’ve already committed to operating on the bleeding edge - so take the extra step needed and ensure you’re continually adhering to Elixir Security best practices as well as adopting every bit of security tooling / process out there that supports your business’ needs. That way when the shoe does inevitably drop; you’re ready to adapt and incorporate the changes needed quickly to protect your business from cyberattack.

Michael: You’re the author of the Elixir Secure Coding Training curriculum. Could you talk about the project and what you want to achieve with it?

Holden: The Elixir Secure Coding Training (ESCT) has been the culmination of a lot of hard work for myself and the Product Security team at Podium to create an all encompassing, hands-on, engaging Secure Coding course for our Engineers to participate in during onboarding. Best of all - it’s completely open source, free to use, and easily adaptable if an organization wishes to make private changes to suit their needs. Education is one of the most powerful tools when it comes to “Shifting Left” and we feel like this is a wonderful application of that principle for businesses running a primarily Elixir codebase.

Built in Elixir Livebook, we have core modules that span all of Elixir Security as well as other focus areas in Web Security as they apply to the Elixir Ecosystem. This includes topics like the OWASP Top 10 or CI/CD Secure Best Practices. Each module is approachable by design and incorporates interactive, Elixir based quiz questions that the participant is required to pass and will receive immediate feedback on whether they got it right or not. For more information on the background of the project, as well as more details surrounding what other modules were included in earlier versions of the project, I was honored to speak about it more thoroughly at ElixirConf 2022.

When we released the ESCT as a public beta back in August of 2022, our last hurdle before declaring it ready for mass adoption was the finalization of our grading mechanism. I’m so happy to announce that in the beginning of Q2 2023 - we are planning our 0.9 release! Complete with a publicly facing grading server that everyone who is using the open source version of the training is free to use and connect to. Simply open the first module Livebook and the built-in API client will connect to the server to check your answers as you complete the course.

For companies wishing to make modifications to the open source curriculum (whether to remove irrelevant sections to your infrastructure or introduce company specific modules), we will also include the grading server code as part of the ESCT repo with specific instructions on how to set up your own grading server/answer key.

Michael: Thank you for talking to me today. If people want to get in touch with you, how should they do that?

Holden: My pleasure! I’m accessible via Twitter @HoldenOullette or you can open an issue on any of the projects I support and tag me @houllette.

Paraxial.io stops data breaches by securing your Elixir and Phoenix apps. Detect and fix critical security issues today.