Posted on 2020-02-20 by Matt Strahan in Business Security
When reading about the Toyota Production System and the Lean Methodology, a remarkably simple technique was talked about called the “Five Whys”. It was used by Toyota to solve the underlying problems, not just the symptoms. The technique was made popular by books such as The Lean Startup.
When there is a production failure, outage, or problem, the “Five Whys” facilitator will bring all the relevant people into a room and ask “why” again and again to try and pull the thread of the full sequence of events that led to the issue. The Wikipedia page for Five Whys gives this example:
- Why? – The battery is dead. (First why)
- Why? – The alternator is not functioning. (Second why)
- Why? – The alternator belt has broken. (Third why)
- Why? – The alternator belt was well beyond its useful service life and not replaced. (Fourth why)
- Why? – The vehicle was not maintained according to the recommended service schedule. (Fifth why, a root cause)
When reading about this technique, I began thinking about security vulnerabilities. How often do we talk about patching the vulnerability without thinking enough about what caused the vulnerability in the first place? And I’m not just saying “we didn’t do the patch”, I’m saying the underlying processes that people don’t even realise are there that made us end up here.
Do you figure out why the security issue occurred?
How did that vulnerability came to be? Even when there’s that “root cause analysis” part of the report, the consultants are often making assumptions or even guesses. They don’t know everything you do in your business and they don’t see you work day to day. They know what can cause it in other organisations but your organisation is your own.
When the root causes aren’t addressed, we end up finding the same sorts of vulnerabilities again and again in slightly different combinations. We get frustrated – why does it feel that the same things are found year after year?
Companies often just patch the issue rather than tackling the root cause because tackling the root cause is harder. The root cause tends to be non-obvious, harder to define, and harder to fix. It’s just easier to patch something and move on then it is to put in the work and fix the underlying issues. This is understandable – IT staff are generally overworked and under resourced, so they need to try and fix things as quickly and easily as possible.
While it could be easier for this one case, though, the symptoms will come out again and again. It’s going to be harder to patch 500 vulnerabilities than it is to fix the one underlying root cause, and it’s definitely easier to fix the root cause than performing incident response would be when that vulnerability is exploited!
Finding the root cause
We often forget that security vulnerabilities are just another type of problem. Given it’s just another type of problem, we can use the same problem solving techniques that the manufacturing industry has been using with great success for decades. Let’s give an example of using the Five Whys for a security vulnerability that might be found in web application penetration testing:
- Why? – There is a remote code execution vulnerability in our web application
- Why? – No-one updated the web framework
- Why? – Web applications and frameworks aren’t part of our patching process
- Why? – It’s too hard and takes too much time to apply updates to web applications
- Why? – We haven’t built processes, procedures, and pipelines to allow for easy updating of web applications
This gets down to the real issue. Even though the web app pentest might have found a code execution vulnerability, patching that vulnerability is not enough. You need to not only apply this patch, but build the processes, procedures, and pipelines to update web applications and their frameworks without it being a huge effort. Building those processes and pipelines won’t just fix this vulnerability, but every vulnerability of that type you might have in your environment.
If you don’t put in the work and address this root cause, you’ll just be finding more and more vulnerabilities that would be addressed by applying those patches. That’s if you’re lucky – if you’re unlucky you’ll be calling in incident response and talking to the media.
Make the organisation better
When you get the results of penetration testing, it becomes extremely tempting to ignore root causes and just focus on the vulnerabilities as a “todo list”. I sometimes feel that at Volkis we might even inadvertently help clients down this path – we will provide Excel spreadsheets or even put in tickets straight into the client’s ITSM system, one for each vulnerability. The clients think that once the vulnerabilities are fixed then everything is done, but should that be the case?
Another way of thinking about the vulnerabilities in a penetration test is that each vulnerability is an opportunity to learn and a challenge to be better. By drilling down to the root causes you will use the vulnerability to learn more about your systems, processes, and where you can improve so that the vulnerability would never have been there in the first place. This is what we do in our root cause workshops.
Using the same principles and techniques that the manufacturing industry have shown to work and provide meaningful and measurable improvement, we can start to tackle these root causes. Doing this over and over again, we can make sure we’re secure not just now but into the future.
About the author
Matthew Strahan is Co-Founder and Managing Director at Volkis. He has over a decade of dedicated cyber security experience, including penetration testing, governance, compliance, incident response, technical security and risk management. You can catch him on Twitter and LinkedIn.
Cover photo by Emily Morter on Unsplash
If you need help with your security,
get in touch with Volkis.
Follow us on Twitter and
LinkedIn