Posted on 2020-03-03 by Matt Strahan in Business Security
At school I was taught that a good piece of writing should “say what you’re going to say, say it, then say what you’ve said”. In that vein, I’m going to talk about the advantages and disadvantages of bug bounties and penetration testing but it will all come down to this:
Penetration testing and bug bounties tend to complement each other extremely well. The disadvantages of penetration testing tend to be the advantages of bug bounties and visa versa. Let’s go through it in more detail.
Why go for bug bounties?
Even though I head up a company that provides penetration testing services, I recommend that organisations have a bug bounty in place. Bug bounties are cheap (that is unless you go for the million dollar bounties provided by Google and Apple) and they are easy to set up. The two main bug bounty managers are Hacker One and Bug Crowd.
They are also ever prevalent. You don’t set up a bug bounty for a specific time, but you set it up and it just keeps going. It takes advantage of the old Eric Raymond adage “given enough eyes, all bugs are shallow”. You can take advantage of all the people looking at your organisation’s internet presence and find these bugs.
I believe the biggest advantage of bug bounties isn’t often put in the headlines. Bug bounty organisations allow a formal way of reporting bugs without risk to the researcher. Back in the day it was common for security researchers to sit on bugs purely because they were afraid of either being sued or having the police called on them if they reported it to the company. “Why did you try and hack us?” the company would argue.
Even nowadays it’s not uncommon for organisations to attack a security researcher who reports a vulnerability. The simple act of verifying the vulnerability is regarded as a malicious hack, and, the companies argue, how could the researcher have found the vulnerability unless the researcher was targeting the company? This stupid argument just ends up with vulnerabilities lying unreported – instead of researchers alerting the vulnerable company, the company ends up being targeted by hackers looking to exploit it for profit.
Bug bounty programmes let security researchers know that it’s safe to report the vulnerability to you. Knowing about the vulnerability will only make you more secure.
Once the vulnerability is reported, bug bounty programmes will assist in triaging and assessing the vulnerability. This makes patching and response easier and more effective for your organisation.
Although they have huge value, bug bounties are not the be-all-and-end-all of external security for organisations. This comes down to the economics for security researchers. When you’re paying a bounty you’re ultimately paying for the time of the researcher. The researcher is only going to spend so much time investigating your environment before they decide it’s not worth continuing and move on to the next bounty. There’ll be no report when this happens and no additional advisory, it’ll just happen. The result ends up with bug bounties providing a wide but shallow look at your external security.
The economics also come into the likelihood that the best researchers will look at your environment. If the bounty isn’t big enough then they are unlikely to be pulled across. Pricing your bounty is perhaps the hardest decision for you to make when building a bug bounty programme. Price it too high and you could be in for big costs if there are plenty of bugs. Price it too low and you won’t get any coverage. The variability of pricing and costs makes it difficult to allocate budgets for the bounty. Luckily the bug bounty programmes are happy to help with the pricing issue and expectation.
For organisations that have authenticated access (which ends up being pretty much every organisation with a VPN) the bug bounties are also unlikely to delve into the authenticated areas. You can easily have huge parts of your environment that are locked off from bounties.
Finally, bug bounties aren’t going to alert you of a risk until after it’s already been in your environment. You have to hope the security researchers find the issue first, because otherwise you’re in trouble.
Why go for penetration testing?
Luckily the problems with bug bounties are what penetration testing excels at.
Penetration testing offers a more structured method of working. The tester will be more comprehensive and thorough, and guarantee a certain level of investigation. The tester will also tell you what is and isn’t tested, and give you visibility over the coverage you are getting. They will also provide evidence including testing logs if you require so you have auditability over their actions.
It is always best to integrate penetration testing into both your project release schedule and have an annual test. This means that the projects will be secure even before you release it to the internet, and prevent those dangerous first moments of exposure.
As part of a project release schedule, penetration testing is usually built into the projects costs. The pricing is consistent and can be budgeted for. Even the annual penetration testing usually has consistent pricing and can be worked into the annual budget without too much variance.
You can also get more than just a test out of penetration testing. With root cause analysis, remediation support, testing of detection and response, and trend analysis, a penetration test from a good company can provide more value than just finding vulnerabilities. The best testers act as much in the advisory role as the tester role.
Penetration testing will also fill your legal and regulatory compliance requirements, key for PCI DSS and the Australian Privacy Act among others.
The testing, however, will be a point in time. With constant changes to the environment along with modern agile CI/CD pipelines there can be new vulnerabilities being introduced and uncovered that might not have been there during the test. These vulnerabilities can be caught with annual testing, but that can leave a large window of opportunity for someone to use the vulnerabilities against you. Luckily, this pervasive testing is just what bug bounties excel at.
Why not both?
As you can see, each of these controls complement each other. Whenever there is a conversation arguing for one or the other, I always take it back to it not being an either-or discussion. Mature security organisations should both perform penetration testing and have bug bounties in place.
About the author
Matthew Strahan is Co-Founder and Managing Director at Volkis. He has over a decade of dedicated cyber security experience, including penetration testing, governance, compliance, incident response, technical security and risk management. You can catch him on Twitter and LinkedIn.
If you need help with your security,
get in touch with Volkis.
Follow us on Twitter and
LinkedIn