In my current smart contract auditing, I sometimes need help setting proper severity as it is more challenging than I remember from Web2 bug bounty or pentesting.
In Web2, we have primarily standardised issues such as XSS, CSRF, SSRF, SQLi, and IDORs. We can easily use the CVSS calculator or Bugcrowd VRT to set severity. Of course, the severity can sometimes be influenced by the triager’s/customer’s subjective perception.
The security in web3 is still at the beginning, and cause of this, you can notice that many guidelines on how to set the proper severity/validity of issues are still changing. Some problems you can see in Solodit 2 years ago are almost entirely invalid for most of the contests, connected with such massive development of EVM, solidity language and other technologies related to Smart contracts—for example, chainlink round completeness. That makes the standardisation of issues harder.
However, we can now see some progress in the standardisation as some low-hanging fruit issues are detected by bots, like using transfer instead of safeTransfer when dealing with ERC20 tokens. As we already know the implications of these issues, we can set proper severity and standardise them.
The problem here lies in the uniqueness of protocols followed by the bugs we can find. The implication of the same bug in the two protocols can differ. Example? Recently I read an article from @DevDacian about attacks on oracles. One issue described that turning off Oracle can DoS the protocol. However, the implication will be different if this situation influences liquidations, putting protocol into insolvency or if the failed oracle prevents users from withdrawing from some vault for a short period.
So the question is, how to set proper severity? The goal here is to understand the protocol you are testing. Understand all possible implications of that issue for the application, users etc. We can remember the instructions from platforms such as Immunefi, Sherlock or Code4Arena, but we still have to judge the severity related to the protocol. Anyway, if funds are lost -> issue.
Keep in mind all factors that influence attack vectors to exploit the issue you are submitting. Ask yourself these questions:
- Is it a privileged operation?
- Can I exploit the vulnerability anytime, or do I need to wait for specific conditions?
- Are funds lost?
- Can I break protocol functionality?
- Is it a developer mistake, and the protocol behaves differently based on it?
- How does it influence users?
Of course, there are many other questions. This is my personal checklist I ask myself when trying to set proper severity. But still, it is sometimes challenging. So if you have an issue, ask the protocol team during an audit what the impact is for them. Sometimes, they are so kind to discuss an issue and can help you set the proper severity.
Now we are in a state where we set proper severity based on these general questions and subjective recognition. But we always need some framework. It will take some time. After we gather enough bugs, we can divide them into categories related to attack vectors, application types etc. As for standardisation, you need perfect statistics, which require large data sets.
In conclusion, we are still in the early days of smart contracts and security around them. In the current situation, try to be fair with your bugs. Do not underestimate them but also do not overestimate them.
Sources
https://immunefi.com/severity-system/is your first post. Edit or delete it, then start writing!
https://docs.sherlock.xyz/audits/judging/judging
https://docs.code4rena.com/awarding/judging-criteria
Note: Originally published August 16, 2023, on Talfao’s blog.