Security-Focused Review Guide

Background

HackerOne offers a limited scope version of PullRequest dedicated to code security scrutiny. Certain customers require that feedback escalated to their development team is security or security-adjacent in nature. This solution aims to address the growing need for developer-friendly security tools, providing on-demand security expertise to accelerate development cycles and reduce vulnerabilities before production. This enhances HackerOne's continuous vulnerability testing framework by adding a robust code review component, ensuring a more comprehensive security posture from development through production.

Customer Goals

• Avoid alert fatigue from noisy code scanners and AI. Human experts will raise only the concerns that matter, reducing the chance for false positives while also adding a layer of scrutiny before Merge.

• Give developers access to the security context they need to ship secure code, faster—without the false positives and bottlenecks.

Reviewer Expectations

• Keep feedback related to security concerns (these could be direct or indirect)

• Raise feedback only on valid issues or bring attention to issues that could be valid, but require some clarification from the development team to determine validity

• Focus on areas highlighted by our security automation and AI hot spots (See next section for more information on these), only raising issues that are valid (to the best of your ability based on context available)

• Focus on other areas not highlighted by our automation that deserve additional scrutiny per your expert judgement

How to Complete a Security-Focused Review

The reviewer interface for this type of review has been designed to ensure that the reviewers engaged have the tools they need to do the job more easily and with the right expectations. The reviewer experience for security-specific reviews is different from our more generalized code review offering that reviewers may be accustomed to. Completing a review can be done with the following three steps in mind.

Step 1: Verify Automated Findings

Review automated results raised by various automated SAST tools provided. The Automation Results Checklist allows reviewers to skip directly to each automated security rule triggered.

Security Automation Checklist example

For any automated results that are found to be invalid or false positives, ignore them. Those that are found to be valid should be escalated using the Convert to a comment button. If you are unsure of the validity of an automated finding or you are missing some relevant context, raise the finding with a clarifying question and provide as much information as to why you think it could be a valid risk for the code author to consider.

SAST example

Review any code specifically outlined in pink. Our AI engine attempts to find risky areas of code and highlights sections to draw attention to them. Each finding highlighted in pink should come with an explainer about what the code was flagged and the ability to convert that explainer to a comment. As with other automated findings, the expectation should be for reviewers to only raise issues related to verified risk using the power of human experts! To report any valid findings here, simply click the icon and edit the to ensure the feedback is accurate and helpful.

Converting an AI finding to an inline comment

In the end, you bring subject matter expertise to review our customer's code. If you feel there are other areas that deserve additional scrutiny in a security context, we encourage going outside of the automation. In the end, this should be a faster and more focused review.

Step 2: Submit the Review

Once the review is complete, make sure to submit it using the Review button in the top right of the review window. Submitting a summary comment is optional and should not be used if nothing of note was found. If something was discovered during the review, feel free to expand on it and leave a summary of one or multiple pieces of feedback found.

Submit Review

Step 3: Follow Up Reviews

Any new commits related to security feedback raised will receive another review. If it makes sense, use the relevant comment thread to engage further to suggest additional changes or indicate that changes made will resolve the issue. This process will continue until the change is ultimately merged or closed by the customer.