BHIS does a lot of outreach via our blog, HackNaked.TV, training, and especially webcasts. In the course of outreach, sometimes folks come to us whom never had a pentest, or interestingly, had a pentest and are unhappy with the results.

When probing customers about those unhappy results, many of those experiences seem to have two common elements: Black Box testing and/or more fundamentally, mismatched expectations.

Now Black Box testing definitely has its place. Examples might include:

Your new application has security reviews baked into the development cycle.
You perform your own internal pentests.
Your site has been pentested in a Grey or Crystal Box fashion recently.
You’re testing whether your SOC or Incident Response team is actually escalating issues.

However, if none of the above applies and you decide your very first pentest will be a Black Box test, you’re actually testing the pentester, and not testing the target.

The more cooperative the test, the more we’re spending time testing the application or target. The more Black Box the test, the more time the pentester will spend on discovery, guesswork, and exploration. Pentesters love the challenge, but here’s where the 2nd piece comes in: mismatched expectations. The problem is when the pentester misses an issue due to concentrating on an area of the application that the customer wasn’t really worried about. In the example of a network pentest with numerous hosts, the pentester will not know to focus on targets that may actually be more valuable to the customer or harbor more risk for the organization.

Another point that often comes up with Black Box testing regarding mismatched expectations are the Lessons Learned resulting from the report. Most engagements are a week. When the target list or application isn’t given the full scope it could have, the the results of the testing may not be representative of the actual risk of the application/target set for the other 360 days of the year. Customers are paying for our help in making them better and for bringing issues to the surface that may be exploited by a real attacker. It’s important to make the most effective use of that valuable time in helping the customer get better. A determined attacker will spend as long as it takes – why give real attackers an advantage by treating your yearly test as an adversarial engagement with little information?

For many of our customers, we recommend doing a hybrid test – Phase 1 will be a Black Box for a very defined time delimited phase, immediately followed by a meeting with the customer to obtain additional information for a cooperative test for Phase 2. This seems to be the best of both worlds and leads to a report that combines both testing styles.

So, if you’re considering a Black box test for your next engagement, be sure you decide ahead of time – what are you actually testing? A reputable penetration testing company will help you define scope items based off of your goals and provide you with tips to maximize the testing time and process.

Ready to learn more?

Level up your skills with affordable classes from Antisyphon!

Pay-What-You-Can Training

Available live/virtual and on-demand

Black Box testing – Are you testing the Pentester, or your target?