How to Not Suck at Reporting (or How to Write Great Pentesting Reports)

Reporting is a penetration testing topic that doesn’t have a whole lot of popularity. People have a hard time being inspired to write about the technical details of their engagements. In some cases, testers skim the surface only identifying the successful portions of their test. In others, testers just regurgitate the output from some scanning tool calling the results the final report.

However, sometimes you find a tester who creates a work of art.  It covers successful exploitation, the things that didn’t work, identifies the impact of a vulnerability in a way that the organization can understand, recommends corrective action, and does so in a manner that tells a story such that the organization can recreate the results or retest directly from the report itself. It is this type of report that we strive to create for every engagement here at BHIS.

The next few paragraphs will explain what I believe are the most important aspects of good reporting. I’ve put them in order of what I believe to be most critical to providing value to your customer.  So let’s go ahead and get started…

1.  The Scientific Method

All of the steps are critical to our success because they feed into the last step – share.

Most people will be familiar with the steps outlined by the scientific method; question, hypothesize, experiment, observe & record, analyze, and share results. Application of this methodology was beaten into me by engineering school as I was forced to write lab reports in this style at least once every week for four years.  When reporting on a penetration test, we can apply this method to each vulnerability that we find in an environment. Let’s break each step down:

• Question – Many of our questions are typically asked up front by a vulnerability scanner. However, we should be prepared to ask additional questions once we’ve processed the scanning results. Questions such as “What are the not-so-common listening ports on the network?”, “Where might I be able to use default credentials?”, or “What happens when…?”
• Hypothesize – Typically, this step involves interpreting scanning results or a response from an application or service that we’re interacting with. This can include selection of a tool or technique that we believe may be successful against a particular service or host.
• Experiment – This is execution of the actual exploit or tool that we’ve identified in the hypothesize step.
• Observe & Record – Here we record the outcome of our exploitative effort.
• Analyze – Did it work, did it fail, why? What can we learn from it? At this point we may change our hypothesis or identify a different approach or tool for the experiment.
• Share – Here we document the results of the process.

Share is the step that is most emphasized in engineering disciplines.  An experiment means nothing if the results cannot be independently verified.  Repeatability is what we strive for in our testing and a critical element is that our customers can independently validate our results.

2. Tell a Story

Each vulnerability that you investigate should have a story attached to it. You should attempt to answer the following questions for each vulnerability you investigate.

• What drew you to this element (vulnerability scanning results, an unidentified listening service, an uncommon open port)?
• What was the vulnerability?
• What impact does it have on the organization?
• Did you attempt to exploit it?
• How difficult was it to construct an exploit?
• What was the result of the exploit attempt?
• Can further exploitation be attained?
• What can the organization do to remediate the issue?
• What can the organization do to mitigate the issue?

Reporting usually has the competing priorities of full coverage and brevity. However, your report should include details on failed attempts at exploitation as much as successful ones.  This shows the target organization where defenses are working and helps them to understand your approach to executing the test.

3. Write as You Go

Instead of collecting artifacts and assembling the story once the test is complete, you should try to write the methodology section of your report as you go. This allows you to record your actions, take screen captures, and identify potential findings as a stream of consciousness. This account of your testing activity does not need to be perfect. However, it goes a long way in completing your report as it will only need a small amount of polish and full write-up of any findings identified within it.

4. Organization

There are several ways to approach organization within a report and none of them are wrong.  The easiest is likely chronological order. By recording as you test you ensure that all of the pertinent details are captured and that none of your testing needs to be accomplished twice. Since most scanning tools record vulnerabilities by a relative risk rating, it is likely that chronological order will coincide with a risk rating order as well. This ensures that the most important material is at the beginning of your methodology section.

However, it may be wise to add content to sections as they make sense.  Many high risk vulnerabilities will be identified in the “Informational” findings while exploring a vulnerability scan. Moving this information to the beginning of the report may make the most sense.  In addition, grouping similar vulnerabilities together may be appropriate. As an example, misconfigurations in TLS, SSH, and RDP services have a similar impact and so it makes sense to keep them in the same area of the report.

5. Illustration

Make sure to include many illustrative screenshots of the condition that you’re investigating, the tools used to exploit it, and the results of exploitation. Within each screenshot you should also zoom in on the point of interest and have indicators highlighting the action. A screenshot doesn’t help if your audience can’t figure out what it is trying to convey. However, a well thought out image of the situation can go a long way in helping to recreate results and driving home the impact of the activity.

As you’re putting these illustrations together, be cautious of the information that you are displaying.  In many cases, the elements of a test that drive business impact are sensitive in nature. Some may be obscure like password hashes, but others are more direct, like personally identifiable information or protected health information. As a penetration tester, we never know where our reports might end up, whose hands they will be in, or the motivation of the individual reading. As a result, we absolutely must make sure that graphics are properly redacted.

6. Voice and Tense

As much as possible, reports should use past tense. Since a penetration test is a point in time assessment, it is appropriate to identify the vulnerabilities that WERE present in the environment at the time of testing. This helps to remind the reader that changes may have occurred within the environment affecting the vulnerabilities identified within the report. These changes could result in degradation or improvement of the overall posture of the environment.

Third person is an almost universal requirement in official technical and engineering writing. Penetration test reporting is no exception.  As a result, you should avoid using first and second person pronouns such as I, we, and you. Instead, replace them with third person nouns and pronouns such as “testers”, “the tester,” he, she, him, her, and it.

Active versus passive voice has a long-standing flame war akin to “vi versus emacs” within the scientific community.  Some argue that active voice is more concise and clear. Others indicate that passive voice helps to avoid first person pronouns and stresses what was accomplished. Just like the question of your favorite editor, this is a personal choice that is up to you. A full treatment on the subject can be found at the following URL.

https://cgi.duke.edu/web/sciwriting/index.php?action=passive_voice

Whatever your choice, just make sure to use it consistently throughout your writing.

7. Consistency

Many penetration test reports don’t end up being authored by a single individual. Usually, just like the penetration tests themselves, the report is composed of separate sub-elements authored by several members of the testing team. Because of this, a single editor should review the report in its entirety to ensure that all of the individual sections apply the same tense, voice, and styles in a consistent manner. Otherwise, the report will feel fragmented when a customer reads it. The individual sections don’t have to be perfect mirrors of one another but they should strive to avoid wild contrasts that clash with one another.

6. Grammar and Spelling

Grammar and spelling may seem to be a bit nit picky. However, we operate in a field that stresses attention to detail. As such, we should demonstrate the same attention to detail in our reporting. Spell checkers are ubiquitous in nearly all applications so misspelled words are rarely forgiven. Some things you should always look carefully for are:

• Misused words – A correctly spelled word that has the wrong meaning given the context.
• Contractions – Don’t use them unless you have to but if you do, make sure that you’re using the right one – your and you’re are not the same thing.
• Acronyms – Make sure you expand them before first use in a report.
• Numbers – consistently use digits or spell out values based on their size or value.

One of the techniques that I use a lot is the old read it backwards trick. When you read a sentence you wrote you will tend to apply the meaning you intend and skip words. By reading backward, you avoid applying your own context and focus on the words that are on the page.

8. Fight for Feedback!!!

After you’re done reading your report and are satisfied with the content, give it to someone else on your team. Ensure that it gets reviewed for technical accuracy and for adherence to your reporting standard (voice, tense, person, grammar, and spelling).

At BHIS we employ a two-tiered review process that mirrors this recommendation. After a tester is happy with their offering, it is sent for peer review to determine technical accuracy. The report is returned to the original tester for edits. Some are incorporated and others discarded. We leave the decision up to the original author.

After the initial editing round is complete, the tester forwards the report to our very own grammar police. They make sure that a real human being (as opposed to a technophile tester) can read and understand the contents of the report. The original author once again accepts or rejects changes and then delivers to the customer.  This process helps to ensure that our reports are top-notch.

Conclusion

It is our responsibility to ensure that our customers can grasp the concepts illustrated in our penetration testing reports. Applying the techniques outlined above can help to ensure that we understand and satisfy that responsibility. Our reports are typically the only artifact that remains after the test is complete. They illustrate business value to the organizations that we test. By distinguishing ourselves through comprehensive and accurate reporting we can keep businesses coming back for service.

If you are interested in an excellent writing reference, check out “The Tongue and Quill.”  It is a very comprehensive writing style guide published by the US Air Force. I have had a copy on my bookshelf since 1993 and I reference it often.  A PDF file is available at:

http://static.e-publishing.af.mil/production/1/saf_cio_a6/publication/afh33-337/afh33-337.pdf