, , , , ,

How SPF, DKIM, and DMARC Authentication Works to Increase Inbox Penetration (Testing) Rates

Kent Ickler //

TL;DR Want a quick fix? 

Almost every marketing platform we’ve seen has decent tutorials on authorizing outbound email with SPF and DKIM authorization.

If you’re a marketing arm of your organization or are a mass marketing company and cherish the absolute brilliance of Sender Policy Framework (SPF), DomainKeys Identified Mail (DKIM), and Domain-based Message Authentication, Reporting and Conformance (DMARC): stop now.  This isn’t for you.  You already know what I have to say.

Bottom line:

This is subject to individual server configuration, but generally speaking, the more you can prove to a recipient mailserver that an email is valid, authorized, and intended, the more likely it is to reach an inbox.  

If you don’t know what I just said, prepare for a ride:  

Your mass marketing emails are hitting my inbox not because you bypassed our spam discernment, but because our clients’ mailservers are sometimes misconfigured and we don’t want to miss an important email from our customers.  At BHIS, some emails that otherwise would go to spam get sent into our inbox and “tagged” to remind our staff that extra scrutiny is warranted because something is amiss and we are calling the email’s origin into suspicion. 

Disclaimer: 

This blog uses the words “Authenticated”, “Authorized”, and “Vetted” somewhat loosely to represent the process and outcomes related to SPF, DKIM, and DMARC validation.  In this context the definitions are analogous and don’t align to how InfoSec would typically define them. Assume unauthenticated, unauthorized, unvetted, untrusted is not preferred.

Let’s Talk about email Authorization.

Back in the day, email was just trusted by everyone.  Remember when it was fun to send email from [email protected]?  Well… nevermind.

Do Hackers Phish?

Yes, hackers phish.

These days email has a problem: big-spam, phishing, malice, tom-foolery, etc. The consequence of the untrustworthiness of bad-intention is that today’s’ email services have migrated to a new default expectation that email should be vetted prior to delivery.  This is a good trend too, but it means emails now need to provide more information about how they were sent to prove their intent and authorization.

A new norm:

Untrusted email is not trustworthy.

Or 

Unauthorized email is not authorized.

Or…

Credentials please…

We’re not saying that unauthenticated, unvetted email disappears into oblivion. We are however saying that the authorization and providence an email can deliver itself with will increase its likelihood of successful delivery.

It’s become important for mail providers to provide a service that protects their end users from various bad-intent threats.  Not all emails are threats, but some are.  Distinguishing between them can be difficult even for humans. Protocols exist that can allow mail servers to validate the authenticity and authorization of email during delivery.  Email platforms have come to rely on those protocols for spam discernment, and it’s a good thing.  

In days past, it was less important for mass-marketing to worry about intent, authorization, or validation.  Emails would all deliver just fine.  Today:, not so.  Between threat intel feeds, deny-lists, and massive spam operations, marketers need a way to ensure they can inform delivery servers that their email isn’t spam, even if the email contains trigger words like “deposit”, “account number”, or “Junior Financial Secretary to the King of Algeria”.

This is where SPF, DKIM, and DMARC come into play.  SPF and DKIM platforms provide a mechanism for the registrant (Domain Administrator) of a domain to authorize (trust/authorize) which email servers are allowed to send email indicating their domain name in email headers (such as the From email address).  Additionally DMARC provides methods for enforcing SPF and DKIM and a mechanism for reporting compliance metrics and logging of authorized email activity.

Sender Policy Framework (SPF)

Sender Policy Framework (SPF), consists of a DNS record that an administrator can add to their domain’s zone file, and a protocol that instructs a recipient mail server to validate the originating mail server’s authorization to use the indicated domain name.

We’ve talked about this before:

Deep Dive:

DomainKey Identified Mail (DKIM)  

DKIM is a public key infrastructure that includes two keys.  The first key is entered in the domain name zone file for public access.  This key is then referenced by name in outbound emails by mail servers.  The second key (private) is stored on the email server and used to cryptographically sign hash computations of the email’s various components. 

DKIM compliant emails include a mail-header (meta-data) that provide the DKIM private key’s matching public key name, the email’s computed hash, the cryptographically signed hash, and instructions to process validate the signed hash.  The comparison of email hash and validation of crytophraphicaly signed hash determines the emails’ DKIM validation.

Deep Dive

Domain-based Message Authentication, Reporting and Conformance (DMARC)

DMARC  is a protocol that instructs recipient mail servers what to do if SPF and DKIM are absent or invalid.  DMARC also provides a method for recipient mail servers to report to the indicated domain owner about current email trends, including potential malicious activity.  This can effectively close the gap of an adversary sending unauthorized emails without the knowledge of the domain owner.

Deep Dive:

Ok, so what’s the deal?

If you’re using a mass-email platform and you didn’t go through the effort to authorize that email platform with SPF, DKIM, and create a ruleset in DMARC, the entirety of the inbox penetration rates will depend not on your marketing efforts, but on the configuration of the recipient mail servers to handle “unauthorized” email.  Unvetted email is, in fact, unvetted…   If you’ve spent the money to make an effective email campaign, Be sure you spent the time to ensure it will hit inboxes.

Marketeering hot tip:

Setup SPF, DKIM, and DMARC.

It will take you less than a half hour to do and your inbox penetration rate will go up.

What!? You want a Demo?

OK.

The Pathology of Email Authorization Validation

Here, I’ve sent an email from my BlackHillsInfoSec.com email address to my DefensiveOrigins.com email account.  On my Defensive Origins account, I have opened the full source of the email that includes the specific mail headers that have been appended by the various origin, transport, and destination mail servers.  

As Sent: 

As Received: 

The email headers that were included in the received email are where we start investigating how the email validation process played out.  Let’s start with SPF.

Sender Policy Framework Autopsy

SPF was around before DKIM.  It’s not absolute that a recipient mail server will first check SPF records, but it’s fair to assume that most recipient mail servers will validate a mail server for authorization according to SPF standards.  Let’s look at the email I sent myself.

Remember, this email is coming from blackhillsinfosec.com.

We can check BlackHillsInfoSec.com’s SPF record using MXToolbox or by querying DNS ourselves.  We will run the DNS query in a moment, but let’s check out MXToolbox’s ability to quickly decode the SPF syntax into common language.  MXToolbox is a great tool for quickly reading SPF records. 

The important bit here is the SPF include:_spf.google.com method in Black Hills’ SPF record.  The include method instructs the recipient mail server to query the SPF record of _spf.google.com and include it in the SPF record of blackhillsinfosec.com.   As we will see in a moment, the _spf.google.com record also has inclusions of more SPF records. Always take great care when adding an include method in an SPF record, you may be authorizing more than you think.  The list can grow pretty quick. 

The domain administrator of BlackHillsInfoSec.com expects (SPF authorizes, via DNS record) mail indicating blackhillsinfosec.com to be originated from the associated Google mail servers.  Did I mention that BlackHillsInfosec.com uses GSuite (Google Suite)?

In the email received by my DefensiveOrigins account, the recipient mail server has included information about the email server that originated the email.  The header below indicates the email was originated by the mail server referenced as mail-ed1-f53.google.com.

With a few DNS queries, we can track down if this is SPF validated for BlackHillsInfoSec.com according to the BlackHillsInfoSec.com SPF record.

The use of the include SPF method was used to validate the originating Google email server.  We manually validated that the email was sent via an SPF authorized server using “nslookup”.  The Defensive Origins mail server came to the same conclusion and documented the authorization result in an appended mail header in the received email named Received-SPF.  This header is defined in RFC 7208 [https://tools.ietf.org/html/rfc7208].

The recipient mail server also included a summary of the various validation checks in the mail header named “Authentication-Results”.  This is a summary of the authentication process. This header is defined in RFC 7601[https://tools.ietf.org/html/rfc7601] (made obsolete by RFC 8601[https://tools.ietf.org/html/rfc8601])

Congrats: 

SPF VALIDATION ACHIEVEMENT BADGE

Next up: Public Key Infrastructure and email authorization.

DKIM Signing Validation

Black Hills also has DKIM validation that authorizes mail servers by use of cryptographic signatures of (portions of) the contents of emails.  This allows recipient mail servers to validate specific email headers and email body content has not been tampered and verify email is authorized by the Domain Administrators of BlackHillsInfosec.com.

DKIM’s use of Public Key Infrastructure (PKI).

Mail administrators create a DKIM compliant key pair. A key (private) is stored on the email server and used to cryptographically sign computed hashes of specific email components.   The public key is posted in public DNS in a special subdomain of the indicated domain name.  It is not good practice to re-use keys-pairs for multiple servers. DKIM allows for use of multiple key-pairs.

When emails are sent with DKIM signing, the originating email server will include an email header that indicates which DNS DKIM public key to validate the email’s cryptographically signed hashes.  The selection of the DKIM public key is called the “selector”.   The selector instructs the recipient mail server which DNS record to query to obtain the DKIM public key. The DKIM public key is procured by a  DNS TXT query of [selector]._domainkey.domain.tld.  The public key found on the DNS TXT record is used to validate the cryptographic signature included in the DKIM authorized email header.  The comparison of the computed email hash and validation of the cryptographic signature determines if the email is DKIM authorized.

In the email I sent from BlackHillsInfosec.com, the originating email server had included the DKIM-Signature mail header.  This header is defined in RFC 6376 [https://tools.ietf.org/html/rfc6376] and includes the cryptographically signed hash as well as instructions for the recipient mail server to create the hash and validate the signature.

The above email header indicates that we are using:

  • v=1 Version one of DKIM protocol
  • a= Hash Algorithm
  • c= Hash input method (accounts for header modification in transit)
  • s= Public Key selector 
  • h= Headers that have been signed in b= 
  • bh = hash of body part(s) (according to “c=”) [base64]
  • b= Cryptographically signed hash of body/headers 

The originating email server used its DKIM private key to cryptographically sign the hash values for the body (bh=[hash]) and store the signed hash (b=hash/signature).  This process is detailed in RFC-6376 Section 3.7 but ultimately the output is a signature that can be validated with the DKIM public key.  

Upon the recipient email server analyzing the DKIM mail header it began its DKIM validation process. It used the information from parameters “a=”, “c”, “and “h=” to compute a hash of the email components.  The hash was then compared to the value provided in parameter “bh”.  If the hashes matched, the validation process continued.  If the hashes did not match, the DKIM validation has failed.

The recipient mail server then queried DNS to retrieve the DKIM public key by using the indicated domain and selector specified in the DKIM header.

  • d=blackhillsinfosec.com
  • s=google
[CODE]
# dig google._domainkey.blackhillsinfosec.com txt
; <<>> DiG 9.11.3 <<>> google._domainkey.blackhillsinfosec.com txt
;; ANSWER SECTION:
google._domainkey.blackhillsinfosec.com. 1800 IN TXT "v=DKIM1; k=rsa; 
p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBi  
QKBgQCVxOyLCpVSrOzHZxkQciLBVhOkDYeMoM0A9A/NVUERAgXA7HDnP8c63tCrq8aKOv
KwcTAXF1EV+BSNQcneCmG/QX9oWPuTBvkX2J        
wG0oRt/oP155KcJBbdczO9mmOwnFxTgiIiuF6oYT92cvcNT6zUh4QtvFmypMkIGrEZeGF9oQI
DAQAB"
[/CODE]

The server then validated the signed hash (=b) using the DKIM public key. The cryptographic signature is validated using the public-key and the email is considered DKIM authorized.  If the signed hash could not be validated or the public key could not be retrieved, the email would have failed DKIM authorized. 

After the DKIM validation process, the recipient mail server included in DKIM validation results in the Authentication-Results header. 

Congrats!

DKIM VALIDATION ACHIEVEMENT.

At this point in validation, we have confirmed that the email was sent via an SPF authorized email system and that the email used DKIM signatures to validate delivery and specific unmodified content between the originating mail server, the transport server, and the delivery server.  But, what happened if that wasn’t the case?  That’s where DMARC comes in!

Question Time!

  • What happens if an email includes a DKIM signature but fails DKIM validation?  
  • What happens if SPF authenticates an email server, but the DKIM signature was missing?  
  • What if the DKIM signature is validated, but SPF indicates the mail server wasn’t authorized?  
  • Can an administrator be notified if someone is sending authorized email?
  • Can an administrator be notified if someone is sending unauthorized email?

All of these questions and related problems are solved by the use of DMARC.  Much like SPF and DKIM, DMARC is a DNS TXT record assigned to the domain zone file of an indicated email domain.  The DMARC record instructs recipient mail servers how received email should be processed based on the results of the SPF and DKIM protocols and additional instructions to create a feedback loop notifying an administrator of the indicated domain’s email use.

Our example email was sent from BlackHillsInfosec.com.  The recipient mail server queried the BlackHillsInfoSec.com domain for its DMARC DNS record.  The DNS record instructed the recipient mail server how to process the email based on its SPF and DKIM results.

The DMARC DNS record is assigned to a special subdomain named _dmarc.  In our case the recipient mail server queried TXT record for _dmarc.blackhillsinfosec.com.

[CODE]
C:\>nslookup -type=txt _dmarc.blackhillsinfosec.com
Non-authoritative answer:
_dmarc.blackhillsinfosec.com    text =

    "v=DMARC1; p=none; rua=mailto:[email protected];
ruf=mailto:[email protected]; fo=1"
[/CODE]

The BHIS Dmarc record above instructs mail servers that the txt record is a valid DMARC txt record.  The BHIS record instructs the recipient mail server to process the email according to this DMARC configuration:

  • v=DMARC1 DMVARC v1 valid TXT record
  • p= none if email fails validation, do nothing additional
  • Rua = Send aggregate mail data o [email protected]
  • Ruf = Send failure forensic reports to [email protected]
  • fo=1 Send forensic upon SPF or DKIM not-pass

The email having been successfully SPF and DKIM validated is processed according to the DMARC ruleset and is delivered to the intended recipient. However, not without being part of an aggregate report sent in compliance with the DMARC record.

Congrats!

DMARC VALIDATION BADGE!

About those DMARC reports

You might be asking a question:

Just how many emails go to those notification addresses we specified in the DMARC record?  Well, in regards to DMARC compliant mail servers, we get:

  • 1 aggregate email per recipient-mail-server per indicated-domain per day
  • 1 forensic report per unauthorized email delivery.

The math here is effectively the number of domains that BHIS sends email to on a daily basis + emails that BHIS doesn’t send to domains that received email indicating BHIS as the origin domain + forensic reports of authorized email.  It’s worth noting that not all recipient mail servers send forensic reports of unauthorized email, but typically will include the failure result in its aggregate report email.

You might be thinking… isn’t that a lot of emails about emails?  You guys actually read those?

Yes, kinda. It is a huge amount of emails that we never read. It’s all automated.  

More on that next time when we ask  “Hey, what’s that spike of invalid mails sent?”

All in all, that’s a long story short about a 1 line email.

Now, get those emails in my inbox (without getting tagged) and start making money 🙂

Post-Blog Banter: Why this blog exists

Every day I get emails from various companies from industries trying to sell their products.  At BHIS we’ve grown accustomed to non-traditional marketing, focusing on client and community relationships over mass-emails.  But we’re not dogging– we don’t run our business like others — our atypical strategies aren’t for everyone and they don’t come without its own type of painful stories and problems.  

Traditional marketing is still important and when we see our competitors (friends, actually) making a potential pricey misstep, we want to help.  

At BHIS, we have a mantra: “Proudly Sucking at Capitalism.”  

Sometimes doing the RIGHT thing isn’t profitable, at all.  

This blog is probably one of those cases. It certainly didn’t take five minutes to write this blog, editors were assigned, peer-reviewers recommended modifications,  someone fixed my tYpos, someone uploaded and published it, and someone else may have notified you.  Helping our competitors’ (friends’) marketing endeavors probably isn’t in the best interest of the BHIS’ bottom line, but it is in our other interests:  We seriously do want to make it (the world) better, and if helping you get your valid idea out there helps… well… good.  

Franky, I’m saddened that I’ve grown fatigued by the number of emails for X product or Y service that we “Tag” as being “suspicious” not on merit of the product or service, but merely on the fact the marketing campaign sent emails that were considered unauthorized.  The result can be costly.  Our visibility in the lack of authorization is not why we built the email-tagging system, it’s just a by-product.  In fact, if we didn’t have that tagging system, I wouldn’t even know I received the email because it would have hit the spam black hole never to be seen except when hunting for an unrelated missing email.

Make your ideas heard.  Make the world a better place.  

Go forth and do good things.

Want to learn more mad skills from the person who wrote this blog?

Check out this class from Kent and Jordan:

Defending the Enterprise

Available live/virtual and on-demand!