Tap Into Your Valuable DNS Data

Joff Thyer //

The Domain Name System (DNS) is the single most important protocol on the Internet. The distributed architecture of DNS name servers and resolvers has resulted in a resilient and highly scalable system that is largely unchanged from the early days of NSFNET. It is a unique solution that allows domain holders to manage their own zone data while still being linked into the global hierarchy of distribution Internet domains across the world.  

In the security context, knowing what our client systems are looking up in the DNS yields valuable information about normal versus abnormal operations in any environment.   

Nearly all organizations will run recursive DNS servers within their environments and almost all communications that your internal client stations engage in are likely to be preceded by one or more DNS lookup requests.   

Logging these requests and responses must be viewed as a necessity as it represents a valuable source of data for both security and operational analysis.

Having said that, logging the data is where things get a little tricky. The challenge is this, most DNS name servers for even a modest-sized organization are subject to considerable network load. They are constantly bombarded with requests and are rightfully implemented as multi-threaded high-performance code.

An internal Microsoft domain controller name server has a feature to “enable debug logging” however it is not recommended to keep it enabled. The reason is that the continuous file I/O operations to write out the log data will slow things down. Any disk write operation is subject to kernel buffers, and associated wait time for the I/O buffers to be flushed to disk.  

The Internet Software Consortium’s BIND server has had a logging feature within the configuration for some time, however, the same issue applies. Enabling this feature will slow things down as the server writes out logging information to disk.

For a small network with about a half dozen endpoints (like my network at home), you probably don’t really care or would even notice the penalty of logging DNS requests with synchronous file I/O. In larger environments, enabling such logging features will impact overall server performance to the detriment of internal users. Furthermore, within the DNS code threads, if there is a choice between servicing a new DNS request, versus writing buffered log information to disk, the DNS requests will likely be serviced as a priority and log entries may well be lost.

One possible solution is to deploy additional systems, such as a Bro sensor or full packet capture solution to listen to network data. This will typically involve mirroring data with a network SPAN port configuration or dedicated optical tap upstream of the DNS server. Clearly, such a solution involves additional expense and complexity.  Why not just collect the data directly from the source DNS server?

DNSTAP

DNSTAP, proposed by Farsight Security solves the I/O performance bottleneck yielding valuable logging information directly from the DNS server itself. DNSTAP logs query and response information in a flexible binary structured log format using Google’s protocol buffer implementation.

The architecture diagram from http://dnstap.info gives a sense of the logical components of DNSTAP. In short, there is an encoding of a copy of the DNS data itself which is then serialized to a reliable byte stream by the sender (DNS server). The resulting serialized data can be read by receiver software and logged to file if needed.

By separating the DNSTAP byte stream I/O from the DNS server operation itself solves the slow logging I/O problem and yields a great deal of granularity given that the resulting data is the actual DNS requests and responses themselves.

DNSTAP is available for Bind, Unbound, and the Knot server implementations. For the remainder of this blog, I am going to focus on the BIND server implementation.

After listening to Paul Vixie speak about DNSTAP at Wild West Hackin’ Fest, and after some additional research, I decided to enable DNSTAP in my home office network. In my case, I am running an Ubuntu 18.04 based router gateway, and DNS server.

The first discovery I made was that the latest Ubuntu 18.04 patched distribution of BIND did not have the DNSTAP code compiled into the distribution. This is going to require manually building the ISC BIND server from source.

Before compiling it, an Ubuntu system with the appropriate source code build tools installed will be needed. Preferably this should not be the target system for final installation!

sudo apt update
sudo apt install build-essential libtool autoconf automake libssl-dev

Additionally, DNSTAP depends upon Google protocol buffers, the protocol file compiler, and the related frame stream tools. These items are actually available from the Ubuntu repositories although you might choose to compile them for the latest/greatest code. Using the Ubuntu repositories, I installed these as follows:

sudo apt install libprotobuf-c-dev libprotobuf-c1
sudo apt install protobuf-c-compiler
sudo apt install fstrm-bin libfstrm0 libfstrm-dev libfstrm0-dbg

Although the frame stream tool is available from Ubuntu’s repositories if you want the latest software you might consider visiting Farsight’s GitHub repository and cloning the source from there for building.

git clone https://github.com/farsightsec/fstrm.git
cd fstrm
./autogen.sh
./configure
make
sudo make install

Once you have the dependencies installed, you should download the latest source code for BIND. I chose Bind 9.14 as it has the code for DNSTAP included and considered a current/stable release as of today’s date.

wget ftp://ftp.isc.org/isc/bind9/cur/9.14/bind-9.14.2.tar.gz
tar -xzf bind-9.14.2.tar.gz
cd bind-9.14.2
./configure --enable-dnstap --sysconfdir=/etc/bind --localstatedir=/ 
--enable-threads --enable-largefile --with-libtool --enable-shared 
--enable-static --with-gnu-ld --enable-dnsrps --prefix=/usr/local
make
sudo make install

At this point, you would probably be wise to remove the Ubuntu BIND distribution so that there is no confusion as to which version of the “named” binary you are using for DNS.

sudo dpkg --remove bind9

Capturing the Frame Stream

For optimal results, you want your BIND name server to write its logging information to a UNIX socket and then use the frame stream capture program to read that socket and create a log file in the binary protocol buffers format. Subsequently, you can use a program called “dnstap-read”, included in the BIND distribution to read the binary format and produce readable output as needed.

It is important to start the frame stream capture process before BIND so that the UNIX socket will be created for you. I opted to first experiment with the command-line options, and then ultimately created a systemd service to ensure that the “fstrm_capture” binary was always started before BIND. My available logging disk space was limited so I decided to rotate the output logging file on a daily basis using 86,400 seconds as the rotate/split parameter.

Systemd Configuration File saved as “/lib/systemd/system/framestream.service”

As can be seen above, when this service is started the UNIX socket named “dnstap.sock” is created in the “/var/cache/bind” directory.

Configuring BIND

The BIND configuration is very straight forward and requires only two lines of configuration in the “options” section of the name server configuration file. It should be noted that DNSTAP configuration options include different message types of data that can be logged. In each case, you can optionally specify “query” or “response”.  If you don’t specify “query” or “response” then both are logged by default.

The message types are:

• Auth
  • Query received from a resolver by an authoritative name server. Response sent will be from an authoritative name server. These queries/responses from the perspective of the authoritative name server.
• Client
  • Query sent from a client (or stub resolver) to a name server. The name server is expected to perform a recursive lookup and send the response back to the client.
• Resolver
  • Query sent from another resolver to an authoritative name server from the perspective of the resolver. Response message received from an authoritative name server by a resolver from the perspective of the resolver.
• Forwarder
  • DNS traffic sent/received and forward to/from downstream and upstream DNS servers.

I decided for my configuration to focus on both client and authoritative traffic. Including the resolver message type can be a little noisy. How you approach the configuration really depends upon your goals. My BIND configuration file “options” section is shown below for reference.

Read and Analyze the Logging Data

The ISC BIND distribution includes a program called “dnstap-read” which allows the user to read data from the binary structured protocol buffer file, and print it to screen. There are several useful options that can be used with “dnstap-read”.

If you don’t specify any options, then “dnstap-read” will simply read the current log and print all of the associated requests/responses in the log along with an additional code indicating what sort of message type is logged. Message types as extracted from the protocol buffer definition file can be:

• AUTH_QUERY (AQ)
• AUTH_RESPONSE (AR)
• RESOLVER_QUERY (RQ)
• RESOLVER_RESPONSE (RR)
• CLIENT_QUERY (CQ)
• CLIENT_RESPONSE (CR)
• FORWARDER_QUERY (FQ)
• FORWARDER_RESPONSE (FR)
• STUB_QUERY (SQ)
• STUB_RESPONSE (SR)
• TOOL_QUERY (TQ)
• TOOL_RESPONSE (TR)

Sample Output from “dnstap-read”

For extra detail, you can print the full DNS message using “-p” which yields a “dig” like output with all of the expected detail. Both a YAML (-y), and hexadecimal dump (-x) form of the data is also available. In all cases, the original single line output, as shown above, is also included.

Sample Output using the “-p” Flag of Single Query/Response

Since reading and learning all about DNSTAP, I have taken the extra step of starting a Python3 implementation that can parse the frame stream log forward, read the data in much the same way as “dnstap-read” but additionally produce some statistics. You can find my work here at https://github.com/yoda66/DNSTAP-FrameStream-Python.

All credit due to Paul Vixie and the Farsight Security team for continuing to spread the word about DNSTAP with the ability to collect valuable DNS data in a scalable fashion for further analysis.

References:

• https://www.youtube.com/watch?reload=9&v=OxFFTxJv1L4
• https://www.vanimpe.eu/2018/12/27/dnstap-for-improved-dns-logging-on-ubuntu/
• http://dnstap.info/Tutorials/NANOG60/
• https://github.com/farsightsec
• https://github.com/yoda66/DNSTAP-FrameStream-Python