, , , , ,

Zabbix Templates for Security Analysts and Systems Administrators – EOY 2021

Kent Ickler //

Background

BHIS uses several tools for monitoring infrastructure. One of the most important tools for us that helps monitor systems health is Zabbix. It’s been a while since I went about creating Zabbix (https://www.zabbix.com/) monitoring templates. Long story short, I took a backseat role to Systems Administration a couple years ago when we hired DRock and later Napalm-Nick to kick butt on our systems team while I transitioned the majority of my time to penetration testing. I helped out the Systems team occasionally throughout the year, including a rebuild of our Zabbix monitoring platform.

The new Zabbix version that had been released since our previous deployment had some bugfixes and new features, especially with how the system captures JSON type data. In the course of the last year, we’ve deployed multiple new Zabbix instances. To help the Systems team monitor the difficult-to-monitor systems and services, I’ve created a handful of Zabbix templates that help BHIS keep an eye on things while we’re asleep or otherwise occupied with other projects. Not everyone will find these useful, but some of you may.

Other security firms using the same API technologies to help us augment data will find some of the templates useful for monitoring API usage, credit allowances, etc.

Here’s a quick rundown of new templates we’ve created as open-source and how you could use them in your own operations.

Interface Bandwidth Aggregates – Bandwidth-vnstat

Link: https://github.com/Relkci/Zabbix_Bandwidth-vnstat

This template uses the Linux tool, “vnstat,” to monitor traffic per interval, day, and month. Alerts can be configured on an interface if network bandwidth will breach certain configurable thresholds. Below are some sample graphs from the template and BHIS’s own www.blackhillsinfosec.com webserver.

FullContact API

Link: https://github.com/Relkci/Zabbix_FullContact_API

It’s not surprising that BHIS uses API service to augment their security research and penetration testing. FullContact (www.fullcontact.com) is a service that aggregates company user information and provides results in JSON format or via their website interface. BHIS created a Zabbix template to monitor the API credit allowance and usage of the FullContact platform.

Sendgrid Email Delivery

Link: https://github.com/Relkci/Zabbix_Sendgrid_Api_Provider

BHIS uses Sendgrid (www.sendgrid.com) for its delivery of some marketing-based emails. We also use it for things like alerting and some transactional-type emails. This Zabbix template monitors email-send credits, account information, spam reports, unsubscribe reports, IP reputation, and the like. Alerts are configured to let us know when we are short on credits or when administrative changes are made to the platform.

Here’s a screenshot of a sample graph showing the suppression over time:

Censys.IO API Monitor

Link: https://github.com/Relkci/Zabbix_Censys.io-API-Status

Much like FullContact, BHIS uses Censys.io to augment some of its data analysis. This Zabbix template monitors API usage and allowances and alerts us if our credits are approaching overages.

WPScan API Monitor

Link: https://github.com/Relkci/Zabbix_WPScanAPIStatus

WPScan is a tool that can perform analysis of WordPress based websites. It can enumerate plugins, users, themes, and even find vulnerabilities. A service related to WPScan is www.wpscan.com that can execute these scans via an API request. This Zabbix template monitors the use of the wpscan.com API account and lets us know if we approach our API allowance.

Shodan.IO API Monitor

Link: https://github.com/Relkci/Zabbix_Shodan-APIStatus

Shodan.io has been around for ages! Shodan is known for scanning the public internet and aggregating services it identifies. A service of Shodan.io allows organizations to monitor their own internet footprint, as well as do adhoc scans of the Shodan.io public internet database. This Zabbix template monitors the API usage of our Shodan.io account and lets us know if we’re running out of credits.

VirusTotal API Monitor

Link: https://github.com/Relkci/Zabbix_VirusTotalAPIStatus

There is a wealth of hunter and malware information at VirusTotal that can assist both red and blue teams be efficient in malware creation, as well as threat hunting and Incident Response. This Zabbix template monitors the VirusTotal API usage across a multitude of VirusTotal’s products. Alerts are configured to let us know if we are about to breach our API allowance and to let us know about account administrative changes.

MailChimp Marketing Monitor

Link: https://github.com/Relkci/Zabbix_MailChimpStats

Much like BHIS uses Sendgrid to send some transactional and marketing emails, we also use MailChimp for some of our more targeted emails to users that have subscribed to our mailing lists. This Zabbix template monitors campaign administration, email send statistics, bounce-backs, and account security. The Zabbix triggers alert us when they see a problem.

Here’s a couple of sample screenshots:

Augmented Geolocation with IPStack

Link: https://github.com/Relkci/Zabbix_GeoLocation-IPStack

Zabbix saves some information about a monitored host or service that can be used to augment information using other services. In this case, we use the public IP of a host with IPStack (www.ipstack.com) to augment geolocation data of the host itself. This allows us to then create groupings and aggregations of statistics based on the geographical locations of user servers, without having to worry about manually updating them. This template also monitors the IPStack account, API usage and lets us know if there are issues.

Here’s a couple of screenshots of the resulting augmented data:

Cloudflare Tunnel Metrics

Link: https://github.com/Relkci/Zabbix_Cloudflared

Cloudflare has some incredible products. We use some. One of their products is Cloudflare Argo Tunnels. The service is probably going to be renamed soon, but in short, you can think of it as a reverse proxy. Anyway, this Zabbix template monitors our Cloudflare Tunnels and lets us know when things don’t look healthy.

Here’s a few screenshots of this template:

Nessus Professional Monitor

Link: https://github.com/Relkci/Zabbix_Nessus-Professional_Monitoring

We use a variety of vulnerability scanners at BHIS. Tenable’s Nessus has been in the game for quite a while. This Zabbix template monitors the health of our Nessus deployments, as well as some operational and administrative information. An alert lets us know when problems are identified.

Here are some screenshots:

Security Scorecard

Link: https://github.com/Relkci/Zabbix_SecurityScorecard

Last, but not least, is a service that we got familiar with over the past year. I’m not posting this as a product announcement for Security Scorecard. CJ and I have gone rounds about their service and how it can be pretty awesome for Third-Party Risk Management departments. That aside, BHIS created a small deployment within the Security Scorecard service to monitor some of our vendor’s attention to security posture… or something. Let’s just say we’re undecided on how meaningful the data is. CJ already asked me for a webcast on this topic, so… someday. Anyway, this Zabbix template monitors the “Security Scorecard” grades and scores for your organizations, your vendors, and your competitors. The template will let you know when your score, or the percentage of your score compared to your competitors’, changes. It can also let you know if your vendors’ scores change.

These screenshots are pretty hard to snag without throwing out our vendors’ and competitors’ names (competitors are fine; we’re like extended family). Anyhow, here’s some of the generalized graphs:

Using Zabbix has allowed us to monitor several of our automated, operational, and administrative processes for both statistics and security oversight. We hope these templates might be of some use to you. To be alerted to future templates we build, subscribe notifications to this repository and you’ll be notified when we make updates. https://github.com/Relkci/Zabbix-Templates

References:

Want to learn more mad skills from the person who wrote this blog?

Check out this class from Kent and Jordan:

Defending the Enterprise

Available live/virtual and on-demand!