How Logging Strategies Can Affect Cyber Investigations w/ Kiersten & James
This webcast was originally published on September 12, 2024.
In this video, Kirsten Gross and James Marrs discuss how logging strategies can affect cyber investigations, specifically focusing on Windows logs. They explore three different logging configurations for a Windows host and examine how these configurations impact the detection of various attacks. The video also introduces a tool called Audit Inspector, which helps manage logging strategies and ensures consistent deployment across an organization.
- The webinar focuses on logging strategies for Windows hosts and their impact on cyber investigations.
- Three different logging configurations are examined: default logging, BHIS Sysmon Config, and advanced audit policies.
- The tool ‘Audit Inspector’ helps manage logging strategies and ensure consistent deployment across an organization.
Highlights
Full Video
Transcript
James Marrs
So this is how logging strategies can affect cyber investigations. And in terms of Windows logs, we are not looking at anything other than Windows logs today.
Linux, Mac OS, too big of a beast for this particular webcast.
Kiersten Gross
but a lot of the same, let’s say investigative, strategies. a lot of the same principles associated with creating a logging strategy or determining if it’s sufficient for your organization applies across the board.
but specifically, we’ll be discussing Windows logs, just because some of you might not have been around when we were doing introductions for pre-show banter.
I’m Kirsten.
James Marrs
And I am. I’m James. We’re both soc analysts here for Bhis.
Kiersten Gross
we spend most of our time looking at Windows logs. So that’s, that’s one reason why we’re able to, primarily focus on these Windows logs.
we’ve, we’ve seen a lot and we’ve done a lot.
James Marrs
All right. so here’s just a brief overview of what we’re going to talk about today. Just first, just a quick introduction.
We’re going to talk about, three different, logging strategies or configurations for a Windows host. Then we’ll be looking at two different attacks and what those attacks look like with those, different logging configurations.
And then we’re going to be talking about a tool that Kirsten wrote, that kind of sparked the idea for this webcast that can help you manage, your logging strategies and manage how your logging is deployed on individual endpoints and make sure that it stays consistent across your organization.
And then of course we’ll have a conclusion and time for questions.
Kiersten Gross
we actually included a little introduction here. Aside from us being soc analysts, I, prefer to program in rust. I’m a little of that mad scientist.
I, generally want to test the limit and generally it’s Did it work? Well, then I don’t know why we wouldn’t do it.
James Marrs
Yeah, yeah, Kirsten is good with those, coming up with those crazy questions like, wow, I definitely never thought of that, so. And I am, I’m also a Soc analyst here.
I mostly learned c sharp and the Windows side of things for programming. Kirsten is lobbying for getting me into rust and I think it’s working slowly but surely.
I work on a bunch of CI cd automation for BHS as well. And yeah, I do enjoy some chess on the off side. But.
Kiersten Gross
Anyway, for our discussion, we stood up a lab and I. In this lab we have a Windows ten Pro machine and a Kali Linux machine, the Kali Linux machine we use to attack the Windows ten Pro.
There’s not going to be any Linux logs or anything that we’re going to be discussing associated with that machine, but we’ll be reviewing the logs associated with the Windows ten Pro.
We performed two attacks against the Windows ten machine to allow us to review logs and the differences associated with three different logging scenarios.
We took the default logging for that machine. We also had a scenario where we stood up the Bhis sysmon config and then we also have a scenario where we set up and configured what we recommend for advanced audit policies.
and we used the tool that James had suggested, or rather that James had introduced that I made that we call audit inspector.
so we’re going to discuss a little bit about audit inspector at the end, but the TLDR for this discussion is that it gives us the ability to measure what an audit policy looks like on a given individual host.
here’s an example of what that output would look like. It allows us to measure what the default logging was associated with that Windows ten pro machine.
As you can see, it measures success and failure configurations for advanced audit policies and a little Easter egg.
If anybody wants to look at something that’s humorous, pay attention to the account lockout and look up the Microsoft docs for that specific audit policy and figure out why that particular configuration is humorous.
But this is specifically how it was set up after doing the whole Windows, setup. This is the default logging on that particular Windows ten Pro straight out of.
James Marrs
The box, just installed from the ISO updates. This is what we got. Then we look at the enhanced logging, this is what you get when you run audit inspector and just give it the Bhis recommended settings.
As you can see, there’s a lot more interesting things being logged, notably process creation, and we can see some Powershell logging module there.
it’s quite a bit more inclusive than just the default logging, as you will see in a few minutes. And just this enhancement from the default Windows configuration lets us do so much more in terms of investigating interesting alerts or scenarios as opposed to just, as opposed to just looking at the default logs.
Kiersten Gross
And audit inspector also has the capability of measuring, a sysmon installation. In this case we’re able to see what the hash is associated with our config, what the file path as identified by Sysmon is.
We’re able to see what version of Sysmon is installed and the current status of the Sysmon service.
James Marrs
This is extremely, extremely helpful if you’re deploying, sysmon across a large organization or whatnot. This makes it so easy to see what exact, Sysmon config you’re running to make sure that all your endpoints are running the same, or if they’re running a specific one that you want.
it just shows the health of Sysmon as well. There have been many times where we’ve gone to an, investigate an alert and it’s like, well, is sysmon running on this?
Is it even the right config? We don’t really know. And so this was part of the, desire for this tool and one of the original ideas that Kirsten had for it so that we could track what is running what in terms of sysmon logging.
Kiersten Gross
So to overemphasize the point, these are the logging scenarios that we set up in the lab. So this is the default.
Now it’s stuck. This is the enhanced logging and then this is the sysmonite.
We performed this password spray against those three individual hosts, or rather those three individual logging scenarios.
James Marrs
Yep, just a simple password spray. A, few attempts we get the password. not a very good password, but you get the idea.
It’s good enough for our purposes anyways.
Kiersten Gross
So reviewing the actual logs that were generated by this separated, out according to which audit policy we were using, the default Windows policy was able to detect this password spray, but Sysmon provides no logging at all associated with this password spray.
There could be some ways to make, assumptions about what might be happening if you look at some of the sysmon logging, but it would be in special circumstances of an account that probably shouldn’t be on that host.
But if you’re trying to use a valid credential, chances are you’re not going to be able to see that using Sysmon logging.
James Marrs
Yeah, and you’ll notice for a password spray, default Windows login, enhanced Windows logging, it’s, they both, they both can see it, right? Just a bunch of 4625s followed by a 4624 successful login.
and that is that this really goes to show that, depending on what you are trying to catch or detect in your environment, not all logs are a, one size fits all.
some logging sources may be more useful to you than others. but this also goes to show that just because you have Sysmon doesn’t mean you can just ignore Windows logs because there are some crucial, attacks that can happen that Sysmon won’t pick up on, but the Windows logging will.
Kiersten Gross
We specifically chose this one, this attack where the default Windows did detect it, but we did that purposely. To also contrast our second attack of the default logging won’t detect attacks that aren’t found in the d, or rather using logs that aren’t found in the default policy.
Just because the default policy detected that attack doesn’t mean it will detect this next one. In this next attack, we performed a file download and renamed the file to cool Dragon wallpaper.
We purposely made sure that wallpaper was spelled wrong and gave it two file extensions, PNG and Exe. And then we executed that binary, on m the left side of the screen.
We’re hosting that file using Python so that we can download it to the Windows machine on the right, and then we’re going to execute it and it will open a new window.
This new window was generated by that binary, or rather spawned by that binary, and it gives us the ability to run as trusted installer.
Trusted installer is effectively a specialized system account that gives you the ability to do things that not even system can do.
James Marrs
And here we can see, some right away, some pretty stark differences between the three, different logging scenarios.
So for the default Windows logging, all we can see is that the system account logged in, that may or may not be normal, and honestly does not stand out too terribly much unless for a fact that no system account should ever be logging.
Kiersten Gross
On, in fact associated with this specific attack. We only know that that’s the logging associated with this. Because of the timestamp, there was other system logons and system associated behaviors related to services and other Windows behaviors.
So the only reason we’re able to identify those two logs as associated with this attack is because we knew that that’s when we ran it.
James Marrs
Yeah, it’s default Windows logging leaves a little bit to be desired. but you can see in the enhanced Windows logging we get a little bit more context as to what actually happened.
Right away we can see a 4688 created process event and the process name should immediately start triggering alarm bells.
Cool dragon wallpaper. PNG exe sounds like somebody, and I don’t know, maybe hr thought they were going to get a cool wallpaper for their desktop and got something nasty instead.
We can see the parent name is Powershell. Immediately after the process starts we see system logging in with the 4600, 24 event there.
And then trusted installer gets fired up. Even more than that, we see trusted installer spawning a Powershell, executable.
This would be a, in my opinion, a pretty strong indicator that hey, something really sketchy is going on, on this host and we should dig in a little further if we can.
because that Powershell exe, that trusted installer spawned, correct me if I’m wrong, kirsten, but that would have, the privileges associated with trusted installer, right?
Yep, yep. So you have basically super system on that host now. then we want to take a look at Sysmon.
This is where things really start to get juicy, so to speak. right away, the very first event we see is a Sysmon event.
Code three. Three is a network connection. We see Powershell connected to this IP address 172. that is our kali attacker, vm.
And so that right away is helpful because we can tell, like, all right, this Windows host went and talked to this unknown, host, shall we say.
And then we see directly after that a syspine id. Eleven and 29, those are both file creation association events.
we can see the file path, where the cool dragon wallpaper executable was written to. And then in the eid 29 we can also see the hashes of that file.
And that is super helpful to know. All right, is this actually a, like a wallpaper and just misnamed or is virustotal or something?
Recognizing this as mimikats with a different file name, you get the idea. These hashes are really helpful in terms of seeing what file is actually the user downloaded.
Kiersten Gross
Yep. And then when we are investigating an alert, especially one that was generated from an EDR or some order, some other auditing source, those don’t typically give us the context associated with this malicious download.
There are a variety of ways that using sysmon and the enhanced auditing configuration, we’re able to gain context associated with what happened in this circumstance.
We can see where it came from after we’ve identified that it’s doing something that we don’t like, and that gives us as the security team an idea of where we need to go and where we need to focus our attention to try to determine the extent of what’s happening here.
in this particular circumstance, because we know that trusted installer shouldn’t be running on this host to spot a Powershell process, we know that this is probably an incident where we need to get ir, involved if we were to, for example, not have trusted installer spotting Powershell, but, other types of circumstances that are, let’s say, not as clear cut.
For example, a member of the IT team is installing a VPN product that is not typical for the IT team.
Are they testing out a new system? Are they testing out a new product? Are they going to be connecting to a third party? For some reason, there’s a variety of different considerations to be made in that circumstance.
So all of the information that we can collect is helpful to determine whether we need to escalate to the people that can make the proper decisions of what needs to happen.
And we can provide the proper context to know where focuses need to start. And then on the flip side, using all of this data, we can also determine when something is a benign alert.
We’re alerting on something that typically looks malicious, but using this type of context we’re able to determine that it’s nothing bad.
James Marrs
all right, and so on this slide we go over some common attacks and what you can likely expect to detect based on what logging strategy you have deployed in your environment.
As you can see with the default policy there’s really not much. A password spray, sure, but malicious download, no.
Obviously we didn’t see the trusted installer escalation and then kerberoasting as rep roasting user account control. None of that stuff in our experience we have found gets picked up by the default Windows policy.
For the enhanced Windows default or enhanced Windows auditing we can see a bit more, and that’s mostly due to those four thousand six hundred and eighty eight s and occasional powershell script, block logging.
we saw the password spray, we saw the process creation for the cool dragon wallpaper PNG.
And you would in theory be able to see the living off the land binary attack as well, just because of those process creation events.
You could see oh, this binary is pulling this dll that it doesn’t normally use. We should maybe investigate that and then.
Kiersten Gross
Go sorry James, for completion. For the complete overall situation here, we, we admit that the enhanced policy applied to this specific machine will not detect things like kerberosing or as rep roasting.
And that’s because we configured that host specifically not to be domain controller. sorry, it doesn’t employ any domain controller logging.
That doesn’t mean that we can’t use enhanced auditing to detect it. if we were to use audit inspector to configure a domain controller, we could detect it.
There are policies that we recommend that will detect those behaviors specifically.
James Marrs
Yep, you can see in the asterisk down there where we have a link to the enhanced domain controller policy that Bhis recommends, for your domain controllers.
But then looking at the sysmon logs, we can see a fair amount more of like executable stuff happening on host stuff that’s really quite valuable to an investigation.
downloads. Where did files come from? That one is huge because those file creation events you can see. Exactly. All right, what host did we download it from?
What is the hash of the file? was there a weird DNS request ahead of this? There’s just so much information in context in using these sysmon logs that we get with.
Instead, we get a lot more context using Sysmon instead of using just Windows logs for those types of events. But as you can see, also like Sysmon, you’re not going to detect the password spray or kerber roasting or as rep roasting.
as Kirsten mentioned, those would be audit policies on your domain controller that would detect those and then disabling user account control that is also detectable, by Sysmon.
We can see events, where processes are modifying things that they shouldn’t or disabling registry strings that should not be disabled.
And I think that’s event id 13, isn’t it, Kirsten?
Kiersten Gross
yes.
James Marrs
Okay, sweet. And then low bus usage as well. So there really is quite a stark difference between what you get for logging in terms of versus, in sysmon versus enhanced versus the default policy.
Kiersten Gross
So effectively we want to use these labs as a illustration that your audit strategy needs to be based on what attacks you want to detect in this specific circumstance.
If we need to be able to monitor for malicious file downloads, if we need to monitor for attempts to escalate into trusted installer, if we have to monitor password spraying, if we have to monitor kerberosing, we need to have both the enhanced auditing policies applied to hosts, we need to have the enhanced audit policy attached to a domain controller, and we should deploy Sysmon in order to detect all of those specific attacks.
If we are reliant on a single individual one of those, we will have what is commonly referred to as a blind spot.
James Marrs
Yeah, mhm. Just to piggyback off of that, I think as Kirsten mentioned earlier, EDR logs are great. They can tell you, all right, something happened, we know something bad might have happened, but what happened before that?
What happened after that? What is going on in the context of this interesting event? And like we, it really is so critical to have those contextual logs because otherwise you’re just going to be kind of stuck in limbo, so to say.
Like, all right, carbon black identified this malicious executable. Quarantine killed it, great. But where did that come from?
How did it even get on your host in the first place? And did anything happen after that? Did that executable do what it was designed to do before it was detected?
And these are questions that these enhanced logging strategies can really, help you answer when you’re doing any kind of investigation into interesting events on your hosts.
Kiersten Gross
Yep. Even if the EDR blocks a given scenario. Right. running a malicious binary, even if it stops it, that doesn’t tell us if there’s an attacker in the environment.
It doesn’t tell us where. We need to go investigate to find out if there’s an attacker in the environment. If it’s a phishing download and it’s blocked, then it’s not as time sensitive.
But if there’s an actual attacker in the environment, and we say, oh, EDR blocked it, it’s okay, then we are essentially playing the ostrich.
We’re putting our head in the sand, just saying, eh, it’s okay. So even though the EDR is stopping inactivity, there is a lot of contextual information that we need to gain.
And that’s why, if we’re living or dying on an EDR, we don’t have the proper context to respond to.
Zach Hill
Hick.
Kiersten Gross
the attackers out of the environment.
James Marrs
Okay, so I think we might, we’ve beaten that horse pretty good by now. So it’s important to log more than just default Windows or just your EDR if you really want to get some good context into events.
This is where we talk about how to maintain your audit strategy. as you can see here, you might ask why. I set it.
I clicked the button. It’s configured great, but stuff happens, stuff breaks, all sorts of these. There’s all kinds of reasons why your, audit policies might differ or stray over time from what you originally set.
and all of these reasons on this slide here are pretty, I mean, we’ve encountered them in the wild before. As to why things aren’t logging as expected.
sure, there’s many, many more scenarios you could think of, but the big one there is, in my opinion, is malicious attackers. Malicious actors attack logging.
If you can, get on a host, kill sysmon, kill edr, you’ve essentially just turned that host into a dark, a dark horse in your environment, really.
And how are you going to know if that actually happened if you just stopped getting logs for this post. Well, great. Is it off? Did it disappear?
Did it get decommissioned? or is it just still running, but with logs tweaked, enabled? Or just logs set back to the default, Windows logging.
So that part of this is, where the, inception for audit inspector came from is the idea that we need a quick and easy way to maintain our audit strategies across the entire domain and to quickly and easily see what actually is being logged.
Kiersten Gross
Yep. we’ve noticed that in distributed environments that there are a lot of things that can break logging.
There are a lot of situations where in the soc we’ve been trying to prove a negative. If logs aren’t coming in, why aren’t they coming in?
as James said, is the host off? did they make an update that changed a little bit of how the logging structured?
If you’re familiar with azure logging, the schema associated with the logs changes all the time. So effectively we use audit inspector to determine an easy way of identifying if those configuration changes have, been manipulated.
We use it to check the health of sysmon. We use it to see if the sysmon service is running. We check it to see if the config file for Sysmon is universal across the environment.
If you have one host specifically that’s running a different sysmon configured, that’s definitely interesting if you’ve deployed it to your entire domain using the same infrastructure.
if you’ve performed a Sysmon update and some of your hosts got the update, there could be some troubleshooting that needs to be done to make sure that all of your hosts are updated.
Here we have the list of what we recommend in Windows environments, particularly associated with active directory, we recommend that you configure your advanced audit policies.
James Marrs
There.
Kiersten Gross
we have a link to the original publication that, bhis made several years ago associated with GPO deployments for audit configurations.
We will be updating that second repo in the coming weeks, if not months, to have a updated version of those gpos, but we do not have them effectively ready.
But we do use gpos that are a little bit more updated than what’s in that event. Logging repo the most up to date. If you want a list of what we recommend for advanced audit policies, you can find on that GitHub, page associated with audit inspector.
audit inspector not only allows us to monitor what is configured on a host, it allows us to modify them into what we desire on a host.
It’s a tool that allows us to push policies. Also, if gpos or active directory is not how your environment works, if you use an RMM tool to deploy software in your environment, this tool will help you accomplish that without needing gpos.
James Marrs
Yep. And then we have our Sysmon config there as well as we continuously, update and work on our sysmon config.
but you can see the OG, as Kirsten said, is linked there. it is a little out of date, but it is still effective.
You’re going to get a lot more information and context into what’s going on on your endpoints with that config. in the future we will be pushing a new up to date Sysmon config to that Windows, auditing repo on the Black Hills infosec GitHub page.
Stay tuned for that because that’s where all that good stuff is going to live in the future. But if you’re wanting to just get started or test things out in a lab, all of these links are a good place to start before deploying things to, your production environment.
And so you’ve heard us talk about audit Inspector quite a bit now. So what here, how do you use the thing?
Here’s the, where do I get it? Well, how do I use it? I’ll let Kirsten answer those questions since he wrote the thing.
Kiersten Gross
So you can download a binary build of audit inspector from our GitHub page.
We recommend that if you’re going to be using the install feature that your organization signs the binary so that that that’s the binary version that you have approved, that you’ve reviewed, that you’ve tested, and that works.
If you run audit inspector with no command line arguments, it will perform the default logging configuration, which means it will turn on all the logging that we suggest for a Windows host, specifically associated with an active directory environment.
Audit inspector was also built with the ability to determine if the host it’s running on is a domain controller, meaning it will configure the domain controller specific policies if it detects that your host is a domain controller.
That way you don’t have to worry about specifying any command line arguments for domain controller specific configurations. If you don’t want audit inspector to perform any sorts of configurations or changes to your host, you can use Z as your command line argument and it will only log the configuration that it detects on the host.
When you run the actual install behavior that I described, it will create a scheduled task on your host and it will copy itself into a, protected directory.
We recommend signing it specifically so that you can identify if there’s any sorts of manipulation going on in your environment. If someone attempts to replace audit inspector, although audit inspector does configure itself and the scheduled task associated with it, that only administrators can manipulate or edit them.
That doesn’t mean that once privilege escalation has already occurred, that someone isn’t going to attempt to replace the binary. And that’s why we recommend you sign it, to ensure that if someone replaces it, that you can both detect that change and that you can potentially prevent it from executing, a manipulated binary.
Every time that you run audit inspector, it will generate one of possibly two binaries or, sorry, event ids. twelve, three, four, five.
Completely original. And this will log the current logging configuration of the host. It will also generate a 12344, which will be specifically what changes audit inspector made to your host.
It will not perform any changes that it does not tell you that it performed. these are specifically intended to allow you to.
If you’re ingesting logs in a distributed environment, you can ingest them into your siem and easily reference all of the configurations associated with all your hosts.
You can use, data parsing and, graphical interfaces to identify which hosts might require troubleshooting or updates.
Or if, like we described before, if Sysmon has been updated on a host, if you need to troubleshoot because Sysmon updates failed, this gives you an easy way of identifying which hosts require the, attention.
Also, if the service is falling over Sysmon’s broken, it won’t record anything. So this gives you a good way of identifying which hosts have broken sysmon installations.
James Marrs
Yep. Can’t plug it enough. I love it. It makes our lives so much easier when troubleshooting things. Like, why aren’t we getting logs?
Or let’s just try out this configuration for a lab that’s, a quick, easy one liner and boom, you’re set.
So I think, is that, about question time, Kirsten?
Kiersten Gross
I, believe so. Let’s see.
James Marrs
All right, does this tool audit domain controller policies?
Zach Hill
Mhm, mhm, mhm.
Kiersten Gross
Going to consume a little bit too much memory if you’re going to basically, slow down a machine in production for a virtual environment where I would say if you are spinning up and spinning down machines consistently to where you have a brand new machine every day that you’re working, there’s no need for you to have all of that logging, because if that machine becomes popped there.
There’s not going to be any real persistence on that machine, although it might be helpful for identifying where, the attack came from.
But typically it’s not a good idea just to turn it all off.
James Marrs
Let’s see here, what elastic integration was used? Just the elastic agent, I think version eight, something along those lines.
How do the enhanced Windows logging and sysmon logging compared to the timeline logging that you get in Microsoft defender in a e five license?
If I have that in defender, do I still need these others? That is a fantastic question. and I will be honest and say I don’t know.
I have not, dug into the Microsoft defender logs all that much.
so, yeah, I do not know. So, that’s actually probably a good, topic for another webcast.
Kiersten Gross
typically I am, unsure if that product will necessarily ingest all the advanced audit Windows policies that we’re discussing, but if they’re not ingested by default, it would probably be helpful to turn them on simply from the fact that if there are attacks happening that affect memory or that are manipulating how a binary is operating.
For example, a in memory attack where you perform a command injection, having various different logging sources that have different mechanisms on the backside help you identify if that kind of thing is happening.
if one logging source tells you that a specific command was ran and a different one tells you on the same process that a different command line was used, that could be an indication that somebody’s attempting to bypass EDR and it will depend solely on how the logging resource is working on the backside on what specifically is logging to determine what command line was run.
So even if that product works really well, and it is a good detection engine and it collects a lot of logging associated with the host, all of the context that you might need after a detection is created could exist in these logs.
Zach Hill
Hey, guys, sorry about that. ran to the bathroom right when you guys were, finishing, I guess. Were you? We all going through the Q and A in zoom or you just answering?
James Marrs
Okay, yep, sure are.
Zach Hill
Where’d you leave off with?
James Marrs
Let’s see, just, talking about how, telemetry uploaded to, defender or e five licenses compares to, sysmon and enhanced logs is what, what we just discussed.
Zach Hill
All right, cool. Sorry, I was trying to find that one. There’s a quite a few different questions here, so try to go through and answer what we can. But thank you guys for sharing your knowledge today. It was very, very helpful.
James Marrs
Yeah, no, no problem. do you have an example of the output for audit inspector so on slides, let’s see here.
I think it’s six. Yeah, five, six and seven. That is the output from audit Inspector just outputs JSON.
Kiersten Gross
So it should be pretty. The first two are most, well, all of them are excerpts really. the thing that’s missing that you would find in an actual audit inspector log would be the basically audit inspector identifying itself as the source that created it.
That’s, that’s effectively the only difference between the sysmon version of it, of it in slide. What did you say the slide was?
James Marrs
It’s five, six and seven, I think.
Kiersten Gross
Right here. This is effectively what an audit inspector log would look like. the audit policy specifically refers to the advanced audit policies that are built into Windows but are generally not turned on.
the file, the process and the service values are associated with cis spawn. And then there would be another, agent reference to audit inspector that tells you that audit inspector created this log.
This is the version of audit inspector that was used, but effectively this is what it would look like. There should be an example found in the wiki of audit inspector on GitHub.
Zach Hill
Thank you, sir. I like this question here from Madison. When you guys have detected an attack, DC attackers running who am m I or other often flagged commands.
Kiersten Gross
No.
James Marrs
Yeah, no, I was trying to think for a minute. No.
Kiersten Gross
It’S very common in labs for them to use the who am I? As the specific event that we’re going to identify as.
Yep, that’s a malicious activity. But in the wild, most attackers, especially ones that are creating their own malicious binaries, are going to use Windows APIs and they’re going to make specific calls and they’re going to more manipulate their tokens than they are going to run.
Who am I?
Zach Hill
Kevin, says they’ve had the discussion of EDR versus Sysmon, as they’re wanting both, but they get asked what’s the same sort of question of what the overlap is. What would your response to that be?
James Marrs
Well, I would say your EDR is likely going to catch a malicious event, but it’s not going to look at what happened before, what led up to that malicious event, or after, if anything else happened afterwards.
We’ve seen before where EDR has quarantined killed a binary. But there’s still other sketchy stuff going on on the host or there’s still beacon activity going that EDR hasn’t caught.
Sysmon would provide some of that telemetry, in context to be able to detect those other types of activity going on, on that host.
So one does not exclude the other, I would say.
Kiersten Gross
Yeah, typically it’s not a good idea to run multiple edrs on the same host, but multiple logging sources are typically very helpful.
They provide us the context needed to confidently know or confidently have an idea of where we need to go, or if ir needs to get involved, or if it’s a phishing attempt that was foiled by EDR, or if somebody downloaded an old game that they thought wouldn’t hurt anything, or, it’s a Sysmon plays well with EDR, aside from the fact that some edrs don’t like what the sysmon config looks like.
So sometimes you have to exclude that from your EDR watch list.
Zach Hill
Thanks guys. And then with all the configurations you guys covered, are there any known gaps that you’re aware of that you wish you could cover?
Kiersten Gross
So from the perspective of Windows registry, if you use advanced audit policies to audit the registry, there’s no way for you to configure and maintain what registry values it’s going to generate logs for.
That’s one reason why by default we don’t recommend turning on the registry audit policy associated with the advanced audit policies.
We recommend using Sysmon because it allows you to configure what registry events are happening and which ones you’re going to audit, simply because the registry is potentially the noisiest thing that you could monitor.
In a perfect world, if we had unlimited resources and all hosts had as many core cpu’s as you would need, we would turn that on too.
Although that might create a needle in a haystack situation where we might blind ourselves. But if there was going to be a specific place that could potentially use a little more research or a little bit more effort in distinguishing between the two, it would be the registry advanced audit policies.
Zach Hill
with a managed EDR service, that probably doesn’t have sysmon context, what are they really providing?
Kiersten Gross
I would probably say they’re performing an alert engine.
James Marrs
Yep.
Kiersten Gross
alert engines are very helpful.
James Marrs
Mhm.
Kiersten Gross
Because it gives you the idea of where to start your investigation. But as soon as you find where to start, if that’s the only source for logging that you have, it’s kind of the equivalent of trying to use, I would say maybe a stick to build an airplane.
Zach Hill
Here’s a little bit of a longer, I don’t know, maybe you guys, did answer this question from rob about the telemetry upload to M 365.
Did you guys get that one before now? Sounds new.
James Marrs
I think so.
Kiersten Gross
Okay.
Zach Hill
just so everybody knows, I’m trying to go through the list, of questions that we have here in Zoom. So if you do have questions, we still have about another six minutes or so left, so go ahead and throw them in the chat. We’ll try to get to them if we can.
Otherwise, James and Kirsten, is there a good way to get ahold of you guys if people have further questions?
James Marrs
probably just at in the discord.
Zach Hill
Perfect.
James Marrs
Simple, easy.
Zach Hill
Kevin asks, in a midsize environment with a small security team, how are you managing the testing and deployments of sysmon to keep configs updated, but also catch performance hogs?
Hmm.
James Marrs
Mhm.
Kiersten Gross
So in the SoC, specifically, if you use active directory, typically it’s easier to perform a deployment using gpos.
We use a similar structure to what you’ll find in the event logging repo in GitHub. But there’s several updates and we’ll be pushing those updates to GitHub in the future.
If you don’t have active directory, there are likely going to be RMM tools or Powershell remoting of some sort.
And that’s one of the things that audit inspector aimed to identify to solve. Because if you’re not using gpos, modifying all of these audit policies is a very mundane and manual task, whereas if you can push a single binary to your individual hosts and execute it to perform that configuration, it is possible to automate the process, to a more universal approach.
So in this situation, audit inspector allows us to see the version of Sysmon that’s running. It allows us to see the Sysmon service is like in our example, 1515, and it’s very common for us to see where pushes of the service don’t perform the update as expected.
So if you’re running audit inspector, you’ll see, version discrepancies if, if, like a config wasn’t pushed properly, or if the update process failed.
But the audit inspector does not measure necessarily, performance on an individual host.
there isn’t anything in the project that will measure how Sysmon has impacted the host. I believe that in the time that we’ve been working with Sysmon, I think that there’s only been one or two instances where we’ve seen Sysmon actually impact the performance of the host.
And it had more to do with the version of Sysmon and what sysmons doing than, any of the surrounding context.
I’m unsure if that necessarily answers your question or not.
Zach Hill
James, do you have anything to add to that?
James Marrs
No, I was pretty much going to share, yeah. we can deploy it through gpos. If you don’t have ad, you’ll likely have some kind of RMM and then you could use audit inspector quite easily with that.
I would think so, yeah. Tldr on Kirsten’s answer.
Zach Hill
Perfect. Thank you, man. M Peter asks, with, those, event auditing repos, would you recommend those for both SoC monitoring and DFIR investigations, or are there more specific events that you think would be useful in a DFIR scenario versus the day to day logging for a SoC?
Kiersten Gross
d for logging and deeper investigations will definitely be a little bit more intense than these types of logs. As an example, you will likely dump memory, the contents of RAM for future investigations.
When you’re performing IR, when you’re doing SoC work, most of it is going to be metadata about what’s happening on the host. It’s not going to actually be a copy of the process that’s being ran.
So if, for example, a process injection attack happens, there might be indicators that we can see in that made metadata.
But for the most part, that’s one reason why an EDR is very helpful, is because it can see those types of memory manipulations. Whereas an IR team, they’re going to actually go onto the host and pull the contents of RAm so they can see how that process injection happened, what it was attempting to perform, if there were any successful attacks performed.
Whereas in the SoC, we’re not going to have the contents of Ram, we’re not going to be touching disk, we’re not going to be pulling virtually the whole host, if that makes sense.
So there is a little bit of difference in how, this logging helps in identifying, compromises or attacks.
But by the time that you’ve identified something or detected something using these logs, typically they’re helpful for context to know where to start or where to send ir to go pull those logs to go, pull ram to go pull all the other artifacts that would be needed to perform a thorough investigation.
Zach Hill
Awesome. Well, thank you guys. we’re at the top of the hour, so just want to say thank you again for being here, sharing your knowledge with us and everybody else. It’s very much appreciated. Do, you have any parting words you’d like to leave with everybody for the day?
James Marrs
thanks for coming, everybody. It’s a blast.
Zach Hill
Kirsten?
Kiersten Gross
I believe that, effectively, like we’ve said, we’ll be pushing more updates to those Windows logging repos, and we’ll probably be contributing more outside of the Windows realm.
But thanks for being part of the community, and we hope to provide more, assistance and knowledge.
Zach Hill
Awesome.
James Marrs
You guys will.
Zach Hill
We always do. We always do. That’s our goal. So thank you again for being here. for everybody else, thank you for, taking time out of your day, to be here with us. We appreciate you. We’ll be here again next week, same time, same place, and we hope to see you then.
So until next week, I hope you all have a, great week, great weekend. Take care, everybody. Ryan, kill a fire for sir. See everybody.