In this post, we are going to review how to detect malware remnants during an incident response investigation.
A fundamental step in any DFIR triage scenario is to determine if there is any form of basic or advanced malware. In the last two posts in my “Intro to IR” series, we looked at how to detect malware starting and running.
Now, we are going to focus on looking for remnants of malware during incident response after it has stopped running. We’ll complete the post with a quick tour of how to use incident response software Cyber Triage to analyze malware-related data.
30-Second Series Refresher
Let’s review the main concepts of this “Intro to IR” series before we dive into the details of this post.
This series is about both a framework for approaching investigations and specific techniques. Digital investigations are about answering questions, and we’ve been breaking up the general “Is this computer compromised?” triage question into three (still big) questions:
- Is there suspicious user activity (such as sensitive files being accessed)?
- Are there malicious programs (such as malware)?
- Are there malicious system changes (such as relaxed security settings)?
And each of those three questions is broken into smaller questions. The process repeats until you get to questions that can be answered with data.
We are now at post #8 in the series and looking at bullet #2 and malicious programs.
In post #6, we broke up our question of “Are there malicious programs on this computer?” into three smaller questions:
- Are there suspicious programs that start based on some trigger (such as when the computer starts, a user logins, scheduled task, etc.)?
- Are there suspicious processes currently running?
- Are there remnants of past executions of known malicious programs?
In other words, we’re going to look at what programs get started, what programs are currently running, and known traces we can find.
We already covered the persistence/startup topic and running malware. We’re now looking for traces of past executions.
Why We Look for Malware Remnants During Incident Response Investigations
We’ve already covered a lot of ways to detect malware, right? So, why do we need more?
There are two main reasons:
- Removal: The malware may have been removed, and the past two categories relied on looking at places that malware starts or while it’s running. Those won’t find things if the malware has been removed. This can happen when the bad guy decides to abandon a host or upgrade his technology but doesn’t want to leave anything behind that may alert the security team to the compromise
- Stealthy: Malware authors are constantly trying to find the most stealthy ways of getting their malware to execute and avoid detection. There is always the risk that your attacker is using a new technique that the previously mentioned techniques do not immediately detect. Such as:
- There is another computer in your enterprise that remotely triggers the malware to periodically run on the endpoint you’re triaging. In this situation, the persistence mechanism is not on this host, and you may not have seen the process run during your collection
- The attacker is using a new persistence mechanism on this host. There are dozens of startup places and maybe they found a new one that no one else is looking at.
It’s always good to look in as many places as possible during endpoint triage so that you find some quick hits before diving too deep into a certain area.
What Malware Remnants Incident Response Professionals Should Know
We can’t answer the question of “Are there remnants of past executions of known malicious programs?” because we haven’t defined what a remnant is yet.
Let’s look at the life cycle of a process (refer to this post for basics on processes):
- It starts and loads libraries
- It runs and “does stuff”
- It shuts down.
So, as a first step in looking for remnants, let’s look at each of these steps.
Startup Remnants
When a process is started, it may cause Windows to create various registry keys and event log entries (depending on settings). With the exception of process logging, many of the data is stored as a benefit for Windows so that it can preload libraries and not waste time in the future.
These locations have names such as:
- Prefetch
- UserAssist
- MUICache.
We covered this topic in a previous post that was focused on investigating what programs a user executed. We won’t cover it again, but you should refer back to the post on investigating user activity if you haven’t read it yet.
Process Running Remnants
Malware can do a lot of stuff when it’s running.
But, there are a few things that we can focus on when we look for remnants:
- Indicator of Compromise (IOC) files or keys: Malware may make files, folders, or registry values to store configuration or data it intercepts. Sometimes, the malware cleanup process misses some of these files or keys. If you have a threat intel feed that provides paths or keys of known malware, then it may give you a clue that something previously ran
- Unallocated Memory: If the process stopped running, but the computer has not rebooted, then you may find remnants of it in unused memory pages. When a process stops running, the memory pages it was using are marked as “free” so that another process can use them. But, until that happens, they contain the data from the previous process. A signature scan of memory may alert you to this
- Network Caches: When malware is running, it may reach out to a server to get commands and instructions. When it does that, it may know only the hostname of the server and, therefore, Windows needs to resolve the hostname to an IP address. Windows will cache this resolution in case it is needed again. It may show that some process reached out to a malicious domain. Note that DNS cache is not associated with a process, so this data alone does not allow you to trace back to anything more specific.
Shutdown Remnants
Generally, there isn’t much evidence left behind when a process shuts down.
But, there are two exceptions:
- Auditing: You can configure Windows to log each time a program exits. This is not the default behavior, but it is possible if the organization has changed log settings or is doing continuous monitoring
- Crash Logs: Sometimes malware crashes and exits unexpectedly. You may find system logs of a process crashing. Reviewing those may reveal a suspiciously named process that was having problems.
Question Breakdown
Based on the above breakdown, we can break the question of “Are there remnants of past executions of known malicious programs?” into smaller questions that can be more easily answered:
- Is there evidence that a malicious program previously started or stopped?
- Is there evidence that a malicious program created known files or registry keys?
- Is there evidence that a malicious program was in memory?
- Is there evidence that malicious network hosts were connected to?
- Is there evidence that a malicious program was crashing?
Where To Find the Malware Remnants During Incident Response
We’re looking at a diverse set of data types when we look for remnants, but let’s summarize where all of this data is:
- Programs Starting: Refer to the past blog on all of the locations that store this information
- IOC: Known file and registry keys will require access to all of the file and registry key names on the system. You can get this data from either a live file system scan, memory capture, or full disk image
- Unallocated Memory: To look for traces of previously run programs in memory, you’ll need access to memory via either a full memory capture or live memory analysis that focuses the unallocated space
- Network Caches: You’ll need to get this data from either OS APIs or a memory capture. It is not stored on disk
- Program Crashing: You can get this data from event logs stored on disk or in memory.
How to Analyze It
Once you get access to this information, you need to do something with it in order to determine if it is suspicious or malicious.
- Programs Starting: For any of the locations that can tie back to an executable, it should be analyzed by static and dynamic analysis techniques. Services such as Reversing Labs and VirusTotal can give you broad coverage to a variety of engines
- IOC: Filename and registry key scanning requires access to a threat intelligence service that provides feeds of known file names
- Unallocated Memory: Tools such as Volatility have modules that focus on unallocated space and additional logic is needed to scan it for known signatures of malware (perhaps using Yara rules)
- Network Caches: A network reputation service is needed to sort through dozens or hundreds of domains that you’ll find on a host. It is not always obvious which domains and IP addresses are being used for malicious reasons.
Many of these steps can be tedious if you are doing them manually. Given the amount of data that needs to be analyzed and compared with threat intel, automation is critical to a fast response.
Using Cyber Triage to Find Malware Remnants During Incident Response
Cyber Triage allows you to search for many of these malware remnant locations, though the exact amount of data depends on which acquisition approach you use.
Its agentless collection tool on a live system will grab data about programs that run, DNS cache, and all file names. Analysis of a disk image will not get volatile data, such as DNS cache:
The navigation menus on the left-hand side of Cyber Triage are organized based on the same ideas that this blog series uses. The first items are all about users. The next set is about malware. The last is about system configuration:
The malware remnants results can be found in:
- Programs Run (which also has data used for user investigations)
- DNS Cache
- Files
Programs Run Area
Refer to the previous post on the details of the “Programs Run” area. The basic idea though, is that Cyber Triage scans many locations that store information about what programs were run. It also collects the associated program that was run and uploads it to Reversing Labs for analysis of 40+ engines.
Programs are marked as “Bad” if they are found to be malware or marked as “Suspicious” if they have other atypical features, such as what folder they were run from:
You can use the bottom area to learn more about the file and its execution history:
Or, the timeline on the right to compare its times with other events in the incident:
You can mark the item as “Bad” or “Suspicious” using the buttons below the table. This ensures that you follow up on them and include them in reports:
DNS Cache Area
The “DNS Cache” section shows you the domain names and IP addresses that were still being retained by the OS. The IP address is also mapped to a Country using the MaxMind GeoIP database.
For this data to be most useful, you should pull in blacklist information associated with IP addresses.
Cyber Triage supports a CSV format for this and the OpenPhish format. Cyber Triage will mark domains as “Suspicious” if they are from Dynamic DNS providers and, therefore, the domain could be associated with high-flux malware.
For any selected DNS Cache entry, the “Host” tab in the bottom area will give you WhoIs information about that domain.
NOTE: We are actively looking into including a domain reputation service with a Cyber Triage subscription like we do with Reversing Labs for file scanning.
Files Area
The “Files” area shows you two things:
- A full directory structure that you can navigate with
- The files marked as “Bad” or “Suspicious.”
When you configure Cyber Triage with a blacklist of file names, then the files that match those rules will be highlighted here. This can be used for IOC scans.
You can make those rules by going to “Options” and choosing the “Blacklist” tab:
The rules should be added with a CSV format such as:
# value,comment
sample/folder1/malware.exe, Known malware
Note that this only scans the files that were found by the collection tool. To get all file names, you need to perform the “Full Scan.” If you did a partial collection then this IOC scan will not be complete.
Conclusion
When you have a limited amount of time to perform endpoint triage, you should first focus on a quick review of all data types. The malware remnants that we talked about in this article provide the complete picture beyond persistence and what is currently running. In the next post in the series, we move on the System Configuration.
If you want to walk through the steps outlined in this post with Cyber Triage, you can try it out for free today.
