This post in the “DFIR Next Steps” series is about what to do when you identify the use of RMM tools on a host. This is the first part of a series on RMM investigations and will start with a broad overview of RMM tools and some of the challenges they present. Later parts of the series will dive in depth on specific tools.
Scenario
The use of a RMM tool is identified on a host. The associated process has been identified, and it is suspected that the tool was used by a threat actor. Since most RMMs are legitimate tools, they are unlikely to be alerted on by standard EDR\SIEM detection rules. Rather detection will be dependent upon custom rules for the local environment, threat hunting, or an IR investigation. Note that Cyber Triage uses a range of techniques to detect RMM tools and will score them as bad or suspicious, depending on how long the exe has been on the system.
When investigating this RMM we are looking to:
- Confirm that the use of the tool is malicious
- Is it a standard install for the environment?
- Was the tool installed via the standard installer?
- How long has the tool been on the system?
- What actions were taken during logon sessions?
- Have malicious logons occurred?
- Has the tool been used for remote access?
- Can logon sessions be identified?
- Can connections to control servers be identified?
- Has malicious activity occurred?
- Is there out of character behavior for the compromised account?
- Unusual file access?
- Creation of large archives?
- Are there connections to bad\suspicious URLs?
- Download\creation of malware?
- Use of network scanning\mapping software?
- High number of failed logons?
- Is there out of character behavior for the compromised account?
- Identify what to look for next
- How was the RMM initially executed?
Why do we care about RMM tools?
Threat actors have been increasing their use of Remote Monitoring and Management tools over the past few years. In January 2023, CISA published an advisory about the malicious use of portable RMM tools in phishing campaigns. They are also extremely popular with ransomware operators (for a great breakdown of what tools different groups are using see the ransomware tool matrix by @BushidoToken https://github.com/BushidoUK/Ransomware-Tool-Matrix/blob/main/Tools/RMM-Tools.md ).
In addition to being legitimate tools and therefore less likely to be detected, many RMMs have portable, or even single binary versions which can be launched from the command line. This makes it easy to pivot from remote code execution to a full remote desktop.
The other big attraction of remote desktop tools is that they act as reverse shells, connecting out to the vendors control servers (frequently to port 443) and appearing as standard browser traffic. Making network detection tools blind to the final destination and circumventing firewalls.
In terms of an attack timeline, the use of a RMM tool is likely to occur early within the attack lifecycle, as it can provide both remote access and a persistence mechanism. However, within the investigation timeline, it is likely to occur after other known malicious behavior has been identified. As such, RMM tools are unlikely to be the first detection of malicious activity except when custom detection rules are in place or threat hunting is taking place.
Confirm that the existence of the tool is malicious
Was the tool run as an installed program?
If a portable version of the tool has been used, it is a good indication that it was malicious, or at least unauthorized. Portable versions do not require administrative privileges, allowing users to circumvent security controls. Things to check:
- Was the program launched from a program files subdirectory?
- Is the program listed in installed programs?
Is it a standard install for the environment?
- Check for instances of the tool being launched on other hosts. Within Cyber Triage, you can run an incident wide search for the executable.
- Always remember to talk to people! On many occasions, I have seen investigators spend hours trying to answer a question that could be addressed in a matter of minutes by talking to someone!
How long has the tool been on the system?
Was it installed well before the period of interest? Does the creation timestamps on program executable and related files significantly pre-date the time period of interest?
Have malicious logons occurred?
Has the tool been used for remote access?
There are a few methods of determining if remote access has occurred. Most full installs of RMM tools will log logon events, either in text log files or the windows event logs. Portable installs are less likely to create logs. Other indicators are connections to the software servers, available in SRUM data, DNS cache, or IDS, proxy or firewall logs.
Can logon sessions be identified?
If the tool has been installed, there is a good chance that log files will be available on the host. These may be text files (most likely in %appdata%) or even event log entries. Note that since the RMM most likely gives the remote user direct access to the already logged on desktop, it is unlikely that any standard windows logon events will be recorded.
Has malicious activity occurred?
The strongest indicator of malicious use of a RMM tool is going to be the actions that were taken while the threat actor had remote access to the host. In order to do this, you will need to identify the user account the tool was launched by and examine user activity during the logon session. Some examples of what to look for are given below.
Is there out of character behavior for the compromised account?
Review user activity and check for out of character activity. For example, unusual file access, creation of large archives, execution of unusual applications (especially anything to do with security, powershell, or command shell).
Are there connections to bad\suspicious\unusual URLs?
There is a high probability that the threat actor will look to transfer toolkits to the host or exfiltrate files from the host. Review browser history, dns cache, and firewall proxy logs for indications of access to suspect sites.
Download\creation\detection of malware?
One thing to always check when starting a case is what defensive software is installed on the host. Far too often, there are cases where EDR is installed alongside local Anti-virus software and some threats are blocked by the local software before detection by the EDR. As a result, there is useful information about the attack in the local AV logs, which have not been reported to the centralized monitoring system. Cyber Triage will collect Microsoft Defender event logs by default.
Keep in mind that RMM tools are highly unlikely to be blocked, unless you are running some custom EDR rules; however, it is possible that once the threat actor has access to the environment they will attempt to upload additional tools that will be detected.
Use of network next stage software?
Once access is gained to a host, it is likely that network mapping and privilege escalation will occur. In order to do this, tools will need to be transferred onto the compromised host or a reverse proxy setup. In both cases, there should be evidence of the execution on the host. Some scanning tools are detected as malicious but others are not. So, in addition to reviewing detection logs, review processes that have run on the host for signs of tools used to map and pivot.
High number of failed logons?
Any scanning of active directory, file shares, and other windows resources is likely to result in authentication failures, which should be logged. These log entries can provide some insight into the type of scanning and any other potentially compromised hosts.
What to look for next?
In terms of the attack timeline, at this stage, you should have already examined most activity that occurred during the RMM session(s). So, now you need to consider how the tool was transferred onto the compromised host and if the threat actor was able to pivot to any other hosts.
How was the RMM placed on the system?
A good source of how the RMM was placed on the system is the creation date of the file, and even better, of the parent directory if it was created at the same time. A timeline of all events shortly before this time is likely to provide some context around how the tool was transferred onto the system.
What level of privilege did the threat actor gain?
In order to determine the potential access to the environment the threat actor had, you should consider the privileges of the compromised account. In particular, if they had any administrator group memberships, at both the local and domain level.
What can be used to identify other compromised hosts?
In order to identify other compromised hosts, you should look for any authentication events from the compromised system, search for files identical to those dropped during the RMM session, as well as the RMM itself, and for any connections to the RMM control servers.
Conclusion
Remote Monitoring and Management tools are a popular resource for threat actors to control compromised hosts. As the tools are legitimate, they are unlikely to be detected as malicious by default detection rulesets, and they provide the threat actor with the same level of access to the host as the account the tool is run under.
Once a RMM tool has been identified, you should determine if the tool was installed or portable and check for the existence of tool specific logs. Ideally, log entries will provide some indication of when the threat actor was active on the host. Keep in mind that at this point there is a high probability that files have been exfiltrated, and the threat actor will have pivoted to other systems.
If you want to make your Incident Response investigations more efficient, try a demo of Cyber Triage today.
