Fileless Malware Attacks Explained: Detection and Defense

Fileless malware is one of the most dangerous and evasive attack techniques in modern cybersecurity. Unlike traditional malware that writes executable files to disk, fileless attacks operate entirely in memory — leaving little to no forensic trace on the filesystem. Security tools that rely on file scanning miss them almost entirely.

What Makes Malware “Fileless”?

The term “fileless” is a bit of a misnomer. Most fileless attacks do involve some file at the initial entry point — a malicious Office document, a phishing PDF, or a drive-by download. The key difference is that the actual malicious payload never touches disk as a standalone executable. Instead, it injects itself into legitimate running processes or uses built-in system tools to execute code directly in RAM.

Fileless malware typically exploits:

Living-off-the-land binaries (LOLBins) like powershell.exe, wscript.exe, mshta.exe, and certutil.exe
Reflective DLL injection — loading a DLL into a process’s memory without writing it to disk
Process hollowing — spawning a legitimate process and replacing its code in memory
Registry-based persistence — storing encoded payloads in registry keys that execute via scheduled tasks or WMI

How a Typical Fileless Attack Unfolds

A real-world fileless attack often follows this chain:

Initial access — The victim opens a malicious Word document with a weaponized macro
Macro execution — The macro spawns powershell.exe with a Base64-encoded command
In-memory payload — PowerShell downloads a script using IEX (Invoke-Expression) and runs it entirely in memory
Persistence — The script registers a WMI event subscription or adds a registry run key pointing back to another PowerShell one-liner
Command and control — The payload connects outbound to a C2 server without ever writing a binary to C:\Windows\Temp

A simplified example of the PowerShell staging technique looks like this:

powershell.exe -NoP -NonI -W Hidden -Enc <Base64EncodedPayload>

The -Enc flag accepts a Base64-encoded command, making the payload invisible to casual log inspection. Tools like Cobalt Strike and Metasploit’s exploit/multi/handler use this technique extensively in red team engagements.

Why Traditional AV Fails Against Fileless Malware

Signature-based antivirus products scan files on disk. If no file exists, there is nothing to scan. Even heuristic engines struggle because the code executing is often powershell.exe — a fully trusted, signed Microsoft binary.

Behavioral detection can catch some patterns, but attackers have adapted:

They fragment payloads across multiple registry keys
They use legitimate scripting engines like wscript.exe and cscript.exe
They encode payloads multiple times (Base64 inside Base64, XOR obfuscation)
They disable Windows Script Block Logging before executing

Detection Techniques That Actually Work

1. Enable PowerShell Script Block Logging

Windows PowerShell 5.0+ supports Script Block Logging, which captures the actual decoded content of executed scripts — even if they were passed Base64-encoded.

Enable it via Group Policy:

Computer Configuration > Administrative Templates > Windows Components > Windows PowerShell
> Turn on PowerShell Script Block Logging = Enabled

Or via registry:

New-Item -Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\PowerShell\ScriptBlockLogging" -Force
Set-ItemProperty -Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\PowerShell\ScriptBlockLogging" -Name "EnableScriptBlockLogging" -Value 1

Logs appear in Event ID 4104 in the Microsoft-Windows-PowerShell/Operational log.

2. Monitor Process Injection with Sysmon

Sysmon (System Monitor) from Sysinternals logs granular process activity including:

Event ID 8 — CreateRemoteThread (classic injection indicator)
Event ID 10 — Process access (one process reading/writing another’s memory)
Event ID 25 — Process tampering (process image changed — hollowing indicator)

A minimal Sysmon config to catch injection:

<RuleGroup name="ProcessInjection" groupRelation="or">
  <CreateRemoteThread onmatch="include">
    <TargetImage condition="is not">C:\Windows\System32\svchost.exe</TargetImage>
  </CreateRemoteThread>
</RuleGroup>

3. Use Memory Forensics Tools

Volatility 3 is the industry-standard memory forensics framework. If you capture a RAM image from a compromised machine, you can analyze it with:

# List running processes and detect hidden ones
python3 vol.py -f memory.raw windows.pslist

# Scan for injected code in process memory
python3 vol.py -f memory.raw windows.malfind

# Dump suspicious process memory for analysis
python3 vol.py -f memory.raw windows.dumpfiles --pid 1234

The malfind plugin specifically looks for memory regions that are executable but not backed by a file on disk — a hallmark of reflective DLL injection.

4. Behavioral EDR Solutions

Endpoint Detection and Response (EDR) platforms like Microsoft Defender for Endpoint, CrowdStrike Falcon, and SentinelOne monitor process behavior in real time. They use kernel-level hooks to catch:

Processes spawning unusual child processes (e.g., winword.exe spawning powershell.exe)
Network connections made by scripting engines
Attempts to disable security tools via taskkill or registry modification

5. WMI Activity Logging

Fileless malware loves WMI for persistence. Enable WMI activity logging:

wevtutil sl Microsoft-Windows-WMI-Activity/Operational /e:true

Watch for Event ID 5861 — WMI permanent event subscription created. Legitimate software rarely creates WMI subscriptions.

Hardening Your Environment

Hardening Control	Technique
Disable PowerShell v2	`Disable-WindowsOptionalFeature -Online -FeatureName MicrosoftWindowsPowerShellV2Root`
Constrained Language Mode	Set `__PSLockdownPolicy` = 4 via AppLocker
Block macro execution	Group Policy: Disable all macros in Office documents
WDAC / AppLocker	Whitelist only signed, approved executables
Attack Surface Reduction rules	Enable via Microsoft Defender ASR rules

Application control via Windows Defender Application Control (WDAC) or AppLocker is the most effective single control. If mshta.exe, wscript.exe, and cscript.exe cannot run arbitrary code, an enormous class of fileless attacks is blocked before they start.

Incident Response When You Suspect Fileless Malware

If your EDR or SIEM fires an alert suggesting fileless activity, act fast — the evidence lives in volatile memory:

Do not reboot the machine — you will lose the payload
Capture a memory image with tools like WinPmem or DumpIt
Isolate the host from the network at the switch level, not just via software
Collect event logs — PowerShell operational, Sysmon, Security, WMI Activity
Analyze memory with Volatility 3 for injected regions and suspicious network connections
Check persistence mechanisms — registry run keys, scheduled tasks, WMI subscriptions, COM hijacking

Conclusion

Fileless malware represents the convergence of stealth and sophistication. By abusing trusted system tools and living entirely in memory, these attacks bypass the defenses that organizations spent years building. The answer is not better antivirus — it is layered detection through script block logging, Sysmon, behavioral EDR, memory forensics, and aggressive application control. Understanding how these attacks work is the first step to stopping them.