Mastering Syscall-Level Reversing pt2; The Art of Unhooking & Direct Syscalls

Intro: The Hook is the problem

In part 1, we dissected a Linux binary that used direct syscall instructions to fly under the radar. We also traced its execution from write() all the way down to the kernel, watching it decrypt and execute a payload entirely in memory.

The thing is, userland hooks are only as trustworthy as the process that contains them.

Now, let’s move to the primary battleground for this technique: Windows. If you’ve spent any time reversing modern malware, you’ve likely encountered the reality that user-mode API hooks are the first line of defense for most EDRs and the first line of attack for sophisticated adversaries who clearly have way too much time on their hands.

Modern EDRs like CrowdStrike, SentinelOne and Microsoft Defender for Endpoint rely heavily on userland API hooking with ntdll.dll. They overwrite the first few bytes of sensitive functions like NtCreateProcess, NtAllocateVirtualMemory or NtWriteVirtualMemory with a JMP instruction that redirects execution to their own monitoring code.

The attacker’s counter? Unhooking or bypassing these hooks entirely by rolling their own syscall invocation. We’ll explore how this is done and crucially, how you can spot it in a binary you’re analyzing because let’s face it, reading other people’s poorly written malware is how some of us pay bills haha.

The hooking landscape: how defenders watch

Before we can understand unhooking, we need to understand hooking. Security tools use 2 primary methods to intercept API calls. Think of them as bouncers at a local pub, except they’ve been paid by attackers.

1. Inline hooking

This technique replaces the first few bytes of a function in memory with a JMP instruction that directs execution to the EDR’s monitoring code. On Windows, the typical syscall stub begins with the bytes 4C 8B D1 B8, which translates to:

mov r10, rcx ; 4C 8B D1 - Save first argument because we're polite
mov eax, <SSN>  ; B8 XX 00 00 00 - Syscall number, the secret handshake

When an EDR hooks a function, it overwrites these bytes. Instead of 4C 8B D1 B8, you’ll see something like E9 XX XX XX XX, a JMP instruction to the EDR’s monitoring code.

2. Import Address Table Hooking

The import address table is a lookup table of function pointers for functions imported from DLLs. IAT hooking replaces the function address in this table with another address, redirecting the program’s execution flow.

3. The Detection Problem

The challenge for defenders is that these hooks operate in userland and userland can be manipulated by any process running with sufficient privileges. As one researcher put it, “user processes do not usually make syscalls directly and may indicate suspicious behavior.” But that’s exactly what sophisticated malware does because apparently nobody told them they’re not supposed to do that.

The attacker’s arsenal: direct syscalls

The most powerful evasion technique is to bypass the Windows API entirely and invoke system calls directly. Instead of calling NtAllocateVirtualMemory through ntdll.dll, where the EDR has placed its hooks, the malware executes the syscall instruction itself. It’s like deciding to build your own car because you don’t trust the mechanic except the car is already weaponized to steal your company’s secrets.

The challenge? Syscall numbers change between windows versions. You can’t hardcode them.

1. The syscall number problem: windows edition

Unlike Linux, where syscall numbers are stable across kernel versions and you can reliably count on sys_write being 1(ft Linus Torvalds),

The syscall number for NtCreateFile changes across Windows builds, making hardcoded values unreliable.

This dynamic nature makes Windows reversing particularly challenging because you must know the exact OS version you’re analyzing or dynamically resolve the SSNs at runtime. Or you could just guess and hope for the best which I’ve definitely never done in a production incident response scenario. Def not.

2. Hell’s Gate: The foundation

This technique is popularized by the security community and by popularized I mean “made everyone’s life more complicated”, parses the loaded image of ntdll.dll in memory to find the syscall stub for a given function and extracts the SSN from the mov eax, <SSN> instruction.

The approach works like this:

1. walk the process environment block (PEB) to find the base address of ntdll.dll, it’s like going to the library except the library is a kernel data structure and the librarian doesn’t like you.
2. parse the PE headers to locate the Export Directory, the table of contents for the library.
3. hash the function name you need because comparing strings is too mainstream
4. loop through exported functions, hash each one and compare. Like finding Waldo but Waldo is a Windows API function and he’s hiding behind 500 other functions.
5. when found, dereference the function pointer’s address in memory. This is the moment of truth
6. Read the bytes. If it starts with 4C 8B D1 B8, the SSN is the byte after B8

The limitation? If the syscall stub is hooked, Hell’s gate can’t extract the SSN because the expected bytes aren’t there. It’s like trying to read a book where someone has torn out the pages with the answers.

3. Halo’s gate: looking next door

Halo’s gate is an evolution of Hell’s gate. If it encounters a hooked syscall stub, it looks at neighboring syscall stubs and adjusts the SSN accordingly. This works because syscall numbers in the SSDT (system service descriptor table) follow each other incrementally.

The logic is elegant: if NtAllocateVirtualMemory is hooked, Halo’s gate checks the next function in the export table, gets its SSN and subtracts the distance to calculate the correct SSN. It’s like borrowing a lighter from your neighbor except the neighbor is a windows API function and the lighter is a syscall number.

What if EDRs hook the second instruction? Some EDRs hook after mov r10, rcx, meaning the first bytes 4C 8B D1 are intact, but the mov eax instruction is replaced. Tartarus’ Gate addresses this by checking for the sequence 4C 8B D1 E9 which translates to mov r10, rcx; jmp <address>. Because apparently, the EDR vendors and the vendors and the malware authors are engaged in a game of “I know you know I know” that would make a soap opera jealous.

4. FreshyCalls

FreshyCalls take a diff approach. Instead of parsing individual functions, it:

1. searches the export directory for functions starting with Nt
2. sorts them by memory address
3. derives syscall IDs from the ordering of exported Nt* functions after sorting them by their addresses in memory, exploiting the layout Microsoft uses internally.

This works because Microsoft exports Nt* functions in order of their syscall numbers, the lowest address is syscall 0, the next syscall 1 and so on. It’s like realizing that your keys were in your pocket the whole time after spending an hour looking for them.

SysWhispers2 takes a different approach. Rather than resolving syscall numbers at runtime, it generates version-specific syscall stubs during compilation. This lets an application invoke system calls directly without relying on the potentially hooked exports in ntdll.dll. Both Nt* and Zw* names ultimately resolve to the same syscall stubs in user mode, so the distinction is mostly one of naming rather than behavior.

5. The Evolution continues: SysWhispers3

Extends the approach with support for:

• x86/WOW64 syscalls (because 32-bit isn’t dead, apparently)
• syscall instruction replacement with na EGG (to be dynamically replaced at runtime because static code is for amateurs)
• direct jumps to syscalls with randomized patterns to avoid signature-based detection. if you can’t beat them, confuse them.

Indirect Syscalls: Spoofing the Call Stack

A significant limitation of direct syscalls is that the syscall instruction originates from the malware’s own code section, not from ntdll.dll. Modern EDRs use instrumentation Callbacks to detect syscalls and check if the return address is inside ntdll.dll.

If a syscall doesn’t return to ntdll.dll, it’s a red flag. It’s like showing up to a fancy car meet with a stock car. You’re there but everyone can see you don’t belong there.

Indirect syscalls address this by jumping to a syscall instruction that exists in ntdll.dll itself. The malware constructs a stub that jumps to the syscall instruction inside ntdll’s .text section, making it appear as though the call came from the legitimate DLL.

This is how the technique works:

; direct syscall - suspicious, return address is in malware code
mov r10, rcx
mov eax, <SSN>
syscall
ret

; indirect syscall - stealthier, jumps to syscall in ntdll
mov r10, rcx
mov eax, <SSN>
jmp r11
; return address appears to be from ntdll

This is why frameworks like RecycledGate and implementations like Artemis offer both direct and indirect modes. Because why have one way to bypass security when you can have two?

The Unhooking Arms Race: Whisper2Shout

Traditional unhooking techniques involve reading a clean copy of ntdll.dll from disk and overwriting the hooked version in memory. This works, but EDRs have evolved to detect and counteract it.

Whisper2Shout represents a significant evolution in unhooking methodology. The technique is based on a key observation: when an AV/EDR hooks a function, it must store the original bytes somewhere in memory, typically in a private memory region, to execute legitimate calls correctly.

The Whisper2Shout approach is elegant:

1. Identify the private memory region where EDR stores its trampolines
2. Trace the arrows - the jump at the function start points to a trampoline in this private region
3. Find the original stub - the trampoline contains both a jump to the AV’s monitoring code and a jump back to the original function
4. Overwrite the hooking stub - instead of restoring the original function bytes(which the EDR might monitor for integrity), overwrite the trampoline in the private region with a jump directly to the original function.

The critical innovation? When the EDR checks its hooks, it sees its JMP still intact at the function start. The EDR’s integrity check passes because the hook is still there, but execution bypasses the monitoring code entirely.

As the researchers noted, “we circumvented the hooking trampoline, guaranteeing the seamless execution of the function as if no hooks were present, even in the presence of a jump at the symbol address”

Whisper2Shout Limitations; because nothing is perfect

Despite its sophistication, Whisper2Shout has limitations:

1. calls to NtProtectVirtualMemory to change memory permissions could still alert some EDRs.
2. The scanning technique itself could be detected by setting guard pages on the trampoline memory because security vendors have also read the whitepaper.
3. Modern EDRs employ hook integrity verification that checksums the entire .text section of ntdll.dll. If Whisper2Shout modifies the trampoline but the EDR also checks the trampoline’s integrity, the technique fails.
4. Some EDR’s don’t store trampolines in predictable private memory regions. They may use kernel-mode hooks instead, rendering userland unhooking techniques completely useless.
5. The timing window. Whisper2Shout must perform its unhooking before the EDR’s self-protection mechanisms detect the tampering. This is like trying to pick a lock while the owner is walking toward the door. These limitations are proof why this cat and mouse game continues to evolve. For every unhooking technique, there’s a countermeasure in development.

The irony is that these techniques, while designed to evade defenders, often become recognizable fingerprints for reverse engineers. Once you know what PEB walking, export hashing, manual SSN resolution, or indirect syscall stubs look like, they stand out immediately in a disassembler even when the malware imports almost nothing from the Windows API.

What’s next?

We’ve spent two articles discussing how direct and indirect syscalls work, how attackers bypass user-mode hooks and why techniques like Hell’s Gate, Halo’s Gate, FreshyCalls and Whisper2Shout continue to evolve.

We’re just halfway there.

In the next part, we’ll shift from concepts to practice. We’ll take a real binary to Ghidra, identify custom syscall implementations, recover syscall numbers, recognize PEB walking and export hashing and learn the patterns that immediately tell you you’re looking at a Hell’s Gate style implementation.

By the end, you should be able to glance at a disassembly and answer if a binary is bypassing a Windows API.