Mastering Syscall-Level Reversing pt1

1. Introduction: Why Syscall-Level Matters

When I first started reverse engineering, I made a critical mistake: I focused exclusively on high-level API calls. printf, ReadFile, socket, these were my comfort zone. But then I encountered my first piece of malware that bypassed every user-mode hook I had set up. It was calling the kernel directly, laughing at my detection mechanisms.

That’s when I realized: if you want to truly understand what a program is doing, you must go to the source, the system call layer.

Syscall-level reversing is the art of analyzing a program’s direct conversations with the operating system kernel. It strips away all abstractions, all library sugar-coating, and reveals the raw requests a program makes to hardware resources. This is the level where:

• malware evades EDR/AV hooks
• packers manipulate memory protections
• rootkits establish persistence
• anti-debugging tricks operate

through studying samples, CTF challenges and binaries, I’ve developed a methodology that I’m going to share with you rn. This isn’t just theory, I’ll walk you through real binaries, show you the exact commands I use and reveal the tricks that have saved me countless hours in the lab.

2. The Foundation: Understanding System Calls

What exactly is a System Call?

A system call(syscall) is a programmatic way for a user-space application to request a service from the kernel. Think of it as the ultimate API, the boundary between user mode and kernel mode.

The Anatomy of a Syscall: Every syscall consists of three components:

1. The syscall number - a unique identifier for the operation
2. The arguments - parameters passed to the kernel
3. The return value - the kernel’s response

The Syscall Journey

Let me trace what happens when you call write() in C:

write(fd, buf, count)
  ↓
libc wrapper (glibc)
  ↓
Assembly: mov rax, 0x1 (syscall number)
          mov rdi, fd
          mov rsi, buf
          mov rdx, count
          syscall (trap to kernel)
  ↓
Kernel: sys_write() handler
  ↓
Return to user mode with result

Why Direct Syscalls Matter: Most programs use libc wrappers. But sophisticated code, especially malware, bypasses these wrappers entirely:

; direct syscall - no libc dependency
mov rax, 1
mov rdi, 1
lea rsi, [message]
mov rdx, 12
syscall

This approach:

• Eliminates library dependencies
• Avoids function hooking
• Reduces binary size
• Increases stealth

The Syscall Tables

Linux x86-64 Syscall Numbers:

0   = sys_read
1   = sys_write
2   = sys_open
3   = sys_close
9   = sys_mmap
12  = sys_brk
83  = sys_mkdir
59  = sys_execve
60  = sys_exit

complete table location: /usr/include/x86_64-linux-gnu/asm/unistd_64.h

Windows Syscall Numbers (Dynamic):

Unlike Linux, Windows syscall numbers change between versions. This is why malware often dynamically resolves them at runtime.

Windows 10 1903: NtCreateFile = 0x55
Windows 10 2004: NtCreateFile = 0x50
Windows 11:      NtCreateFile = 0x52

This dynamic nature makes Windows reversing particularly challenging, you must know the exact OS version you’re analyzing.

3. Linux vs Windows: The Architectural Differences

Linux Architecture

Syscall Invocation Methods:

1. syscall instruction(x86-64) - modern, faster
2. int 0x80 (x86) - legacy, slower
3. vsyscall/vDSO - special fast-path for certain calls

Here’s an example - sys_execve:

mov rax, 59       ; sys_execve
mov rdi, path     ; filename
mov rsi, argv     ; argv array
mov rdx, envp     ; envp array
syscall

Windows Architecture

Syscall Invocation Methods:

1. syscall (x64) - primary method
2. sysenter (x86) - fast system call
3. int 2e (x86) - legacy interrupt

Register Convention(x64):

Example - NtWriteFile:

mov eax, 0x50        ; Syscall number (Windows 10 2004)
mov r10, rcx         ; Save first argument
mov rdx, Handle      ; File handle
mov r8, Event        ; Event
mov r9, ApcRoutine   ; APC routine
; Additional args on stack
syscall

Key Differences for Reversing

4. Essential Tools for the Journey

Over time, I have curated a toolkit that works for every syscall reversing scenario. Here’s what I use:

Linux Tools

1. strace - the syscall tracer

   strace -e trace=file -o syscalls.log ./binary

my go-to for initial analysis. The -e flag filters specific syscall categories.

2. ltrace - library call tracer

   ltrace -e printf,write ./binary

essential for differentiating library calls from raw syscalls.

3. GDB with Syscall Breakpoints

(gdb) catch syscall
(gdb) catch syscall execve
(gdb) info syscalls

I use this for deep inspection of syscall arguments

4. objdump - static analysis

   objdump -d -M intel binary | grep -A 10 syscall

find all syscall instructions in the binary

5. sysdig - kernel level tracing

   sysdig proc.name=binary

for when you need kernel-level visibility.

Linux Deep Dive: step by step analysis

let me walk you through a real analysis(demonstration). It’s for a suspicious binary that was bypassing a file-monitoring agent

Phase 1: Initial reconnaissance file information:

$ file suspicious_binary
suspicious_binary: ELF 64-bit LSB executable, x86-64, statically linked

statically linked means no external libc. Likely using direct syscalls

String analysis:

$ strings suspicious_binary | grep -i syscall
# No output

Phase 2: Dynamic tracing with strace

$ strace -o full_trace.log ./suspicious_binary
$ head -20 full_trace.log

output:

execve("./suspicious_binary", ["./suspicious_binary"], 0x7ffe... ) = 0
arch_prctl(ARCH_SET_FS, 0xdeadbeef...) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f...
mprotect(0xdead... , 0x1000, PROT_READ|PROT_EXEC) = 0
write(1, "Decrypting...\n", 14) = 14
openat(AT_FDCWD, "/tmp/.cache", O_RDWR|O_CREAT, 0644) = 3
read(3, "..." , 0x1000) = 0x1000
write(3, "..." , 0x2000) = 0x2000
close(3) = 0
ptrace(PTRACE_TRACEME, 0, 0, 0) = -1 EPERM (Operation not permitted)

Red flags:

1. arch_prctl - setting up thread-local storage which is suspicious for a simple binary
2. mprotect with PROT_EXEC - likely self modifying code
3. openat in /tmp/.cache - hidden file operations
4. ptrace call - anti-debugging attempt

Phase 3: Focusing on suspicious syscalls

$ strace -e trace=open, openat, read, write, mmap, mprotect, ptrace ./suspicious_binary

Detailed analysis of openat:

openat(AT_FDCWD, "/tmp/.cache", O_RDWR|O_CREAT, 0644) = 3

the file descriptor 3 is returned. Let’s see what happens to it.

Phase 4: Deep inspection with GDB

$ gdb ./suspicious_binary
(gdb) catch syscall openat
Catchpoint 1 (syscall 'openat')
(gdb) catch syscall mmap
Catchpoint 2 (syscall 'mmap')
(gdb) run

Breakpoint hit - openat:

Catchpoint 1 (call to syscall openat), 0x0000000000400c50
(gdb) info registers
rax            0x101     257  ; syscall number for openat
rdi            0xffffffffffffff9c  ; AT_FDCWD (-100)
rsi            0x600d20  "/tmp/.cache"
rdx            0x42      66  ; O_RDWR|O_CREAT
r10            0x1b6     438  ; 0666? Actually 438 decimal
(gdb) x/s $rsi
0x600d20: "/tmp/.cache"

next breakpoint - mmap:

Catchpoint 2 (call to syscall mmap), 0x0000000000400c80
(gdb) info registers
rax            0x9       9  ; sys_mmap
rdi            0x0       NULL  ; Let kernel choose address
rsi            0x1000    4096  ; 1 page
rdx            0x3       PROT_READ|PROT_WRITE
r10            0x22      MAP_PRIVATE|MAP_ANONYMOUS
r8             0xffffffffffffffff  ; -1 (no file backing)
r9             0x0       0  ; offset

from the text, the programming is mmapping anonymous memory, then presumably writing decrypted code there.

Phase 5: Static Analysis with objdump

$ objdump -d -M intel suspicious_binary | grep -B5 -A5 syscall

snippet:

  400c50:  b8 01 01 00 00        mov    eax,0x101
  400c55:  bf 9c ff ff ff        mov    edi,0xffffff9c
  400c5a:  48 be 20 0d 60 00 00  movabs rsi,0x600d20
  400c61:  00 00 00
  400c64:  ba 42 00 00 00        mov    edx,0x42
  400c69:  0f 05                 syscall

key observation: the binary uses mov eax, 0x257 instead of mov rax, 0x257 - this is common optimization. The syscall number 0x257 = 599

Phase 6: Building the execution flow Using the trace and assembly, I mapped the execution:

1. write “Decrypting…” to stdout
2. open /tmp/.cache for reading/writing
3. read encrypted data from the file
4. mmap anonymous memory
5. decrypt data (custom algorithm based on XOR)
6. write decrypted data to mmap region
7. mprotect region to PROT_EXEC
8. jump to decrypted code
9. ptrace check (anti-debug)
10. payload execution

Phase 7: bypassing anti-debug The ptrace anti-debug was simple:

mov rax, 101       ; ptrace
mov rdi, 0         ; PTRACE_TRACEME
syscall
test rax, rax
jnz exit

bypass in GDB:

(gdb) catch syscall ptrace
(gdb) commands
> set rax = 0
> continue
> end

Phase 8: Extracting the Payload at the mmap breakpoint:

(gdb) x/100xb $rax  ; After mmap returns
0x7ffff7ff0000: 0x90 0x90 0x90 0x90 0x48 0xb8 0x2f 0x62
...

I dumped the payload:

(gdb) dump memory payload.bin 0x7ffff7ff0000 0x7ffff7ff1000

What we learned: The binary was a downloader that:

• checked for debuggers
• decrypted a second-stage payload
• executed it from memory(fileless) this is an example why syscall-level analysis is critical. The high-level API calls would have shown nothing.