CTF Writeup: Private Binary | RE | Crypto | BSidesSF 2026

Description

I managed to exfiltrate a highly classified, encrypted executable, but no tool I throw at it can parse it. It doesn't even have an ELF header!

Fortunately, I also managed to exfiltrate the custom Linux Kernel Module (loader.ko) that the target system uses to execute these files.

How does the executable run without an ELF header? Does the kernel module have something to do with it?

We're given: loader.ko and flag.enc

---

Overview

A Linux kernel module (loader.ko) registers a custom binary format via the binfmt subsystem, allowing the kernel to recognize and execute .enc files that lack a standard ELF header. When run, the loader maps flag.enc into memory and decrypts it in-place using RC4 with a hardcoded key (BSIDES_SF_2026) before transferring execution to the decrypted payload. The solution is to replicate this decryption in user space — no need to run the kernel module at all.

---

Solution

Step 1 — Initial Observations

Running standard tools on flag.enc shows it is not a recognized ELF binary. The presence of loader.ko is the key clue: a custom kernel module must be handling execution.

Step 2 — Reverse the Kernel Module

Reversing loader.ko reveals several important details:

• The module registers a custom binary format using __register_binfmt(&encfile_fmt, 1), hooking into Linux's binfmt subsystem.

• It only handles files with a .enc extension, checked via strrchr + strcmp.

• On execution, it calls begin_new_exec() to initialize a new process context, then maps the file into memory at base address 0x10000 with RWX permissions using vm_mmap.

Step 3 — Identify the Encryption

After mapping, the module decrypts the file in-place. The logic matches RC4:

• Key: BSIDES_SF_2026 (hardcoded)

• KSA: Initializes a 256-byte state array S, then scrambles it using the key.

• PRGA: XORs each byte of the mapped file with a keystream byte derived from S.

After decryption, execution is transferred to 0x10000 via start_thread.

Step 4 — Replicate Decryption in User Space

def rc4_decrypt(data, key):
    S = list(range(256))
    j = 0

    # KSA
    for i in range(256):
        j = (j + S[i] + key[i % len(key)]) % 256
        S[i], S[j] = S[j], S[i]

    # PRGA
    i = j = 0
    out = bytearray()

    for byte in data:
        i = (i + 1) % 256
        j = (j + S[i]) % 256
        S[i], S[j] = S[j], S[i]
        k = S[(S[i] + S[j]) % 256]
        out.append(byte ^ k)

    return out


key = b"BSIDES_SF_2026"

with open("flag.enc", "rb") as f:
    encrypted = f.read()

decrypted = rc4_decrypt(encrypted, key)

with open("flag_dec", "wb") as f:
    f.write(decrypted)

Step 5 — Execute the Decrypted Binary

$ cat flag_dec
??H?5?!?<1?CTF{fl4t_b1n4r13s_4r3_m3t4_c00l}

This reveals the flag.

---

Flag

CTF{fl4t_b1n4r13s_4r3_m3t4_c00l}

---

Key Takeaways

• The binfmt subsystem is powerful. Kernel modules can completely redefine how executables are loaded, allowing non-ELF binaries to run transparently.

• A missing ELF header points to a custom loader. When standard tools fail on an "executable," look for a registered binary format handler.

• Hardcoded keys are common in CTF loaders. Always check for embedded strings when reversing decryption routines.

• You don't need to run the kernel module. Replicating the decryption algorithm in user space is simpler, safer, and sufficient.

• Recognizing RC4 matters. The KSA + PRGA pattern is distinctive — learning to spot it by inspection speeds up reverse engineering significantly.

---

Tools Used

Tool	Purpose
file / readelf / objdump	Confirm flag.enc is not a standard ELF binary
Ghidra / IDA / Binary Ninja	Reverse engineer loader.ko to recover the decryption logic
Python	Replicate RC4 decryption and produce flag_dec