The Last Bitbender: Reversing Heaven's Gate with Binary Ninja

COMPFEST CTF 2026 · Reverse Engineering

chall.exe is a 3,072 byte PE32 that barely does anything on its own. It allocates RWX memory, copies 674 bytes into it and calls the result.

That little blob is the actual challenge. It starts in x86, escapes into x86-64 through Heaven’s Gate, decrypts the next x86 stage and switches back. Then it does it again.

i spent much less time on the crypto than on making sure i had Binary Ninja in the right architecture. The same bytes look valid in both modes. Wrong mode, plausible assembly, complete bullshit.

This was also my first time using Binary Ninja Cloud. i honestly expected a restricted demo. It handled the challenge just fine.

What the blob does

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
chall.exe (PE32)
    |
    | VirtualAlloc(RWX, 0x2a2), memcpy, call
    v
blob + 0x000  x86 bootstrap
    |          retf to CS + 0x10
    v
blob + 0x017  x86-64 stage 1
    |          decrypts [0x0d4, 0x196)
    |          retf to CS - 0x10
    v
blob + 0x0d4  x86 stage 2
    |          decrypts [0x196, 0x230)
    |          retf to CS + 0x10
    v
blob + 0x196  x86-64 stage 3
    |          decrypts [0x230, 0x282)
    |          retf to CS - 0x10
    v
blob + 0x230  x86 stage 4
               writes 16 output bytes and returns

The service throws 16 random bytes at us and expects another 16 bytes as lowercase hex. Once the mixer is translated, all of the architecture hopping can stay in the write-up. The solver is plain Python.

Starting with the boring part

i started with the original chall.exe. No patching and no extracted stage yet.

There is one relevant function at 0x401000:

Address What happens
0x40102a VirtualAlloc(0, 0x2a2, 0x3000, 0x40)
0x401065 copy 0x2a2 bytes from 0x402000
0x40107d input pointer into ECX, output pointer into EDX
0x40108c call the new allocation
0x4010a3 compare 16 output bytes with memcmp
Binary Ninja HLIL graph showing the original PE wrapper allocating, copying, calling and comparing the payload

The complete wrapper: allocate, copy, call, compare.

The .data section starts at file offset 0x600 and virtual address 0x402000. The first 0x2a2 bytes are the blob.

Directly behind the blob sits this:

1
2
input:    9c41e07db2f5361a8ad30c47e961b5f2
expected: 023a3db6ab0ec7efd2babd484c91f80f

This is great. The challenge ships its own known-answer test.

i used that as my stop sign for the live service. As long as my Python did not produce 023a...f80f, it was wrong. There is no reason to debug this through a socket.

The original challenge hash is:

1
493d571c6c2b15f8815d21c2e9cad9d711d8b9f9bc2966a44e41efdf31675bc9

The first retf

The blob entry is normal x86:

1
2
3
4
5
6
7
8
9
10
11
00000000  push ebx
00000001  push esi
00000002  push edi
00000003  push ebp
00000004  xor  eax, eax
00000006  mov  ax, cs
00000009  add  eax, 0x10
0000000c  push eax
0000000d  call 0x12
00000012  add  dword [esp], 5
00000016  retf

call leaves 0x12 on the stack. The code turns that into 0x17, pushes CS + 0x10 and executes retf.

So 0x17 is not more x86. This is the classic Heaven’s Gate switch into 64-bit mode.

Binary Ninja x86 disassembly showing the bootstrap and far return at address 0x402016

The x86 bootstrap stops at 0x402016. Do not let the disassembler continue in x86 mode.

The seven bytes at 0x17 are the whole joke:

1
48 8d 05 64 02 00 00

As x86:

1
2
dec eax
lea eax, [0x264]

As x86-64:

1
lea rax, [rip + 0x264]    ; blob + 0x282

In x86, 0x48 is dec eax. In x86-64 it is the REX.W prefix for the following lea.

The wrong listing did not contain invalid instructions. It just slowly stopped making sense. Loading the same bytes with an x86-64 entry at 0x402017 resolved the first lea to the scratch area at 0x402282. That was the confirmation i needed.

Binary Ninja Cloud does not take raw blobs

Binary Ninja Cloud wants PE, ELF, Mach-O or BNDB. It does not want 674 raw bytes.

i could have analyzed five little slices and adjusted the addresses in my head. No thanks. The stages share scratch memory and the 64-bit code uses RIP-relative addressing, so the original layout is useful.

build_analysis_containers.py puts the entire blob into a minimal one-section PE at 0x402000.

Only three things differ between the containers:

  • PE32 or PE32+
  • the entry RVA
  • the filename

The bootstrap and stage 1 containers use the untouched bytes. Stages 2 through 4 use the output from decrypt_stage.py. No patched opcodes and no edited screenshots.

Entry Architecture What it shows
0x402000 x86 bootstrap
0x402017 x86-64 stage 1
0x4020d4 x86 decrypted stage 2
0x402196 x86-64 decrypted stage 3
0x402230 x86 decrypted stage 4

This also keeps the scratch area in the correct place:

Range Size Use
[0x000, 0x017) 0x17 x86 bootstrap
[0x017, 0x0d4) 0xbd x86-64 stage 1
[0x0d4, 0x196) 0xc2 x86 stage 2
[0x196, 0x230) 0x9a x86-64 stage 3
[0x230, 0x282) 0x52 x86 stage 4
[0x282, 0x2a2) 0x20 pointers, left, right

x86-64 decrypts the first x86 stage

Stage 1 starts at 0x17.

It stores the input and output pointers from ECX and EDX, then reads the request as two little-endian 64-bit values. Those become left and right in my code. Register names are not useful names.

The decryptor itself is a 64-bit LCG:

1
2
3
4
5
6
7
8
9
10
11
state = rol64(
    (0x46662DE2AE713EE0 * 0xD1B54A32D192ED03) & MASK64,
    17,
) ^ 0x6E7A5380F8318187

for offset in range(0xD4, 0xD4 + 0xC2):
    state = (
        state * 0x9E6C63C6A3C4B1D1
        + 0x2545F4914F6CDD1D
    ) & MASK64
    blob[offset] ^= state >> 56

It decrypts 0xc2 bytes starting at 0xd4. The loop stops at 0x196, exactly where the next mode starts.

Stage 1 also performs the first request-dependent operation:

1
left ^= 0xA6F1C0D93B5E2748

Then another far return subtracts 0x10 from the code segment and we are back in x86.

Binary Ninja HLIL showing the stage 1 decrypt loop and return from x86-64 to x86

Decrypt loop on the left, shared state and the return to x86 on the right.

This is where i separated unpacking from solving. The LCG reveals the next code stage. The XOR modifies the request. They happen next to each other in assembly, but they have no business living in the same Python function.

The carry i lost in stage 2

The decrypted x86 stage uses call and pop to find its base. Right after that is the part i got wrong:

1
2
3
4
5
6
7
mov eax, dword [left]
mov ecx, dword [right]
mul ecx
add eax, dword [left]
mov edi, edx
mov edx, dword [left + 4]
adc edx, edi

i translated it one instruction at a time. Low 32 bits looked right, high 32 bits did not. i had thrown away half of mul.

mul ecx writes its result to EDX:EAX. The following add and adc add all 64 bits to left.

So the operation is:

1
2
3
4
left = (
    left
    + (left & 0xFFFFFFFF) * (right & 0xFFFFFFFF)
) & MASK64

Afterwards, two shld instructions form a 64-bit rotate:

1
2
right = rol64(right, 13)
left ^= right

Then a 32-bit LCG decrypts 0x9a bytes at 0x196:

1
2
3
4
5
6
7
8
state = 0x5F3A19C7

for offset in range(0x196, 0x196 + 0x9A):
    state = (
        state * 0x2C9277B5
        + 0xAC564B05
    ) & 0xFFFFFFFF
    blob[offset] ^= state >> 24

The stage ends in another retf, back to x86-64.

Binary Ninja x86 disassembly showing mul, add, adc, shld and the stage 3 decryptor

Raw disassembly wins here. mul, add, adc and both shld instructions are all visible.

The last two stages

Stage 3 is x86-64 again and finishes the mixer:

1
2
3
right = rol64((right + left) & MASK64, 29)
right = (right * 0xFF51AFD7ED558CCD) & MASK64
left = rol64((left + right) & MASK64, 17)

It also decrypts the last 0x52 bytes at 0x230:

1
2
3
4
5
6
7
8
state = 0xB5297A4D2C1F60E9

for offset in range(0x230, 0x230 + 0x52):
    state = (
        state * 0x2545F4914F6CDD1D
        + 0x9E6C63C6A3C4B1D1
    ) & MASK64
    blob[offset] ^= state >> 56

The two constants traded places compared with stage 1. i kept both loops separate in the script because making one clever generic LCG here only makes it easier to pass the wrong arguments.

Binary Ninja x86-64 disassembly showing the mixer, final decrypt loop and return to x86

Mixer at the top, decrypt loop on the right, final trip back to x86 at the bottom.

Stage 4 is x86 and finally writes the answer:

1
2
output0 = left ^ right
output1 = (left + right) & MASK64

The addition is another add and adc pair. Both values go out little-endian.

Then ebp, edi, esi and ebx are popped. Those match the four pushes at blob entry. A normal ret takes us back to the wrapper.

Binary Ninja x86 disassembly showing the final output stores, restored registers and return

Two results, four restored registers and one completely normal ret.

What remains for the solver

The live service does not care how its code was packed. It only needs this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
from struct import pack, unpack

MASK64 = (1 << 64) - 1


def rol64(value: int, count: int) -> int:
    return ((value << count) | (value >> (64 - count))) & MASK64


def transform(block: bytes) -> bytes:
    left, right = unpack("<QQ", block)

    left ^= 0xA6F1C0D93B5E2748
    left = (
        left
        + (left & 0xFFFFFFFF) * (right & 0xFFFFFFFF)
    ) & MASK64

    right = rol64(right, 13)
    left ^= right

    right = rol64((right + left) & MASK64, 29)
    right = (right * 0xFF51AFD7ED558CCD) & MASK64
    left = rol64((left + right) & MASK64, 17)

    return pack(
        "<QQ",
        left ^ right,
        (left + right) & MASK64,
    )

The LCGs are gone because they only reveal code. transform() contains only operations involving the request.

Checking my work

decrypt_stage.py extracts the blob, writes every decrypted stage and runs the test vector from the PE:

1
2
3
4
5
6
7
$ python3 decrypt_stage.py
.../bitbender.bin
.../stage2.bin
.../stage3.bin
.../stage4.bin
.../bitbender.decrypted.bin
self-test transform reproduced

The relevant assertion is:

1
2
3
assert transform(bytes.fromhex(
    "9c41e07db2f5361a8ad30c47e961b5f2"
)).hex() == "023a3db6ab0ec7efd2babd484c91f80f"

This caught my broken mul translation. It would also catch all of these:

  • entering x86-64 at 0x18 and losing the REX.W byte at 0x17
  • translating the stage 2 mul as a 32-bit result
  • losing the carry from add into adc
  • packing the final values in the wrong byte order

The same goes for the Binary Ninja containers:

1
2
3
4
5
6
$ python3 build_analysis_containers.py
37c039b7076c...  02-original-blob-x86-bootstrap.exe
2dfef4c3ba8c...  03-original-blob-x64-stage1.exe
af783a2dd57f...  04-decrypted-blob-x86-stage2.exe
239a3b104c33...  05-decrypted-blob-x64-stage3.exe
cdb9c0097400...  06-decrypted-blob-x86-stage4.exe

The service

There is not much left. The service asks for a CTFd token and prints a request:

1
2
3
CTFd access token: ...
request: aa22af6b80451cb4279fa1117dcfe2f5
response:

solve.py parses that request, calls transform() and sends the hex result:

1
2
request = bytes.fromhex(request_hex)
response = transform(request).hex()

Run it like this:

1
2
3
$ CTFD_TOKEN='...' python3 solve.py
ok
COMPFEST18{...}

After all that, the network part is two lines.

Scripts and hashes

Everything needed to reproduce it:

And the hashes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
chall.exe
493d571c6c2b15f8815d21c2e9cad9d711d8b9f9bc2966a44e41efdf31675bc9

bitbender.bin
6846131af3e6763bd877fb47aa18ad55300e85fff5cd6297a30d76d934834bdb

bitbender.decrypted.bin
db2516eab6235dd87bd55199d35ab658a243eee90b984b97eea4da57b782cb12

stage2.bin
57e7138e28285e7df47b2cf29e3fa2012351d716aa77e2123bb14aa114c96527

stage3.bin
201c3dc0b1cd90c0ec94cb3cd4e98ca498471f7a90c64ef617f0437f5d4e1908

stage4.bin
c6da629fbb0c5138841162f47908d554cec28d63f41112ae45b4766e86e05fac

Flag

1
COMPFEST18{0nly_th3_av4t4r_m4st3r3d_4ll_th3m_b1ts_iR2Yth2DpF0F8hzD}

i liked this one. Basic crypto, tiny solver, horrible disassembly if you miss one architecture switch.

0x48 at offset 0x17 is basically the challenge. x86 sees dec eax. x86-64 sees the prefix that makes the next instruction work.

Binary Ninja will happily show either version. Choosing the correct one is still your job.