A (project of a) native code debugger

Architecture

The debugger works like this: you need a byteArray object with arbitrary you want to execute. This is the already assembled code. You can generate this via nativeBoost, waterfall, handwriting bytes, or whatever form you imagine, just get your assembly into a byteArray.

To work with ptrace you need a separate process, and it needs to be marked as being traced. In our case, we will start the new process and tell the OS we want to trace it. After that, we'll have a paused traceable process which we can manipulate through ptrace calls.

To aid in our work you have an assembler and a disassembler. You should look at the tests, for example:

testForkAndContinue
    | debugger assembler disassembled status |
    assembler := self newAssembler.
    assembler mov: 9 to: AL; ret.
    debugger := WpDebugger forBinary: assembler generate.
    debugger continue.
    status := debugger waitUpdate.
    self assert: status exited.
    self assert: status exitCode equals: 9.

this creates the code, then opens the debugger passing the bytearray. After that point, the process was created and is paused. So you make it continue and wait until the process does something useful for ptrace (an update can be process exited or process caused a segfault or process called for an int3 or many other things). In this case, the process should have exited and its exit code should be 9.

Infrastructure

Warp consists of a small C helper part, a nativeboost ptrace wrapper and after that is all smalltalk.

C part

We use C because we need two things: a program (executor) that executes a bytearray you pass as parameter, and function that can be called from smalltalk and that forks into the executor (remember in unix you don't create processes from nothing, you fork existing ones).

To do this, the lib takes the bytearray and writes it to a file in /tmp/. Then forks into the executor passing as arg the name of the just created file. We do it that way because we can have a small look at the generated code if needed. From console objdump -D -Mintel -b binary -m i8086 /tmp/file.

This is all put in a small folder with a makefile (run make all) to compile.

The functions we have are these, you can read their contents from debuglib.c:

void printCode(char *code, unsigned int size)
void printArgs(int argc, char *argv[])
int executeCode(char* filename)
void writeInFile(unsigned char *code, unsigned int size, char *filename)
int forkAndExecuteTracing(unsigned char *code, unsigned int size, int tracing)
int forkAndExecute(unsigned char *code, unsigned int size)
int forkAndPause(unsigned char *code, unsigned int size)

last update: Sept 19, 2013