Ways to get caught: Buffer Overflows

Buffer Overflows are a class of techniques used by bad guys to do everything they want. I attempt to give a simplified description of how it works.

In native languages like C and C++, buffers a.k.a arrays can be filled with more data than they are made to hold. (This is not strictly true. Modern OSes and compilers have ways to check this even for native code).

If the array is declared as a local array with a fixed size, just like any other local variable, a space equal to the size of the array is pushed into the stack. This is nothing new. All local variables are allocated on the stack segment. The compiler generated machine code automatically pushes all local variables onto the stack segment.

Did I say segment? Yes. For the uninitiated, let me explain: Every executable in most OSes are organized into segments during runtime. There is the data segment, code segment, stack segment and heap segment. Stack and heap segments may share the same memory space and they grow into each other. Keep in mind that stack and heap segments are not the data structures we studied in CS1102, we are talking about sections of memory allocated for an executable during runtime by the OS.

OK so back to local variables being pushed onto the machine stack. There is one other thing that get pushed onto the stack besides local variables: Return Addresses! The stack stores resturn addresses for the processor to return to after a function call. The funny thing is that even main() has a return address.

So here is what the bad guys do; Since most input to any program, be it a web server or a nuclear detonation device, is stored in arrays (remember char** argv?) the bad guys will give the program specially designed input values which overflow a local buffer and overwrite the return address of the current function to point to the address of the beginning of the array. (There are some technnicalities involved like NOP sleds because we do not know the exact address of the beginning of the array. I do not cover these). So the processor returns to the begnning of the array to execute whatever is written there. And guess what? The array contains the machine code bad guys want to execute on your system.

Lets say the bad guys targeted a web server that was running with administrator privileges. Then they would be able to execute code only an administrator would be allowed to execute. Thus buffer overflows could potentially give unlimited power for the attacker over the system.

Believe me, there are ways to upload entire VNC Server DLLs into certain Windows machines because of a buffer overflow in one of the network services a Windows machine opens for the whole world to connect to. And the attacker would be able to hijack your mouse and screen just like in the movies.

Fortunately, with the introduction of the NX bit most buffer overflows are impossible because the NX bit can be enabled to make a stack segment non-executable.