You would think that, after all of these years of working as a BIOS engineer, in a company stuffed-full of assembly-language expertise, there would be nothing to catch us off guard. But it happened with a single line of assembly.
Appears simple. The 'call' instruction pushes the instruction pointer of the instruction following the call and jumps to the address specified by the RAX CPU register. But the called function was corrupting the stack. Here's what we saw (simplified):
sub rsp, 40
lea rdx, QWORD PTR [rsp + 48] ; points to local variable
mov QWORD PTR [rdx], 1
What is going on?!? A quick glance shows why there was stack corruption. The function only allocates 40 bytes of local storage (sub rsp, 40), but is writing a '1' to a part of the stack above the return function. There were no parameters on the stack, so it destroyed local variables of the calling assembly language function.
- The variable pointed to by edx is a local variable (not a passed in parameter).
- The function is declared as VOID x (VOID);
- This function was generated by a C/C++ compiler.
- The function was called by an assembly language.
- The same pattern occurred in other compiled code without problems.
At this point, it was time to look up the "X64 Calling Conventions" which describes how functions are called. Here is a typical stack frame used by C/C++ compilers on X64-capable processors:
So, looking at this stack frame, it is clear that the code is pointing to "RCX Home"? What is "RCX Home"? Why is it getting a pointer to it?
Let's take these one at a time. On X64 processors, the first four parameters passed to any function are passed in CPU registers RCX, RDX, R8 and R9. Any parameters after those are passed on the stack. But, even though the parameters are passed in registers, the caller also reserves space, as if the parameters were passed on the stack. This reserved space is called the "Home" or "Backing Store" for those register-passed parameters.
Ok. So far, so good. But this function was declared VOID. So there wouldn't be any parameters, right? Correct. There are no parameters. However, according to the specification, the "Home" space must still be allocated for all four registers even if there are no parameters. This is required in order to support un-prototyped functions in C.
So that answers part of the mystery. The 'lea' instruction is actually pointing to "RCX Home". Normally, when the C/C++ compiler is generating a function call, it automatically allocates space on the stack for this. But since our caller (call eax) was in assembly language, the author forgot to allocate the space, thinking it was just like IA-32 assembly. So the quick answer is:
sub rsp, 4 * sizeof (QWORD)
add rsp, 4 * sizeof (QWORD)
This code creates the "Home" area on the stack (even though the function has no parameters), calls the function and then deletes the "Home" area.
One More Mystery
But it doesn't answer all of my questions. Why was the lea edx pointing to the "RCX Home" (offset 48)? My examination of the C/C++ source code showed that this instruction was actually referring to a local variable. Aren't local variables supposed to be below RSP?
It took some detailed reading of the specification to find this little gem:
Even if the called function has fewer than 4 parameters, these 4 stack locations are effectively owned by the called function, and may be used by the called function for other purposes besides saving parameter register values.
So, the C/C++ compiler is using the extra stack space as temporary storage. It can do this because it knows that there were no function parameters. Rather using more stack, it just reuses the space already allocated by the caller. Smart stack usage. But confusing to old-time assembly language programmers.
That was one mystery, but not the only one. It turns out there is another caveat when calling from assembly to C/C++ functions. Prior to the function call, RSP must be aligned on a 16-byte boundary. By doing this, called functions can use aligned XMM instructions to access 16-byte instructions.
Looks like us old-time assembly language programmers need to keep up with the latest in compiler technology.