Indirect Calling of GCC Nested Functions Without Executable Stack for Older Versions

Addressing Executable Stack Requirements

The article discusses a technique for using nested functions in GCC without needing an executable stack, specifically targeting older GCC versions. While an executable stack is sometimes accepted, this method offers an alternative for environments where it is not preferred or allowed. This builds on previous discussions about using nested functions for callbacks without an executable stack in GCC 17 and Clang.

GCC's Nested Function Implementation

When taking the address of a nested function, GCC generates assembly code that places a trampoline on the stack. This trampoline is immediately invoked and loads the static frame register to a structure containing captured variables from the parent function, then jumps to the local function. The trampoline's code includes immediate constants for the static chain and the code address.

Indirect Calling Mechanism

Instead of directly invoking the trampoline, the code address and static chain can be extracted from it. These extracted pointers can then be used with the `__builtin_call_with_static_chain` built-in function to call the local function indirectly. An example using `noplate`'s `peek` and `array_slice` macros demonstrates this extraction for x86_64 architectures.

Comparison to Future GCC Features

The information obtained through this manual extraction process is equivalent to what will be provided by the new built-in functions `__builtin_call_static_chain` and `__builtin_call_code_address` in the upcoming GCC 17. This method therefore serves as a workaround for achieving similar functionality in older GCC versions before these built-ins are widely available.