Thread-stack isolation on RTEMS

After isolating memory blocks on RTEMS with ARMv7-A MMU, it is time we isolate thread stacks.

But, before we do that, there are some concepts about threads on RTEMS that we should understand.

• Stack Allocation -   In RTEMS, the stack allocation mechanism is user-configurable, this means that the user can have a custom mechanism for allocating stacks for their BSP. This is done by defining CONFIGURE_TASK_ALLOCATOR_INIT with the allocation function in the application code. 

       

• Context Initialization -  Each thread has its own set of registers( stack pointer, program counter) and relevant attributes ( thread id ) for its execution context that need to be assigned to it during thread initialization. In RTEMS, context initialization is done through cpu-specific _CPU_Context_Initialize() function. The Context_Control structure stores the thread register and related attributes, it is initialized by a call to _CPU_Context_Iniitalize().

Context_Control structure

• Context switch and restoration - In any context switch procedure we need to save the register state of the thread including the stack pointer and the program counter of the executing thread and then switch to the heir thread by loading the program counter with the address of the heir thread stack pointer.                                                                          In RTEMS this is done through CPU-specific assembly code. During context-switch, we save the registers of the executing thread, load the register values of the heir thread and switch to  _Thread_Do_Dispatch() by loading the program counter with the pointer to the handler function.  For restoration, we simply restore the register details by loading them from the 'R0' register and similarly branch to _Thread_Handler() function.

                                     

Context-Switching Code

Isolating Thread Stacks - 

• Setting memory attributes dynamically -  We have already isolated memory blocks in the previous post by setting different access permission to different memory regions. Now we need to provide a high-level mechanism to isolate these blocks so that the same framework can be used across all the architectures. The cpukit/include/rtems/score/memorymanagement.h provides the set of APIs that can be defined for the target architecture for setting memory attributes of address space. Here, we will see its implementation for ARMv7 MMU

• Flag Translation -  At the high-level implementation we need to pass generic memory attributes to memory_attributes_set() function. These attributes are translated to the architecture-specific implementation for the defined BSP architecture i.e. Suppose we pass the 'READ/WRITE' flag to the function, at the architecture level it needs to translate to the bit 'combination and position' of the access-permission bit in the control register. This is why we map the high-level flags to the low-level implementation by defining the memory_translate_flags() for the target architecture. You can check the implementation for ARM MMU here

• Allocating protected stacks -  As we had discussed above,  in RTEMS, we can have a custom stack allocation mechanism. This will be useful to us, as we can utilize this feature to allocate stack with specified memory permissions. Another utility of this feature, as we will see further ahead in the discussion, is that we can register the allocated stack to a chain (doubly linked-list) for tracking them effectively. We define our custom stack allocation mechanism in bsp/shrared/start/stackalloc.c. After we have defined this, the user should always configure the application as discussed earlier to allocate protected stacks.

• Tracking protected stacks - We must keep track of all the allocated stacks and their memory access attributes because,  during context switch, we need to unset the memory attribute of the current stack and set the memory attributes of the heir stacks, now this can be done only when we have a track of the thread stack attribute that is currently executing and the heir thread stack.

• Protected stack attributes - Every allocated stack has some attributes that we need to track for setting memory attributes for the stack space. These include the stack size, stack address, access flags, execution status, and in the case of stack-sharing, shared stack attributes. The stack-management APIs are declared in cpukit/include/rtems/score/stackmanagement.h. The stack management structure looks like this - 

• Adding allocated stacks to a chain - A very simple way of tracking stacks is by adding each allocated stack to a linked list. In RTEMS, we already have chains that are implemented as a doubly-linked list. We can set the current_stack attribute to 'true' for the most recently allocated stack and set all the other nodes to 'false' and append the stack attribute structure to the list. This way we can keep track of all the allocated stacks and their allocation status. The implementation of adding stack attributes to a chain can be found here.

• Context initialization of protected stacks - We must register/initialize the stack attributes of a particular to its Context_Control structure because the members of this structure are saved and restored during a context switch. We call prot_stack_context_initialize() from _CPU_Context_Iniitalize() register the stack attributes to the control structure.

• Context switching of protected stacks - For switching context of protected stacks, we follow the pre-existing model in RTEMS. We save the relevant registers and attributes and call the Thread_Do_Dispatch() function by loading the program counter with the address of the function. The only difference is that,  for protected stacks, we call the prot_stack_context_switch function,  which unsets the current memory attributes, from the assembly code by passing the stack attribute structure as a parameter. We load this parameter to the 'R0' register through 'LDR' instruction by specifying the proper offset into the context control structure.

• Context restoration - We follow the same approach,  as that with switching, with context restoration.  We restore the relevant registers and attributes and call the Thread_Handler() function by loading the program counter with the address of the function. The difference here is that we call prot_stack_context_restore that sets the memory attributes of the thread stack and marks the current_stack attribute as 'true'. We pass the stack attribute to the function by loading this parameter to the 'R0' register through 'LDR' instruction by specifying the proper offset into the context control structure.

This completes our thread isolation implementation. Clone and build this repo for trying out the implementation with various cases where you try to access the stack address of a dormant thread from an executing thread. Be ready to have the OS throw exceptions your way!

Note - This implementation is only tested for POSIX threads, classical RTEMS threads have not yet been tested and the implementation may leak memory when trying to isolate them.