High level design and implementation of thread-stack sharing

In this post, we will be discussing the high-level design for sharing thread stacks. Our focus would be to make the design as much POSIX compliant as possible. But first - 

Why do we need to share thread stacks?

There are certain operations in RTEMS in which a thread writes/reads to/from the stack of another thread. This includes IPC mechanisms such as message queues, in fact, all blocking reads ( sockets, files, etc.) read/write to the stack of a different thread. Now if we have completely isolated thread stacks from each other, these valid operations will give fatal exceptions whenever they read/write to the stack of a different thread. Hence we need to share thread stacks for enabling these operations.

The mechanism for sharing thread-stacks - In the last two posts, we discussed the strict-isolation of thread-stacks. We saw that when a thread is executing, it only has access to its stack and the global data. This is made possible by unsetting (set to NO-ACCESS) the memory attributes of the previous thread during a context-switch.

Now if we want our target thread to have access to the stack of a given thread, we need to set the memory entries of the thread-stack we want to share in the 'context' of the target thread. There are a couple of important things to consider here - 

1. The thread-stack that will be shared may have memory access permission different than its intrinsic permission, i.e. if the thread-stack has R/W permission in its 'context' it is possible that its access permission while sharing maybe Read-only.
2. We need to keep track of all the memory regions with a thread along with their access permission. This is important because we need to set/unset all these memory regions during each context switch/restoration( set with proper access permission ).  

Determining a POSIX compliant way of sharing thread-stacks - Since sharing stacks, at its core, is a mapping operation the obvious call for sharing stacks is mmap(), the problem is, mmap usually maps a file to the address space of the currently executing process, but in our case, we need to map a memory region to a thread of our choice. To do this, we need to tailor our mmap operation around different calls to fulfill our needs. This can be achieved by the following sequence of calls -

1. Get the file descriptor of the memory to be shared by opening a shared memory object through shm_open. Here we provide the access permission of the memory region to be shared. We also provide a fixed pattern of naming to the object (More on this in the next section).
2. Make a call to ftruncate that truncates the file size to the size of the stack and so that the shared memory object handler points to the stack address.
3. Now we share this file to the target thread by making a call to mmap(). Here it is important to understand the various parameters we need to pass to mmap() for a successful mmap operation. This call is usually defined as mmap( void* addr, size_t length, int prot, in flags, int fd, off_t offset ). For our operations we need to do the following - 

                           - addr - We pass the address of the target thread stack to indicate the thread with which we want to share the memory region with,i.e suppose we want to share stack space of T2 thread with that of the T1 thread we pass the address of T1 thread.

                            - length- This is the stack size of the sharing thread.

                            - prot - This is the memory access attribute of the region. We have four options -                             PROT_EXEC Pages may be executed

                     PROT_READ Pages may be read.

                     PROT_WRITE Pages may be written

                     PROT_NONE Pages may not be accessed.

                            - flags -  For stack sharing operation we necessarily need to provide the MAP_SHARED option.

                             - fd - This will be the file descriptor to the shared memory object we discussed above.

                             - offset - Since we want to share the complete stack space we keep the offset to zero.

Application requirements for sharing thread stacks-

  The following are some of the requirements that an application writer has to follow for sharing thread stacks - 

1.  Naming for shared memory objects is done in the application and the name follows a fixed naming pattern ( "/taskfs/" ), this is used to differentiate between a normal mmap operation and a stack sharing operation.
2. We need to explicitly allocate stack memory from the application for stack sharing, and then set through pthread_attr_setstack*().
3. This one is a possible improvement that has not been integrated yet-  Any application has to specify a series of repetitive steps (shm_open, ftruncate, mmap) for sharing a particular thread-stack. Maybe this can be wrapped under a function ((rtems_share_stack() ?) ) and we only make a call to that function every time we have to share a thread stack.
4. For an example of how this is done refer to this test application.