\chapter{Extending GDB for \uC} \section{Introduction} A sequential program has a single call stack. A debugger knows about this call stack and provides commands to walk up/down the call frames to examine the values of local variables, as well as global variables. A concurrent program has multiple call stacks (for coroutines/tasks), so a debugger must be extended to locate these stacks for examination, similar to a sequential stack. For example, when a concurrent program deadlocks, looking at the task's call stack can locate the resource and the blocking cycle that resulted in the deadlock. Hence, it is very useful to display the call stack of each task to know where it is executing and what values it is currently computing. Because each programming language's concurrency is different, GDB has to be specifically extended for \uC. \section{Design Constraints} As mentioned in Chapter \ref{GDB}, there are several ways to extend GDB. However, there are a few design constraints on the selected mechanism. All the functions implemented should maintain similar functionality to existing GDB commands. In addition to functional requirements, usability and flexibility are requirements for this project. These final requirements enable developers to be productive quickly and do more with the extensions. The extensions created for \uC are simple to use and versatile. The following new GDB command are all implemented through the Python API for GDB. Python is a scripting language with built-in data structures and functions that enables the development of more complex operations and saves time on development. \section{\uC source-code example} Listing \ref{uC-src-code} shows a \uC program that implicitly creates two clusters, system and user, which implicitly have a processor (kernel thread) and processor task. The program explicitly creates three additional processors and ten tasks on the user cluster. \begin{figure} \begin{lstlisting}[numbers=left, xleftmargin=4.0ex, style=C++, caption={\uC source code used for GDB commands},label={uC-src-code}, basicstyle=\small] _Task T { const int tid; std::string name; void f(int param) { if ( param != 0 ) f( param - 1 ); // recursion for ( volatile size_t i = 0; i < 100000000; i += 1 ); // delay int x = 3; std::string y = "example"; } // breakpoint void main() { if ( tid != 0 ) // T0 goes first for ( volatile size_t i = 0; i < 1000000000; i += 1 ) // delay if ( i % 10000000 == 0 ) yield(); // execute other tasks f(3); } public: T(const int tid) : tid( tid ) { name = "T" + std::to_string(tid); setName(name.c_str()); } }; int main() { uProcessor procs[3]; // extra processors const int numTasks = 10; T * tasks[numTasks]; // extra tasks // allocate tasks with different names for (int id = 0; id < numTasks; id += 1) { tasks[id] = new T(id); } // deallocate tasks for (int id = 0; id < numTasks; id += 1) { delete tasks[id]; } } \end{lstlisting} \end{figure} \section{Existing User-defined GDB Commands} Listing \ref{uC-callstack} shows the GDB output at the base case of the recursion for one of the tasks created in the \uC program in listing \ref{uC-src-code}. The task is stopped at line 10. The backtrace shows the three calls to function \verb|f|, started in the task's \verb|main|. The top two frames (5 and 6) are administrative frames from \uC. The values of the argument and local variables are printed. \begin{lstlisting}[caption={Call stack of function \texttt{a} in the \uC program from listing \ref{uC-src-code}}, label={uC-callstack}, basicstyle=\small\tt] (gdb) backtrace #0 T::f (this=0xa4f950, param=0) at test.cc:10 #1 0x000000000041e509 in T::f (this=0xa4f950, param=1) at test.cc:6 #2 0x000000000041e509 in T::f (this=0xa4f950, param=2) at test.cc:6 #3 0x000000000041e509 in T::f (this=0xa4f950, param=3) at test.cc:6 #4 0x000000000041e654 in T::main (this=0xa4f950) at test.cc:15 #5 0x0000000000428de2 in ...::invokeTask (This=...) at ... #6 0x0000000000000000 in ?? () (gdb) info args this = 0xa4f950 param = 0 (gdb) info locals x = 3 y = "example" \end{lstlisting} \subsection{Listing all clusters in a \uC program} Listing \ref{clusters-command} shows the new command \verb|clusters| to list all program clusters along with their associated address. The output shows the two \uC implicitly created clusters. \begin{lstlisting}[caption={clusters command}, label={clusters-command}, basicstyle=\small\tt] (gdb) clusters Name Address systemCluster 0x65a300 userCluster 0x7ca300 \end{lstlisting} \subsection{Listing all processors in a cluster} Listing \ref{cluster-procs} shows the new command \verb|processors|, which requires a cluster argument to show all the processors in that cluster. In particular, this example shows that there are four processors in the \verb|userCluster|, with their associated address, PID, preemption and spin. \begin{lstlisting}[caption={processors command}, label={cluster-procs}, basicstyle=\small\tt] (gdb) processors Address PID Preemption Spin 0x7ccc30 8421504 10 1000 0x8c9b50 9478272 10 1000 0x8c9d10 10002560 10 1000 0x8c9ed0 10530944 10 1000 \end{lstlisting} \subsection{Listing all tasks in all clusters} Listing \ref{tasks} shows the new command \verb|task| with the \verb|all| argument to list all the tasks in a \uC program at this point in the execution snapshot. The internal \uC threads (implicitly created) are numbered with negative identifiers, while those created by the application are numbered with zero/positive. The \verb|*| indicates the \uC thread (\verb|T0|) that encountered the breakpoint at line 10. GDB stops all execution and the states of the other threads are ready, running, or blocked. If the argument \verb|all| is removed, only internal information about the \verb|userCluster| and its implicitly created threads is printed, which is sufficient for most applications. \begin{lstlisting}[caption={task command for displaying all tasks for all clusters}, label={tasks}, basicstyle=\footnotesize\tt] (gdb) task all Cluster Name Address systemCluster 0x65a300 ID Task Name Address State -1 uProcessorTask 0x6c99c0 uBaseTask::Blocked -2 uSystemTask 0x789f40 uBaseTask::Blocked userCluster 0x7ca300 ID Task Name Address State -1 uProcessorTask 0x80ced0 uBaseTask::Blocked -2 uBootTask 0x659dd0 uBaseTask::Blocked 0 main 0x7fffffffe490 uBaseTask::Blocked -3 uProcessorTask 0x90e810 uBaseTask::Blocked -4 uProcessorTask 0x98ee00 uBaseTask::Blocked -5 uProcessorTask 0xa0f3f0 uBaseTask::Blocked * 1 T0 0xa4f950 uBaseTask::Running 2 T1 0xa8fce0 uBaseTask::Running 3 T2 0xad0070 uBaseTask::Running 4 T3 0xb10400 uBaseTask::Ready 5 T4 0xb50790 uBaseTask::Ready 6 T5 0xb90b20 uBaseTask::Ready 7 T6 0xbd0eb0 uBaseTask::Ready 8 T7 0xc11240 uBaseTask::Ready 9 T8 0xc515d0 uBaseTask::Running 10 T9 0xc91960 uBaseTask::Ready \end{lstlisting} \subsection{Listing all tasks in a cluster} Listing \ref{cluster-tasks} shows the new command \verb|task| with a cluster argument to list only the names of the tasks on that cluster. In this version of the command \verb|task|, the associated address for each task and its state is displayed. \begin{lstlisting}[caption={task command for displaying all tasks in a cluster}, label={cluster-tasks}, basicstyle=\small\tt] (gdb) task systemCluster ID Task Name Address State -1 uProcessorTask 0x6c99c0 uBaseTask::Blocked -2 uSystemTask 0x789f40 uBaseTask::Blocked \end{lstlisting} \section{Changing Stacks} The next extension for displaying information is writing new commands that allow stepping from one \uC task to another. Each switching remembers the task tour in a LIFO way. This semantics means push and pop commands are needed. The push is performed by the \verb|task| command with a task argument. The pop is performed by the new command \verb|prevtask| or shorthand \verb|prev|. \subsection{Task Switching} The task argument for pushing is a relative id within a cluster or absolute address on any cluster. For instance, to switch from any task to task \verb|T2| seen in listing \ref{tasks}, the first command in listing \ref{task-addr-arguments} uses relative id (3) implicitly from the \verb|userCluster|, the second command uses an absolute address (\verb|0xad0070|), and the third command uses relative id (3) with the explicit \verb|userCluster|. Core functionality of these approaches is the same. Finally, the \verb|prevtask| command is used to unwind the stack until it is empty. \begin{lstlisting}[caption={task command arguments}, label={task-addr-arguments}, basicstyle=\small\tt] (gdb) task 3 (gdb) task 0xad0070 (gdb) task 3 userCluster (gdb) prevtask ... (gdb) prev ... (gdb) prev ... (gdb) prev empty stack \end{lstlisting} \subsection{Switching Implementation} To implement the task tour, it is necessary to store the context information for every context switching. This requirement means the \verb|task| command needs to store this information every time it is invoked. \begin{figure} \centering \includegraphics[width=8cm]{uContext_stack} \caption{Machine context (uMachContext) for each task} \label{machine-context} \vspace*{0.5in} \begin{lstlisting}[style=Python, caption={Abridged \texttt{push\_task} source code}, label={pushtask-code}, basicstyle=\small\tt] # get GDB type of uContext_t * uContext_t_ptr_type = gdb.lookup_type('UPP::uMachContext::uContext_t').pointer() # retrieve the context object from a task and cast it to the type uContext_t * task_context = task['context'].cast(uContext_t_ptr_type) # the offset where sp would be starting from uSwitch function sp_address_offset = 48 # lookup the value of stack pointer (sp), frame pointer (fp), # program counter (pc) xsp = task_context['SP'] + sp_address_offset xfp = task_context['FP'] if not gdb.lookup_symbol('uSwitch'): print('uSwitch symbol is unavailable') return # This value is calculated here because we always here when the task is in # blocked state xpc = get_addr(gdb.parse_and_eval('uSwitch').address + 28) # must switch back to frame-0 to set 'pc' register with the value of xpc gdb.execute('select-frame 0') # retrieve register values and push sp, fp, pc into a global stack global STACK sp = gdb.parse_and_eval('$sp') fp = gdb.parse_and_eval('$fp') pc = gdb.parse_and_eval('$pc') stack_info = StackInfo(sp = sp, fp = fp, pc = pc) STACK.append(stack_info) # update registers for new task gdb.execute('set $rsp={}'.format(xsp)) gdb.execute('set $rbp={}'.format(xfp)) gdb.execute('set $pc={}'.format(xpc)) \end{lstlisting} \end{figure} Figure \ref{machine-context} shows a task points to a structure containing a \verb|uContext_t| data structure, storing the stack and frame pointer, and the stack pointer. Listing \ref{pushtask-code} shows these pointers are copied into an instance of the Python tuple \verb|StackInfo| for every level of task switching. This tuple also stores information about the program counter that is calculated from the address of the \verb|uSwitch| assembly function because a task always stops in \verb|uSwitch| when its state is blocked. Similarly, switching commands retrieve this context information but from the task that a user wants to switch to, and sets the equivalent registers to the new values. To push using the \verb|task| command, the value of the hardware stack pointer \verb|rsp| register, frame pointer \verb|rbp| register, and program counter register \verb|pc| are copied from the blocked task's save-area to the Python stack. To pop using the \verb|prevtask| command, the three registers are moved from the Python stack to the appropriate hardware registers. Popping an empty stack prints a warning. Note, for tasks running when a breakpoint is encountered, the task's save-area is out-of-date; i.e., the save area is only updated on a context switch, and a running task's stack represents the current unstored state for that task, which will be stored at the next context switch. Hence, to examine running tasks, it is necessary to use the GDB \verb|info threads| and \verb|thread| commands to examine and then step onto running tasks. Listing \ref{rr-tasks} shows how to examine ready and running tasks. Task \verb|T3| is ready (see Listing \ref{tasks}) because it was forced to context switch because of a time-slice preemption. Switching to \verb|T3|, which is relative id 4, and listing its the backtrace (stack frames) shows frames 0--6, which are the execution sequence for a time-slice preemption (and can be ignored), and frames 7--9, which are the frames at the point of preemption. Frame 7 shows \verb|T3| is at line 14 in the test program (see Listing \ref{uC-src-code}). Switching to running task \verb|T1|, which is relative id 2 (see Listing \ref{tasks}), and listing its backtrace shows a similar backtrace to ready task \verb|T3|. However, this backtrace contains stale information. The GDB command \verb|info threads| shows the status of each kernel thread in the application, which represents the true location of each running thread. By observation, it can be seen that thread 2 is executing task \verb|0xa8fce0|, which is task \verb|T1|. Switching to kernel thread 2 via GDB command \verb|thread 2| and listing its backtrace show that task \verb|T1| is current executing at line 13 in the test program. \begin{figure} \begin{lstlisting}[numbers=left, xleftmargin=3.0ex, caption={Examine ready/running tasks}, label={rr-tasks}, basicstyle=\footnotesize\tt] (gdb) task 4 #0 T::f (this=0xa4f950, param=0) at test.cc:10 (gdb) backtrace #0 uSwitch () at /u0/usystem/software/u++-7.0.0/src/kernel/uSwitch-x86_64.S:64 #1 0x000000000042bd5c in uBaseCoroutine::taskCxtSw (this=0x8c9d28) ... #2 0x000000000042fff4 in UPP::uProcessorKernel::scheduleInternal ... #3 0x000000000042d4b6 in uBaseTask::uYieldInvoluntary ... #4 0x000000000042172f in uKernelModule::rollForward ... #5 0x000000000042f4fe in UPP::uSigHandlerModule::sigAlrmHandler ... #6 #7 0x000000000041e620 in T::main (`this=0xb10400`) at test.cc:14 #8 0x0000000000428de2 in UPP::uMachContext::invokeTask (This=...) ... #9 0x0000000000000000 in ?? () (gdb) task 2 (gdb) backtrace #0 uSwitch () at /u0/usystem/software/u++-7.0.0/src/kernel/uSwitch-x86_64.S:64 #1 0x000000000042bd70 in uBaseCoroutine::taskCxtSw (this=0x8c9b68) ... #2 0x000000000042fff4 in UPP::uProcessorKernel::scheduleInternal ... #3 0x000000000042d4b6 in uBaseTask::uYieldInvoluntary ... #4 0x000000000042172f in uKernelModule::rollForward ... #5 0x000000000042f50c in UPP::uSigHandlerModule::sigAlrmHandler ... #6 #7 0x000000000041e620 in T::main (`this=0xa8fce0`) at test.cc:14 #8 0x0000000000428de2 in UPP::uMachContext::invokeTask (This=...) ... #9 0x0000000000000000 in ?? () (gdb) info threads Id Target Id Frame 1 Thread 0x7ffff7fc8780 (LWP 7425) "a.out" 0x00007ffff6d74826 in ... 2 Thread 0x808080 (LWP 7923) "a.out" 0x41e5fc in T::main `(this=0xa8fce0`) at test.cc:13 * 3 Thread 0x90a080 (LWP 7926) "a.out" uSwitch () ... 4 Thread 0x98a080 (LWP 7929) "a.out" T::main (this=0xad0070) at test.cc:14 5 Thread 0xa0b080 (LWP 7931) "a.out" 0x41e629 in T::main (this=0xc515d0) at test.cc:14 (gdb) thread 2 #1 0x000000000041e509 in T::f (this=0xa4f950, param=1) at test.cc:6 6 if ( param != 0 ) f( param - 1 ); // recursion [Switching to thread 2 (Thread 0x808080 (LWP 7923))] #0 0x000000000041e5fc in T::main (`this=0xa8fce0`) at test.cc:13 13 for ( volatile size_t i = 0; i < 1000000000; i += 1 ) // delay (gdb) backtrace #0 0x000000000041e5fc in T::main (`this=0xa8fce0`) at test.cc:13 #1 0x0000000000428de2 in UPP::uMachContext::invokeTask (This=...) ... #2 0x0000000000000000 in ?? () \end{lstlisting} \end{figure} \subsection{Continuing Implementation} When a breakpoint or error is encountered, all concurrent execution stops. The state of the program can now be examined and changed; after which the program may be continued. Continuation must always occur from the top of the stack (current call) for each task, and at the specific task where GDB stopped execution. However, during a task tour, the new GDB commands change the notion of the task where execution stopped to make it possible to walk other stacks. Hence, it is a requirement for continuation that the task walk always return to frame-0 of the original stopped task before any program continuation \cite{Reference11}. % For every new function call, a new stack frame is created and the values of all the registers are % changed for that frame. Therefore, in order to see the true value of hardware registers, innermost % frame that is frame-0 must be selected \cite{Reference11}. However, it is possible to not be in % frame-0, so prior to setting these values, the command must switch back to the innermost % (currently executing) frame first. To provide for this requirement, the original stop task is implicitly remembered, and there is a new \verb|reset| command that \emph{must} be explicitly executed before any continuation to restore the locate state. To prevent errors from forgetting to call the \verb|reset| command, additional hooks are added to the existing built-in GDB continuation commands to implicitly call \verb|reset|. The following list of these commands results from GDB documentation \cite{Reference15} and similar work done for KOS \cite{Reference14}. \begin{lstlisting}[caption={Built-in GDB commands that allow continuation of a program}, label={continue-cmds}, basicstyle=\small\tt] continue,next,nexti,step,stepi,finish,advance,jump,signal,until,run,thread, reverse-next,reverse-step,reverse-stepi,reverse-continue,reverse-finish \end{lstlisting} % These hooks call a new command called \verb|reset| prior to executing the command to enable % continuation of a program to ensure that the program's context is automatically switched back to % the context of the task that initiates the first context switch. The \verb|reset| command behaves % as same as the command \verb|prevtask|, however, it goes back directly to where the task is when % the program last stops, which is the first task in the task tour. \section{Result} The current implementation successfully allows users to display a snapshot of \uC execution with respect to clusters, processors, and tasks. With this information it is possible to tour the call stacks of the tasks to see execution locations and data values. Additionally, users are allowed to continue the execution where the program last pauses assuming that the program has not crashed. The continuation of execution is done by automatically reversing the task walk from any existing GDB commands such as \verb|continue|, or a user can manually reverse the task walk using the command \verb|prevtask| and then continue.