Changeset 95b3a9c for doc/theses

doc/theses/andrew_beach_MMath/existing.tex

-              r5e99a9a
+              r95b3a9c
 \chapter{\texorpdfstring{\CFA Existing Features}{Cforall Existing Features}}
+\chapter{\CFA Existing Features}
 \CFA (C-for-all)~\cite{Cforall} is an open-source project extending ISO C with
 …
 obvious to the reader.
 \section{\texorpdfstring{Overloading and \lstinline|extern|}{Overloading and extern}}
+\section{Overloading and \lstinline{extern}}
 \CFA has extensive overloading, allowing multiple definitions of the same name
 to be defined.~\cite{Moss18}
 …
 \section{Reference Type}
 \CFA adds a rebindable reference type to C, but more expressive than the \CC
+\CFA adds a rebindable reference type to C, but more expressive than the \Cpp
 reference.  Multi-level references are allowed and act like auto-dereferenced
 pointers using the ampersand (@&@) instead of the pointer asterisk (@*@). \CFA
 …
 Both constructors and destructors are operators, which means they are just
 functions with special operator names rather than type names in \CC. The
+functions with special operator names rather than type names in \Cpp. The
 special operator names may be used to call the functions explicitly (not
 allowed in \CC for constructors).
+allowed in \Cpp for constructors).
 In general, operator names in \CFA are constructed by bracketing an operator
 …
 matching overloaded destructor @void ^?{}(T &);@ is called.  Without explicit
 definition, \CFA creates a default and copy constructor, destructor and
 assignment (like \CC). It is possible to define constructors/destructors for
+assignment (like \Cpp). It is possible to define constructors/destructors for
 basic and existing types.
 …
 \CFA uses parametric polymorphism to create functions and types that are
 defined over multiple types. \CFA polymorphic declarations serve the same role
 as \CC templates or Java generics. The ``parametric'' means the polymorphism is
+as \Cpp templates or Java generics. The ``parametric'' means the polymorphism is
 accomplished by passing argument operations to associate \emph{parameters} at
 the call site, and these parameters are used in the function to differentiate
 …
 Note, a function named @do_once@ is not required in the scope of @do_twice@ to
 compile it, unlike \CC template expansion. Furthermore, call-site inferencing
+compile it, unlike \Cpp template expansion. Furthermore, call-site inferencing
 allows local replacement of the most specific parametric functions needs for a
 call.
 …
+}
 \end{cfa}
 The generic type @node(T)@ is an example of a polymorphic-type usage.  Like \CC
+The generic type @node(T)@ is an example of a polymorphic-type usage.  Like \Cpp
 templates usage, a polymorphic-type usage must specify a type parameter.

doc/theses/andrew_beach_MMath/features.tex

-              r5e99a9a
+              r95b3a9c
 \section{Virtuals}
+Virtual types and casts are not part of the exception system nor are they
+required for an exception system. But an object-oriented style hierarchy is a
+great way of organizing exceptions so a minimal virtual system has been added
+to \CFA.
+The pattern of a simple hierarchy was borrowed from object-oriented
+programming was chosen for several reasons.
+The first is that it allows new exceptions to be added in user code
+and in libraries independently of each other. Another is it allows for
+different levels of exception grouping (all exceptions, all IO exceptions or
+a particular IO exception). Also it also provides a simple way of passing
+data back and forth across the throw.
 Virtual types and casts are not required for a basic exception-system but are
 useful for advanced exception features. However, \CFA is not object-oriented so
 there is no obvious concept of virtuals.  Hence, to create advanced exception
 features for this work, I needed to designed and implemented a virtual-like
+there is no obvious concept of virtuals. Hence, to create advanced exception
+features for this work, I needed to design and implement a virtual-like
 system for \CFA.
+% NOTE: Maybe we should but less of the rational here.
 Object-oriented languages often organized exceptions into a simple hierarchy,
 \eg Java.
 …
 \end{center}
 The hierarchy provides the ability to handle an exception at different degrees
 of specificity (left to right).  Hence, it is possible to catch a more general
+of specificity (left to right). Hence, it is possible to catch a more general
 exception-type in higher-level code where the implementation details are
 unknown, which reduces tight coupling to the lower-level implementation.
 …
 While much of the virtual infrastructure is created, it is currently only used
 internally for exception handling. The only user-level feature is the virtual
 cast, which is the same as the \CC \lstinline[language=C++]|dynamic_cast|.
+cast, which is the same as the \Cpp \lstinline[language=C++]|dynamic_cast|.
 \label{p:VirtualCast}
 \begin{cfa}
 (virtual TYPE)EXPRESSION
 \end{cfa}
+Note, the syntax and semantics matches a C-cast, rather than the unusual \CC
+syntax for special casts. Both the type of @EXPRESSION@ and @TYPE@ must be a
+pointer to a virtual type. The cast dynamically checks if the @EXPRESSION@ type
+is the same or a subtype of @TYPE@, and if true, returns a pointer to the
+Note, the syntax and semantics matches a C-cast, rather than the function-like
+\Cpp syntax for special casts. Both the type of @EXPRESSION@ and @TYPE@ must be
+a pointer to a virtual type.
+The cast dynamically checks if the @EXPRESSION@ type is the same or a subtype
+of @TYPE@, and if true, returns a pointer to the
 @EXPRESSION@ object, otherwise it returns @0p@ (null pointer).
 …
 Exceptions are defined by the trait system; there are a series of traits, and
 if a type satisfies them, then it can be used as an exception.  The following
+if a type satisfies them, then it can be used as an exception. The following
 is the base trait all exceptions need to match.
 \begin{cfa}
 trait is_exception(exceptT &, virtualT &) {
         virtualT const & @get_exception_vtable@(exceptT *);
+        virtualT const & get_exception_vtable(exceptT *);
 };
 \end{cfa}
+The function takes any pointer, including the null pointer, and returns a
+reference to the virtual-table object. Defining this function also establishes
+the virtual type and a virtual-table pair to the \CFA type-resolver and
+promises @exceptT@ is a virtual type and a child of the base exception-type.
+\PAB{I do not understand this paragraph.}
+One odd thing about @get_exception_vtable@ is that it should always be a
+constant function, returning the same value regardless of its argument.  A
+pointer or reference to the virtual table instance could be used instead,
+however using a function has some ease of implementation advantages and allows
+for easier disambiguation because the virtual type name (or the address of an
+instance that is in scope) can be used instead of the mangled virtual table
+name.  Also note the use of the word ``promise'' in the trait
+description. Currently, \CFA cannot check to see if either @exceptT@ or
+@virtualT@ match the layout requirements. This is considered part of
+@get_exception_vtable@'s correct implementation.
+\section{Raise}
+\CFA provides two kinds of exception raise: termination
+\see{\VRef{s:Termination}} and resumption \see{\VRef{s:Resumption}}, which are
+specified with the following traits.
+The trait is defined over two types, the exception type and the virtual table
+type. This should be one-to-one, each exception type has only one virtual
+table type and vice versa. The only assertion in the trait is
+@get_exception_vtable@, which takes a pointer of the exception type and
+returns a reference to the virtual table type instance.
+The function @get_exception_vtable@ is actually a constant function.
+Recardless of the value passed in (including the null pointer) it should
+return a reference to the virtual table instance for that type.
+The reason it is a function instead of a constant is that it make type
+annotations easier to write as you can use the exception type instead of the
+virtual table type; which usually has a mangled name.
+% Also \CFA's trait system handles functions better than constants and doing
+% it this way reduce the amount of boiler plate we need.
+% I did have a note about how it is the programmer's responsibility to make
+% sure the function is implemented correctly. But this is true of every
+% similar system I know of (except Agda's I guess) so I took it out.
+There are two more traits for exceptions @is_termination_exception@ and
+@is_resumption_exception@. They are defined as follows:
 \begin{cfa}
 trait is_termination_exception(
                 exceptT &, virtualT & | is_exception(exceptT, virtualT)) {
         void @defaultTerminationHandler@(exceptT &);
+        void defaultTerminationHandler(exceptT &);
 };
+\end{cfa}
+The function is required to allow a termination raise, but is only called if a
+termination raise does not find an appropriate handler.
+Allowing a resumption raise is similar.
+\begin{cfa}
 trait is_resumption_exception(
                 exceptT &, virtualT & | is_exception(exceptT, virtualT)) {
         void @defaultResumptionHandler@(exceptT &);
+        void defaultResumptionHandler(exceptT &);
 };
 \end{cfa}
+The function is required to allow a resumption raise, but is only called if a
+resumption raise does not find an appropriate handler.
+Finally there are three convenience macros for referring to the these traits:
+@IS_EXCEPTION@, @IS_TERMINATION_EXCEPTION@ and @IS_RESUMPTION_EXCEPTION@.  Each
+takes the virtual type's name, and for polymorphic types only, the
+parenthesized list of polymorphic arguments. These macros do the name mangling
+to get the virtual-table name and provide the arguments to both sides
+\PAB{What's a ``side''?}
+In other words they make sure that a given type and virtual type is an
+exception and defines one of the two default handlers. These default handlers
+are used in the main exception handling operations \see{Exception Handling}
+and their use will be detailed there.
+However all three of these traits can be trickly to use directly.
+There is a bit of repetition required but
+the largest issue is that the virtual table type is mangled and not in a user
+facing way. So there are three macros that can be used to wrap these traits
+when you need to refer to the names:
+@IS_EXCEPTION@, @IS_TERMINATION_EXCEPTION@ and @IS_RESUMPTION_EXCEPTION@.
+All take one or two arguments. The first argument is the name of the
+exception type. Its unmangled and mangled form are passed to the trait.
+The second (optional) argument is a parenthesized list of polymorphic
+arguments. This argument should only with polymorphic exceptions and the
+list will be passed to both types.
+In the current set-up the base name and the polymorphic arguments have to
+match so these macros can be used without losing flexability.
+For example consider a function that is polymorphic over types that have a
+defined arithmetic exception:
+\begin{cfa}
+forall(Num | IS_EXCEPTION(Arithmetic, (Num)))
+void some_math_function(Num & left, Num & right);
+\end{cfa}
+\section{Exception Handling}
+\CFA provides two kinds of exception handling, termination and resumption.
+These twin operations are the core of the exception handling mechanism and
+are the reason for the features of exceptions.
+This section will cover the general patterns shared by the two operations and
+then go on to cover the details each individual operation.
+Both operations follow the same set of steps to do their operation. They both
+start with the user preforming a throw on an exception.
+Then there is the search for a handler, if one is found than the exception
+is caught and the handler is run. After that control returns to normal
+execution.
+If the search fails a default handler is run and then control
+returns to normal execution immediately. That is where the default handlers
+@defaultTermiationHandler@ and @defaultResumptionHandler@ are used.
 \subsection{Termination}
 \label{s:Termination}
+Termination raise, called ``throw'', is familiar and used in most programming
+languages with exception handling. The semantics of termination is: search the
+stack for a matching handler, unwind the stack frames to the matching handler,
+execute the handler, and continue execution after the handler. Termination is
+used when execution \emph{cannot} return to the throw. To continue execution,
+the program must \emph{recover} in the handler from the failed (unwound)
+execution at the raise to safely proceed after the handler.
+A termination raise is started with the @throw@ statement:
+Termination handling is more familiar kind and used in most programming
+languages with exception handling.
+It is dynamic, non-local goto. If a throw is successful then the stack will
+be unwound and control will (usually) continue in a different function on
+the call stack. They are commonly used when an error has occured and recovery
+is impossible in the current function.
+% (usually) Control can continue in the current function but then a different
+% control flow construct should be used.
+A termination throw is started with the @throw@ statement:
 \begin{cfa}
 throw EXPRESSION;
 \end{cfa}
 The expression must return a termination-exception reference, where the
 termination exception has a type with a @void defaultTerminationHandler(T &)@
+(default handler) defined. The handler is found at the call site using \CFA's
+trait system and passed into the exception system along with the exception
+itself.
+At runtime, a representation of the exception type and an instance of the
+exception type is copied into managed memory (heap) to ensure it remains in
+scope during unwinding. It is the user's responsibility to ensure the original
+exception object at the throw is freed when it goes out of scope. Being
+allocated on the stack is sufficient for this.
+Then the exception system searches the stack starting from the throw and
+proceeding towards the base of the stack, from callee to caller. At each stack
+frame, a check is made for termination handlers defined by the @catch@ clauses
 of a @try@ statement.
+The expression must return a reference to a termination exception, where the
+termination exception is any type that satifies @is_termination_exception@
+at the call site.
+Through \CFA's trait system the functions in the traits are passed into the
+throw code. A new @defaultTerminationHandler@ can be defined in any scope to
+change the throw's behavior (see below).
+The throw will copy the provided exception into managed memory. It is the
+user's responcibility to ensure the original exception is cleaned up if the
+stack is unwound (allocating it on the stack should be sufficient).
+Then the exception system searches the stack using the copied exception.
+It starts starts from the throw and proceeds to the base of the stack,
+from callee to caller.
+At each stack frame, a check is made for resumption handlers defined by the
+@catch@ clauses of a @try@ statement.
 \begin{cfa}
 try {
         GUARDED_BLOCK
 } @catch (EXCEPTION_TYPE$\(_1\)$ * NAME)@ { // termination handler 1
+} catch (EXCEPTION_TYPE$\(_1\)$ * NAME$\(_1\)$) {
         HANDLER_BLOCK$\(_1\)$
 } @catch (EXCEPTION_TYPE$\(_2\)$ * NAME)@ { // termination handler 2
+} catch (EXCEPTION_TYPE$\(_2\)$ * NAME$\(_2\)$) {
         HANDLER_BLOCK$\(_2\)$
+}
 \end{cfa}
+The statements in the @GUARDED_BLOCK@ are executed. If those statements, or any
+functions invoked from those statements, throws an exception, and the exception
+When viewed on its own a try statement will simply exceute the statements in
+@GUARDED_BLOCK@ and when those are finished the try statement finishes.
+However, while the guarded statements are being executed, including any
+functions they invoke, all the handlers following the try block are now
+or any functions invoked from those
+statements, throws an exception, and the exception
 is not handled by a try statement further up the stack, the termination
 handlers are searched for a matching exception type from top to bottom.
 …
 Exception matching checks the representation of the thrown exception-type is
 the same or a descendant type of the exception types in the handler clauses. If
+there is a match, a pointer to the exception object created at the throw is
+bound to @NAME@ and the statements in the associated @HANDLER_BLOCK@ are
+executed. If control reaches the end of the handler, the exception is freed,
+and control continues after the try statement.
+The default handler visible at the throw statement is used if no matching
+termination handler is found after the entire stack is searched. At that point,
+the default handler is called with a reference to the exception object
+generated at the throw. If the default handler returns, the system default
+action is executed, which often terminates the program. This feature allows
+each exception type to define its own action, such as printing an informative
+error message, when an exception is not handled in the program.
+it is the same of a descendent of @EXCEPTION_TYPE@$_i$ then @NAME@$_i$ is
+bound to a pointer to the exception and the statements in @HANDLER_BLOCK@$_i$
+are executed. If control reaches the end of the handler, the exception is
+freed and control continues after the try statement.
+If no handler is found during the search then the default handler is run.
+Through \CFA's trait system the best match at the throw sight will be used.
+This function is run and is passed the copied exception. After the default
+handler is run control continues after the throw statement.
+There is a global @defaultTerminationHandler@ that cancels the current stack
+with the copied exception. However it is generic over all exception types so
+new default handlers can be defined for different exception types and so
+different exception types can have different default handlers.
 \subsection{Resumption}
 \label{s:Resumption}
+Resumption raise, called ``resume'', is as old as termination
+raise~\cite{Goodenough75} but is less popular. In many ways, resumption is
+simpler and easier to understand, as it is simply a dynamic call (as in
+Lisp). The semantics of resumption is: search the stack for a matching handler,
+execute the handler, and continue execution after the resume. Notice, the stack
+cannot be unwound because execution returns to the raise point. Resumption is
+used used when execution \emph{can} return to the resume. To continue
+execution, the program must \emph{correct} in the handler for the failed
+execution at the raise so execution can safely continue after the resume.
+Resumption exception handling is a less common form than termination but is
+just as old~\cite{Goodenough75} and is in some sense simpler.
+It is a dynamic, non-local function call. If the throw is successful a
+closure will be taken from up the stack and executed, after which the throwing
+function will continue executing.
+These are most often used when an error occured and if the error is repaired
+then the function can continue.
 A resumption raise is started with the @throwResume@ statement:
 …
 \end{cfa}
 The semantics of the @throwResume@ statement are like the @throw@, but the
+expression has a type with a @void defaultResumptionHandler(T &)@ (default
+handler) defined, where the handler is found at the call site by the type
+system.  At runtime, a representation of the exception type and an instance of
+the exception type is \emph{not} copied because the stack is maintained during
+the handler search.
+Then the exception system searches the stack starting from the resume and
+proceeding towards the base of the stack, from callee to caller. At each stack
+frame, a check is made for resumption handlers defined by the @catchResume@
+clauses of a @try@ statement.
+expression has return a reference a type that satifies the trait
+@is_resumption_exception@. The assertions from this trait are available to
+the exception system while handling the exception.
+At runtime, no copies are made. As the stack is not unwound the exception and
+any values on the stack will remain in scope while the resumption is handled.
+Then the exception system searches the stack using the provided exception.
+It starts starts from the throw and proceeds to the base of the stack,
+from callee to caller.
+At each stack frame, a check is made for resumption handlers defined by the
+@catchResume@ clauses of a @try@ statement.
 \begin{cfa}
 try {
         GUARDED_BLOCK
 } @catchResume (EXCEPTION_TYPE$\(_1\)$ * NAME)@ { // resumption handler 1
+} catchResume (EXCEPTION_TYPE$\(_1\)$ * NAME$\(_1\)$) {
         HANDLER_BLOCK$\(_1\)$
 } @catchResume (EXCEPTION_TYPE$\(_2\)$ * NAME)@ { // resumption handler 2
+} catchResume (EXCEPTION_TYPE$\(_2\)$ * NAME$\(_2\)$) {
         HANDLER_BLOCK$\(_2\)$
+}
 \end{cfa}
+The statements in the @GUARDED_BLOCK@ are executed. If those statements, or any
+functions invoked from those statements, resumes an exception, and the
+exception is not handled by a try statement further up the stack, the
+resumption handlers are searched for a matching exception type from top to
+bottom. (Note, termination and resumption handlers may be intermixed in a @try@
+statement but the kind of raise (throw/resume) only matches with the
+corresponding kind of handler clause.)
+The exception search and matching for resumption is the same as for
+termination, including exception inheritance. The difference is when control
+reaches the end of the handler: the resumption handler returns after the resume
 rather than after the try statement. The resume point assumes the handler has
 corrected the problem so execution can safely continue.
+If the handlers are not involved in a search this will simply execute the
+@GUARDED_BLOCK@ and then continue to the next statement.
+Its purpose is to add handlers onto the stack.
+(Note, termination and resumption handlers may be intermixed in a @try@
+statement but the kind of throw must be the same as the handler for it to be
+considered as a possible match.)
+If a search for a resumption handler reaches a try block it will check each
+@catchResume@ clause, top-to-bottom.
+At each handler if the thrown exception is or is a child type of
+@EXCEPTION_TYPE@$_i$ then the a pointer to the exception is bound to
+@NAME@$_i$ and then @HANDLER_BLOCK@$_i$ is executed. After the block is
+finished control will return to the @throwResume@ statement.
 Like termination, if no resumption handler is found, the default handler
+visible at the resume statement is called, and the system default action is
+executed.
+For resumption, the exception system uses stack marking to partition the
+resumption search. If another resumption exception is raised in a resumption
+handler, the second exception search does not start at the point of the
+original raise. (Remember the stack is not unwound and the current handler is
+at the top of the stack.) The search for the second resumption starts at the
+current point on the stack because new try statements may have been pushed by
+the handler or functions called from the handler. If there is no match back to
+the point of the current handler, the search skips\label{p:searchskip} the stack frames already
+searched by the first resume and continues after the try statement. The default
+handler always continues from default handler associated with the point where
+the exception is created.
+visible at the throw statement is called. It will use the best match at the
+call sight according to \CFA's overloading rules. The default handler is
+passed the exception given to the throw. When the default handler finishes
+execution continues after the throw statement.
+There is a global @defaultResumptionHandler@ is polymorphic over all
+termination exceptions and preforms a termination throw on the exception.
+The @defaultTerminationHandler@ for that throw is matched at the original
+throw statement (the resumption @throwResume@) and it can be customized by
+introducing a new or better match as well.
+% \subsubsection?
+A key difference between resumption and termination is that resumption does
+not unwind the stack. A side effect that is that when a handler is matched
+and run it's try block (the guarded statements) and every try statement
+searched before it are still on the stack. This can lead to the recursive
+resumption problem.
+The recursive resumption problem is any situation where a resumption handler
+ends up being called while it is running.
+Consider a trivial case:
+\begin{cfa}
+try {
+        throwResume (E &){};
+} catchResume(E *) {
+        throwResume (E &){};
+}
+\end{cfa}
+When this code is executed the guarded @throwResume@ will throw, start a
+search and match the handler in the @catchResume@ clause. This will be
+call and placed on the stack on top of the try-block. The second throw then
+throws and will seach the same try block and put call another instance of the
+same handler leading to an infinite loop.
+This situation is trivial and easy to avoid, but much more complex cycles
+can form with multiple handlers and different exception types.
+To prevent all of these cases we mask sections of the stack, or equvilantly
+the try statements on the stack, so that the resumption seach skips over
+them and continues with the next unmasked section of the stack.
+A section of the stack is marked when it is searched to see if it contains
+a handler for an exception and unmarked when that exception has been handled
+or the search was completed without finding a handler.
 % This might need a diagram. But it is an important part of the justification
 …
 \end{verbatim}
+This resumption search-pattern reflect the one for termination, which matches
+with programmer expectations. However, it avoids the \emph{recursive
+resumption} problem. If parts of the stack are searched multiple times, loops
+can easily form resulting in infinite recursion.
+Consider the trivial case:
+\begin{cfa}
+try {
+        throwResume$\(_1\)$ (E &){};
+} catch( E * ) {
+        throwResume;
+}
+\end{cfa}
+Based on termination semantics, programmer expectation is for the re-resume to
+continue searching the stack frames after the try statement. However, the
+current try statement is still on the stack below the handler issuing the
+reresume \see{\VRef{s:Reraise}}. Hence, the try statement catches the re-raise
+again and does another re-raise \emph{ad infinitum}, which is confusing and
+difficult to debug. The \CFA resumption search-pattern skips the try statement
+so the reresume search continues after the try, mathcing programmer
+expectation.
+The rules can be remembered as thinking about what would be searched in
+termination. So when a throw happens in a handler; a termination handler
+skips everything from the original throw to the original catch because that
+part of the stack has been unwound, a resumption handler skips the same
+section of stack because it has been masked.
+A throw in a default handler will preform the same search as the original
+throw because; for termination nothing has been unwound, for resumption
+the mask will be the same.
+The symmetry with termination is why this pattern was picked. Other patterns,
+such as marking just the handlers that caught, also work but lack the
+symmetry whih means there is more to remember.
 \section{Conditional Catch}
+Both termination and resumption handler-clauses may perform conditional matching:
+\begin{cfa}
+catch (EXCEPTION_TYPE * NAME ; @CONDITION@)
+Both termination and resumption handler clauses can be given an additional
+condition to further control which exceptions they handle:
+\begin{cfa}
+catch (EXCEPTION_TYPE * NAME ; CONDITION)
 \end{cfa}
 First, the same semantics is used to match the exception type. Second, if the
 exception matches, @CONDITION@ is executed. The condition expression may
 reference all names in scope at the beginning of the try block and @NAME@
 introduced in the handler clause.  If the condition is true, then the handler
 matches. Otherwise, the exception search continues at the next appropriate kind
 of handler clause in the try block.
+introduced in the handler clause. If the condition is true, then the handler
+matches. Otherwise, the exception search continues as if the exception type
+did not match.
 \begin{cfa}
 try {
 …
 remaining handlers in the current try statement.
+\section{Reraise}
+\label{s:Reraise}
+\section{Rethrowing}
+\colour{red}{From Andrew: I recomend we talk about why the language doesn't
+have rethrows/reraises instead.}
+\label{s:Rethrowing}
 Within the handler block or functions called from the handler block, it is
 possible to reraise the most recently caught exception with @throw@ or
+@throwResume@, respective.
+\begin{cfa}
+catch( ... ) {
+        ... throw; // rethrow
+@throwResume@, respectively.
+\begin{cfa}
+try {
+        ...
+} catch( ... ) {
+        ... throw;
 } catchResume( ... ) {
         ... throwResume; // reresume
+        ... throwResume;
+}
 \end{cfa}
 …
 handler is generated that does a program-level abort.
 \section{Finally Clauses}
+A @finally@ clause may be placed at the end of a @try@ statement.
+Finally clauses are used to preform unconditional clean-up when leaving a
+scope. They are placed at the end of a try statement:
 \begin{cfa}
 try {
         GUARDED_BLOCK
 } ...   // any number or kind of handler clauses
 } finally {
+} ... // any number or kind of handler clauses
+... finally {
         FINALLY_BLOCK
+}
 \end{cfa}
+The @FINALLY_BLOCK@ is executed when the try statement is unwound from the
+stack, \ie when the @GUARDED_BLOCK@ or any handler clause finishes. Hence, the
+finally block is always executed.
+The @FINALLY_BLOCK@ is executed when the try statement is removed from the
+stack, including when the @GUARDED_BLOCK@ finishes, any termination handler
+finishes or during an unwind.
+The only time the block is not executed is if the program is exited before
+the stack is unwound.
 Execution of the finally block should always finish, meaning control runs off
 the end of the block. This requirement ensures always continues as if the
 finally clause is not present, \ie finally is for cleanup not changing control
 flow.  Because of this requirement, local control flow out of the finally block
 is forbidden.  The compiler precludes any @break@, @continue@, @fallthru@ or
+flow. Because of this requirement, local control flow out of the finally block
+is forbidden. The compiler precludes any @break@, @continue@, @fallthru@ or
 @return@ that causes control to leave the finally block. Other ways to leave
 the finally block, such as a long jump or termination are much harder to check,
+and at best requiring additional run-time overhead, and so are discouraged.
+and at best requiring additional run-time overhead, and so are mearly
+discouraged.
+Not all languages with exceptions have finally clauses. Notably \Cpp does
+without it as descructors serve a similar role. Although destructors and
+finally clauses can be used in many of the same areas they have their own
+use cases like top-level functions and lambda functions with closures.
+Destructors take a bit more work to set up but are much easier to reuse while
+finally clauses are good for once offs and can include local information.
 \section{Cancellation}
 …
 possible forwards the cancellation exception to a different stack.
+Cancellation is not an exception operation like termination or resumption.
 There is no special statement for starting a cancellation; instead the standard
 library function @cancel_stack@ is called passing an exception.  Unlike a
 raise, this exception is not used in matching only to pass information about
+library function @cancel_stack@ is called passing an exception. Unlike a
+throw, this exception is not used in matching only to pass information about
 the cause of the cancellation.
+Handling of a cancellation depends on which stack is being cancelled.
+(This also means matching cannot fail so there is no default handler either.)
+After @cancel_stack@ is called the exception is copied into the exception
+handling mechanism's memory. Then the entirety of the current stack is
+unwound. After that it depends one which stack is being cancelled.
 \begin{description}
 \item[Main Stack:]
 The main stack is the one used by the program main at the start of execution,
+and is the only stack in a sequential program.  Hence, when cancellation is
+forwarded to the main stack, there is no other forwarding stack, so after the
+stack is unwound, there is a program-level abort.
+and is the only stack in a sequential program. Even in a concurrent program
+the main stack is only dependent on the environment that started the program.
+Hence, when the main stack is cancelled there is nowhere else in the program
+to notify. After the stack is unwound, there is a program-level abort.
 \item[Thread Stack:]
 A thread stack is created for a @thread@ object or object that satisfies the
 @is_thread@ trait.  A thread only has two points of communication that must
+@is_thread@ trait. A thread only has two points of communication that must
 happen: start and join. As the thread must be running to perform a
+cancellation, it must occur after start and before join, so join is a
+cancellation point.  After the stack is unwound, the thread halts and waits for
+another thread to join with it. The joining thread, checks for a cancellation,
+cancellation, it must occur after start and before join, so join is used
+for communication here.
+After the stack is unwound, the thread halts and waits for
+another thread to join with it. The joining thread checks for a cancellation,
 and if present, resumes exception @ThreadCancelled@.
 …
 the exception is not caught. The implicit join does a program abort instead.
+This semantics is for safety. One difficult problem for any exception system is
+defining semantics when an exception is raised during an exception search:
+which exception has priority, the original or new exception? No matter which
+exception is selected, it is possible for the selected one to disrupt or
+destroy the context required for the other. \PAB{I do not understand the
+following sentences.} This loss of information can happen with join but as the
+thread destructor is always run when the stack is being unwound and one
+termination/cancellation is already active. Also since they are implicit they
+are easier to forget about.
+This semantics is for safety. If an unwind is triggered while another unwind
+is underway only one of them can proceed as they both want to ``consume'' the
+stack. Letting both try to proceed leads to very undefined behaviour.
+Both termination and cancellation involve unwinding and, since the default
+@defaultResumptionHandler@ preforms a termination that could more easily
+happen in an implicate join inside a destructor. So there is an error message
+and an abort instead.
+\todo{Perhaps have a more general disucssion of unwind collisions before
+this point.}
+The recommended way to avoid the abort is to handle the intial resumption
+from the implicate join. If required you may put an explicate join inside a
+finally clause to disable the check and use the local
+@defaultResumptionHandler@ instead.
 \item[Coroutine Stack:] A coroutine stack is created for a @coroutine@ object
+or object that satisfies the @is_coroutine@ trait.  A coroutine only knows of
+two other coroutines, its starter and its last resumer.  The last resumer has
+the tightest coupling to the coroutine it activated.  Hence, cancellation of
+the active coroutine is forwarded to the last resumer after the stack is
+unwound, as the last resumer has the most precise knowledge about the current
+execution. When the resumer restarts, it resumes exception
+or object that satisfies the @is_coroutine@ trait. A coroutine only knows of
+two other coroutines, its starter and its last resumer. Of the two the last
+resumer has the tightest coupling to the coroutine it activated and the most
+up-to-date information.
+Hence, cancellation of the active coroutine is forwarded to the last resumer
+after the stack is unwound. When the resumer restarts, it resumes exception
 @CoroutineCancelled@, which is polymorphic over the coroutine type and has a
 pointer to the cancelled coroutine.

doc/theses/andrew_beach_MMath/future.tex

-              r5e99a9a
+              r95b3a9c
 \item
 The implementation of termination is not portable because it includes
 hand-crafted assembly statements. These sections must be generalized to support
 more hardware architectures, \eg ARM processor.
+hand-crafted assembly statements. These sections must be ported by hand to
+support more hardware architectures, such as the ARM processor.
 \item
 Due to a type-system problem, the catch clause cannot bind the exception to a
 …
 scope of the @try@ statement, where the local control-flow transfers are
 meaningful.
+\item
+There is no detection of colliding unwinds. It is possible for clean-up code
+run during an unwind to trigger another unwind that escapes the clean-up code
+itself; such as a termination exception caught further down the stack or a
+cancellation. There do exist ways to handle this but currently they are not
+even detected and the first unwind will simply be forgotten, often leaving
+it in a bad state.
+\item
+Also the exception system did not have a lot of time to be tried and tested.
+So just letting people use the exception system more will reveal new
+quality of life upgrades that can be made with time.
 \end{itemize}

doc/theses/andrew_beach_MMath/implement.tex

-              r5e99a9a
+              r95b3a9c
 @_URC_END_OF_STACK@.
+Second, when a handler is matched, raise exception continues onto the cleanup phase.
+Second, when a handler is matched, raise exception continues onto the cleanup
+phase.
 Once again, it calls the personality functions of each stack frame from newest
 to oldest. This pass stops at the stack frame containing the matching handler.

doc/theses/andrew_beach_MMath/thesis-frontpgs.tex

-              r5e99a9a
+              r95b3a9c
         A thesis \\
         presented to the University of Waterloo \\
+        presented to the University of Waterloo \\
         in fulfillment of the \\
         thesis requirement for the degree of \\
 …
 \cleardoublepage
 %----------------------------------------------------------------------
 % EXAMINING COMMITTEE (Required for Ph.D. theses only)
 …
 \begin{center}\textbf{Examining Committee Membership}\end{center}
   \noindent
+The following served on the Examining Committee for this thesis. The decision of the Examining Committee is by majority vote.
+  \bigskip
+  \noindent
+\begin{tabbing}
+Internal-External Member: \=  \kill % using longest text to define tab length
+External Examiner: \>  Bruce Bruce \\
+The following served on the Examining Committee for this thesis. The decision
+of the Examining Committee is by majority vote.
+  \bigskip
+  \noindent
+\begin{tabbing}
+Internal-External Member: \=  \kill % using longest text to define tab length
+External Examiner: \>  Bruce Bruce \\
 \> Professor, Dept. of Philosophy of Zoology, University of Wallamaloo \\
 \end{tabbing}
   \bigskip
+\end{tabbing}
+  \bigskip
   \noindent
 \begin{tabbing}
 …
 \end{tabbing}
   \bigskip
   \noindent
   \begin{tabbing}
 …
 \end{tabbing}
   \bigskip
   \noindent
 \begin{tabbing}
 …
 \end{tabbing}
   \bigskip
   \noindent
 \begin{tabbing}
 …
   % December 13th, 2006.  It is designed for an electronic thesis.
   \noindent
+I hereby declare that I am the sole author of this thesis. This is a true copy of the thesis, including any required final revisions, as accepted by my examiners.
+  \bigskip
+I hereby declare that I am the sole author of this thesis. This is a true copy
+of the thesis, including any required final revisions, as accepted by my
+examiners.
+  \bigskip
   \noindent
 I understand that my thesis may be made electronically available to the public.

doc/theses/andrew_beach_MMath/thesis.tex

-              r5e99a9a
+              r95b3a9c
 % FRONT MATERIAL
 %----------------------------------------------------------------------
 \input{thesis-frontpgs}
+\input{thesis-frontpgs}
 %----------------------------------------------------------------------
 …
 A \gls{computer} could compute $\pi$ all day long. In fact, subsets of digits
 of $\pi$'s decimal approximation would make a good source for psuedo-random
 vectors, \gls{rvec} .
+vectors, \gls{rvec} .
 %----------------------------------------------------------------------
 …
 \begin{itemize}
 \item A well-prepared PDF should be
+\item A well-prepared PDF should be
   \begin{enumerate}
     \item Of reasonable size, {\it i.e.} photos cropped and compressed.
     \item Scalable, to allow enlargment of text and drawings.
   \end{enumerate}
+    \item Scalable, to allow enlargment of text and drawings.
+  \end{enumerate}
 \item Photos must be bit maps, and so are not scaleable by definition. TIFF and
 BMP are uncompressed formats, while JPEG is compressed. Most photos can be
 compressed without losing their illustrative value.
 \item Drawings that you make should be scalable vector graphics, \emph{not}
+\item Drawings that you make should be scalable vector graphics, \emph{not}
 bit maps. Some scalable vector file formats are: EPS, SVG, PNG, WMF. These can
 all be converted into PNG or PDF, that pdflatex recognizes. Your drawing
 package probably can export to one of these formats directly. Otherwise, a
 common procedure is to print-to-file through a Postscript printer driver to
 create a PS file, then convert that to EPS (encapsulated PS, which has a
 bounding box to describe its exact size rather than a whole page).
+all be converted into PNG or PDF, that pdflatex recognizes. Your drawing
+package probably can export to one of these formats directly. Otherwise, a
+common procedure is to print-to-file through a Postscript printer driver to
+create a PS file, then convert that to EPS (encapsulated PS, which has a
+bounding box to describe its exact size rather than a whole page).
 Programs such as GSView (a Ghostscript GUI) can create both EPS and PDF from
 PS files. Appendix~\ref{AppendixA} shows how to generate properly sized Matlab
 plots and save them as PDF.
 \item It's important to crop your photos and draw your figures to the size that
 you want to appear in your thesis. Scaling photos with the
 includegraphics command will cause loss of resolution. And scaling down
+you want to appear in your thesis. Scaling photos with the
+includegraphics command will cause loss of resolution. And scaling down
 drawings may cause any text annotations to become too small.
 \end{itemize}
 For more information on \LaTeX\, see the uWaterloo Skills for the
 Academic Workplace \href{https://uwaterloo.ca/information-systems-technology/services/electronic-thesis-preparation-and-submission-support/ethesis-guide/creating-pdf-version-your-thesis/creating-pdf-files-using-latex/latex-ethesis-and-large-documents}{course notes}.
+Academic Workplace \href{https://uwaterloo.ca/information-systems-technology/services/electronic-thesis-preparation-and-submission-support/ethesis-guide/creating-pdf-version-your-thesis/creating-pdf-files-using-latex/latex-ethesis-and-large-documents}{course notes}.
 \footnote{
 Note that while it is possible to include hyperlinks to external documents,
 it is not wise to do so, since anything you can't control may change over time.
 It \emph{would} be appropriate and necessary to provide external links to
 additional resources for a multimedia ``enhanced'' thesis.
 But also note that if the \package{hyperref} package is not included,
 as for the print-optimized option in this thesis template, any \cmmd{href}
+it is not wise to do so, since anything you can't control may change over time.
+It \emph{would} be appropriate and necessary to provide external links to
+additional resources for a multimedia ``enhanced'' thesis.
+But also note that if the \package{hyperref} package is not included,
+as for the print-optimized option in this thesis template, any \cmmd{href}
 commands in your logical document are no longer defined.
 A work-around employed by this thesis template is to define a dummy
 \cmmd{href} command (which does nothing) in the preamble of the document,
 before the \package{hyperref} package is included.
+\cmmd{href} command (which does nothing) in the preamble of the document,
+before the \package{hyperref} package is included.
 The dummy definition is then redifined by the
 \package{hyperref} package when it is included.
 …
 The classic book by Leslie Lamport \cite{lamport.book}, author of \LaTeX , is
 worth a look too, and the many available add-on packages are described by
+worth a look too, and the many available add-on packages are described by
 Goossens \textit{et al} \cite{goossens.book}.
 …
 Export Setup button in the figure Property Editor.
 \section{From the Command Line}
+\section{From the Command Line}
 All figure properties can also be manipulated from the command line. Here's an
 example:
+example:
 \begin{verbatim}
 x=[0:0.1:pi];

doc/theses/andrew_beach_MMath/unwinding.tex

-              r5e99a9a
+              r95b3a9c
 \chapter{\texorpdfstring{Unwinding in \CFA}{Unwinding in Cforall}}
+\chapter{Unwinding in \CFA}
 Stack unwinding is the process of removing stack frames (activations) from the
 …
 alternate transfers of control.
 \section{\texorpdfstring{\CFA Implementation}{Cforall Implementation}}
+\section{\CFA Implementation}
 To use libunwind, \CFA provides several wrappers, its own storage, personality

doc/theses/andrew_beach_MMath/uw-ethesis-frontpgs.tex

-              r5e99a9a
+              r95b3a9c
         \vspace*{1.0cm}
+        \Huge
+        {\bf Exception Handling in \CFA}
+        {\Huge\bf Exception Handling in \CFA}
         \vspace*{1.0cm}
-        \normalsize
         by \\
         \vspace*{1.0cm}
+        \Large
+        Andrew James Beach \\
+        {\Large Andrew James Beach} \\
         \vspace*{3.0cm}
-        \normalsize
         A thesis \\
         presented to the University of Waterloo \\
+        presented to the University of Waterloo \\
         in fulfillment of the \\
         thesis requirement for the degree of \\
 …
         \vspace*{1.0cm}
         \copyright\ Andrew James Beach \the\year \\
+        \copyright{} Andrew James Beach \the\year \\
         \end{center}
 \end{titlepage}
+% The rest of the front pages should contain no headers and be numbered using Roman numerals starting with `ii'
+% The rest of the front pages should contain no headers and be numbered using
+% Roman numerals starting with `ii'.
 \pagestyle{plain}
 \setcounter{page}{2}
+\cleardoublepage % Ends the current page and causes all figures and tables that have so far appeared in the input to be printed.
+% In a two-sided printing style, it also makes the next page a right-hand (odd-numbered) page, producing a blank page if necessary.
+\cleardoublepage % Ends the current page and causes all figures and tables
+% that have so far appeared in the input to be printed. In a two-sided
+% printing style, it also makes the next page a right-hand (odd-numbered)
+% page, producing a blank page if necessary.
 \begin{comment}
+\begin{comment}
 % E X A M I N I N G   C O M M I T T E E (Required for Ph.D. theses only)
 % Remove or comment out the lines below to remove this page
 \begin{center}\textbf{Examining Committee Membership}\end{center}
   \noindent
+The following served on the Examining Committee for this thesis. The decision of the Examining Committee is by majority vote.
+The following served on the Examining Committee for this thesis.
+The decision of the Examining Committee is by majority vote.
   \bigskip
   \noindent
 \begin{tabbing}
 Internal-External Member: \=  \kill % using longest text to define tab length
 External Examiner: \>  Bruce Bruce \\
+External Examiner: \>  Bruce Bruce \\
 \> Professor, Dept. of Philosophy of Zoology, University of Wallamaloo \\
 \end{tabbing}
+\end{tabbing}
   \bigskip
   \noindent
 \begin{tabbing}
 …
 \end{tabbing}
   \bigskip
   \noindent
   \begin{tabbing}
 …
 \end{tabbing}
   \bigskip
   \noindent
 \begin{tabbing}
 …
 \end{tabbing}
   \bigskip
   \noindent
 \begin{tabbing}
 …
   % December 13th, 2006.  It is designed for an electronic thesis.
  \begin{center}\textbf{Author's Declaration}\end{center}
  \noindent
+I hereby declare that I am the sole author of this thesis. This is a true copy of the thesis, including any required final revisions, as accepted by my examiners.
+I hereby declare that I am the sole author of this thesis. This is a true copy
+of the thesis, including any required final revisions, as accepted by my
+examiners.
   \bigskip
   \noindent
 I understand that my thesis may be made electronically available to the public.

doc/theses/andrew_beach_MMath/uw-ethesis.tex

-              r5e99a9a
+              r95b3a9c
 %======================================================================
 % University of Waterloo Thesis Template for LaTeX
 % Last Updated November, 2020
 % by Stephen Carr, IST Client Services,
+% University of Waterloo Thesis Template for LaTeX
+% Last Updated November, 2020
+% by Stephen Carr, IST Client Services,
 % University of Waterloo, 200 University Ave. W., Waterloo, Ontario, Canada
 % FOR ASSISTANCE, please send mail to request@uwaterloo.ca
 % DISCLAIMER
+% To the best of our knowledge, this template satisfies the current uWaterloo thesis requirements.
+% However, it is your responsibility to assure that you have met all requirements of the University and your particular department.
+% Many thanks for the feedback from many graduates who assisted the development of this template.
+% Also note that there are explanatory comments and tips throughout this template.
+% To the best of our knowledge, this template satisfies the current uWaterloo
+% thesis requirements. However, it is your responsibility to assure that you
+% have met all requirements of the University and your particular department.
+% Many thanks for the feedback from many graduates who assisted the
+% development of this template. Also note that there are explanatory comments
+% and tips throughout this template.
 %======================================================================
 % Some important notes on using this template and making it your own...
+% The University of Waterloo has required electronic thesis submission since October 2006.
+% See the uWaterloo thesis regulations at
+% https://uwaterloo.ca/graduate-studies/thesis.
+% This thesis template is geared towards generating a PDF version optimized for viewing on an electronic display, including hyperlinks within the PDF.
+% DON'T FORGET TO ADD YOUR OWN NAME AND TITLE in the "hyperref" package configuration below.
+% THIS INFORMATION GETS EMBEDDED IN THE PDF FINAL PDF DOCUMENT.
+% You can view the information if you view properties of the PDF document.
+% Many faculties/departments also require one or more printed copies.
+% This template attempts to satisfy both types of output.
+% The University of Waterloo has required electronic thesis submission since
+% October 2006. See the uWaterloo thesis regulations at:
+%   https://uwaterloo.ca/graduate-studies/thesis.
+% This thesis template is geared towards generating a PDF version optimized
+% for viewing on an electronic display, including hyperlinks within the PDF.
+% DON'T FORGET TO ADD YOUR OWN NAME AND TITLE in the "hyperref" package
+% configuration below. THIS INFORMATION GETS EMBEDDED IN THE FINAL PDF
+% DOCUMENT. You can view the information if you view properties of the PDF.
+% Many faculties/departments also require one or more printed copies.
+% This template attempts to satisfy both types of output.
 % See additional notes below.
+% It is based on the standard "book" document class which provides all necessary sectioning structures and allows multi-part theses.
+% If you are using this template in Overleaf (cloud-based collaboration service), then it is automatically processed and previewed for you as you edit.
+% For people who prefer to install their own LaTeX distributions on their own computers, and process the source files manually, the following notes provide the sequence of tasks:
+% It is based on the standard "book" document class which provides all
+% necessary sectioning structures and allows multi-part theses.
+% If you are using this template in Overleaf (cloud-based collaboration
+% service), then it is automatically processed and previewed for you as you
+% edit.
+% For people who prefer to install their own LaTeX distributions on their own
+% computers, and process the source files manually, the following notes
+% provide the sequence of tasks:
 % E.g. to process a thesis called "mythesis.tex" based on this template, run:
 % pdflatex mythesis     -- first pass of the pdflatex processor
 % bibtex mythesis       -- generates bibliography from .bib data file(s)
+% makeindex         -- should be run only if an index is used
+% pdflatex mythesis     -- fixes numbering in cross-references, bibliographic references, glossaries, index, etc.
+% pdflatex mythesis     -- it takes a couple of passes to completely process all cross-references
+% If you use the recommended LaTeX editor, Texmaker, you would open the mythesis.tex file, then click the PDFLaTeX button. Then run BibTeX (under the Tools menu).
+% Then click the PDFLaTeX button two more times.
+% If you have an index as well,you'll need to run MakeIndex from the Tools menu as well, before running pdflatex
+% the last two times.
+% N.B. The "pdftex" program allows graphics in the following formats to be included with the "\includegraphics" command: PNG, PDF, JPEG, TIFF
+% Tip: Generate your figures and photos in the size you want them to appear in your thesis, rather than scaling them with \includegraphics options.
+% Tip: Any drawings you do should be in scalable vector graphic formats: SVG, PNG, WMF, EPS and then converted to PNG or PDF, so they are scalable in the final PDF as well.
+% makeindex         -- should be run only if an index is used
+% pdflatex mythesis     -- fixes numbering in cross-references, bibliographic
+%                      references, glossaries, index, etc.
+% pdflatex mythesis     -- it takes a couple of passes to completely process all
+%                      cross-references
+% If you use the recommended LaTeX editor, Texmaker, you would open the
+% mythesis.tex file, then click the PDFLaTeX button. Then run BibTeX (under
+% the Tools menu). Then click the PDFLaTeX button two more times.
+% If you have an index as well, you'll need to run MakeIndex from the Tools
+% menu as well, before running pdflatex the last two times.
+% N.B. The "pdftex" program allows graphics in the following formats to be
+% included with the "\includegraphics" command: PNG, PDF, JPEG, TIFF
+% Tip: Generate your figures and photos in the size you want them to appear
+% in your thesis, rather than scaling them with \includegraphics options.
+% Tip: Any drawings you do should be in scalable vector graphic formats: SVG,
+% PNG, WMF, EPS and then converted to PNG or PDF, so they are scalable in the
+% final PDF as well.
 % Tip: Photographs should be cropped and compressed so as not to be too large.
+% To create a PDF output that is optimized for double-sided printing:
+% 1) comment-out the \documentclass statement in the preamble below, and un-comment the second \documentclass line.
+% 2) change the value assigned below to the boolean variable "PrintVersion" from " false" to "true".
+%======================================================================
+% To create a PDF output that is optimized for double-sided printing:
+% 1) comment-out the \documentclass statement in the preamble below, and
+%    un-comment the second \documentclass line.
+% 2) change the value assigned below to the boolean variable "PrintVersion"
+%    from "false" to "true".
+% ======================================================================
 %   D O C U M E N T   P R E A M B L E
 % Specify the document class, default style attributes, and page dimensions, etc.
+% Specify the document class, default style attributes, page dimensions, etc.
 % For hyperlinked PDF, suitable for viewing on a computer, use this:
 \documentclass[letterpaper,12pt,titlepage,oneside,final]{book}
+% For PDF, suitable for double-sided printing, change the PrintVersion variable below to "true" and use this \documentclass line instead of the one above:
+% For PDF, suitable for double-sided printing, change the PrintVersion
+% variable below to "true" and use this \documentclass line instead of the
+% one above:
 %\documentclass[letterpaper,12pt,titlepage,openright,twoside,final]{book}
+\usepackage{etoolbox}
 % Some LaTeX commands I define for my own nomenclature.
+% If you have to, it's easier to make changes to nomenclature once here than in a million places throughout your thesis!
+% If you have to, it's easier to make changes to nomenclature once here than
+% in a million places throughout your thesis!
 \newcommand{\package}[1]{\textbf{#1}} % package names in bold text
 \newcommand{\cmmd}[1]{\textbackslash\texttt{#1}} % command name in tt font
 \newcommand{\href}[1]{#1} % does nothing, but defines the command so the print-optimized version will ignore \href tags (redefined by hyperref pkg).
 %\newcommand{\texorpdfstring}[2]{#1} % does nothing, but defines the command
+\newcommand{\cmmd}[1]{\textbackslash\texttt{#1}} % command name in tt font
+\newcommand{\href}[1]{#1} % does nothing, but defines the command so the
+% print-optimized version will ignore \href tags (redefined by hyperref pkg).
 % Anything defined here may be redefined by packages added below...
 …
 \newboolean{PrintVersion}
 \setboolean{PrintVersion}{false}
+% CHANGE THIS VALUE TO "true" as necessary, to improve printed results for hard copies by overriding some options of the hyperref package, called below.
+% CHANGE THIS VALUE TO "true" as necessary, to improve printed results for
+% hard copies by overriding some options of the hyperref package, called below.
 %\usepackage{nomencl} % For a nomenclature (optional; available from ctan.org)
+\usepackage{amsmath,amssymb,amstext} % Lots of math symbols and environments
+\usepackage[pdftex]{graphicx} % For including graphics N.B. pdftex graphics driver
+% Lots of math symbols and environments
+\usepackage{amsmath,amssymb,amstext}
+% For including graphics N.B. pdftex graphics driver
+\usepackage[pdftex]{graphicx}
+% Removes large sections of the document.
+\usepackage{comment}
+% Adds todos (Must be included after comment.)
+\usepackage{todonotes}
 % Hyperlinks make it very easy to navigate an electronic document.
+% In addition, this is where you should specify the thesis title and author as they appear in the properties of the PDF document.
+% In addition, this is where you should specify the thesis title and author as
+% they appear in the properties of the PDF document.
 % Use the "hyperref" package
 % N.B. HYPERREF MUST BE THE LAST PACKAGE LOADED; ADD ADDITIONAL PKGS ABOVE
 \usepackage[pdftex,pagebackref=true]{hyperref} % with basic options
 %\usepackage[pdftex,pagebackref=true]{hyperref}
+                % N.B. pagebackref=true provides links back from the References to the body text. This can cause trouble for printing.
+% N.B. pagebackref=true provides links back from the References to the body
+% text. This can cause trouble for printing.
 \hypersetup{
     plainpages=false,       % needed if Roman numbers in frontpages
 …
     pdffitwindow=false,     % window fit to page when opened
     pdfstartview={FitH},    % fits the width of the page to the window
 %    pdftitle={uWaterloo\ LaTeX\ Thesis\ Template},    % title: CHANGE THIS TEXT!
+%    pdftitle={uWaterloo\ LaTeX\ Thesis\ Template}, % title: CHANGE THIS TEXT!
 %    pdfauthor={Author},    % author: CHANGE THIS TEXT! and uncomment this line
 %    pdfsubject={Subject},  % subject: CHANGE THIS TEXT! and uncomment this line
 %    pdfkeywords={keyword1} {key2} {key3}, % list of keywords, and uncomment this line if desired
+%    pdfkeywords={keyword1} {key2} {key3}, % optional list of keywords
     pdfnewwindow=true,      % links in new window
     colorlinks=true,        % false: boxed links; true: colored links
 …
     urlcolor=cyan           % color of external links
+}
+\ifthenelse{\boolean{PrintVersion}}{   % for improved print quality, change some hyperref options
+% for improved print quality, change some hyperref options
+\ifthenelse{\boolean{PrintVersion}}{
 \hypersetup{    % override some previously defined hyperref options
 %    colorlinks,%
 …
 }{} % end of ifthenelse (no else)
+\usepackage[automake,toc,abbreviations]{glossaries-extra} % Exception to the rule of hyperref being the last add-on package
+% If glossaries-extra is not in your LaTeX distribution, get it from CTAN (http://ctan.org/pkg/glossaries-extra),
+% although it's supposed to be in both the TeX Live and MikTeX distributions. There are also documentation and
+% installation instructions there.
+% Exception to the rule of hyperref being the last add-on package
+\usepackage[automake,toc,abbreviations]{glossaries-extra}
+% If glossaries-extra is not in your LaTeX distribution, get it from CTAN
+% (http://ctan.org/pkg/glossaries-extra), although it's supposed to be in
+% both the TeX Live and MikTeX distributions. There are also documentation
+% and installation instructions there.
 % Setting up the page margins...
+\setlength{\textheight}{9in}\setlength{\topmargin}{-0.45in}\setlength{\headsep}{0.25in}
+% uWaterloo thesis requirements specify a minimum of 1 inch (72pt) margin at the
+% top, bottom, and outside page edges and a 1.125 in. (81pt) gutter margin (on binding side).
+% While this is not an issue for electronic viewing, a PDF may be printed, and so we have the same page layout for both printed and electronic versions, we leave the gutter margin in.
+% Set margins to minimum permitted by uWaterloo thesis regulations:
+\setlength{\textheight}{9in}
+\setlength{\topmargin}{-0.45in}
+\setlength{\headsep}{0.25in}
+% uWaterloo thesis requirements specify a minimum of 1 inch (72pt) margin at
+% the top, bottom, and outside page edges and a 1.125 in. (81pt) gutter margin
+% (on binding side). While this is not an issue for electronic viewing, a PDF
+% may be printed, and so we have the same page layout for both printed and
+% electronic versions, we leave the gutter margin in. Set margins to minimum
+% permitted by uWaterloo thesis regulations:
 \setlength{\marginparwidth}{0pt} % width of margin notes
 % N.B. If margin notes are used, you must adjust \textwidth, \marginparwidth
 % and \marginparsep so that the space left between the margin notes and page
 % edge is less than 15 mm (0.6 in.)
+\setlength{\marginparsep}{0pt} % width of space between body text and margin notes
+\setlength{\evensidemargin}{0.125in} % Adds 1/8 in. to binding side of all
+% width of space between body text and margin notes
+\setlength{\marginparsep}{0pt}
+% Adds 1/8 in. to binding side of all
 % even-numbered pages when the "twoside" printing option is selected
+\setlength{\oddsidemargin}{0.125in} % Adds 1/8 in. to the left of all pages when "oneside" printing is selected, and to the left of all odd-numbered pages when "twoside" printing is selected
+\setlength{\textwidth}{6.375in} % assuming US letter paper (8.5 in. x 11 in.) and side margins as above
+\setlength{\evensidemargin}{0.125in}
+% Adds 1/8 in. to the left of all pages when "oneside" printing is selected,
+% and to the left of all odd-numbered pages when "twoside" printing is selected
+\setlength{\oddsidemargin}{0.125in}
+% assuming US letter paper (8.5 in. x 11 in.) and side margins as above
+\setlength{\textwidth}{6.375in}
 \raggedbottom
+% The following statement specifies the amount of space between paragraphs. Other reasonable specifications are \bigskipamount and \smallskipamount.
+% The following statement specifies the amount of space between paragraphs.
+% Other reasonable specifications are \bigskipamount and \smallskipamount.
 \setlength{\parskip}{\medskipamount}
+% The following statement controls the line spacing.
+% The default spacing corresponds to good typographic conventions and only slight changes (e.g., perhaps "1.2"), if any, should be made.
+% The following statement controls the line spacing.
+% The default spacing corresponds to good typographic conventions and only
+% slight changes (e.g., perhaps "1.2"), if any, should be made.
 \renewcommand{\baselinestretch}{1} % this is the default line space setting
 % By default, each chapter will start on a recto (right-hand side) page.
+% We also force each section of the front pages to start on a recto page by inserting \cleardoublepage commands.
+% In many cases, this will require that the verso (left-hand) page be blank, and while it should be counted, a page number should not be printed.
+% The following statements ensure a page number is not printed on an otherwise blank verso page.
+% We also force each section of the front pages to start on a recto page by
+% inserting \cleardoublepage commands. In many cases, this will require that
+% the verso (left-hand) page be blank, and while it should be counted, a page
+% number should not be printed. The following statements ensure a page number
+% is not printed on an otherwise blank verso page.
 \let\origdoublepage\cleardoublepage
 \newcommand{\clearemptydoublepage}{%
 …
 \let\cleardoublepage\clearemptydoublepage
+% Define Glossary terms (This is properly done here, in the preamble and could also be \input{} from a separate file...)
+% Define Glossary terms (This is properly done here, in the preamble and
+% could also be \input{} from a separate file...)
 \input{glossaries}
 \makeglossaries
-\usepackage{comment}
 % cfa macros used in the document
 %\usepackage{cfalab}
+% I'm going to bring back eventually.
+\makeatletter
+% Combines all \CC* commands:
+\newrobustcmd*\Cpp[1][\xspace]{\cfalab@Cpp#1}
+\newcommand\cfalab@Cpp{C\kern-.1em\hbox{+\kern-.25em+}}
+% Optional arguments do not work with pdf string. (Some fix-up required.)
+\pdfstringdefDisableCommands{\def\Cpp{C++}}
+% Colour text, formatted in LaTeX style instead of TeX style.
+\newcommand*\colour[2]{{\color{#1}#2}}
+\makeatother
 \input{common}
+\CFAStyle                                               % CFA code-style for all languages
+\lstset{language=CFA,basicstyle=\linespread{0.9}\tt}    % CFA default lnaguage
+% CFA code-style for all languages
+\CFAStyle
+% CFA default lnaguage
+\lstset{language=CFA,basicstyle=\linespread{0.9}\tt}
+% Annotations from Peter:
 \newcommand{\PAB}[1]{{\color{blue}PAB: #1}}
+% Change the style of abbreviations:
+\renewcommand{\abbrevFont}{}
 %======================================================================
 %   L O G I C A L    D O C U M E N T
 % The logical document contains the main content of your thesis.
+% Being a large document, it is a good idea to divide your thesis into several files, each one containing one chapter or other significant chunk of content, so you can easily shuffle things around later if desired.
+% Being a large document, it is a good idea to divide your thesis into several
+% files, each one containing one chapter or other significant chunk of content,
+% so you can easily shuffle things around later if desired.
 %======================================================================
 \begin{document}
 …
 % FRONT MATERIAL
 % title page,declaration, borrowers' page, abstract, acknowledgements,
+% dedication, table of contents, list of tables, list of figures, nomenclature, etc.
+%----------------------------------------------------------------------
+\input{uw-ethesis-frontpgs}
+% dedication, table of contents, list of tables, list of figures,
+% nomenclature, etc.
+%----------------------------------------------------------------------
+\input{uw-ethesis-frontpgs}
 %----------------------------------------------------------------------
 % MAIN BODY
 % We suggest using a separate file for each chapter of your thesis.
 % Start each chapter file with the \chapter command.
 % Only use \documentclass or \begin{document} and \end{document} commands in this master document.
+% Start each chapter file with the \chapter command. Only use \documentclass,
+% \begin{document} and \end{document} commands in this master document.
 % Tip: Putting each sentence on a new line is a way to simplify later editing.
 %----------------------------------------------------------------------
 …
 % Bibliography
+% The following statement selects the style to use for references.
+% It controls the sort order of the entries in the bibliography and also the formatting for the in-text labels.
+% The following statement selects the style to use for references.
+% It controls the sort order of the entries in the bibliography and also the
+% formatting for the in-text labels.
 \bibliographystyle{plain}
+% This specifies the location of the file containing the bibliographic information.
+% It assumes you're using BibTeX to manage your references (if not, why not?).
+\cleardoublepage % This is needed if the "book" document class is used, to place the anchor in the correct page, because the bibliography will start on its own page.
+% Use \clearpage instead if the document class uses the "oneside" argument
+\phantomsection  % With hyperref package, enables hyperlinking from the table of contents to bibliography
+% The following statement causes the title "References" to be used for the bibliography section:
+% This specifies the location of the file containing the bibliographic
+% information. It assumes you're using BibTeX to manage your references (if
+% not, why not?).
+\cleardoublepage % This is needed if the "book" document class is used, to
+% place the anchor in the correct page, because the bibliography will start
+% on its own page.
+% Use \clearpage instead if the document class uses the "oneside" argument.
+\phantomsection  % With hyperref package, enables hyperlinking from the table
+% of contents to bibliography.
+% The following statement causes the title "References" to be used for the
+% bibliography section:
 \renewcommand*{\bibname}{References}
 …
 \bibliography{uw-ethesis,pl}
+% Tip: You can create multiple .bib files to organize your references.
+% Just list them all in the \bibliogaphy command, separated by commas (no spaces).
+% The following statement causes the specified references to be added to the bibliography even if they were not cited in the text.
+% The asterisk is a wildcard that causes all entries in the bibliographic database to be included (optional).
+% Tip: You can create multiple .bib files to organize your references. Just
+% list them all in the \bibliogaphy command, separated by commas (no spaces).
+% The following statement causes the specified references to be added to the
+% bibliography even if they were not cited in the text. The asterisk is a
+% wildcard that causes all entries in the bibliographic database to be
+% included (optional).
 % \nocite{*}
 %----------------------------------------------------------------------
 …
 % The \appendix statement indicates the beginning of the appendices.
 \appendix
+% Add an un-numbered title page before the appendices and a line in the Table of Contents
+% Add an un-numbered title page before the appendices and a line in the Table
+% of Contents
 % \chapter*{APPENDICES}
 % \addcontentsline{toc}{chapter}{APPENDICES}
+% Appendices are just more chapters, with different labeling (letters instead of numbers).
+% Appendices are just more chapters, with different labeling (letters instead
+% of numbers).
 % \input{appendix-matlab_plots.tex}
+% GLOSSARIES (Lists of definitions, abbreviations, symbols, etc. provided by the glossaries-extra package)
+% GLOSSARIES (Lists of definitions, abbreviations, symbols, etc.
+% provided by the glossaries-extra package)
 % -----------------------------
 \printglossaries

doc/theses/fangren_yu_COOP_F20/Report.tex

-              r5e99a9a
+              r95b3a9c
 \CFA language, developed by the Programming Language Group at the University of Waterloo, has a long history, with the initial language design in 1992 by Glen Ditchfield~\cite{Ditchfield92} and the first proof-of-concept compiler built in 2003 by Richard Bilson~\cite{Bilson03}. Many new features have been added to the language over time, but the core of \CFA's type-system --- parametric functions introduced by the @forall@ clause (hence the name of the language) providing parametric overloading --- remains mostly unchanged.
 The current \CFA reference compiler, @cfa-cc@, is designed using the visitor pattern~\cite{vistorpattern} over an abstract syntax tree (AST), where multiple passes over the AST modify it for subsequent passes. @cfa-cc@ still includes many parts taken directly from the original Bilson implementation, which served as the starting point for this enhancement work to the type system. Unfortunately, the prior implementation did not provide the efficiency required for the language to be practical: a \CFA source file of approximately 1000 lines of code can take a multiple minutes to compile. The cause of the problem is that the old compiler used inefficient data structures and algorithms for expression resolution, which involved significant copying and redundant work.
+The current \CFA reference compiler, @cfa-cc@, is designed using the visitor pattern~\cite{vistorpattern} over an abstract syntax tree (AST), where multiple passes over the AST modify it for subsequent passes. @cfa-cc@ still includes many parts taken directly from the original Bilson implementation, which served as the starting point for this enhancement work to the type system. Unfortunately, the prior implementation did not provide the efficiency required for the language to be practical: a \CFA source file of approximately 1000 lines of code can take multiple minutes to compile. The cause of the problem is that the old compiler used inefficient data structures and algorithms for expression resolution, which involved significant copying and redundant work.
 This report presents a series of optimizations to the performance-critical parts of the resolver, with a major rework of the compiler data-structures using a functional-programming approach to reduce memory complexity. The improvements were suggested by running the compiler builds with a performance profiler against the \CFA standard-library source-code and a test suite to find the most underperforming components in the compiler algorithm.
 …
 \end{itemize}
 The resolver algorithm, designed for overload resolution, uses a significant amount of reused, and hence copying, for the intermediate representations, especially in the following two places:
+The resolver algorithm, designed for overload resolution, allows a significant amount of code reused, and hence copying, for the intermediate representations, especially in the following two places:
 \begin{itemize}
 \item
 …
 forall( dtype T | sized( T ) )
 T * malloc( void ) { return (T *)malloc( sizeof(T) ); } // call C malloc
 int * i = malloc();  // type deduced from left-hand size $\Rightarrow$ no size argument or return cast
+int * i = malloc();  // type deduced from left-hand size $\(\Rightarrow\)$ no size argument or return cast
 \end{cfa}
 An unbound return-type is problematic in resolver complexity because a single match of a function call with an unbound return type may create multiple candidates. In the worst case, consider a function declared that returns any @otype@ (defined \VPageref{otype}):
 …
 \begin{cfa}
 void f( int );
 double g$_1$( int );
 int g$_2$( long );
+double g$\(_1\)$( int );
+int g$\(_2\)$( long );
 f( g( 42 ) );
 \end{cfa}

doc/theses/thierry_delisle_PhD/thesis/Makefile

-              r5e99a9a
+              r95b3a9c
 BibTeX = BIBINPUTS=${TeXLIB} && export BIBINPUTS && bibtex
 MAKEFLAGS = --no-print-directory --silent
+MAKEFLAGS = --no-print-directory # --silent
 VPATH = ${Build} ${Figures}
 …
         emptytree \
         fairness \
+        io_uring \
+        pivot_ring \
         system \
+}
 …
 ## Define the documents that need to be made.
 all: thesis.pdf
 thesis.pdf: ${TEXTS} ${FIGURES} ${PICTURES} glossary.tex local.bib
+thesis.pdf: ${TEXTS} ${FIGURES} ${PICTURES} thesis.tex glossary.tex local.bib
 DOCUMENT = thesis.pdf
 …
 # Directives #
+.NOTPARALLEL:                                           # cannot make in parallel
 .PHONY : all clean                                      # not file names
 …
         ${LaTeX} $<
-build/fairness.svg : fig/fairness.py | ${Build}
-        python3 $< $@
 ## Define the default recipes.
 …
         sed -i 's/$@/${Build}\/$@/g' ${Build}/$@_t
+build/fairness.svg : fig/fairness.py | ${Build}
+        python3 $< $@
 ## pstex with inverted colors
 %.dark.pstex : fig/%.fig Makefile | ${Build}

doc/theses/thierry_delisle_PhD/thesis/local.bib

-              r5e99a9a
+              r95b3a9c
+}
+@manual{MAN:bsd/kqueue,
+  title = {KQUEUE(2) - FreeBSD System Calls Manual},
+  url   = {https://www.freebsd.org/cgi/man.cgi?query=kqueue},
+  year  = {2020},
+  month = {may}
+}
 % Apple's MAC OS X
 @manual{MAN:apple/scheduler,
 …
 % --------------------------------------------------
+% Man Pages
+@manual{MAN:open,
+  key        = "open",
+  title      = "open(2) Linux User's Manual",
+  year       = "2020",
+  month      = "February",
+}
+@manual{MAN:accept,
+  key        = "accept",
+  title      = "accept(2) Linux User's Manual",
+  year       = "2019",
+  month      = "March",
+}
+@manual{MAN:select,
+  key        = "select",
+  title      = "select(2) Linux User's Manual",
+  year       = "2019",
+  month      = "March",
+}
+@manual{MAN:poll,
+  key        = "poll",
+  title      = "poll(2) Linux User's Manual",
+  year       = "2019",
+  month      = "July",
+}
+@manual{MAN:epoll,
+  key        = "epoll",
+  title      = "epoll(7) Linux User's Manual",
+  year       = "2019",
+  month      = "March",
+}
+@manual{MAN:aio,
+  key        = "aio",
+  title      = "aio(7) Linux User's Manual",
+  year       = "2019",
+  month      = "March",
+}
+@misc{MAN:io_uring,
+  title   = {Efficient IO with io\_uring},
+  author  = {Axboe, Jens},
+  year    = "2019",
+  month   = "March",
+  version = {0,4},
+  howpublished = {\url{https://kernel.dk/io_uring.pdf}}
+}
+% --------------------------------------------------
 % Wikipedia Entries
 @misc{wiki:taskparallel,
 …
   note = "[Online; accessed 2-January-2021]"
+}
+@misc{wiki:future,
+  author = "{Wikipedia contributors}",
+  title = "Futures and promises --- {W}ikipedia{,} The Free Encyclopedia",
+  year = "2020",
+  url = "https://en.wikipedia.org/wiki/Futures_and_promises",
+  note = "[Online; accessed 9-February-2021]"
+}

doc/theses/thierry_delisle_PhD/thesis/text/core.tex

-              r5e99a9a
+              r95b3a9c
 \section{Design}
 In general, a na\"{i}ve \glsxtrshort{fifo} ready-queue does not scale with increased parallelism from \glspl{hthrd}, resulting in decreased performance. The problem is adding/removing \glspl{thrd} is a single point of contention. As shown in the evaluation sections, most production schedulers do scale when adding \glspl{hthrd}. The common solution to the single point of contention is to shard the ready-queue so each \gls{hthrd} can access the ready-queue without contention, increasing performance though lack of contention.
+In general, a na\"{i}ve \glsxtrshort{fifo} ready-queue does not scale with increased parallelism from \glspl{hthrd}, resulting in decreased performance. The problem is adding/removing \glspl{thrd} is a single point of contention. As shown in the evaluation sections, most production schedulers do scale when adding \glspl{hthrd}. The common solution to the single point of contention is to shard the ready-queue so each \gls{hthrd} can access the ready-queue without contention, increasing performance.
 \subsection{Sharding} \label{sec:sharding}
 An interesting approach to sharding a queue is presented in \cit{Trevors paper}. This algorithm presents a queue with a relaxed \glsxtrshort{fifo} guarantee using an array of strictly \glsxtrshort{fifo} sublists as shown in Figure~\ref{fig:base}. Each \emph{cell} of the array has a timestamp for the last operation and a pointer to a linked-list with a lock and each node in the list is marked with a timestamp indicating when it is added to the list. A push operation is done by picking a random cell, acquiring the list lock, and pushing to the list. If the cell is locked, the operation is simply retried on another random cell until a lock is acquired. A pop operation is done in a similar fashion except two random cells are picked. If both cells are unlocked with non-empty lists, the operation pops the node with the oldest cell timestamp. If one of the cells is unlocked and non-empty, the operation pops from that cell. If both cells are either locked or empty, the operation picks two new random cells and tries again.
+An interesting approach to sharding a queue is presented in \cit{Trevors paper}. This algorithm presents a queue with a relaxed \glsxtrshort{fifo} guarantee using an array of strictly \glsxtrshort{fifo} sublists as shown in Figure~\ref{fig:base}. Each \emph{cell} of the array has a timestamp for the last operation and a pointer to a linked-list with a lock. Each node in the list is marked with a timestamp indicating when it is added to the list. A push operation is done by picking a random cell, acquiring the list lock, and pushing to the list. If the cell is locked, the operation is simply retried on another random cell until a lock is acquired. A pop operation is done in a similar fashion except two random cells are picked. If both cells are unlocked with non-empty lists, the operation pops the node with the oldest timestamp. If one of the cells is unlocked and non-empty, the operation pops from that cell. If both cells are either locked or empty, the operation picks two new random cells and tries again.
 \begin{figure}
 …
 \paragraph{Local Information} Figure~\ref{fig:emptytls} shows an approach using dense information, similar to the bitmap, but each \gls{hthrd} keeps its own independent copy. While this approach can offer good scalability \emph{and} low latency, the liveliness and discovery of the information can become a problem. This case is made worst in systems with few processors where even blind random picks can find \glspl{thrd} in a few tries.
 I built a prototype of these approaches and none of these techniques offer satisfying performance when few threads are present. All of these approach hit the same 2 problems. First, randomly picking sub-queues is very fast but means any improvement to the hit rate can easily be countered by a slow-down in look-up speed when there are empty lists. Second, the array is already as sharded to avoid contention bottlenecks, so any denser data structure tends to become a bottleneck. In all cases, these factors meant the best cases scenario, \ie many threads, would get worst throughput, and the worst-case scenario, few threads, would get a better hit rate, but an equivalent poor throughput. As a result I tried an entirely different approach.
+I built a prototype of these approaches and none of these techniques offer satisfying performance when few threads are present. All of these approach hit the same 2 problems. First, randomly picking sub-queues is very fast. That speed means any improvement to the hit rate can easily be countered by a slow-down in look-up speed, whether or not there are empty lists. Second, the array is already sharded to avoid contention bottlenecks, so any denser data structure tends to become a bottleneck. In all cases, these factors meant the best cases scenario, \ie many threads, would get worst throughput, and the worst-case scenario, few threads, would get a better hit rate, but an equivalent poor throughput. As a result I tried an entirely different approach.
 \subsection{Dynamic Entropy}\cit{https://xkcd.com/2318/}
 In the worst-case scenario there are only few \glspl{thrd} ready to run, or more precisely given $P$ \glspl{proc}\footnote{For simplicity, this assumes there is a one-to-one match between \glspl{proc} and \glspl{hthrd}.}, $T$ \glspl{thrd} and $\epsilon$ a very small number, than the worst case scenario can be represented by $\epsilon \ll P$, than $T = P + \epsilon$. It is important to note in this case that fairness is effectively irrelevant. Indeed, this case is close to \emph{actually matching} the model of the ``Ideal multi-tasking CPU'' on page \pageref{q:LinuxCFS}. In this context, it is possible to use a purely internal-locality based approach and still meet the fairness requirements. This approach simply has each \gls{proc} running a single \gls{thrd} repeatedly. Or from the shared ready-queue viewpoint, each \gls{proc} pushes to a given sub-queue and then popes from the \emph{same} subqueue. In cases where $T \gg P$, the scheduler should also achieves similar performance without affecting the fairness guarantees.
+In the worst-case scenario there are only few \glspl{thrd} ready to run, or more precisely given $P$ \glspl{proc}\footnote{For simplicity, this assumes there is a one-to-one match between \glspl{proc} and \glspl{hthrd}.}, $T$ \glspl{thrd} and $\epsilon$ a very small number, than the worst case scenario can be represented by $T = P + \epsilon$, with $\epsilon \ll P$. It is important to note in this case that fairness is effectively irrelevant. Indeed, this case is close to \emph{actually matching} the model of the ``Ideal multi-tasking CPU'' on page \pageref{q:LinuxCFS}. In this context, it is possible to use a purely internal-locality based approach and still meet the fairness requirements. This approach simply has each \gls{proc} running a single \gls{thrd} repeatedly. Or from the shared ready-queue viewpoint, each \gls{proc} pushes to a given sub-queue and then pops from the \emph{same} subqueue. The challenge is for the the scheduler to achieve good performance in both the $T = P + \epsilon$ case and the $T \gg P$ case, without affecting the fairness guarantees in the later.
 To handle this case, I use a pseudo random-number generator, \glsxtrshort{prng} in a novel way. When the scheduler uses a \glsxtrshort{prng} instance per \gls{proc} exclusively, the random-number seed effectively starts an encoding that produces a list of all accessed subqueues, from latest to oldest. The novel approach is to be able to ``replay'' the \glsxtrshort{prng} backwards and there exist \glsxtrshort{prng}s that are fast, compact \emph{and} can be run forward and backwards. Linear congruential generators~\cite{wiki:lcg} are an example of \glsxtrshort{prng}s that match these requirements.
+To handle this case, I use a \glsxtrshort{prng}\todo{Fix missing long form} in a novel way. There exist \glsxtrshort{prng}s that are fast, compact and can be run forward \emph{and} backwards.  Linear congruential generators~\cite{wiki:lcg} are an example of \glsxtrshort{prng}s of such \glsxtrshort{prng}s. The novel approach is to use the ability to run backwards to ``replay'' the \glsxtrshort{prng}. The scheduler uses an exclusive \glsxtrshort{prng} instance per \gls{proc}, the random-number seed effectively starts an encoding that produces a list of all accessed subqueues, from latest to oldest. Replaying the \glsxtrshort{prng} to identify cells accessed recently and which probably have data still cached.
 The algorithm works as follows:

doc/theses/thierry_delisle_PhD/thesis/text/intro.tex

r5e99a9a	r95b3a9c
7	7	While previous work on the concurrent package of \CFA focused on features and interfaces, this thesis focuses on performance, introducing \glsxtrshort{api} changes only when required by performance considerations. More specifically, this thesis concentrates on scheduling and \glsxtrshort{io}. Prior to this work, the \CFA runtime used a strictly \glsxtrshort{fifo} \gls{rQ}.
8	8
9		This work exclusively concentrates on Linux as it's operating system since the existing \CFA runtime and compiler does not already support other operating systems. Furthermore, as \CFA is yet to be released, supporting version of Linux older that the latest version is not a goal of this work.
	9	This work exclusively concentrates on Linux as it's operating system since the existing \CFA runtime and compiler does not already support other operating systems. Furthermore, as \CFA is yet to be released, supporting version of Linux older than the latest version is not a goal of this work.

doc/theses/thierry_delisle_PhD/thesis/text/io.tex

-              r5e99a9a
+              r95b3a9c
 \chapter{User Level \glsxtrshort{io}}
 As mentionned in Section~\ref{prev:io}, User-Level \glsxtrshort{io} requires multiplexing the \glsxtrshort{io} operations of many \glspl{thrd} onto fewer \glspl{proc} using asynchronous \glsxtrshort{io} operations. Various operating systems offer various forms of asynchronous operations and as mentioned in Chapter~\ref{intro}, this work is exclusively focuesd on Linux.
+\chapter{User Level \io}
+As mentioned in Section~\ref{prev:io}, User-Level \io requires multiplexing the \io operations of many \glspl{thrd} onto fewer \glspl{proc} using asynchronous \io operations. Different operating systems offer various forms of asynchronous operations and as mentioned in Chapter~\ref{intro}, this work is exclusively focused on the Linux operating-system.
 \section{Existing options}
 Since \glsxtrshort{io} operations are generally handled by the
+\section{Kernel Interface}
+Since this work fundamentally depends on operating-system support, the first step of any design is to discuss the available interfaces and pick one (or more) as the foundations of the non-blocking \io subsystem.
+\subsection{\lstinline|epoll|, \lstinline|poll| and \lstinline|select|}
+\subsection{\lstinline{O_NONBLOCK}}
+In Linux, files can be opened with the flag @O_NONBLOCK@~\cite{MAN:open} (or @SO_NONBLOCK@~\cite{MAN:accept}, the equivalent for sockets) to use the file descriptors in ``nonblocking mode''. In this mode, ``Neither the @open()@ nor any subsequent \io operations on the [opened file descriptor] will cause the calling
+process to wait''~\cite{MAN:open}. This feature can be used as the foundation for the non-blocking \io subsystem. However, for the subsystem to know when an \io operation completes, @O_NONBLOCK@ must be use in conjunction with a system call that monitors when a file descriptor becomes ready, \ie, the next \io operation on it does not cause the process to wait\footnote{In this context, ready means \emph{some} operation can be performed without blocking. It does not mean an operation returning \lstinline{EAGAIN} succeeds on the next try. For example, a ready read may only return a subset of bytes and the read must be issues again for the remaining bytes, at which point it may return \lstinline{EAGAIN}.}.
+This mechanism is also crucial in determining when all \glspl{thrd} are blocked and the application \glspl{kthrd} can now block.
+\subsection{Linux's AIO}
+There are three options to monitor file descriptors in Linux\footnote{For simplicity, this section omits \lstinline{pselect} and \lstinline{ppoll}. The difference between these system calls and \lstinline{select} and \lstinline{poll}, respectively, is not relevant for this discussion.}, @select@~\cite{MAN:select}, @poll@~\cite{MAN:poll} and @epoll@~\cite{MAN:epoll}. All three of these options offer a system call that blocks a \gls{kthrd} until at least one of many file descriptors becomes ready. The group of file descriptors being waited is called the \newterm{interest set}.
+\paragraph{\lstinline{select}} is the oldest of these options, it takes as an input a contiguous array of bits, where each bits represent a file descriptor of interest. On return, it modifies the set in place to identify which of the file descriptors changed status. This destructive change means that calling select in a loop requires re-initializing the array each time and the number of file descriptors supported has a hard limit. Another limit of @select@ is that once the call is started, the interest set can no longer be modified. Monitoring a new file descriptor generally requires aborting any in progress call to @select@\footnote{Starting a new call to \lstinline{select} is possible but requires a distinct kernel thread, and as a result is not an acceptable multiplexing solution when the interest set is large and highly dynamic unless the number of parallel calls to \lstinline{select} can be strictly bounded.}.
+\paragraph{\lstinline{poll}} is an improvement over select, which removes the hard limit on the number of file descriptors and the need to re-initialize the input on every call. It works using an array of structures as an input rather than an array of bits, thus allowing a more compact input for small interest sets. Like @select@, @poll@ suffers from the limitation that the interest set cannot be changed while the call is blocked.
+\paragraph{\lstinline{epoll}} further improves these two functions by allowing the interest set to be dynamically added to and removed from while a \gls{kthrd} is blocked on an @epoll@ call. This dynamic capability is accomplished by creating an \emph{epoll instance} with a persistent interest set, which is used across multiple calls. This capability significantly reduces synchronization overhead on the part of the caller (in this case the \io subsystem), since the interest set can be modified when adding or removing file descriptors without having to synchronize with other \glspl{kthrd} potentially calling @epoll@.
+However, all three of these system calls have limitations. The @man@ page for @O_NONBLOCK@ mentions that ``[@O_NONBLOCK@] has no effect for regular files and block devices'', which means none of these three system calls are viable multiplexing strategies for these types of \io operations. Furthermore, @epoll@ has been shown to have problems with pipes and ttys~\cit{Peter's examples in some fashion}. Finally, none of these are useful solutions for multiplexing \io operations that do not have a corresponding file descriptor and can be awkward for operations using multiple file descriptors.
+\subsection{POSIX asynchronous I/O (AIO)}
+An alternative to @O_NONBLOCK@ is the AIO interface. Its interface lets programmers enqueue operations to be performed asynchronously by the kernel. Completions of these operations can be communicated in various ways: either by spawning a new \gls{kthrd}, sending a Linux signal, or by polling for completion of one or more operation. For this work, spawning a new \gls{kthrd} is counter-productive but a related solution is discussed in Section~\ref{io:morethreads}. Using interrupts handlers can also lead to fairly complicated interactions between subsystems. Leaving polling for completion, which is similar to the previous system calls. While AIO only supports read and write operations to file descriptors, it does not have the same limitation as @O_NONBLOCK@, \ie, the file descriptors can be regular files and blocked devices. It also supports batching multiple operations in a single system call.
+AIO offers two different approach to polling: @aio_error@ can be used as a spinning form of polling, returning @EINPROGRESS@ until the operation is completed, and @aio_suspend@ can be used similarly to @select@, @poll@ or @epoll@, to wait until one or more requests have completed. For the purpose of \io multiplexing, @aio_suspend@ is the best interface. However, even if AIO requests can be submitted concurrently, @aio_suspend@ suffers from the same limitation as @select@ and @poll@, \ie, the interest set cannot be dynamically changed while a call to @aio_suspend@ is in progress. AIO also suffers from the limitation of specifying which requests have completed, \ie programmers have to poll each request in the interest set using @aio_error@ to identify the completed requests. This limitation means that, like @select@ and @poll@ but not @epoll@, the time needed to examine polling results increases based on the total number of requests monitored, not the number of completed requests.
+Finally, AIO does not seem to be a popular interface, which I believe is due in part to this poor polling interface. Linus Torvalds talks about this interface as follows:
 \begin{displayquote}
         AIO is a horrible ad-hoc design, with the main excuse being "other,
+        AIO is a horrible ad-hoc design, with the main excuse being ``other,
         less gifted people, made that design, and we are implementing it for
         compatibility because database people - who seldom have any shred of
         taste - actually use it".
+        taste - actually use it''.
         But AIO was always really really ugly.
 …
 \end{displayquote}
 Interestingly, in this e-mail answer, Linus goes on to describe
+Interestingly, in this e-mail, Linus goes on to describe
 ``a true \textit{asynchronous system call} interface''
 that does
 …
 in
 ``some kind of arbitrary \textit{queue up asynchronous system call} model''.
 This description is actually quite close to the interface of the interface described in the next section.
+This description is actually quite close to the interface described in the next section.
+\subsection{\texttt{io\_uring}}
+A very recent addition to Linux, @io_uring@\cit{io\_uring} is a framework that aims to solve many of the problems listed with the above mentioned solutions.
+\subsection{\lstinline{io_uring}}
+A very recent addition to Linux, @io_uring@~\cite{MAN:io_uring}, is a framework that aims to solve many of the problems listed in the above interfaces. Like AIO, it represents \io operations as entries added to a queue. But like @epoll@, new requests can be submitted while a blocking call waiting for requests to complete is already in progress. The @io_uring@ interface uses two ring buffers (referred to simply as rings) at its core: a submit ring to which programmers push \io requests and a completion ring from which programmers poll for completion.
+One of the big advantages over the prior interfaces is that @io_uring@ also supports a much wider range of operations. In addition to supporting reads and writes to any file descriptor like AIO, it supports other operations like @open@, @close@, @fsync@, @accept@, @connect@, @send@, @recv@, @splice@, \etc.
+On top of these, @io_uring@ adds many extras like avoiding copies between the kernel and user-space using shared memory, allowing different mechanisms to communicate with device drivers, and supporting chains of requests, \ie, requests that automatically trigger followup requests on completion.
 \subsection{Extra Kernel Threads}\label{io:morethreads}
 Finally, if the operating system does not offer any satisfying forms of asynchronous \glsxtrshort{io} operations, a solution is to fake it by creating a pool of \glspl{kthrd} and delegating operations to them in order to avoid blocking \glspl{proc}.
+Finally, if the operating system does not offer a satisfactory form of asynchronous \io operations, an ad-hoc solution is to create a pool of \glspl{kthrd} and delegate operations to it to avoid blocking \glspl{proc}, which is a compromise for multiplexing. In the worst case, where all \glspl{thrd} are consistently blocking on \io, it devolves into 1-to-1 threading. However, regardless of the frequency of \io operations, it achieves the fundamental goal of not blocking \glspl{proc} when \glspl{thrd} are ready to run. This approach is used by languages like Go\cit{Go} and frameworks like libuv\cit{libuv}, since it has the advantage that it can easily be used across multiple operating systems. This advantage is especially relevant for languages like Go, which offer a homogeneous \glsxtrshort{api} across all platforms. As opposed to C, which has a very limited standard api for \io, \eg, the C standard library has no networking.
 \subsection{Discussion}
+These options effectively fall into two broad camps: waiting for \io to be ready versus waiting for \io to complete. All operating systems that support asynchronous \io must offer an interface along one of these lines, but the details vary drastically. For example, Free BSD offers @kqueue@~\cite{MAN:bsd/kqueue}, which behaves similarly to @epoll@, but with some small quality of use improvements, while Windows (Win32)~\cit{https://docs.microsoft.com/en-us/windows/win32/fileio/synchronous-and-asynchronous-i-o} offers ``overlapped I/O'', which handles submissions similarly to @O_NONBLOCK@ with extra flags on the synchronous system call, but waits for completion events, similarly to @io_uring@.
+For this project, I selected @io_uring@, in large parts because to its generality. While @epoll@ has been shown to be a good solution for socket \io (\cite{DBLP:journals/pomacs/KarstenB20}), @io_uring@'s transparent support for files, pipes, and more complex operations, like @splice@ and @tee@, make it a better choice as the foundation for a general \io subsystem.
 \section{Event-Engine}
+An event engine's responsibility is to use the kernel interface to multiplex many \io operations onto few \glspl{kthrd}. In concrete terms, this means \glspl{thrd} enter the engine through an interface, the event engines then starts the operation and parks the calling \glspl{thrd}, returning control to the \gls{proc}. The parked \glspl{thrd} are then rescheduled by the event engine once the desired operation has completed.
+\subsection{\lstinline{io_uring} in depth}
+Before going into details on the design of my event engine, more details on @io_uring@ usage are presented, each important in the design of the engine.
+Figure~\ref{fig:iouring} shows an overview of an @io_uring@ instance.
+Two ring buffers are used to communicate with the kernel: one for submissions~(left) and one for completions~(right).
+The submission ring contains entries, \newterm{Submit Queue Entries} (SQE), produced (appended) by the application when an operation starts and then consumed by the kernel.
+The completion ring contains entries, \newterm{Completion Queue Entries} (CQE), produced (appended) by the kernel when an operation completes and then consumed by the application.
+The submission ring contains indexes into the SQE array (denoted \emph{S}) containing entries describing the I/O operation to start;
+the completion ring contains entries for the completed I/O operation.
+Multiple @io_uring@ instances can be created, in which case they each have a copy of the data structures in the figure.
+\begin{figure}
+        \centering
+        \input{io_uring.pstex_t}
+        \caption{Overview of \lstinline{io_uring}}
+%       \caption[Overview of \lstinline{io_uring}]{Overview of \lstinline{io_uring} \smallskip\newline Two ring buffer are used to communicate with the kernel, one for completions~(right) and one for submissions~(left). The completion ring contains entries, \newterm{CQE}s: Completion Queue Entries, that are produced by the kernel when an operation completes and then consumed by the application. On the other hand, the application produces \newterm{SQE}s: Submit Queue Entries, which it appends to the submission ring for the kernel to consume. Unlike the completion ring, the submission ring does not contain the entries directly, it indexes into the SQE array (denoted \emph{S}) instead.}
+        \label{fig:iouring}
+\end{figure}
+New \io operations are submitted to the kernel following 4 steps, which use the components shown in the figure.
+\begin{enumerate}
+\item
+An SQE is allocated from the pre-allocated array (denoted \emph{S} in Figure~\ref{fig:iouring}). This array is created at the same time as the @io_uring@ instance, is in kernel-locked memory visible by both the kernel and the application, and has a fixed size determined at creation. How these entries are allocated is not important for the functioning of @io_uring@, the only requirement is that no entry is reused before the kernel has consumed it.
+\item
+The SQE is filled according to the desired operation. This step is straight forward, the only detail worth mentioning is that SQEs have a @user_data@ field that must be filled in order to match submission and completion entries.
+\item
+The SQE is submitted to the submission ring by appending the index of the SQE to the ring following regular ring buffer steps: \lstinline{buffer[head] = item; head++}. Since the head is visible to the kernel, some memory barriers may be required to prevent the compiler from reordering these operations. Since the submission ring is a regular ring buffer, more than one SQE can be added at once and the head is updated only after all entries are updated.
+\item
+The kernel is notified of the change to the ring using the system call @io_uring_enter@. The number of elements appended to the submission ring is passed as a parameter and the number of elements consumed is returned. The @io_uring@ instance can be constructed so this step is not required, but this requires elevated privilege.% and an early version of @io_uring@ had additional restrictions.
+\end{enumerate}
+\begin{sloppypar}
+The completion side is simpler: applications call @io_uring_enter@ with the flag @IORING_ENTER_GETEVENTS@ to wait on a desired number of operations to complete. The same call can be used to both submit SQEs and wait for operations to complete. When operations do complete, the kernel appends a CQE to the completion ring and advances the head of the ring. Each CQE contains the result of the operation as well as a copy of the @user_data@ field of the SQE that triggered the operation. It is not necessary to call @io_uring_enter@ to get new events because the kernel can directly modify the completion ring. The system call is only needed if the application wants to block waiting for operations to complete.
+\end{sloppypar}
+The @io_uring_enter@ system call is protected by a lock inside the kernel. This protection means that concurrent call to @io_uring_enter@ using the same instance are possible, but there is no performance gained from parallel calls to @io_uring_enter@. It is possible to do the first three submission steps in parallel, however, doing so requires careful synchronization.
+@io_uring@ also introduces constraints on the number of simultaneous operations that can be ``in flight''. Obviously, SQEs are allocated from a fixed-size array, meaning that there is a hard limit to how many SQEs can be submitted at once. In addition, the @io_uring_enter@ system call can fail because ``The  kernel [...] ran out of resources to handle [a request]'' or ``The application is attempting to overcommit the number of requests it can  have  pending.''. This restriction means \io request bursts may have to be subdivided and submitted in chunks at a later time.
+\subsection{Multiplexing \io: Submission}
+The submission side is the most complicated aspect of @io_uring@ and its design largely dictates the completion side.
+While it is possible to do the first steps of submission in parallel, the duration of the system call scales with number of entries submitted. The consequence is that the amount of parallelism used to prepare submissions for the next system call is limited. Beyond this limit, the length of the system call is the throughput limiting factor. I concluded from early experiments that preparing submissions seems to take about as long as the system call itself, which means that with a single @io_uring@ instance, there is no benefit in terms of \io throughput to having more than two \glspl{hthrd}. Therefore the design of the submission engine must manage multiple instances of @io_uring@ running in parallel, effectively sharding @io_uring@ instances. Similarly to scheduling, this sharding can be done privately, \ie, one instance per \glspl{proc}, or in decoupled pools, \ie, a pool of \glspl{proc} use a pool of @io_uring@ instances without one-to-one coupling between any given instance and any given \gls{proc}.
+\subsubsection{Pool of Instances}
+One approach is to have multiple shared instances. \Glspl{thrd} attempting \io operations pick one of the available instances and submits operations to that instance. Since the completion will be sent to the same instance, all instances with pending operations must be polled continuously\footnote{As will be described in Chapter~\ref{practice}, this does not translate into constant CPU usage.}. Since there is no coupling between \glspl{proc} and @io_uring@ instances in this approach, \glspl{thrd} running on more than one \gls{proc} can attempt to submit to the same instance concurrently. Since @io_uring@ effectively sets the amount of sharding needed to avoid contention on its internal locks, performance in this approach is based on two aspects: the synchronization needed to submit does not induce more contention than @io_uring@ already does and the scheme to route \io requests to specific @io_uring@ instances does not introduce contention. This second aspect has an oversized importance because it comes into play before the sharding of instances, and as such, all \glspl{hthrd} can contend on the routing algorithm.
+Allocation in this scheme can be handled fairly easily. Free SQEs, \ie, SQEs that aren't currently being used to represent a request, can be written to safely and have a field called @user_data@ which the kernel only reads to copy to CQEs. Allocation also requires no ordering guarantee as all free SQEs are interchangeable. This requires a simple concurrent bag. The only added complexity is that the number of SQEs is fixed, which means allocation can fail. This failure needs to be pushed up to the routing algorithm, \glspl{thrd} attempting \io operations must not be directed to @io_uring@ instances without any available SQEs. Ideally, the routing algorithm would block operations up-front if none of the instances have available SQEs.
+Once an SQE is allocated, \glspl{thrd} can fill them normally, they simply need to keep track of the SQE index and which instance it belongs to.
+Once an SQE is filled in, what needs to happen is that the SQE must be added to the submission ring buffer, an operation that is not thread-safe on itself, and the kernel must be notified using the @io_uring_enter@ system call. The submission ring buffer is the same size as the pre-allocated SQE buffer, therefore pushing to the ring buffer cannot fail\footnote{This is because it is invalid to have the same \lstinline{sqe} multiple times in the ring buffer.}. However, as mentioned, the system call itself can fail with the expectation that it will be retried once some of the already submitted operations complete. Since multiple SQEs can be submitted to the kernel at once, it is important to strike a balance between batching and latency. Operations that are ready to be submitted should be batched together in few system calls, but at the same time, operations should not be left pending for long period of times before being submitted. This can be handled by either designating one of the submitting \glspl{thrd} as the being responsible for the system call for the current batch of SQEs or by having some other party regularly submitting all ready SQEs, \eg, the poller \gls{thrd} mentioned later in this section.
+In the case of designating a \gls{thrd}, ideally, when multiple \glspl{thrd} attempt to submit operations to the same @io_uring@ instance, all requests would be batched together and one of the \glspl{thrd} would do the system call on behalf of the others, referred to as the \newterm{submitter}. In practice however, it is important that the \io requests are not left pending indefinitely and as such, it may be required to have a current submitter and a next submitter. Indeed, as long as there is a ``next'' submitter, \glspl{thrd} submitting new \io requests can move on, knowing that some future system call will include their request. Once the system call is done, the submitter must also free SQEs so that the allocator can reused them.
+Finally, the completion side is much simpler since the @io_uring@ system call enforces a natural synchronization point. Polling simply needs to regularly do the system call, go through the produced CQEs and communicate the result back to the originating \glspl{thrd}. Since CQEs only own a signed 32 bit result, in addition to the copy of the @user_data@ field, all that is needed to communicate the result is a simple future~\cite{wiki:future}. If the submission side does not designate submitters, polling can also submit all SQEs as it is polling events.  A simple approach to polling is to allocate a \gls{thrd} per @io_uring@ instance and simply let the poller \glspl{thrd} poll their respective instances when scheduled. This design is especially convenient for reasons explained in Chapter~\ref{practice}.
+With this pool of instances approach, the big advantage is that it is fairly flexible. It does not impose restrictions on what \glspl{thrd} submitting \io operations can and cannot do between allocations and submissions. It also can gracefully handle running out of resources, SQEs or the kernel returning @EBUSY@. The down side to this is that many of the steps used for submitting need complex synchronization to work properly. The routing and allocation algorithm needs to keep track of which ring instances have available SQEs, block incoming requests if no instance is available, prevent barging if \glspl{thrd} are already queued up waiting for SQEs and handle SQEs being freed. The submission side needs to safely append SQEs to the ring buffer, make sure no SQE is dropped or left pending forever, notify the allocation side when SQEs can be reused and handle the kernel returning @EBUSY@. Sharding the @io_uring@ instances should alleviate much of the contention caused by this, but all this synchronization may still have non-zero cost.
+\subsubsection{Private Instances}
+Another approach is to simply create one ring instance per \gls{proc}. This alleviate the need for synchronization on the submissions, requiring only that \glspl{thrd} are not interrupted in between two submission steps. This is effectively the same requirement as using @thread_local@ variables. Since SQEs that are allocated must be submitted to the same ring, on the same \gls{proc}, this effectively forces the application to submit SQEs in allocation order\footnote{The actual requirement is that \glspl{thrd} cannot context switch between allocation and submission. This requirement means that from the subsystem's point of view, the allocation and submission are sequential. To remove this requirement, a \gls{thrd} would need the ability to ``yield to a specific \gls{proc}'', \ie, park with the promise that it will be run next on a specific \gls{proc}, the \gls{proc} attached to the correct ring. This is not a current or planned feature of \CFA.}, greatly simplifying both allocation and submission. In this design, allocation and submission form a ring partitioned ring buffer as shown in Figure~\ref{fig:pring}. Once added to the ring buffer, the attached \gls{proc} has a significant amount of flexibility with regards to when to do the system call. Possible options are: when the \gls{proc} runs out of \glspl{thrd} to run, after running a given number of threads \glspl{thrd}, etc.
+\begin{figure}
+        \centering
+        \input{pivot_ring.pstex_t}
+        \caption[Partitioned ring buffer]{Partitioned ring buffer \smallskip\newline Allocated sqes are appending to the first partition. When submitting, the partition is simply advanced to include all the sqes that should be submitted. The kernel considers the partition as the head of the ring.}
+        \label{fig:pring}
+\end{figure}
+This approach has the advantage that it does not require much of the synchronization needed in the shared approach. This comes at the cost that \glspl{thrd} submitting \io operations have less flexibility, they cannot park or yield, and several exceptional cases are handled poorly. Instances running out of SQEs cannot run \glspl{thrd} wanting to do \io operations, in such a case the \gls{thrd} needs to be moved to a different \gls{proc}, the only current way of achieving this would be to @yield()@ hoping to be scheduled on a different \gls{proc}, which is not guaranteed. Another problematic case is that \glspl{thrd} that do not park for long periods of time will delay the submission of any SQE not already submitted. This issue is similar to fairness issues which schedulers that use work-stealing mentioned in the previous chapter.
 \section{Interface}
+Finally, the last important part of the \io subsystem is it's interface. There are multiple approaches that can be offered to programmers, each with advantages and disadvantages. The new \io subsystem can replace the C runtime's API or extend it. And in the later case the interface can go from very similar to vastly different. The following sections discuss some useful options using @read@ as an example. The standard Linux interface for C is :
+@ssize_t read(int fd, void *buf, size_t count);@.
+\subsection{Replacement}
+Replacing the C \glsxtrshort{api}
+\subsection{Synchronous Extension}
+\subsection{Asynchronous Extension}
+\subsection{Interface directly to \lstinline{io_uring}}

doc/theses/thierry_delisle_PhD/thesis/text/runtime.tex

-              r5e99a9a
+              r95b3a9c
 \section{Clusters}
 \CFA allows the option to group user-level threading, in the form of clusters. Both \glspl{thrd} and \glspl{proc} belong to a specific cluster. \Glspl{thrd} are only be scheduled onto \glspl{proc} in the same cluster and scheduling is done independently of other clusters. Figure~\ref{fig:system} shows an overview of the \CFA runtime, which allows programmers to tightly control parallelism. It also opens the door to handling effects like NUMA, by pining clusters to a specific NUMA node\footnote{This is not currently implemented in \CFA, but the only hurdle left is creating a generic interface for cpu masks.}.
+\CFA allows the option to group user-level threading, in the form of clusters. Both \glspl{thrd} and \glspl{proc} belong to a specific cluster. \Glspl{thrd} are only scheduled onto \glspl{proc} in the same cluster and scheduling is done independently of other clusters. Figure~\ref{fig:system} shows an overview of the \CFA runtime, which allows programmers to tightly control parallelism. It also opens the door to handling effects like NUMA, by pining clusters to a specific NUMA node\footnote{This is not currently implemented in \CFA, but the only hurdle left is creating a generic interface for cpu masks.}.
 \begin{figure}
 …
 \section{\glsxtrshort{io}}\label{prev:io}
 Prior to this work, the \CFA runtime did not add any particular support for \glsxtrshort{io} operations. %\CFA being built on C, this means that,
+While all I/O operations available in C are available in \CFA, \glsxtrshort{io} operations are designed for the POSIX threading model~\cite{pthreads}. Using these 1:1 threading operations in an M:N threading model means I/O operations block \glspl{proc} instead of \glspl{thrd}. While this can work in certain cases, it limits the number of concurrent operations to the number of \glspl{proc} rather than \glspl{thrd}. It also means deadlock can occur because all \glspl{proc} are blocked even if at least one \gls{thrd} is ready to run. A simple example of this type of deadlock would be as follows:
+Prior to this work, the \CFA runtime did not add any particular support for \glsxtrshort{io} operations. While all \glsxtrshort{io} operations available in C are available in \CFA, \glsxtrshort{io} operations are designed for the POSIX threading model~\cite{pthreads}. Using these 1:1 threading operations in an M:N threading model means \glsxtrshort{io} operations block \glspl{proc} instead of \glspl{thrd}. While this can work in certain cases, it limits the number of concurrent operations to the number of \glspl{proc} rather than \glspl{thrd}. It also means deadlock can occur because all \glspl{proc} are blocked even if at least one \gls{thrd} is ready to run. A simple example of this type of deadlock would be as follows:
 \begin{quote}
 Given a simple network program with 2 \glspl{thrd} and a single \gls{proc}, one \gls{thrd} sends network requests to a server and the other \gls{thrd} waits for a response from the server. If the second \gls{thrd} races ahead, it may wait for responses to requests that have not been sent yet. In theory, this should not be a problem, even if the second \gls{thrd} waits, because the first \gls{thrd} is still ready to run and should be able to get CPU time to send the request. With M:N threading, while the first \gls{thrd} is ready, the lone \gls{proc} \emph{cannot} run the first \gls{thrd} if it is blocked in the \glsxtrshort{io} operation of the second \gls{thrd}. If this happen, the system is in a synchronization deadlock\footnote{In this example, the deadlocked could be resolved if the server sends unprompted messages to the client. However, this solution is not general and may not be appropriate even in this simple case.}.
 \end{quote}
-Therefore, one of the objective of this work is to introduce \emph{User-Level \glsxtrshort{io}}, like \glslink{uthrding}{User-Level \emph{Threading}} blocks \glspl{thrd} rather than \glspl{proc} when doing \glsxtrshort{io} operations, which entails multiplexing the \glsxtrshort{io} operations of many \glspl{thrd} onto fewer \glspl{proc}. This multiplexing requires that a single \gls{proc} be able to execute multiple I/O operations in parallel. This requirement cannot be done with operations that block \glspl{proc}, \ie \glspl{kthrd}, since the first operation would prevent starting new operations for its blocking duration. Executing I/O operations in parallel requires \emph{asynchronous} \glsxtrshort{io}, sometimes referred to as \emph{non-blocking}, since the \gls{kthrd} does not block.
+\section{Interoperating with C}
+Therefore, one of the objective of this work is to introduce \emph{User-Level \glsxtrshort{io}}, like \glslink{uthrding}{User-Level \emph{Threading}} blocks \glspl{thrd} rather than \glspl{proc} when doing \glsxtrshort{io} operations, which entails multiplexing the \glsxtrshort{io} operations of many \glspl{thrd} onto fewer \glspl{proc}. This multiplexing requires that a single \gls{proc} be able to execute multiple \glsxtrshort{io} operations in parallel. This requirement cannot be done with operations that block \glspl{proc}, \ie \glspl{kthrd}, since the first operation would prevent starting new operations for its blocking duration. Executing \glsxtrshort{io} operations in parallel requires \emph{asynchronous} \glsxtrshort{io}, sometimes referred to as \emph{non-blocking}, since the \gls{kthrd} does not block.
+\section{Interoperating with \texttt{C}}
 While \glsxtrshort{io} operations are the classical example of operations that block \glspl{kthrd}, the non-blocking challenge extends to all blocking system-calls. The POSIX standard states~\cite[\S~2.9.1]{POSIX17}:
 \begin{quote}
 …
 \begin{enumerate}
         \item Precisely identifying blocking C calls is difficult.
         \item Introducing new code can have a significant impact on general performance.
+        \item Introducing control points code can have a significant impact on general performance.
 \end{enumerate}
 Because of these consequences, this work does not attempt to ``sandbox'' calls to C. Therefore, it is possible for an unidentified library calls to block a \gls{kthrd} leading to deadlocks in \CFA's M:N threading model, which would not occur in a traditional 1:1 threading model. Currently, all M:N thread systems interacting with UNIX without sandboxing suffer from this problem but manage to work very well in the majority of applications. Therefore, a complete solution to this problem is outside the scope of this thesis.
+Because of these consequences, this work does not attempt to ``sandbox'' calls to C. Therefore, it is possible calls from an unidentified library will block a \gls{kthrd} leading to deadlocks in \CFA's M:N threading model, which would not occur in a traditional 1:1 threading model. Currently, all M:N thread systems interacting with UNIX without sandboxing suffer from this problem but manage to work very well in the majority of applications. Therefore, a complete solution to this problem is outside the scope of this thesis.

doc/theses/thierry_delisle_PhD/thesis/thesis.tex

-              r5e99a9a
+              r95b3a9c
+% uWaterloo Thesis Template for LaTeX
+% Last Updated June 14, 2017 by Stephen Carr, IST Client Services
+% FOR ASSISTANCE, please send mail to rt-IST-CSmathsci@ist.uwaterloo.ca
+% Effective October 2006, the University of Waterloo
+% requires electronic thesis submission. See the uWaterloo thesis regulations at
+%======================================================================
+% University of Waterloo Thesis Template for LaTeX
+% Last Updated November, 2020
+% by Stephen Carr, IST Client Services,
+% University of Waterloo, 200 University Ave. W., Waterloo, Ontario, Canada
+% FOR ASSISTANCE, please send mail to request@uwaterloo.ca
+% DISCLAIMER
+% To the best of our knowledge, this template satisfies the current uWaterloo thesis requirements.
+% However, it is your responsibility to assure that you have met all requirements of the University and your particular department.
+% Many thanks for the feedback from many graduates who assisted the development of this template.
+% Also note that there are explanatory comments and tips throughout this template.
+%======================================================================
+% Some important notes on using this template and making it your own...
+% The University of Waterloo has required electronic thesis submission since October 2006.
+% See the uWaterloo thesis regulations at
 % https://uwaterloo.ca/graduate-studies/thesis.
+% DON'T FORGET TO ADD YOUR OWN NAME AND TITLE in the "hyperref" package
+% configuration below. THIS INFORMATION GETS EMBEDDED IN THE PDF FINAL PDF DOCUMENT.
+% You can view the information if you view Properties of the PDF document.
+% Many faculties/departments also require one or more printed
+% copies. This template attempts to satisfy both types of output.
+% It is based on the standard "book" document class which provides all necessary
+% sectioning structures and allows multi-part theses.
+% DISCLAIMER
+% To the best of our knowledge, this template satisfies the current uWaterloo requirements.
+% However, it is your responsibility to assure that you have met all
+% requirements of the University and your particular department.
+% Many thanks for the feedback from many graduates that assisted the development of this template.
+% -----------------------------------------------------------------------
+% By default, output is produced that is geared toward generating a PDF
+% version optimized for viewing on an electronic display, including
+% hyperlinks within the PDF.
+% This thesis template is geared towards generating a PDF version optimized for viewing on an electronic display, including hyperlinks within the PDF.
+% DON'T FORGET TO ADD YOUR OWN NAME AND TITLE in the "hyperref" package configuration below.
+% THIS INFORMATION GETS EMBEDDED IN THE PDF FINAL PDF DOCUMENT.
+% You can view the information if you view properties of the PDF document.
+% Many faculties/departments also require one or more printed copies.
+% This template attempts to satisfy both types of output.
+% See additional notes below.
+% It is based on the standard "book" document class which provides all necessary sectioning structures and allows multi-part theses.
+% If you are using this template in Overleaf (cloud-based collaboration service), then it is automatically processed and previewed for you as you edit.
+% For people who prefer to install their own LaTeX distributions on their own computers, and process the source files manually, the following notes provide the sequence of tasks:
 % E.g. to process a thesis called "mythesis.tex" based on this template, run:
 % pdflatex mythesis     -- first pass of the pdflatex processor
 % bibtex mythesis       -- generates bibliography from .bib data file(s)
 % makeindex         -- should be run only if an index is used
+% makeindex         -- should be run only if an index is used
 % pdflatex mythesis     -- fixes numbering in cross-references, bibliographic references, glossaries, index, etc.
+% pdflatex mythesis     -- fixes numbering in cross-references, bibliographic references, glossaries, index, etc.
+% If you use the recommended LaTeX editor, Texmaker, you would open the mythesis.tex
+% file, then click the PDFLaTeX button. Then run BibTeX (under the Tools menu).
+% Then click the PDFLaTeX button two more times. If you have an index as well,
+% you'll need to run MakeIndex from the Tools menu as well, before running pdflatex
+% pdflatex mythesis     -- it takes a couple of passes to completely process all cross-references
+% If you use the recommended LaTeX editor, Texmaker, you would open the mythesis.tex file, then click the PDFLaTeX button. Then run BibTeX (under the Tools menu).
+% Then click the PDFLaTeX button two more times.
+% If you have an index as well,you'll need to run MakeIndex from the Tools menu as well, before running pdflatex
 % the last two times.
+% N.B. The "pdftex" program allows graphics in the following formats to be
+% included with the "\includegraphics" command: PNG, PDF, JPEG, TIFF
+% Tip 1: Generate your figures and photos in the size you want them to appear
+% in your thesis, rather than scaling them with \includegraphics options.
+% Tip 2: Any drawings you do should be in scalable vector graphic formats:
+% SVG, PNG, WMF, EPS and then converted to PNG or PDF, so they are scalable in
+% the final PDF as well.
+% Tip 3: Photographs should be cropped and compressed so as not to be too large.
+% To create a PDF output that is optimized for double-sided printing:
+%
+% 1) comment-out the \documentclass statement in the preamble below, and
+% un-comment the second \documentclass line.
+%
+% 2) change the value assigned below to the boolean variable
+% "PrintVersion" from "false" to "true".
+% --------------------- Start of Document Preamble -----------------------
+% Specify the document class, default style attributes, and page dimensions
+% N.B. The "pdftex" program allows graphics in the following formats to be included with the "\includegraphics" command: PNG, PDF, JPEG, TIFF
+% Tip: Generate your figures and photos in the size you want them to appear in your thesis, rather than scaling them with \includegraphics options.
+% Tip: Any drawings you do should be in scalable vector graphic formats: SVG, PNG, WMF, EPS and then converted to PNG or PDF, so they are scalable in the final PDF as well.
+% Tip: Photographs should be cropped and compressed so as not to be too large.
+% To create a PDF output that is optimized for double-sided printing:
+% 1) comment-out the \documentclass statement in the preamble below, and un-comment the second \documentclass line.
+% 2) change the value assigned below to the boolean variable "PrintVersion" from " false" to "true".
+%======================================================================
+%   D O C U M E N T   P R E A M B L E
+% Specify the document class, default style attributes, and page dimensions, etc.
 % For hyperlinked PDF, suitable for viewing on a computer, use this:
 \documentclass[letterpaper,12pt,titlepage,oneside,final]{book}
+% For PDF, suitable for double-sided printing, change the PrintVersion variable below
+% to "true" and use this \documentclass line instead of the one above:
+% For PDF, suitable for double-sided printing, change the PrintVersion variable below to "true" and use this \documentclass line instead of the one above:
 %\documentclass[letterpaper,12pt,titlepage,openright,twoside,final]{book}
+\newcommand{\href}[1]{#1} % does nothing, but defines the command so the
+    % print-optimized version will ignore \href tags (redefined by hyperref pkg).
+% Some LaTeX commands I define for my own nomenclature.
+% If you have to, it's easier to make changes to nomenclature once here than in a million places throughout your thesis!
+\newcommand{\package}[1]{\textbf{#1}} % package names in bold text
+\newcommand{\cmmd}[1]{\textbackslash\texttt{#1}} % command name in tt font
+\newcommand{\href}[1]{#1} % does nothing, but defines the command so the print-optimized version will ignore \href tags (redefined by hyperref pkg).
+%\newcommand{\texorpdfstring}[2]{#1} % does nothing, but defines the command
+% Anything defined here may be redefined by packages added below...
 % This package allows if-then-else control structures.
 …
 \newboolean{PrintVersion}
 \setboolean{PrintVersion}{false}
+% CHANGE THIS VALUE TO "true" as necessary, to improve printed results for hard copies
+% by overriding some options of the hyperref package below.
+% CHANGE THIS VALUE TO "true" as necessary, to improve printed results for hard copies by overriding some options of the hyperref package, called below.
 %\usepackage{nomencl} % For a nomenclature (optional; available from ctan.org)
 \usepackage{amsmath,amssymb,amstext} % Lots of math symbols and environments
+\usepackage{xcolor}
 \usepackage{graphicx} % For including graphics
 % Hyperlinks make it very easy to navigate an electronic document.
+% In addition, this is where you should specify the thesis title
+% and author as they appear in the properties of the PDF document.
+% In addition, this is where you should specify the thesis title and author as they appear in the properties of the PDF document.
 % Use the "hyperref" package
 % N.B. HYPERREF MUST BE THE LAST PACKAGE LOADED; ADD ADDITIONAL PKGS ABOVE
 \usepackage[pagebackref=false]{hyperref} % with basic options
+                % N.B. pagebackref=true provides links back from the References to the body text. This can cause trouble for printing.
+%\usepackage[pdftex,pagebackref=true]{hyperref}
+% N.B. pagebackref=true provides links back from the References to the body text. This can cause trouble for printing.
 \hypersetup{
         plainpages=false,       % needed if Roman numbers in frontpages
         unicode=false,          % non-Latin characters in Acrobat’s bookmarks
         pdftoolbar=true,        % show Acrobat’s toolbar?
         pdfmenubar=true,        % show Acrobat’s menu?
+        unicode=false,          % non-Latin characters in Acrobat's bookmarks
+        pdftoolbar=true,        % show Acrobat's toolbar?
+        pdfmenubar=true,        % show Acrobat's menu?
         pdffitwindow=false,     % window fit to page when opened
         pdfstartview={FitH},    % fits the width of the page to the window
 …
 \ifthenelse{\boolean{PrintVersion}}{   % for improved print quality, change some hyperref options
 \hypersetup{    % override some previously defined hyperref options
         citecolor=black,
         filecolor=black,
         linkcolor=black,
+        citecolor=black,%
+        filecolor=black,%
+        linkcolor=black,%
         urlcolor=black
 }}{} % end of ifthenelse (no else)
 …
 % although it's supposed to be in both the TeX Live and MikTeX distributions. There are also documentation and
 % installation instructions there.
+\renewcommand*{\glstextformat}[1]{\textsf{#1}}
+\makeatletter
+\newcommand*{\glsplainhyperlink}[2]{%
+  \colorlet{currenttext}{.}% store current text color
+  \colorlet{currentlink}{\@linkcolor}% store current link color
+  \hypersetup{linkcolor=currenttext}% set link color
+  \hyperlink{#1}{#2}%
+  \hypersetup{linkcolor=currentlink}% reset to default
+}
+\let\@glslink\glsplainhyperlink
+\makeatother
 \usepackage{csquotes}
 …
 % Setting up the page margins...
+\setlength{\textheight}{9in}\setlength{\topmargin}{-0.45in}\setlength{\headsep}{0.25in}
+\setlength{\textheight}{9in}
+\setlength{\topmargin}{-0.45in}
+\setlength{\headsep}{0.25in}
 % uWaterloo thesis requirements specify a minimum of 1 inch (72pt) margin at the
+% top, bottom, and outside page edges and a 1.125 in. (81pt) gutter
+% margin (on binding side). While this is not an issue for electronic
+% viewing, a PDF may be printed, and so we have the same page layout for
+% both printed and electronic versions, we leave the gutter margin in.
+% top, bottom, and outside page edges and a 1.125 in. (81pt) gutter margin (on binding side).
+% While this is not an issue for electronic viewing, a PDF may be printed, and so we have the same page layout for both printed and electronic versions, we leave the gutter margin in.
 % Set margins to minimum permitted by uWaterloo thesis regulations:
 \setlength{\marginparwidth}{0pt} % width of margin notes
 …
 \setlength{\evensidemargin}{0.125in} % Adds 1/8 in. to binding side of all
 % even-numbered pages when the "twoside" printing option is selected
+\setlength{\oddsidemargin}{0.125in} % Adds 1/8 in. to the left of all pages
+% when "oneside" printing is selected, and to the left of all odd-numbered
+% pages when "twoside" printing is selected
+\setlength{\textwidth}{6.375in} % assuming US letter paper (8.5 in. x 11 in.) and
+% side margins as above
+\setlength{\oddsidemargin}{0.125in} % Adds 1/8 in. to the left of all pages when "oneside" printing is selected, and to the left of all odd-numbered pages when "twoside" printing is selected
+\setlength{\textwidth}{6.375in} % assuming US letter paper (8.5 in. x 11 in.) and side margins as above
 \raggedbottom
+% The following statement specifies the amount of space between
+% paragraphs. Other reasonable specifications are \bigskipamount and \smallskipamount.
+% The following statement specifies the amount of space between paragraphs. Other reasonable specifications are \bigskipamount and \smallskipamount.
 \setlength{\parskip}{\medskipamount}
+% The following statement controls the line spacing.  The default
+% spacing corresponds to good typographic conventions and only slight
+% changes (e.g., perhaps "1.2"), if any, should be made.
+% The following statement controls the line spacing.
+% The default spacing corresponds to good typographic conventions and only slight changes (e.g., perhaps "1.2"), if any, should be made.
 \renewcommand{\baselinestretch}{1} % this is the default line space setting
+% By default, each chapter will start on a recto (right-hand side)
+% page.  We also force each section of the front pages to start on
+% a recto page by inserting \cleardoublepage commands.
+% In many cases, this will require that the verso page be
+% blank and, while it should be counted, a page number should not be
+% printed.  The following statements ensure a page number is not
+% printed on an otherwise blank verso page.
+% By default, each chapter will start on a recto (right-hand side) page.
+% We also force each section of the front pages to start on a recto page by inserting \cleardoublepage commands.
+% In many cases, this will require that the verso (left-hand) page be blank, and while it should be counted, a page number should not be printed.
+% The following statements ensure a page number is not printed on an otherwise blank verso page.
 \let\origdoublepage\cleardoublepage
 \newcommand{\clearemptydoublepage}{%
 …
 \input{common}
 \CFAStyle                                               % CFA code-style for all languages
 \lstset{basicstyle=\linespread{0.9}\tt}
+\lstset{language=CFA,basicstyle=\linespread{0.9}\tt}    % CFA default language
 % glossary of terms to use
 …
 \makeindex
+%======================================================================
+%   L O G I C A L    D O C U M E N T -- the content of your thesis
+\newcommand\io{\glsxtrshort{io}\xspace}%
+%======================================================================
+%   L O G I C A L    D O C U M E N T
+% The logical document contains the main content of your thesis.
+% Being a large document, it is a good idea to divide your thesis into several files, each one containing one chapter or other significant chunk of content, so you can easily shuffle things around later if desired.
 %======================================================================
 \begin{document}
-% For a large document, it is a good idea to divide your thesis
-% into several files, each one containing one chapter.
-% To illustrate this idea, the "front pages" (i.e., title page,
-% declaration, borrowers' page, abstract, acknowledgements,
-% dedication, table of contents, list of tables, list of figures,
-% nomenclature) are contained within the file "uw-ethesis-frontpgs.tex" which is
-% included into the document by the following statement.
 %----------------------------------------------------------------------
 % FRONT MATERIAL
+% title page,declaration, borrowers' page, abstract, acknowledgements,
+% dedication, table of contents, list of tables, list of figures, nomenclature, etc.
 %----------------------------------------------------------------------
 \input{text/front.tex}
 %----------------------------------------------------------------------
 % MAIN BODY
 %----------------------------------------------------------------------
 % Because this is a short document, and to reduce the number of files
 % needed for this template, the chapters are not separate
 % documents as suggested above, but you get the idea. If they were
 % separate documents, they would each start with the \chapter command, i.e,
+% do not contain \documentclass or \begin{document} and \end{document} commands.
+% We suggest using a separate file for each chapter of your thesis.
+% Start each chapter file with the \chapter command.
+% Only use \documentclass or \begin{document} and \end{document} commands in this master document.
+% Tip: Putting each sentence on a new line is a way to simplify later editing.
+%----------------------------------------------------------------------
 \part{Introduction}
 \input{text/intro.tex}
 …
 \part{Design}
 \input{text/core.tex}
+\input{text/io.tex}
 \input{text/practice.tex}
-\input{text/io.tex}
 \part{Evaluation}
 \label{Evaluation}
 …
 %----------------------------------------------------------------------
 % END MATERIAL
+%----------------------------------------------------------------------
+% B I B L I O G R A P H Y
+% -----------------------
+% The following statement selects the style to use for references.  It controls the sort order of the entries in the bibliography and also the formatting for the in-text labels.
+% Bibliography, Appendices, Index, etc.
+%----------------------------------------------------------------------
+% Bibliography
+% The following statement selects the style to use for references.
+% It controls the sort order of the entries in the bibliography and also the formatting for the in-text labels.
 \bibliographystyle{plain}
 % This specifies the location of the file containing the bibliographic information.
+% It assumes you're using BibTeX (if not, why not?).
+\cleardoublepage % This is needed if the book class is used, to place the anchor in the correct page,
+                 % because the bibliography will start on its own page.
+                 % Use \clearpage instead if the document class uses the "oneside" argument
+% It assumes you're using BibTeX to manage your references (if not, why not?).
+\cleardoublepage % This is needed if the "book" document class is used, to place the anchor in the correct page, because the bibliography will start on its own page.
+% Use \clearpage instead if the document class uses the "oneside" argument
 \phantomsection  % With hyperref package, enables hyperlinking from the table of contents to bibliography
 % The following statement causes the title "References" to be used for the bibliography section:
 …
 \bibliography{local,pl}
 % Tip 5: You can create multiple .bib files to organize your references.
+% Tip: You can create multiple .bib files to organize your references.
 % Just list them all in the \bibliogaphy command, separated by commas (no spaces).
 % % The following statement causes the specified references to be added to the bibliography% even if they were not
 % % cited in the text. The asterisk is a wildcard that causes all entries in the bibliographic database to be included (optional).
+% The following statement causes the specified references to be added to the bibliography even if they were not cited in the text.
+% The asterisk is a wildcard that causes all entries in the bibliographic database to be included (optional).
 % \nocite{*}
+%----------------------------------------------------------------------
+% Appendices
 % The \appendix statement indicates the beginning of the appendices.
 \appendix
 % Add a title page before the appendices and a line in the Table of Contents
+% Add an un-numbered title page before the appendices and a line in the Table of Contents
 \chapter*{APPENDICES}
 \addcontentsline{toc}{chapter}{APPENDICES}
+% Appendices are just more chapters, with different labeling (letters instead of numbers).
 %======================================================================
 \chapter[PDF Plots From Matlab]{Matlab Code for Making a PDF Plot}
 …
 %\input{thesis.ind}                             % index
+\phantomsection
+\end{document}
+\phantomsection         % allows hyperref to link to the correct page
+%----------------------------------------------------------------------
+\end{document} % end of logical document

Context Navigation

Legend:

doc/theses/andrew_beach_MMath/existing.tex

doc/theses/andrew_beach_MMath/features.tex

doc/theses/andrew_beach_MMath/future.tex

doc/theses/andrew_beach_MMath/implement.tex

doc/theses/andrew_beach_MMath/thesis-frontpgs.tex

doc/theses/andrew_beach_MMath/thesis.tex

doc/theses/andrew_beach_MMath/unwinding.tex

doc/theses/andrew_beach_MMath/uw-ethesis-frontpgs.tex

doc/theses/andrew_beach_MMath/uw-ethesis.tex

doc/theses/fangren_yu_COOP_F20/Report.tex

doc/theses/thierry_delisle_PhD/thesis/Makefile

doc/theses/thierry_delisle_PhD/thesis/local.bib

doc/theses/thierry_delisle_PhD/thesis/text/core.tex

doc/theses/thierry_delisle_PhD/thesis/text/intro.tex

doc/theses/thierry_delisle_PhD/thesis/text/io.tex

doc/theses/thierry_delisle_PhD/thesis/text/runtime.tex

doc/theses/thierry_delisle_PhD/thesis/thesis.tex

Download in other formats: