Context Navigation

← Previous Change
Next Change →

Changeset f6664bf2 for doc/theses

Timestamp:

Feb 16, 2021, 1:32:24 PM (5 years ago)

Author:

Thierry Delisle <tdelisle@…>

Branches:

ADT, arm-eh, ast-experimental, enum, forall-pointer-decay, jacob/cs343-translation, master, new-ast-unique-expr, pthread-emulation, qualifiedEnum

Children:

feacef9

Parents:

14533d4 (diff), 1830a86 (diff)
Note: this is a merge changeset, the changes displayed below correspond to the merge itself.
Use the (diff) links above to see all the changes relative to each parent.

Message:

Merge branch 'master' of plg.uwaterloo.ca:software/cfa/cfa-cc

Location:

doc/theses

Files:

: 5 edited

andrew_beach_MMath/features.tex (modified) (19 diffs)
andrew_beach_MMath/uw-ethesis.tex (modified) (2 diffs)
thierry_delisle_PhD/thesis/Makefile (modified) (4 diffs)
thierry_delisle_PhD/thesis/text/io.tex (modified) (6 diffs)
thierry_delisle_PhD/thesis/thesis.tex (modified) (11 diffs)

Legend:

: Unmodified
: Added
: Removed

doc/theses/andrew_beach_MMath/features.tex

-              r14533d4
+              rf6664bf2
 virtual table type; which usually has a mangled name.
 % Also \CFA's trait system handles functions better than constants and doing
 % it this way
+% it this way reduce the amount of boiler plate we need.
 % I did have a note about how it is the programmer's responsibility to make
 …
 % similar system I know of (except Agda's I guess) so I took it out.
+\section{Raise}
+\CFA provides two kinds of exception raise: termination
+\see{\VRef{s:Termination}} and resumption \see{\VRef{s:Resumption}}, which are
+specified with the following traits.
+There are two more traits for exceptions @is_termination_exception@ and
+@is_resumption_exception@. They are defined as follows:
 \begin{cfa}
 trait is_termination_exception(
 …
         void defaultTerminationHandler(exceptT &);
 };
+\end{cfa}
+The function is required to allow a termination raise, but is only called if a
+termination raise does not find an appropriate handler.
+Allowing a resumption raise is similar.
+\begin{cfa}
 trait is_resumption_exception(
                 exceptT &, virtualT & | is_exception(exceptT, virtualT)) {
 …
 };
 \end{cfa}
+The function is required to allow a resumption raise, but is only called if a
+resumption raise does not find an appropriate handler.
+Finally there are three convenience macros for referring to the these traits:
+In other words they make sure that a given type and virtual type is an
+exception and defines one of the two default handlers. These default handlers
+are used in the main exception handling operations \see{Exception Handling}
+and their use will be detailed there.
+However all three of these traits can be trickly to use directly.
+There is a bit of repetition required but
+the largest issue is that the virtual table type is mangled and not in a user
+facing way. So there are three macros that can be used to wrap these traits
+when you need to refer to the names:
 @IS_EXCEPTION@, @IS_TERMINATION_EXCEPTION@ and @IS_RESUMPTION_EXCEPTION@.
+All three traits are hard to use while naming the virtual table as it has an
+internal mangled name. These macros take the exception name as their first
+argument and do the mangling. They all take a second argument for polymorphic
+types which is the parenthesized list of polymorphic arguments. These
+arguments are passed to both the exception type and the virtual table type as
+the arguments do have to match.
+All take one or two arguments. The first argument is the name of the
+exception type. Its unmangled and mangled form are passed to the trait.
+The second (optional) argument is a parenthesized list of polymorphic
+arguments. This argument should only with polymorphic exceptions and the
+list will be passed to both types.
+In the current set-up the base name and the polymorphic arguments have to
+match so these macros can be used without losing flexability.
 For example consider a function that is polymorphic over types that have a
 …
 \end{cfa}
+\section{Exception Handling}
+\CFA provides two kinds of exception handling, termination and resumption.
+These twin operations are the core of the exception handling mechanism and
+are the reason for the features of exceptions.
+This section will cover the general patterns shared by the two operations and
+then go on to cover the details each individual operation.
+Both operations follow the same set of steps to do their operation. They both
+start with the user preforming a throw on an exception.
+Then there is the search for a handler, if one is found than the exception
+is caught and the handler is run. After that control returns to normal
+execution.
+If the search fails a default handler is run and then control
+returns to normal execution immediately. That is where the default handlers
+@defaultTermiationHandler@ and @defaultResumptionHandler@ are used.
 \subsection{Termination}
 \label{s:Termination}
+Termination raise, called ``throw'', is familiar and used in most programming
+languages with exception handling. The semantics of termination is: search the
+stack for a matching handler, unwind the stack frames to the matching handler,
+execute the handler, and continue execution after the handler. Termination is
+used when execution \emph{cannot} return to the throw. To continue execution,
+the program must \emph{recover} in the handler from the failed (unwound)
+execution at the raise to safely proceed after the handler.
+A termination raise is started with the @throw@ statement:
+Termination handling is more familiar kind and used in most programming
+languages with exception handling.
+It is dynamic, non-local goto. If a throw is successful then the stack will
+be unwound and control will (usually) continue in a different function on
+the call stack. They are commonly used when an error has occured and recovery
+is impossible in the current function.
+% (usually) Control can continue in the current function but then a different
+% control flow construct should be used.
+A termination throw is started with the @throw@ statement:
 \begin{cfa}
 throw EXPRESSION;
 …
 change the throw's behavior (see below).
+At runtime, the exception returned by the expression
+is copied into managed memory (heap) to ensure it remains in
+scope during unwinding. It is the user's responsibility to ensure the original
+exception object at the throw is freed when it goes out of scope. Being
+allocated on the stack is sufficient for this.
+Then the exception system searches the stack starting from the throw and
+proceeding towards the base of the stack, from callee to caller. At each stack
+frame, a check is made for termination handlers defined by the @catch@ clauses
+of a @try@ statement.
+The throw will copy the provided exception into managed memory. It is the
+user's responcibility to ensure the original exception is cleaned up if the
+stack is unwound (allocating it on the stack should be sufficient).
+Then the exception system searches the stack using the copied exception.
+It starts starts from the throw and proceeds to the base of the stack,
+from callee to caller.
+At each stack frame, a check is made for resumption handlers defined by the
+@catch@ clauses of a @try@ statement.
 \begin{cfa}
 try {
         GUARDED_BLOCK
 } catch (EXCEPTION_TYPE$\(_1\)$ * NAME$\(_1\)$) { // termination handler 1
+} catch (EXCEPTION_TYPE$\(_1\)$ * NAME$\(_1\)$) {
         HANDLER_BLOCK$\(_1\)$
 } catch (EXCEPTION_TYPE$\(_2\)$ * NAME$\(_2\)$) { // termination handler 2
+} catch (EXCEPTION_TYPE$\(_2\)$ * NAME$\(_2\)$) {
         HANDLER_BLOCK$\(_2\)$
+}
 \end{cfa}
+The statements in the @GUARDED_BLOCK@ are executed. If those statements, or any
+functions invoked from those statements, throws an exception, and the exception
+When viewed on its own a try statement will simply exceute the statements in
+@GUARDED_BLOCK@ and when those are finished the try statement finishes.
+However, while the guarded statements are being executed, including any
+functions they invoke, all the handlers following the try block are now
+or any functions invoked from those
+statements, throws an exception, and the exception
 is not handled by a try statement further up the stack, the termination
 handlers are searched for a matching exception type from top to bottom.
 …
 freed and control continues after the try statement.
+The default handler visible at the throw statement is used if no matching
+termination handler is found after the entire stack is searched. At that point,
 the default handler is called with a reference to the exception object
+generated at the throw. If the default handler returns, control continues
+from after the throw statement. This feature allows
+each exception type to define its own action, such as printing an informative
+error message, when an exception is not handled in the program.
+However the default handler for all exception types triggers a cancellation
 using the exception.
+If no handler is found during the search then the default handler is run.
+Through \CFA's trait system the best match at the throw sight will be used.
+This function is run and is passed the copied exception. After the default
+handler is run control continues after the throw statement.
+There is a global @defaultTerminationHandler@ that cancels the current stack
+with the copied exception. However it is generic over all exception types so
+new default handlers can be defined for different exception types and so
+different exception types can have different default handlers.
 \subsection{Resumption}
 \label{s:Resumption}
+Resumption raise, called ``resume'', is as old as termination
+raise~\cite{Goodenough75} but is less popular. In many ways, resumption is
+simpler and easier to understand, as it is simply a dynamic call.
+The semantics of resumption is: search the stack for a matching handler,
+execute the handler, and continue execution after the resume. Notice, the stack
+cannot be unwound because execution returns to the raise point. Resumption is
+used used when execution \emph{can} return to the resume. To continue
+execution, the program must \emph{correct} in the handler for the failed
+execution at the raise so execution can safely continue after the resume.
+Resumption exception handling is a less common form than termination but is
+just as old~\cite{Goodenough75} and is in some sense simpler.
+It is a dynamic, non-local function call. If the throw is successful a
+closure will be taken from up the stack and executed, after which the throwing
+function will continue executing.
+These are most often used when an error occured and if the error is repaired
+then the function can continue.
 A resumption raise is started with the @throwResume@ statement:
 …
 The semantics of the @throwResume@ statement are like the @throw@, but the
 expression has return a reference a type that satifies the trait
 @is_resumption_exception@. Like with termination the exception system can
 use these assertions while (throwing/raising/handling) the exception.
+@is_resumption_exception@. The assertions from this trait are available to
+the exception system while handling the exception.
 At runtime, no copies are made. As the stack is not unwound the exception and
 any values on the stack will remain in scope while the resumption is handled.
+Then the exception system searches the stack starting from the resume and
+proceeding to the base of the stack, from callee to caller. At each stack
+frame, a check is made for resumption handlers defined by the @catchResume@
+clauses of a @try@ statement.
+Then the exception system searches the stack using the provided exception.
+It starts starts from the throw and proceeds to the base of the stack,
+from callee to caller.
+At each stack frame, a check is made for resumption handlers defined by the
+@catchResume@ clauses of a @try@ statement.
 \begin{cfa}
 try {
 …
+}
 \end{cfa}
+The statements in the @GUARDED_BLOCK@ are executed. If those statements, or any
+functions invoked from those statements, resumes an exception, and the
+exception is not handled by a try statement further up the stack, the
+resumption handlers are searched for a matching exception type from top to
+bottom. (Note, termination and resumption handlers may be intermixed in a @try@
+statement but the kind of raise (throw/resume) only matches with the
+corresponding kind of handler clause.)
+The exception search and matching for resumption is the same as for
+termination, including exception inheritance. The difference is when control
+reaches the end of the handler: the resumption handler returns after the resume
 rather than after the try statement. The resume point assumes the handler has
 corrected the problem so execution can safely continue.
+If the handlers are not involved in a search this will simply execute the
+@GUARDED_BLOCK@ and then continue to the next statement.
+Its purpose is to add handlers onto the stack.
+(Note, termination and resumption handlers may be intermixed in a @try@
+statement but the kind of throw must be the same as the handler for it to be
+considered as a possible match.)
+If a search for a resumption handler reaches a try block it will check each
+@catchResume@ clause, top-to-bottom.
+At each handler if the thrown exception is or is a child type of
+@EXCEPTION_TYPE@$_i$ then the a pointer to the exception is bound to
+@NAME@$_i$ and then @HANDLER_BLOCK@$_i$ is executed. After the block is
+finished control will return to the @throwResume@ statement.
 Like termination, if no resumption handler is found, the default handler
+visible at the resume statement is called, and the system default action is
+executed.
+For resumption, the exception system uses stack marking to partition the
+resumption search. If another resumption exception is raised in a resumption
+handler, the second exception search does not start at the point of the
+original raise. (Remember the stack is not unwound and the current handler is
+at the top of the stack.) The search for the second resumption starts at the
+current point on the stack because new try statements may have been pushed by
+the handler or functions called from the handler. If there is no match back to
+the point of the current handler, the search skips\label{p:searchskip} the
+stack frames already searched by the first resume and continues after
+the try statement. The default handler always continues from default
+handler associated with the point where the exception is created.
+visible at the throw statement is called. It will use the best match at the
+call sight according to \CFA's overloading rules. The default handler is
+passed the exception given to the throw. When the default handler finishes
+execution continues after the throw statement.
+There is a global @defaultResumptionHandler@ is polymorphic over all
+termination exceptions and preforms a termination throw on the exception.
+The @defaultTerminationHandler@ for that throw is matched at the original
+throw statement (the resumption @throwResume@) and it can be customized by
+introducing a new or better match as well.
+% \subsubsection?
+A key difference between resumption and termination is that resumption does
+not unwind the stack. A side effect that is that when a handler is matched
+and run it's try block (the guarded statements) and every try statement
+searched before it are still on the stack. This can lead to the recursive
+resumption problem.
+The recursive resumption problem is any situation where a resumption handler
+ends up being called while it is running.
+Consider a trivial case:
+\begin{cfa}
+try {
+        throwResume (E &){};
+} catchResume(E *) {
+        throwResume (E &){};
+}
+\end{cfa}
+When this code is executed the guarded @throwResume@ will throw, start a
+search and match the handler in the @catchResume@ clause. This will be
+call and placed on the stack on top of the try-block. The second throw then
+throws and will seach the same try block and put call another instance of the
+same handler leading to an infinite loop.
+This situation is trivial and easy to avoid, but much more complex cycles
+can form with multiple handlers and different exception types.
+To prevent all of these cases we mask sections of the stack, or equvilantly
+the try statements on the stack, so that the resumption seach skips over
+them and continues with the next unmasked section of the stack.
+A section of the stack is marked when it is searched to see if it contains
+a handler for an exception and unmarked when that exception has been handled
+or the search was completed without finding a handler.
 % This might need a diagram. But it is an important part of the justification
 …
 \end{verbatim}
+This resumption search pattern reflects the one for termination, and so
+should come naturally to most programmers.
+However, it avoids the \emph{recursive resumption} problem.
+If parts of the stack are searched multiple times, loops
+can easily form resulting in infinite recursion.
+Consider the trivial case:
+\begin{cfa}
+try {
+        throwResume (E &){}; // first
+} catchResume(E *) {
+        throwResume (E &){}; // second
+}
+\end{cfa}
+If this handler is ever used it will be placed on top of the stack above the
+try statement. If the stack was not masked than the @throwResume@ in the
+handler would always be caught by the handler, leading to an infinite loop.
+Masking avoids this problem and other more complex versions of it involving
+multiple handlers and exception types.
+Other masking stratagies could be used; such as masking the handlers that
+have caught an exception. This one was choosen because it creates a symmetry
+with termination (masked sections of the stack would be unwound with
+termination) and having only one pattern to learn is easier.
+The rules can be remembered as thinking about what would be searched in
+termination. So when a throw happens in a handler; a termination handler
+skips everything from the original throw to the original catch because that
+part of the stack has been unwound, a resumption handler skips the same
+section of stack because it has been masked.
+A throw in a default handler will preform the same search as the original
+throw because; for termination nothing has been unwound, for resumption
+the mask will be the same.
+The symmetry with termination is why this pattern was picked. Other patterns,
+such as marking just the handlers that caught, also work but lack the
+symmetry whih means there is more to remember.
 \section{Conditional Catch}
 …
 condition to further control which exceptions they handle:
 \begin{cfa}
 catch (EXCEPTION_TYPE * NAME ; @CONDITION@)
+catch (EXCEPTION_TYPE * NAME ; CONDITION)
 \end{cfa}
 First, the same semantics is used to match the exception type. Second, if the
 …
 reference all names in scope at the beginning of the try block and @NAME@
 introduced in the handler clause. If the condition is true, then the handler
 matches. Otherwise, the exception search continues at the next appropriate kind
 of handler clause in the try block.
+matches. Otherwise, the exception search continues as if the exception type
+did not match.
 \begin{cfa}
 try {
 …
 remaining handlers in the current try statement.
 \section{Reraise}
 \color{red}{From Andrew: I recomend we talk about why the language doesn't
+\section{Rethrowing}
+\colour{red}{From Andrew: I recomend we talk about why the language doesn't
 have rethrows/reraises instead.}
 \label{s:Reraise}
+\label{s:Rethrowing}
 Within the handler block or functions called from the handler block, it is
 possible to reraise the most recently caught exception with @throw@ or
 @throwResume@, respective.
+@throwResume@, respectively.
 \begin{cfa}
 try {
         ...
 } catch( ... ) {
         ... throw; // rethrow
+        ... throw;
 } catchResume( ... ) {
         ... throwResume; // reresume
+        ... throwResume;
+}
 \end{cfa}
 …
 \section{Finally Clauses}
+A @finally@ clause may be placed at the end of a @try@ statement.
+Finally clauses are used to preform unconditional clean-up when leaving a
+scope. They are placed at the end of a try statement:
 \begin{cfa}
 try {
 …
 \end{cfa}
 The @FINALLY_BLOCK@ is executed when the try statement is removed from the
 stack, including when the @GUARDED_BLOCK@ or any handler clause finishes or
 during an unwind.
+stack, including when the @GUARDED_BLOCK@ finishes, any termination handler
+finishes or during an unwind.
 The only time the block is not executed is if the program is exited before
 that happens.
+the stack is unwound.
 Execution of the finally block should always finish, meaning control runs off
 …
 @return@ that causes control to leave the finally block. Other ways to leave
 the finally block, such as a long jump or termination are much harder to check,
+and at best requiring additional run-time overhead, and so are discouraged.
+and at best requiring additional run-time overhead, and so are mearly
+discouraged.
+Not all languages with exceptions have finally clauses. Notably \Cpp does
+without it as descructors serve a similar role. Although destructors and
+finally clauses can be used in many of the same areas they have their own
+use cases like top-level functions and lambda functions with closures.
+Destructors take a bit more work to set up but are much easier to reuse while
+finally clauses are good for once offs and can include local information.
 \section{Cancellation}
 …
 There is no special statement for starting a cancellation; instead the standard
 library function @cancel_stack@ is called passing an exception. Unlike a
 raise, this exception is not used in matching only to pass information about
+throw, this exception is not used in matching only to pass information about
 the cause of the cancellation.
+Handling of a cancellation depends on which stack is being cancelled.
+(This also means matching cannot fail so there is no default handler either.)
+After @cancel_stack@ is called the exception is copied into the exception
+handling mechanism's memory. Then the entirety of the current stack is
+unwound. After that it depends one which stack is being cancelled.
 \begin{description}
 \item[Main Stack:]
 …
 happen in an implicate join inside a destructor. So there is an error message
 and an abort instead.
+\todo{Perhaps have a more general disucssion of unwind collisions before
+this point.}
 The recommended way to avoid the abort is to handle the intial resumption
 …
 \item[Coroutine Stack:] A coroutine stack is created for a @coroutine@ object
 or object that satisfies the @is_coroutine@ trait. A coroutine only knows of
+two other coroutines, its starter and its last resumer. The last resumer has
+the tightest coupling to the coroutine it activated. Hence, cancellation of
+the active coroutine is forwarded to the last resumer after the stack is
+unwound, as the last resumer has the most precise knowledge about the current
+execution. When the resumer restarts, it resumes exception
+two other coroutines, its starter and its last resumer. Of the two the last
+resumer has the tightest coupling to the coroutine it activated and the most
+up-to-date information.
+Hence, cancellation of the active coroutine is forwarded to the last resumer
+after the stack is unwound. When the resumer restarts, it resumes exception
 @CoroutineCancelled@, which is polymorphic over the coroutine type and has a
 pointer to the cancelled coroutine.

doc/theses/andrew_beach_MMath/uw-ethesis.tex

-              r14533d4
+              rf6664bf2
 % Removes large sections of the document.
 \usepackage{comment}
+% Adds todos (Must be included after comment.)
+\usepackage{todonotes}
 % Hyperlinks make it very easy to navigate an electronic document.
 …
 % Optional arguments do not work with pdf string. (Some fix-up required.)
 \pdfstringdefDisableCommands{\def\Cpp{C++}}
+% Colour text, formatted in LaTeX style instead of TeX style.
+\newcommand*\colour[2]{{\color{#1}#2}}
 \makeatother

doc/theses/thierry_delisle_PhD/thesis/Makefile

-              r14533d4
+              rf6664bf2
 BibTeX = BIBINPUTS=${TeXLIB} && export BIBINPUTS && bibtex
 MAKEFLAGS = --no-print-directory --silent
+MAKEFLAGS = --no-print-directory # --silent
 VPATH = ${Build} ${Figures}
 …
 # Directives #
+.NOTPARALLEL:                                           # cannot make in parallel
 .PHONY : all clean                                      # not file names
 …
         ${LaTeX} $<
-build/fairness.svg : fig/fairness.py | ${Build}
-        python3 $< $@
 ## Define the default recipes.
 …
         sed -i 's/$@/${Build}\/$@/g' ${Build}/$@_t
 build/fairness.svg: fig/fairness.py | ${Build}
         python3 fig/fairness.py build/fairness.svg
+build/fairness.svg : fig/fairness.py | ${Build}
+        python3 $< $@
 ## pstex with inverted colors

doc/theses/thierry_delisle_PhD/thesis/text/io.tex

-              r14533d4
+              rf6664bf2
 \chapter{User Level \io}
 As mentionned in Section~\ref{prev:io}, User-Level \io requires multiplexing the \io operations of many \glspl{thrd} onto fewer \glspl{proc} using asynchronous \io operations. Various operating systems offer various forms of asynchronous operations and as mentioned in Chapter~\ref{intro}, this work is exclusively focuesd on Linux.
+As mentioned in Section~\ref{prev:io}, User-Level \io requires multiplexing the \io operations of many \glspl{thrd} onto fewer \glspl{proc} using asynchronous \io operations. Different operating systems offer various forms of asynchronous operations and as mentioned in Chapter~\ref{intro}, this work is exclusively focused on the Linux operating-system.
 \section{Kernel Interface}
 Since this work fundamentally depends on operating system support, the first step of any design is to discuss the available interfaces and pick one (or more) as the foundations of the \io subsystem.
 \subsection{\lstinline|O_NONBLOCK|}
 In Linux, files can be opened with the flag @O_NONBLOCK@~\cite{MAN:open} (or @SO_NONBLOCK@~\cite{MAN:accept}, the equivalent for sockets) to use the file descriptors in ``nonblocking mode''. In this mode, ``Neither the open() nor any subsequent \io operations on the [opened file descriptor] will cause the calling
 process to wait.'' This feature can be used as the foundation for the \io subsystem. However, for the subsystem to be able to block \glspl{thrd} until an operation completes, @O_NONBLOCK@ must be use in conjunction with a system call that monitors when a file descriptor becomes ready, \ie, the next \io operation on it will not cause the process to wait\footnote{In this context, ready means to \emph{some} operation can be performed without blocking. It does not mean that the last operation that return \lstinline|EAGAIN| will succeed on the next try. A file that is ready to read but has only 1 byte available would be an example of this distinction.}.
+There are three options to monitor file descriptors in Linux\footnote{For simplicity, this section omits to mention \lstinline|pselect| and \lstinline|ppoll|. The difference between these system calls and \lstinline|select| and \lstinline|poll| respectively is not relevant for this discussion.}, @select@~\cite{MAN:select}, @poll@~\cite{MAN:poll} and @epoll@~\cite{MAN:epoll}. All three of these options offer a system call that blocks a \gls{kthrd} until at least one of many file descriptor becomes ready. The group of file descriptors being waited on is often referred to as the \newterm{interest set}.
+\paragraph{\lstinline|select|} is the oldest of these options, it takes as an input a contiguous array of bits, where each bits represent a file descriptor of interest. On return, it modifies the set in place to identify which of the file descriptors changed status. This means that calling select in a loop requires re-initializing the array each time and the number of file descriptors supported has a hard limit. Another limit of @select@ is that once the call is started, the interest set can no longer be modified. Monitoring a new file descriptor generally requires aborting any in progress call to @select@\footnote{Starting a new call to \lstinline|select| in this case is possible but requires a distinct kernel thread, and as a result is not a acceptable multiplexing solution when the interest set is large and highly dynamic unless the number of parallel calls to select can be strictly bounded.}.
+\paragraph{\lstinline|poll|} is an improvement over select, which removes the hard limit on the number of file descriptors and the need to re-initialize the input on every call. It works using an array of structures as an input rather than an array of bits, thus allowing a more compact input for small interest sets. Like @select@, @poll@ suffers from the limitation that the interest set cannot be changed while the call is blocked.
+\paragraph{\lstinline|epoll|} further improves on these two functions, by allowing the interest set to be dynamically added to and removed from while a \gls{kthrd} is blocked on a call to @epoll@. This is done by creating an \emph{epoll instance} with a persistent intereset set and that is used across multiple calls. This advantage significantly reduces synchronization overhead on the part of the caller (in this case the \io subsystem) since the interest set can be modified when adding or removing file descriptors without having to synchronize with other \glspl{kthrd} potentially calling @epoll@.
+However, all three of these system calls suffer from generality problems to some extent. The man page for @O_NONBLOCK@ mentions that ``[@O_NONBLOCK@] has no effect for regular files and block devices'', which means none of these three system calls are viable multiplexing strategies for these types of \io operations. Furthermore, @epoll@ has been shown to have some problems with pipes and ttys\cit{Peter's examples in some fashion}. Finally, none of these are useful solutions for multiplexing \io operations that do not have a corresponding file descriptor and can be awkward for operations using multiple file descriptors.
+\subsection{The POSIX asynchronous I/O (AIO)}
+An alternative to using @O_NONBLOCK@ is to use the AIO interface. Its interface lets programmers enqueue operations to be performed asynchronously by the kernel. Completions of these operations can be communicated in various ways, either by sending a Linux signal, spawning a new \gls{kthrd} or by polling for completion of one or more operation. For the purpose multiplexing operations, spawning a new \gls{kthrd} is counter-productive but a related solution is discussed in Section~\ref{io:morethreads}. Since using interrupts handlers can also lead to fairly complicated interactions between subsystems, I will concentrate on the different polling methods. AIO only supports read and write operations to file descriptors and those do not have the same limitation as @O_NONBLOCK@, \ie, the file descriptors can be regular files and blocked devices. It also supports batching more than one of these operations in a single system call.
+AIO offers two different approach to polling. @aio_error@ can be used as a spinning form of polling, returning @EINPROGRESS@ until the operation is completed, and @aio_suspend@ can be used similarly to @select@, @poll@ or @epoll@, to wait until one or more requests have completed. For the purpose of \io multiplexing, @aio_suspend@ is the intended interface. Even if AIO requests can be submitted concurrently, @aio_suspend@ suffers from the same limitation as @select@ and @poll@, \ie, the interest set cannot be dynamically changed while a call to @aio_suspend@ is in progress. Unlike @select@ and @poll@ however, it also suffers from the limitation that it does not specify which requests have completed, meaning programmers then have to poll each request in the interest set using @aio_error@ to identify which requests have completed. This means that, like @select@ and @poll@ but not @epoll@, the time needed to examine polling results increases based in the total number of requests monitored, not the number of completed requests.
 AIO does not seem to be a particularly popular interface, which I believe is in part due to this less than ideal polling interface. Linus Torvalds talks about this interface as follows :
+Since this work fundamentally depends on operating-system support, the first step of any design is to discuss the available interfaces and pick one (or more) as the foundations of the non-blocking \io subsystem.
+\subsection{\lstinline{O_NONBLOCK}}
+In Linux, files can be opened with the flag @O_NONBLOCK@~\cite{MAN:open} (or @SO_NONBLOCK@~\cite{MAN:accept}, the equivalent for sockets) to use the file descriptors in ``nonblocking mode''. In this mode, ``Neither the @open()@ nor any subsequent \io operations on the [opened file descriptor] will cause the calling
+process to wait''~\cite{MAN:open}. This feature can be used as the foundation for the non-blocking \io subsystem. However, for the subsystem to know when an \io operation completes, @O_NONBLOCK@ must be use in conjunction with a system call that monitors when a file descriptor becomes ready, \ie, the next \io operation on it does not cause the process to wait\footnote{In this context, ready means \emph{some} operation can be performed without blocking. It does not mean an operation returning \lstinline{EAGAIN} succeeds on the next try. For example, a ready read may only return a subset of bytes and the read must be issues again for the remaining bytes, at which point it may return \lstinline{EAGAIN}.}.
+This mechanism is also crucial in determining when all \glspl{thrd} are blocked and the application \glspl{kthrd} can now block.
+There are three options to monitor file descriptors in Linux\footnote{For simplicity, this section omits \lstinline{pselect} and \lstinline{ppoll}. The difference between these system calls and \lstinline{select} and \lstinline{poll}, respectively, is not relevant for this discussion.}, @select@~\cite{MAN:select}, @poll@~\cite{MAN:poll} and @epoll@~\cite{MAN:epoll}. All three of these options offer a system call that blocks a \gls{kthrd} until at least one of many file descriptors becomes ready. The group of file descriptors being waited is called the \newterm{interest set}.
+\paragraph{\lstinline{select}} is the oldest of these options, it takes as an input a contiguous array of bits, where each bits represent a file descriptor of interest. On return, it modifies the set in place to identify which of the file descriptors changed status. This destructive change means that calling select in a loop requires re-initializing the array each time and the number of file descriptors supported has a hard limit. Another limit of @select@ is that once the call is started, the interest set can no longer be modified. Monitoring a new file descriptor generally requires aborting any in progress call to @select@\footnote{Starting a new call to \lstinline{select} is possible but requires a distinct kernel thread, and as a result is not an acceptable multiplexing solution when the interest set is large and highly dynamic unless the number of parallel calls to \lstinline{select} can be strictly bounded.}.
+\paragraph{\lstinline{poll}} is an improvement over select, which removes the hard limit on the number of file descriptors and the need to re-initialize the input on every call. It works using an array of structures as an input rather than an array of bits, thus allowing a more compact input for small interest sets. Like @select@, @poll@ suffers from the limitation that the interest set cannot be changed while the call is blocked.
+\paragraph{\lstinline{epoll}} further improves these two functions by allowing the interest set to be dynamically added to and removed from while a \gls{kthrd} is blocked on an @epoll@ call. This dynamic capability is accomplished by creating an \emph{epoll instance} with a persistent interest set, which is used across multiple calls. This capability significantly reduces synchronization overhead on the part of the caller (in this case the \io subsystem), since the interest set can be modified when adding or removing file descriptors without having to synchronize with other \glspl{kthrd} potentially calling @epoll@.
+However, all three of these system calls have limitations. The @man@ page for @O_NONBLOCK@ mentions that ``[@O_NONBLOCK@] has no effect for regular files and block devices'', which means none of these three system calls are viable multiplexing strategies for these types of \io operations. Furthermore, @epoll@ has been shown to have problems with pipes and ttys~\cit{Peter's examples in some fashion}. Finally, none of these are useful solutions for multiplexing \io operations that do not have a corresponding file descriptor and can be awkward for operations using multiple file descriptors.
+\subsection{POSIX asynchronous I/O (AIO)}
+An alternative to @O_NONBLOCK@ is the AIO interface. Its interface lets programmers enqueue operations to be performed asynchronously by the kernel. Completions of these operations can be communicated in various ways: either by spawning a new \gls{kthrd}, sending a Linux signal, or by polling for completion of one or more operation. For this work, spawning a new \gls{kthrd} is counter-productive but a related solution is discussed in Section~\ref{io:morethreads}. Using interrupts handlers can also lead to fairly complicated interactions between subsystems. Leaving polling for completion, which is similar to the previous system calls. While AIO only supports read and write operations to file descriptors, it does not have the same limitation as @O_NONBLOCK@, \ie, the file descriptors can be regular files and blocked devices. It also supports batching multiple operations in a single system call.
+AIO offers two different approach to polling: @aio_error@ can be used as a spinning form of polling, returning @EINPROGRESS@ until the operation is completed, and @aio_suspend@ can be used similarly to @select@, @poll@ or @epoll@, to wait until one or more requests have completed. For the purpose of \io multiplexing, @aio_suspend@ is the best interface. However, even if AIO requests can be submitted concurrently, @aio_suspend@ suffers from the same limitation as @select@ and @poll@, \ie, the interest set cannot be dynamically changed while a call to @aio_suspend@ is in progress. AIO also suffers from the limitation of specifying which requests have completed, \ie programmers have to poll each request in the interest set using @aio_error@ to identify the completed requests. This limitation means that, like @select@ and @poll@ but not @epoll@, the time needed to examine polling results increases based on the total number of requests monitored, not the number of completed requests.
+Finally, AIO does not seem to be a popular interface, which I believe is due in part to this poor polling interface. Linus Torvalds talks about this interface as follows:
 \begin{displayquote}
         AIO is a horrible ad-hoc design, with the main excuse being "other,
+        AIO is a horrible ad-hoc design, with the main excuse being ``other,
         less gifted people, made that design, and we are implementing it for
         compatibility because database people - who seldom have any shred of
         taste - actually use it".
+        taste - actually use it''.
         But AIO was always really really ugly.
 …
 \end{displayquote}
 Interestingly, in this e-mail answer, Linus goes on to describe
+Interestingly, in this e-mail, Linus goes on to describe
 ``a true \textit{asynchronous system call} interface''
 that does
 …
 This description is actually quite close to the interface described in the next section.
 \subsection{\lstinline|io_uring|}
 A very recent addition to Linux, @io_uring@\cite{MAN:io_uring} is a framework that aims to solve many of the problems listed with the above mentioned interfaces. Like AIO, it represents \io operations as entries added on a queue. But like @epoll@, new requests can be submitted while a blocking call waiting for requests to complete is already in progress. The @io_uring@ interface uses two ring buffers (referred to simply as rings) as its core, a submit ring to which programmers push \io requests and a completion buffer which programmers poll for completion.
 One of the big advantages over the interfaces listed above is that it also supports a much wider range of operations. In addition to supporting reads and writes to any file descriptor like AIO, it supports other operations like @open@, @close@, @fsync@, @accept@, @connect@, @send@, @recv@, @splice@, \etc.
 On top of these, @io_uring@ adds many ``bells and whistles'' like avoiding copies between the kernel and user-space with shared memory, allowing different mechanisms to communicate with device drivers and supporting chains of requests, \ie, requests that automatically trigger followup requests on completion.
+\subsection{\lstinline{io_uring}}
+A very recent addition to Linux, @io_uring@~\cite{MAN:io_uring}, is a framework that aims to solve many of the problems listed in the above interfaces. Like AIO, it represents \io operations as entries added to a queue. But like @epoll@, new requests can be submitted while a blocking call waiting for requests to complete is already in progress. The @io_uring@ interface uses two ring buffers (referred to simply as rings) at its core: a submit ring to which programmers push \io requests and a completion ring from which programmers poll for completion.
+One of the big advantages over the prior interfaces is that @io_uring@ also supports a much wider range of operations. In addition to supporting reads and writes to any file descriptor like AIO, it supports other operations like @open@, @close@, @fsync@, @accept@, @connect@, @send@, @recv@, @splice@, \etc.
+On top of these, @io_uring@ adds many extras like avoiding copies between the kernel and user-space using shared memory, allowing different mechanisms to communicate with device drivers, and supporting chains of requests, \ie, requests that automatically trigger followup requests on completion.
 \subsection{Extra Kernel Threads}\label{io:morethreads}
 Finally, if the operating system does not offer any satisfying forms of asynchronous \io operations, a solution is to fake it by creating a pool of \glspl{kthrd} and delegating operations to them in order to avoid blocking \glspl{proc}. The is a compromise on multiplexing. In the worst case, where all \glspl{thrd} are consistently blocking on \io, it devolves into 1-to-1 threading. However, regardless of the frequency of \io operations, it achieves the fundamental goal of not blocking \glspl{proc} when \glspl{thrd} are ready to run. This approach is used by languages like Go\cit{Go} and frameworks like libuv\cit{libuv}, since it has the advantage that it can easily be used across multiple operating systems. This advantage is especially relevant for languages like Go, which offer an homogenous \glsxtrshort{api} across all platforms. As opposed to C, which has a very limited standard api for \io, \eg, the C standard library has no networking.
+Finally, if the operating system does not offer a satisfactory form of asynchronous \io operations, an ad-hoc solution is to create a pool of \glspl{kthrd} and delegate operations to it to avoid blocking \glspl{proc}, which is a compromise for multiplexing. In the worst case, where all \glspl{thrd} are consistently blocking on \io, it devolves into 1-to-1 threading. However, regardless of the frequency of \io operations, it achieves the fundamental goal of not blocking \glspl{proc} when \glspl{thrd} are ready to run. This approach is used by languages like Go\cit{Go} and frameworks like libuv\cit{libuv}, since it has the advantage that it can easily be used across multiple operating systems. This advantage is especially relevant for languages like Go, which offer a homogeneous \glsxtrshort{api} across all platforms. As opposed to C, which has a very limited standard api for \io, \eg, the C standard library has no networking.
 \subsection{Discussion}
 These options effectively fall into two broad camps of solutions, waiting for \io to be ready versus waiting for \io to be completed. All operating systems that support asynchronous \io must offer an interface along one of these lines, but the details can vary drastically. For example, Free BSD offers @kqueue@~\cite{MAN:bsd/kqueue} which behaves similarly to @epoll@ but with some small quality of life improvements, while Windows (Win32)~\cit{https://docs.microsoft.com/en-us/windows/win32/fileio/synchronous-and-asynchronous-i-o} offers ``overlapped I/O'' which handles submissions similarly to @O_NONBLOCK@, with extra flags on the synchronous system call, but waits for completion events, similarly to @io_uring@.
 For this project, I have chosen to use @io_uring@, in large parts due to its generality. While @epoll@ has been shown to be a good solution to socket \io (\cite{DBLP:journals/pomacs/KarstenB20}), @io_uring@'s transparent support for files, pipes and more complex operations, like @splice@ and @tee@, make it a better choice as the foundation for a general \io subsystem.
+These options effectively fall into two broad camps: waiting for \io to be ready versus waiting for \io to complete. All operating systems that support asynchronous \io must offer an interface along one of these lines, but the details vary drastically. For example, Free BSD offers @kqueue@~\cite{MAN:bsd/kqueue}, which behaves similarly to @epoll@, but with some small quality of use improvements, while Windows (Win32)~\cit{https://docs.microsoft.com/en-us/windows/win32/fileio/synchronous-and-asynchronous-i-o} offers ``overlapped I/O'', which handles submissions similarly to @O_NONBLOCK@ with extra flags on the synchronous system call, but waits for completion events, similarly to @io_uring@.
+For this project, I selected @io_uring@, in large parts because to its generality. While @epoll@ has been shown to be a good solution for socket \io (\cite{DBLP:journals/pomacs/KarstenB20}), @io_uring@'s transparent support for files, pipes, and more complex operations, like @splice@ and @tee@, make it a better choice as the foundation for a general \io subsystem.
 \section{Event-Engine}
+The event engines reponsibility is to use the kernel interface to multiplex many \io operations onto few \glspl{kthrd}. In concrete terms, this means that \glspl{thrd} enter the engine through an interface, the event engines then starts the operation and parks the calling \glspl{thrd}, returning control to the \gls{proc}. The parked \glspl{thrd} are then rescheduled by the event engine once the desired operation has completed.
+\subsection{\lstinline|io_uring| in depth}
+Before going into details on the design of the event engine, I will present some more details on the usage of @io_uring@ which are important for the design of the engine.
+An event engine's responsibility is to use the kernel interface to multiplex many \io operations onto few \glspl{kthrd}. In concrete terms, this means \glspl{thrd} enter the engine through an interface, the event engines then starts the operation and parks the calling \glspl{thrd}, returning control to the \gls{proc}. The parked \glspl{thrd} are then rescheduled by the event engine once the desired operation has completed.
+\subsection{\lstinline{io_uring} in depth}
+Before going into details on the design of my event engine, more details on @io_uring@ usage are presented, each important in the design of the engine.
+Figure~\ref{fig:iouring} shows an overview of an @io_uring@ instance.
+Two ring buffers are used to communicate with the kernel: one for submissions~(left) and one for completions~(right).
+The submission ring contains entries, \newterm{Submit Queue Entries} (SQE), produced (appended) by the application when an operation starts and then consumed by the kernel.
+The completion ring contains entries, \newterm{Completion Queue Entries} (CQE), produced (appended) by the kernel when an operation completes and then consumed by the application.
+The submission ring contains indexes into the SQE array (denoted \emph{S}) containing entries describing the I/O operation to start;
+the completion ring contains entries for the completed I/O operation.
+Multiple @io_uring@ instances can be created, in which case they each have a copy of the data structures in the figure.
 \begin{figure}
         \centering
         \input{io_uring.pstex_t}
+        \caption[Overview of \lstinline|io_uring|]{Overview of \lstinline|io_uring| \smallskip\newline Two ring buffer are used to communicate with the kernel, one for completions~(right) and one for submissions~(left). The completion ring contains entries, \newterm{CQE}s: Completion Queue Entries, that are produced by the kernel when an operation completes and then consumed by the application. On the other hand, the application produces \newterm{SQE}s: Submit Queue Entries, which it appends to the submission ring for the kernel to consume. Unlike the completion ring, the submission ring does not contain the entries directly, it indexes into the SQE array (denoted \emph{S}) instead.}
+        \caption{Overview of \lstinline{io_uring}}
+%       \caption[Overview of \lstinline{io_uring}]{Overview of \lstinline{io_uring} \smallskip\newline Two ring buffer are used to communicate with the kernel, one for completions~(right) and one for submissions~(left). The completion ring contains entries, \newterm{CQE}s: Completion Queue Entries, that are produced by the kernel when an operation completes and then consumed by the application. On the other hand, the application produces \newterm{SQE}s: Submit Queue Entries, which it appends to the submission ring for the kernel to consume. Unlike the completion ring, the submission ring does not contain the entries directly, it indexes into the SQE array (denoted \emph{S}) instead.}
         \label{fig:iouring}
 \end{figure}
+Figure~\ref{fig:iouring} shows an overview of an @io_uring@ instance. Multiple @io_uring@ instances can be created, in which case they each have a copy of the data structures in the figure. New \io operations are submitted to the kernel following 4 steps which use the components shown in the figure.
+\paragraph{First} an @sqe@ must be allocated from the pre-allocated array (denoted \emph{S} in Figure~\ref{fig:iouring}). This array is created at the same time as the @io_uring@ instance, is in kernel-locked memory, which means it is both visible by the kernel and the application, and has a fixed size determined at creation. How these entries are allocated is not important for the functionning of @io_uring@, the only requirement is that no entry is reused before the kernel has consumed it.
+\paragraph{Secondly} the @sqe@ must be filled according to the desired operation. This step is straight forward, the only detail worth mentionning is that @sqe@s have a @user_data@ field that must be filled in order to match submission and completion entries.
+\paragraph{Thirdly} the @sqe@ must be submitted to the submission ring, this requires appending the index of the @sqe@ to the ring following regular ring buffer steps: \lstinline|{ buffer[head] = item; head++ }|. Since the head is visible to the kernel, some memory barriers may be required to prevent the compiler from reordering these operations. Since the submission ring is a regular ring buffer, more than one @sqe@ can be added at once and the head can be updated only after the entire batch has been updated.
+\paragraph{Finally} the kernel must be notified of the change to the ring using the system call @io_uring_enter@. The number of elements appended to the submission ring is passed as a parameter and the number of elements consumed is returned. The @io_uring@ instance can be constructed so that this step is not required, but this requires elevated privilege and early version of @io_uring@ had additionnal restrictions.
+The completion side is simpler, applications call @io_uring_enter@ with the flag @IORING_ENTER_GETEVENTS@ to wait on a desired number of operations to complete. The same call can be used to both submit @sqe@s and wait for operations to complete. When operations do complete the kernel appends a @cqe@ to the completion ring and advances the head of the ring. Each @cqe@ contains the result of the operation as well as a copy of the @user_data@ field of the @sqe@ that triggered the operation. It is not necessary to call @io_uring_enter@ to get new events, the kernel can directly modify the completion ring, the system call is only needed if the application wants to block waiting on operations to complete.
+The @io_uring_enter@ system call is protected by a lock inside the kernel. This means that concurrent call to @io_uring_enter@ using the same instance are possible, but there is can be no performance gained from parallel calls to @io_uring_enter@. It is possible to do the first three submission steps in parallel, however, doing so requires careful synchronization.
+@io_uring@ also introduces some constraints on what the number of operations that can be ``in flight'' at the same time. Obviously, @sqe@s are allocated from a fixed-size array, meaning that there is a hard limit to how many @sqe@s can be submitted at once. In addition, the @io_uring_enter@ system call can fail because ``The  kernel [...] ran out of resources to handle [a request]'' or ``The application is attempting to overcommit the number of requests it can  have  pending.''. This requirement means that it can be required to handle bursts of \io requests by holding back some of the requests so they can be submitted at a later time.
+New \io operations are submitted to the kernel following 4 steps, which use the components shown in the figure.
+\begin{enumerate}
+\item
+An SQE is allocated from the pre-allocated array (denoted \emph{S} in Figure~\ref{fig:iouring}). This array is created at the same time as the @io_uring@ instance, is in kernel-locked memory visible by both the kernel and the application, and has a fixed size determined at creation. How these entries are allocated is not important for the functioning of @io_uring@, the only requirement is that no entry is reused before the kernel has consumed it.
+\item
+The SQE is filled according to the desired operation. This step is straight forward, the only detail worth mentioning is that SQEs have a @user_data@ field that must be filled in order to match submission and completion entries.
+\item
+The SQE is submitted to the submission ring by appending the index of the SQE to the ring following regular ring buffer steps: \lstinline{buffer[head] = item; head++}. Since the head is visible to the kernel, some memory barriers may be required to prevent the compiler from reordering these operations. Since the submission ring is a regular ring buffer, more than one SQE can be added at once and the head is updated only after all entries are updated.
+\item
+The kernel is notified of the change to the ring using the system call @io_uring_enter@. The number of elements appended to the submission ring is passed as a parameter and the number of elements consumed is returned. The @io_uring@ instance can be constructed so this step is not required, but this requires elevated privilege.% and an early version of @io_uring@ had additional restrictions.
+\end{enumerate}
+\begin{sloppypar}
+The completion side is simpler: applications call @io_uring_enter@ with the flag @IORING_ENTER_GETEVENTS@ to wait on a desired number of operations to complete. The same call can be used to both submit SQEs and wait for operations to complete. When operations do complete, the kernel appends a CQE to the completion ring and advances the head of the ring. Each CQE contains the result of the operation as well as a copy of the @user_data@ field of the SQE that triggered the operation. It is not necessary to call @io_uring_enter@ to get new events because the kernel can directly modify the completion ring. The system call is only needed if the application wants to block waiting for operations to complete.
+\end{sloppypar}
+The @io_uring_enter@ system call is protected by a lock inside the kernel. This protection means that concurrent call to @io_uring_enter@ using the same instance are possible, but there is no performance gained from parallel calls to @io_uring_enter@. It is possible to do the first three submission steps in parallel, however, doing so requires careful synchronization.
+@io_uring@ also introduces constraints on the number of simultaneous operations that can be ``in flight''. Obviously, SQEs are allocated from a fixed-size array, meaning that there is a hard limit to how many SQEs can be submitted at once. In addition, the @io_uring_enter@ system call can fail because ``The  kernel [...] ran out of resources to handle [a request]'' or ``The application is attempting to overcommit the number of requests it can  have  pending.''. This restriction means \io request bursts may have to be subdivided and submitted in chunks at a later time.
 \subsection{Multiplexing \io: Submission}
+The submission side is the most complicated aspect of @io_uring@ and the completion side effectively follows from the design decisions made in the submission side.
+While it is possible to do the first steps of submission in parallel, the duration of the system call scales with number of entries submitted. The consequence of this is that how much parallelism can be used to prepare submissions for the next system call is limited. Beyond this limit, the length of the system call will be the throughput limiting factor. I have concluded from early experiments that preparing submissions seems to take about as long as the system call itself, which means that with a single @io_uring@ instance, there is no benefit in terms of \io throughput to having more than two \glspl{hthrd}. Therefore the design of the submission engine must manage multiple instances of @io_uring@ running in parallel, effectively sharding @io_uring@ instances. Similarly to scheduling, this sharding can be done privately, \ie, one instance per \glspl{proc}, in decoupled pools, \ie, a pool of \glspl{proc} use a pool of @io_uring@ instances without one-to-one coupling between any given instance and any given \gls{proc}, or some mix of the two. Since completions are sent to the instance where requests were submitted, all instances with pending operations must be polled continously\footnote{As will be described in Chapter~\ref{practice}, this does not translate into constant cpu usage.}.
+The submission side is the most complicated aspect of @io_uring@ and the completion side effectively follows from the design decisions made in the submission side. While it is possible to do the first steps of submission in parallel, the duration of the system call scales with number of entries submitted. The consequence is that the amount of parallelism used to prepare submissions for the next system call is limited.
+Beyond this limit, the length of the system call is the throughput limiting factor. I concluded from early experiments that preparing submissions seems to take about as long as the system call itself, which means that with a single @io_uring@ instance, there is no benefit in terms of \io throughput to having more than two \glspl{hthrd}. Therefore the design of the submission engine must manage multiple instances of @io_uring@ running in parallel, effectively sharding @io_uring@ instances. Similarly to scheduling, this sharding can be done privately, \ie, one instance per \glspl{proc}, in decoupled pools, \ie, a pool of \glspl{proc} use a pool of @io_uring@ instances without one-to-one coupling between any given instance and any given \gls{proc}, or some mix of the two. Since completions are sent to the instance where requests were submitted, all instances with pending operations must be polled continously\footnote{As will be described in Chapter~\ref{practice}, this does not translate into constant cpu usage.}.
 \subsubsection{Shared Instances}
 …
 Allocation failures need to be pushed up to the routing algorithm: \glspl{thrd} attempting \io operations must not be directed to @io_uring@ instances without sufficient @sqe@s available. Furthermore, the routing algorithm should block operations up-front if none of the instances have available @sqe@s.
+Once an @sqe@ is allocated, \glspl{thrd} can fill them normally, they simply need to keep trac of the @sqe@ index and which instance it belongs to.
+Once an @sqe@ is filled in, what needs to happen is that the @sqe@ must be added to the submission ring buffer, an operation that is not thread-safe on itself, and the kernel must be notified using the @io_uring_enter@ system call. The submission ring buffer is the same size as the pre-allocated @sqe@ buffer, therefore pushing to the ring buffer cannot fail\footnote{This is because it is invalid to have the same \lstinline|sqe| multiple times in the ring buffer.}. However, as mentioned, the system call itself can fail with the expectation that it will be retried once some of the already submitted operations complete. Since multiple @sqe@s can be submitted to the kernel at once, it is important to strike a balance between batching and latency. Operations that are ready to be submitted should be batched together in few system calls, but at the same time, operations should not be left pending for long period of times before being submitted. This can be handled by either designating one of the submitting \glspl{thrd} as the being responsible for the system call for the current batch of @sqe@s or by having some other party regularly submitting all ready @sqe@s, \eg, the poller \gls{thrd} mentionned later in this section.
+In the case of designating a \gls{thrd}, ideally, when multiple \glspl{thrd} attempt to submit operations to the same @io_uring@ instance, all requests would be batched together and one of the \glspl{thrd} would do the system call on behalf of the others, referred to as the \newterm{submitter}. In practice however, it is important that the \io requests are not left pending indefinately and as such, it may be required to have a current submitter and a next submitter. Indeed, as long as there is a ``next'' submitter, \glspl{thrd} submitting new \io requests can move on, knowing that some future system call will include their request. Once the system call is done, the submitter must also free @sqe@s so that the allocator can reused them.
+Finally, the completion side is much simpler since the @io_uring@ system call enforces a natural synchronization point. Polling simply needs to regularly do the system call, go through the produced @cqe@s and communicate the result back to the originating \glspl{thrd}. Since @cqe@s only own a signed 32 bit result, in addition to the copy of the @user_data@ field, all that is needed to communicate the result is a simple future~\cite{wiki:future}. If the submission side does not designate submitters, polling can also submit all @sqe@s as it is polling events.  A simple approach to polling is to allocate a \gls{thrd} per @io_uring@ instance and simply let the poller \glspl{thrd} poll their respective instances when scheduled. This design is especially convinient for reasons explained in Chapter~\ref{practice}.
+Once an SQE is allocated, \glspl{thrd} can fill them normally, they simply need to keep track of the SQE index and which instance it belongs to.
+Once an SQE is filled in, what needs to happen is that the SQE must be added to the submission ring buffer, an operation that is not thread-safe on itself, and the kernel must be notified using the @io_uring_enter@ system call. The submission ring buffer is the same size as the pre-allocated SQE buffer, therefore pushing to the ring buffer cannot fail\footnote{This is because it is invalid to have the same \lstinline{sqe} multiple times in the ring buffer.}. However, as mentioned, the system call itself can fail with the expectation that it will be retried once some of the already submitted operations complete. Since multiple SQEs can be submitted to the kernel at once, it is important to strike a balance between batching and latency. Operations that are ready to be submitted should be batched together in few system calls, but at the same time, operations should not be left pending for long period of times before being submitted. This can be handled by either designating one of the submitting \glspl{thrd} as the being responsible for the system call for the current batch of SQEs or by having some other party regularly submitting all ready SQEs, \eg, the poller \gls{thrd} mentioned later in this section.
+In the case of designating a \gls{thrd}, ideally, when multiple \glspl{thrd} attempt to submit operations to the same @io_uring@ instance, all requests would be batched together and one of the \glspl{thrd} would do the system call on behalf of the others, referred to as the \newterm{submitter}. In practice however, it is important that the \io requests are not left pending indefinitely and as such, it may be required to have a current submitter and a next submitter. Indeed, as long as there is a ``next'' submitter, \glspl{thrd} submitting new \io requests can move on, knowing that some future system call will include their request. Once the system call is done, the submitter must also free SQEs so that the allocator can reused them.
+Finally, the completion side is much simpler since the @io_uring@ system call enforces a natural synchronization point. Polling simply needs to regularly do the system call, go through the produced CQEs and communicate the result back to the originating \glspl{thrd}. Since CQEs only own a signed 32 bit result, in addition to the copy of the @user_data@ field, all that is needed to communicate the result is a simple future~\cite{wiki:future}. If the submission side does not designate submitters, polling can also submit all SQEs as it is polling events.  A simple approach to polling is to allocate a \gls{thrd} per @io_uring@ instance and simply let the poller \glspl{thrd} poll their respective instances when scheduled. This design is especially convenient for reasons explained in Chapter~\ref{practice}.
+<<<<<<< HEAD
 With this pool of instances approach, the big advantage is that it is fairly flexible. It does not impose restrictions on what \glspl{thrd} submitting \io operations can and cannot do between allocations and submissions. It also can gracefully handles running out of ressources, @sqe@s or the kernel returning @EBUSY@. The down side to this is that many of the steps used for submitting need complex synchronization to work properly. The routing and allocation algorithm needs to keep track of which ring instances have available @sqe@s, block incoming requests if no instance is available, prevent barging if \glspl{thrd} are already queued up waiting for @sqe@s and handle @sqe@s being freed. The submission side needs to safely append @sqe@s to the ring buffer, make sure no @sqe@ is dropped or left pending forever, notify the allocation side when @sqe@s can be reused and handle the kernel returning @EBUSY@. All this synchronization may have a significant cost and, compare to the next approach presented, this synchronization is entirely overhead.
 \subsubsection{Private Instances}
 Another approach is to simply create one ring instance per \gls{proc}. This alleviate the need for synchronization on the submissions, requiring only that \glspl{thrd} are not interrupted in between two submission steps. This is effectively the same requirement as using @thread_local@ variables. Since @sqe@s that are allocated must be submitted to the same ring, on the same \gls{proc}, this effectively forces the application to submit @sqe@s in allocation order\footnote{The actual requirement is that \glspl{thrd} cannot context switch between allocation and submission. This requirement means that from the subsystem's point of view, the allocation and submission are sequential. To remove this requirement, a \gls{thrd} would need the ability to ``yield to a specific \gls{proc}'', \ie, park with the promise that it will be run next on a specific \gls{proc}, the \gls{proc} attached to the correct ring.}, greatly simplifying both allocation and submission. In this design, allocation and submission form a ring partitionned ring buffer as shown in Figure~\ref{fig:pring}. Once added to the ring buffer, the attached \gls{proc} has a significant amount of flexibility with regards to when to do the system call. Possible options are: when the \gls{proc} runs out of \glspl{thrd} to run, after running a given number of threads \glspl{thrd}, etc.
+=======
+With this pool of instances approach, the big advantage is that it is fairly flexible. It does not impose restrictions on what \glspl{thrd} submitting \io operations can and cannot do between allocations and submissions. It also can gracefully handle running out of resources, SQEs or the kernel returning @EBUSY@. The down side to this is that many of the steps used for submitting need complex synchronization to work properly. The routing and allocation algorithm needs to keep track of which ring instances have available SQEs, block incoming requests if no instance is available, prevent barging if \glspl{thrd} are already queued up waiting for SQEs and handle SQEs being freed. The submission side needs to safely append SQEs to the ring buffer, make sure no SQE is dropped or left pending forever, notify the allocation side when SQEs can be reused and handle the kernel returning @EBUSY@. Sharding the @io_uring@ instances should alleviate much of the contention caused by this, but all this synchronization may still have non-zero cost.
+\subsubsection{Private Instances}
+Another approach is to simply create one ring instance per \gls{proc}. This alleviate the need for synchronization on the submissions, requiring only that \glspl{thrd} are not interrupted in between two submission steps. This is effectively the same requirement as using @thread_local@ variables. Since SQEs that are allocated must be submitted to the same ring, on the same \gls{proc}, this effectively forces the application to submit SQEs in allocation order\footnote{The actual requirement is that \glspl{thrd} cannot context switch between allocation and submission. This requirement means that from the subsystem's point of view, the allocation and submission are sequential. To remove this requirement, a \gls{thrd} would need the ability to ``yield to a specific \gls{proc}'', \ie, park with the promise that it will be run next on a specific \gls{proc}, the \gls{proc} attached to the correct ring. This is not a current or planned feature of \CFA.}, greatly simplifying both allocation and submission. In this design, allocation and submission form a ring partitioned ring buffer as shown in Figure~\ref{fig:pring}. Once added to the ring buffer, the attached \gls{proc} has a significant amount of flexibility with regards to when to do the system call. Possible options are: when the \gls{proc} runs out of \glspl{thrd} to run, after running a given number of threads \glspl{thrd}, etc.
+>>>>>>> 1830a8657cb302a89a7ca045bee06baa48b18101
 \begin{figure}
         \centering
         \input{pivot_ring.pstex_t}
         \caption[Partitionned ring buffer]{Partitionned ring buffer \smallskip\newline Allocated sqes are appending to the first partition. When submitting, the partition is simply advanced to include all the sqes that should be submitted. The kernel considers the partition as the head of the ring.}
+        \caption[Partitioned ring buffer]{Partitioned ring buffer \smallskip\newline Allocated sqes are appending to the first partition. When submitting, the partition is simply advanced to include all the sqes that should be submitted. The kernel considers the partition as the head of the ring.}
         \label{fig:pring}
 \end{figure}
+<<<<<<< HEAD
 This approach has the advantage that it does not require much of the synchronization needed in the shared approach. This comes at the cost that \glspl{thrd} submitting \io operations have less flexibility, they cannot park or yield, and several exceptional cases are handled poorly. Instances running out of @sqe@s cannot run \glspl{thrd} wanting to do \io operations, in such a case the \gls{thrd} needs to be moved to a different \gls{proc}, the only current way of achieving this would be to @yield()@ hoping to be scheduled on a different \gls{proc}, which is not guaranteed.
 …
 %       if cltr.io.flag || proc.io != alloc.io || proc.io->flag:
 %               return submit_slow(cltr.io)
+=======
+This approach has the advantage that it does not require much of the synchronization needed in the shared approach. This comes at the cost that \glspl{thrd} submitting \io operations have less flexibility, they cannot park or yield, and several exceptional cases are handled poorly. Instances running out of SQEs cannot run \glspl{thrd} wanting to do \io operations, in such a case the \gls{thrd} needs to be moved to a different \gls{proc}, the only current way of achieving this would be to @yield()@ hoping to be scheduled on a different \gls{proc}, which is not guaranteed. Another problematic case is that \glspl{thrd} that do not park for long periods of time will delay the submission of any SQE not already submitted. This issue is similar to fairness issues which schedulers that use work-stealing mentioned in the previous chapter.
+>>>>>>> 1830a8657cb302a89a7ca045bee06baa48b18101
 %       submit_fast(proc.io, a)
 …
 \subsection{Asynchronous Extension}
 \subsection{Interface directly to \lstinline|io_uring|}
+\subsection{Interface directly to \lstinline{io_uring}}

doc/theses/thierry_delisle_PhD/thesis/thesis.tex

-              r14533d4
+              rf6664bf2
+% uWaterloo Thesis Template for LaTeX
+% Last Updated June 14, 2017 by Stephen Carr, IST Client Services
+% FOR ASSISTANCE, please send mail to rt-IST-CSmathsci@ist.uwaterloo.ca
+% Effective October 2006, the University of Waterloo
+% requires electronic thesis submission. See the uWaterloo thesis regulations at
+%======================================================================
+% University of Waterloo Thesis Template for LaTeX
+% Last Updated November, 2020
+% by Stephen Carr, IST Client Services,
+% University of Waterloo, 200 University Ave. W., Waterloo, Ontario, Canada
+% FOR ASSISTANCE, please send mail to request@uwaterloo.ca
+% DISCLAIMER
+% To the best of our knowledge, this template satisfies the current uWaterloo thesis requirements.
+% However, it is your responsibility to assure that you have met all requirements of the University and your particular department.
+% Many thanks for the feedback from many graduates who assisted the development of this template.
+% Also note that there are explanatory comments and tips throughout this template.
+%======================================================================
+% Some important notes on using this template and making it your own...
+% The University of Waterloo has required electronic thesis submission since October 2006.
+% See the uWaterloo thesis regulations at
 % https://uwaterloo.ca/graduate-studies/thesis.
+% DON'T FORGET TO ADD YOUR OWN NAME AND TITLE in the "hyperref" package
+% configuration below. THIS INFORMATION GETS EMBEDDED IN THE PDF FINAL PDF DOCUMENT.
+% You can view the information if you view Properties of the PDF document.
+% Many faculties/departments also require one or more printed
+% copies. This template attempts to satisfy both types of output.
+% It is based on the standard "book" document class which provides all necessary
+% sectioning structures and allows multi-part theses.
+% DISCLAIMER
+% To the best of our knowledge, this template satisfies the current uWaterloo requirements.
+% However, it is your responsibility to assure that you have met all
+% requirements of the University and your particular department.
+% Many thanks for the feedback from many graduates that assisted the development of this template.
+% -----------------------------------------------------------------------
+% By default, output is produced that is geared toward generating a PDF
+% version optimized for viewing on an electronic display, including
+% hyperlinks within the PDF.
+% This thesis template is geared towards generating a PDF version optimized for viewing on an electronic display, including hyperlinks within the PDF.
+% DON'T FORGET TO ADD YOUR OWN NAME AND TITLE in the "hyperref" package configuration below.
+% THIS INFORMATION GETS EMBEDDED IN THE PDF FINAL PDF DOCUMENT.
+% You can view the information if you view properties of the PDF document.
+% Many faculties/departments also require one or more printed copies.
+% This template attempts to satisfy both types of output.
+% See additional notes below.
+% It is based on the standard "book" document class which provides all necessary sectioning structures and allows multi-part theses.
+% If you are using this template in Overleaf (cloud-based collaboration service), then it is automatically processed and previewed for you as you edit.
+% For people who prefer to install their own LaTeX distributions on their own computers, and process the source files manually, the following notes provide the sequence of tasks:
 % E.g. to process a thesis called "mythesis.tex" based on this template, run:
 % pdflatex mythesis     -- first pass of the pdflatex processor
 % bibtex mythesis       -- generates bibliography from .bib data file(s)
 % makeindex         -- should be run only if an index is used
+% makeindex         -- should be run only if an index is used
 % pdflatex mythesis     -- fixes numbering in cross-references, bibliographic references, glossaries, index, etc.
+% pdflatex mythesis     -- fixes numbering in cross-references, bibliographic references, glossaries, index, etc.
+% If you use the recommended LaTeX editor, Texmaker, you would open the mythesis.tex
+% file, then click the PDFLaTeX button. Then run BibTeX (under the Tools menu).
+% Then click the PDFLaTeX button two more times. If you have an index as well,
+% you'll need to run MakeIndex from the Tools menu as well, before running pdflatex
+% pdflatex mythesis     -- it takes a couple of passes to completely process all cross-references
+% If you use the recommended LaTeX editor, Texmaker, you would open the mythesis.tex file, then click the PDFLaTeX button. Then run BibTeX (under the Tools menu).
+% Then click the PDFLaTeX button two more times.
+% If you have an index as well,you'll need to run MakeIndex from the Tools menu as well, before running pdflatex
 % the last two times.
+% N.B. The "pdftex" program allows graphics in the following formats to be
+% included with the "\includegraphics" command: PNG, PDF, JPEG, TIFF
+% Tip 1: Generate your figures and photos in the size you want them to appear
+% in your thesis, rather than scaling them with \includegraphics options.
+% Tip 2: Any drawings you do should be in scalable vector graphic formats:
+% SVG, PNG, WMF, EPS and then converted to PNG or PDF, so they are scalable in
+% the final PDF as well.
+% Tip 3: Photographs should be cropped and compressed so as not to be too large.
+% To create a PDF output that is optimized for double-sided printing:
+%
+% 1) comment-out the \documentclass statement in the preamble below, and
+% un-comment the second \documentclass line.
+%
+% 2) change the value assigned below to the boolean variable
+% "PrintVersion" from "false" to "true".
+% --------------------- Start of Document Preamble -----------------------
+% Specify the document class, default style attributes, and page dimensions
+% N.B. The "pdftex" program allows graphics in the following formats to be included with the "\includegraphics" command: PNG, PDF, JPEG, TIFF
+% Tip: Generate your figures and photos in the size you want them to appear in your thesis, rather than scaling them with \includegraphics options.
+% Tip: Any drawings you do should be in scalable vector graphic formats: SVG, PNG, WMF, EPS and then converted to PNG or PDF, so they are scalable in the final PDF as well.
+% Tip: Photographs should be cropped and compressed so as not to be too large.
+% To create a PDF output that is optimized for double-sided printing:
+% 1) comment-out the \documentclass statement in the preamble below, and un-comment the second \documentclass line.
+% 2) change the value assigned below to the boolean variable "PrintVersion" from " false" to "true".
+%======================================================================
+%   D O C U M E N T   P R E A M B L E
+% Specify the document class, default style attributes, and page dimensions, etc.
 % For hyperlinked PDF, suitable for viewing on a computer, use this:
 \documentclass[letterpaper,12pt,titlepage,oneside,final]{book}
+% For PDF, suitable for double-sided printing, change the PrintVersion variable below
+% to "true" and use this \documentclass line instead of the one above:
+% For PDF, suitable for double-sided printing, change the PrintVersion variable below to "true" and use this \documentclass line instead of the one above:
 %\documentclass[letterpaper,12pt,titlepage,openright,twoside,final]{book}
+\newcommand{\href}[1]{#1} % does nothing, but defines the command so the
+    % print-optimized version will ignore \href tags (redefined by hyperref pkg).
+% Some LaTeX commands I define for my own nomenclature.
+% If you have to, it's easier to make changes to nomenclature once here than in a million places throughout your thesis!
+\newcommand{\package}[1]{\textbf{#1}} % package names in bold text
+\newcommand{\cmmd}[1]{\textbackslash\texttt{#1}} % command name in tt font
+\newcommand{\href}[1]{#1} % does nothing, but defines the command so the print-optimized version will ignore \href tags (redefined by hyperref pkg).
+%\newcommand{\texorpdfstring}[2]{#1} % does nothing, but defines the command
+% Anything defined here may be redefined by packages added below...
 % This package allows if-then-else control structures.
 …
 \newboolean{PrintVersion}
 \setboolean{PrintVersion}{false}
+% CHANGE THIS VALUE TO "true" as necessary, to improve printed results for hard copies
+% by overriding some options of the hyperref package below.
+% CHANGE THIS VALUE TO "true" as necessary, to improve printed results for hard copies by overriding some options of the hyperref package, called below.
 %\usepackage{nomencl} % For a nomenclature (optional; available from ctan.org)
 …
 % Hyperlinks make it very easy to navigate an electronic document.
+% In addition, this is where you should specify the thesis title
+% and author as they appear in the properties of the PDF document.
+% In addition, this is where you should specify the thesis title and author as they appear in the properties of the PDF document.
 % Use the "hyperref" package
 % N.B. HYPERREF MUST BE THE LAST PACKAGE LOADED; ADD ADDITIONAL PKGS ABOVE
 \usepackage[pagebackref=false]{hyperref} % with basic options
+                % N.B. pagebackref=true provides links back from the References to the body text. This can cause trouble for printing.
+%\usepackage[pdftex,pagebackref=true]{hyperref}
+% N.B. pagebackref=true provides links back from the References to the body text. This can cause trouble for printing.
 \hypersetup{
         plainpages=false,       % needed if Roman numbers in frontpages
         unicode=false,          % non-Latin characters in Acrobat’s bookmarks
         pdftoolbar=true,        % show Acrobat’s toolbar?
         pdfmenubar=true,        % show Acrobat’s menu?
+        unicode=false,          % non-Latin characters in Acrobat's bookmarks
+        pdftoolbar=true,        % show Acrobat's toolbar?
+        pdfmenubar=true,        % show Acrobat's menu?
         pdffitwindow=false,     % window fit to page when opened
         pdfstartview={FitH},    % fits the width of the page to the window
 …
 \ifthenelse{\boolean{PrintVersion}}{   % for improved print quality, change some hyperref options
 \hypersetup{    % override some previously defined hyperref options
         citecolor=black,
         filecolor=black,
         linkcolor=black,
+        citecolor=black,%
+        filecolor=black,%
+        linkcolor=black,%
         urlcolor=black
 }}{} % end of ifthenelse (no else)
 …
 % Setting up the page margins...
+\setlength{\textheight}{9in}\setlength{\topmargin}{-0.45in}\setlength{\headsep}{0.25in}
+\setlength{\textheight}{9in}
+\setlength{\topmargin}{-0.45in}
+\setlength{\headsep}{0.25in}
 % uWaterloo thesis requirements specify a minimum of 1 inch (72pt) margin at the
+% top, bottom, and outside page edges and a 1.125 in. (81pt) gutter
+% margin (on binding side). While this is not an issue for electronic
+% viewing, a PDF may be printed, and so we have the same page layout for
+% both printed and electronic versions, we leave the gutter margin in.
+% top, bottom, and outside page edges and a 1.125 in. (81pt) gutter margin (on binding side).
+% While this is not an issue for electronic viewing, a PDF may be printed, and so we have the same page layout for both printed and electronic versions, we leave the gutter margin in.
 % Set margins to minimum permitted by uWaterloo thesis regulations:
 \setlength{\marginparwidth}{0pt} % width of margin notes
 …
 \setlength{\evensidemargin}{0.125in} % Adds 1/8 in. to binding side of all
 % even-numbered pages when the "twoside" printing option is selected
+\setlength{\oddsidemargin}{0.125in} % Adds 1/8 in. to the left of all pages
+% when "oneside" printing is selected, and to the left of all odd-numbered
+% pages when "twoside" printing is selected
+\setlength{\textwidth}{6.375in} % assuming US letter paper (8.5 in. x 11 in.) and
+% side margins as above
+\setlength{\oddsidemargin}{0.125in} % Adds 1/8 in. to the left of all pages when "oneside" printing is selected, and to the left of all odd-numbered pages when "twoside" printing is selected
+\setlength{\textwidth}{6.375in} % assuming US letter paper (8.5 in. x 11 in.) and side margins as above
 \raggedbottom
+% The following statement specifies the amount of space between
+% paragraphs. Other reasonable specifications are \bigskipamount and \smallskipamount.
+% The following statement specifies the amount of space between paragraphs. Other reasonable specifications are \bigskipamount and \smallskipamount.
 \setlength{\parskip}{\medskipamount}
+% The following statement controls the line spacing.  The default
+% spacing corresponds to good typographic conventions and only slight
+% changes (e.g., perhaps "1.2"), if any, should be made.
+% The following statement controls the line spacing.
+% The default spacing corresponds to good typographic conventions and only slight changes (e.g., perhaps "1.2"), if any, should be made.
 \renewcommand{\baselinestretch}{1} % this is the default line space setting
+% By default, each chapter will start on a recto (right-hand side)
+% page.  We also force each section of the front pages to start on
+% a recto page by inserting \cleardoublepage commands.
+% In many cases, this will require that the verso page be
+% blank and, while it should be counted, a page number should not be
+% printed.  The following statements ensure a page number is not
+% printed on an otherwise blank verso page.
+% By default, each chapter will start on a recto (right-hand side) page.
+% We also force each section of the front pages to start on a recto page by inserting \cleardoublepage commands.
+% In many cases, this will require that the verso (left-hand) page be blank, and while it should be counted, a page number should not be printed.
+% The following statements ensure a page number is not printed on an otherwise blank verso page.
 \let\origdoublepage\cleardoublepage
 \newcommand{\clearemptydoublepage}{%
 …
 \input{common}
 \CFAStyle                                               % CFA code-style for all languages
 \lstset{basicstyle=\linespread{0.9}\tt}
+\lstset{language=CFA,basicstyle=\linespread{0.9}\tt}    % CFA default language
 % glossary of terms to use
 …
 \makeindex
+\newcommand\io{\glsxtrshort{io}}%
+%======================================================================
+%   L O G I C A L    D O C U M E N T -- the content of your thesis
+\newcommand\io{\glsxtrshort{io}\xspace}%
+%======================================================================
+%   L O G I C A L    D O C U M E N T
+% The logical document contains the main content of your thesis.
+% Being a large document, it is a good idea to divide your thesis into several files, each one containing one chapter or other significant chunk of content, so you can easily shuffle things around later if desired.
 %======================================================================
 \begin{document}
-% For a large document, it is a good idea to divide your thesis
-% into several files, each one containing one chapter.
-% To illustrate this idea, the "front pages" (i.e., title page,
-% declaration, borrowers' page, abstract, acknowledgements,
-% dedication, table of contents, list of tables, list of figures,
-% nomenclature) are contained within the file "uw-ethesis-frontpgs.tex" which is
-% included into the document by the following statement.
 %----------------------------------------------------------------------
 % FRONT MATERIAL
+% title page,declaration, borrowers' page, abstract, acknowledgements,
+% dedication, table of contents, list of tables, list of figures, nomenclature, etc.
 %----------------------------------------------------------------------
 \input{text/front.tex}
 %----------------------------------------------------------------------
 % MAIN BODY
 %----------------------------------------------------------------------
 % Because this is a short document, and to reduce the number of files
 % needed for this template, the chapters are not separate
 % documents as suggested above, but you get the idea. If they were
 % separate documents, they would each start with the \chapter command, i.e,
+% do not contain \documentclass or \begin{document} and \end{document} commands.
+% We suggest using a separate file for each chapter of your thesis.
+% Start each chapter file with the \chapter command.
+% Only use \documentclass or \begin{document} and \end{document} commands in this master document.
+% Tip: Putting each sentence on a new line is a way to simplify later editing.
+%----------------------------------------------------------------------
 \part{Introduction}
 \input{text/intro.tex}
 …
 %----------------------------------------------------------------------
 % END MATERIAL
+%----------------------------------------------------------------------
+% B I B L I O G R A P H Y
+% -----------------------
+% The following statement selects the style to use for references.  It controls the sort order of the entries in the bibliography and also the formatting for the in-text labels.
+% Bibliography, Appendices, Index, etc.
+%----------------------------------------------------------------------
+% Bibliography
+% The following statement selects the style to use for references.
+% It controls the sort order of the entries in the bibliography and also the formatting for the in-text labels.
 \bibliographystyle{plain}
 % This specifies the location of the file containing the bibliographic information.
+% It assumes you're using BibTeX (if not, why not?).
+\cleardoublepage % This is needed if the book class is used, to place the anchor in the correct page,
+                 % because the bibliography will start on its own page.
+                 % Use \clearpage instead if the document class uses the "oneside" argument
+% It assumes you're using BibTeX to manage your references (if not, why not?).
+\cleardoublepage % This is needed if the "book" document class is used, to place the anchor in the correct page, because the bibliography will start on its own page.
+% Use \clearpage instead if the document class uses the "oneside" argument
 \phantomsection  % With hyperref package, enables hyperlinking from the table of contents to bibliography
 % The following statement causes the title "References" to be used for the bibliography section:
 …
 \bibliography{local,pl}
 % Tip 5: You can create multiple .bib files to organize your references.
+% Tip: You can create multiple .bib files to organize your references.
 % Just list them all in the \bibliogaphy command, separated by commas (no spaces).
 % % The following statement causes the specified references to be added to the bibliography% even if they were not
 % % cited in the text. The asterisk is a wildcard that causes all entries in the bibliographic database to be included (optional).
+% The following statement causes the specified references to be added to the bibliography even if they were not cited in the text.
+% The asterisk is a wildcard that causes all entries in the bibliographic database to be included (optional).
 % \nocite{*}
+%----------------------------------------------------------------------
+% Appendices
 % The \appendix statement indicates the beginning of the appendices.
 \appendix
 % Add a title page before the appendices and a line in the Table of Contents
+% Add an un-numbered title page before the appendices and a line in the Table of Contents
 \chapter*{APPENDICES}
 \addcontentsline{toc}{chapter}{APPENDICES}
+% Appendices are just more chapters, with different labeling (letters instead of numbers).
 %======================================================================
 \chapter[PDF Plots From Matlab]{Matlab Code for Making a PDF Plot}
 …
 %\input{thesis.ind}                             % index
+\phantomsection
+\end{document}
+\phantomsection         % allows hyperref to link to the correct page
+%----------------------------------------------------------------------
+\end{document} % end of logical document

Note: See TracChangeset for help on using the changeset viewer.