Index: doc/theses/thierry_delisle_PhD/thesis/Makefile =================================================================== --- doc/theses/thierry_delisle_PhD/thesis/Makefile (revision d95969a75f9a2102e5866017daba99c54b1bb621) +++ doc/theses/thierry_delisle_PhD/thesis/Makefile (revision c292244d4c299c228d7e40cd8a91071f5066c92f) @@ -32,4 +32,5 @@ emptytree \ fairness \ + io_uring \ system \ } Index: doc/theses/thierry_delisle_PhD/thesis/fig/io_uring.fig =================================================================== --- doc/theses/thierry_delisle_PhD/thesis/fig/io_uring.fig (revision c292244d4c299c228d7e40cd8a91071f5066c92f) +++ doc/theses/thierry_delisle_PhD/thesis/fig/io_uring.fig (revision c292244d4c299c228d7e40cd8a91071f5066c92f) @@ -0,0 +1,95 @@ +#FIG 3.2 Produced by xfig version 3.2.7b +Landscape +Center +Metric +A4 +100.00 +Single +-2 +1200 2 +6 180 3240 2025 3510 +2 1 0 1 0 7 40 -1 -1 0.000 0 0 -1 0 0 2 + 720 3240 720 3510 +2 1 0 1 0 7 40 -1 -1 0.000 0 0 -1 0 0 2 + 450 3240 450 3510 +2 2 0 1 0 7 45 -1 20 0.000 0 0 -1 0 0 5 + 180 3240 1260 3240 1260 3510 180 3510 180 3240 +2 1 0 1 0 7 40 -1 -1 0.000 0 0 -1 0 0 2 + 990 3240 990 3510 +4 0 0 40 -1 0 12 0.0000 2 165 990 1035 3420 {\\small S3}\001 +4 0 0 40 -1 0 12 0.0000 2 165 990 765 3420 {\\small S2}\001 +4 0 0 40 -1 0 12 0.0000 2 165 990 225 3420 {\\small S0}\001 +4 0 0 40 -1 0 12 0.0000 2 165 990 495 3420 {\\small S1}\001 +-6 +6 1530 2610 3240 4140 +5 1 0 1 0 7 35 -1 -1 0.000 0 1 1 0 2455.714 3375.000 1890 2700 1575 3375 1890 4050 + 1 1 1.00 60.00 120.00 +1 3 0 1 0 7 40 -1 20 0.000 1 0.0000 2475 3375 315 315 2475 3375 2790 3375 +1 3 0 1 0 7 50 -1 20 0.000 1 0.0000 2475 3375 765 765 2475 3375 3240 3375 +2 1 0 1 0 7 45 -1 -1 0.000 0 0 -1 0 0 2 + 2475 3375 2133 2690 +2 1 0 1 0 7 45 -1 -1 4.000 0 0 -1 0 0 2 + 2475 3375 1769 3093 +2 1 0 1 0 7 45 -1 -1 4.000 0 0 -1 0 0 2 + 2475 3375 1769 3661 +2 1 0 1 0 7 45 -1 -1 4.000 0 0 -1 0 0 2 + 2475 3375 2133 4057 +2 1 1 1 0 7 35 -1 0 4.000 0 0 -1 0 0 2 + 2205 3375 2745 3375 +-6 +6 585 2250 1485 2610 +4 2 0 50 -1 0 12 0.0000 2 135 900 1485 2385 Submission\001 +4 2 0 50 -1 0 12 0.0000 2 165 360 1485 2580 Ring\001 +-6 +6 3600 2610 5265 4140 +5 1 0 1 0 7 35 -1 -1 0.000 0 1 1 0 4384.000 3375.000 4950 4050 5265 3375 4950 2700 + 1 1 1.00 60.00 120.00 +1 3 0 1 0 7 40 -1 20 0.000 1 3.1416 4365 3375 315 315 4365 3375 4050 3375 +1 3 0 1 0 7 50 -1 20 0.000 1 3.1416 4365 3375 765 765 4365 3375 3600 3375 +2 1 0 1 0 7 45 -1 -1 0.000 0 0 -1 0 0 2 + 4365 3375 4707 4060 +2 1 0 1 0 7 45 -1 -1 4.000 0 0 -1 0 0 2 + 4365 3375 5071 3657 +2 1 0 1 0 7 45 -1 -1 4.000 0 0 -1 0 0 2 + 4365 3375 5071 3089 +2 1 0 1 0 7 45 -1 -1 4.000 0 0 -1 0 0 2 + 4365 3375 4707 2693 +2 1 1 1 0 7 35 -1 0 4.000 0 0 -1 0 0 2 + 4635 3375 4095 3375 +-6 +6 5355 2250 6255 2610 +4 0 0 50 -1 0 12 0.0000 2 165 360 5355 2580 Ring\001 +4 0 0 50 -1 0 12 0.0000 2 165 900 5355 2385 Completion\001 +-6 +2 1 0 1 0 7 50 -1 -1 0.000 0 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 2925 2025 2550 2486 +2 1 0 1 0 7 50 -1 -1 4.000 0 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 4275 2475 3825 2025 +2 1 0 1 0 7 50 -1 -1 4.000 0 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 2751 4268 3066 4538 +2 1 0 1 0 7 50 -1 -1 4.000 0 0 -1 1 0 2 + 1 1 1.00 60.00 120.00 + 3780 4545 4275 4230 +2 1 1 1 0 7 55 -1 -1 4.000 0 0 -1 0 0 2 + 0 3375 6255 3375 +4 0 0 35 -1 0 12 0.0000 2 165 1170 1845 3060 {\\small \\&S2}\001 +4 0 0 35 -1 0 12 0.0000 2 165 1170 1755 3420 {\\small \\&S3}\001 +4 0 0 35 -1 0 12 0.0000 2 165 1170 1890 3735 {\\small \\&S0}\001 +4 0 0 50 -1 0 12 0.0000 6 135 360 2790 2565 Push\001 +4 0 0 50 -1 0 12 0.0000 6 165 270 2880 4230 Pop\001 +4 0 0 50 -1 0 12 0.0000 6 135 360 2025 4275 Head\001 +4 0 0 50 -1 0 12 0.0000 6 135 360 2025 2565 Tail\001 +4 0 0 35 -1 0 12 0.0000 2 165 990 4635 3060 {\\small C0}\001 +4 0 0 35 -1 0 12 0.0000 2 165 990 4815 3420 {\\small C1}\001 +4 0 0 35 -1 0 12 0.0000 2 165 990 4635 3780 {\\small C2}\001 +4 0 0 50 -1 0 12 0.0000 4 135 360 4725 4275 Tail\001 +4 0 0 50 -1 0 12 0.0000 6 135 360 4590 2565 Head\001 +4 0 0 50 -1 0 12 0.0000 2 135 990 5535 3285 Kernel Line\001 +4 1 0 50 -1 0 12 0.0000 2 180 1350 3375 4815 {\\Large Kernel}\001 +4 1 0 50 -1 0 12 0.0000 2 180 1800 3375 1845 {\\Large Application}\001 +4 0 0 50 -1 0 12 0.0000 6 165 270 3690 2565 Pop\001 +4 0 0 50 -1 0 12 0.0000 4 135 360 3465 4230 Push\001 +4 0 0 50 -1 0 12 0.0000 2 135 90 0 3285 S\001 Index: doc/theses/thierry_delisle_PhD/thesis/local.bib =================================================================== --- doc/theses/thierry_delisle_PhD/thesis/local.bib (revision d95969a75f9a2102e5866017daba99c54b1bb621) +++ doc/theses/thierry_delisle_PhD/thesis/local.bib (revision c292244d4c299c228d7e40cd8a91071f5066c92f) @@ -512,4 +512,11 @@ } +@manual{MAN:bsd/kqueue, + title = {KQUEUE(2) - FreeBSD System Calls Manual}, + url = {https://www.freebsd.org/cgi/man.cgi?query=kqueue}, + year = {2020}, + month = {may} +} + % Apple's MAC OS X @manual{MAN:apple/scheduler, @@ -577,4 +584,57 @@ % -------------------------------------------------- +% Man Pages +@manual{MAN:open, + key = "open", + title = "open(2) Linux User's Manual", + year = "2020", + month = "February", +} + +@manual{MAN:accept, + key = "accept", + title = "accept(2) Linux User's Manual", + year = "2019", + month = "March", +} + +@manual{MAN:select, + key = "select", + title = "select(2) Linux User's Manual", + year = "2019", + month = "March", +} + +@manual{MAN:poll, + key = "poll", + title = "poll(2) Linux User's Manual", + year = "2019", + month = "July", +} + +@manual{MAN:epoll, + key = "epoll", + title = "epoll(7) Linux User's Manual", + year = "2019", + month = "March", +} + +@manual{MAN:aio, + key = "aio", + title = "aio(7) Linux User's Manual", + year = "2019", + month = "March", +} + +@misc{MAN:io_uring, + title = {Efficient IO with io\_uring}, + author = {Axboe, Jens}, + year = "2019", + month = "March", + version = {0,4}, + howpublished = {\url{https://kernel.dk/io_uring.pdf}} +} + +% -------------------------------------------------- % Wikipedia Entries @misc{wiki:taskparallel, Index: doc/theses/thierry_delisle_PhD/thesis/text/intro.tex =================================================================== --- doc/theses/thierry_delisle_PhD/thesis/text/intro.tex (revision d95969a75f9a2102e5866017daba99c54b1bb621) +++ doc/theses/thierry_delisle_PhD/thesis/text/intro.tex (revision c292244d4c299c228d7e40cd8a91071f5066c92f) @@ -7,3 +7,3 @@ While previous work on the concurrent package of \CFA focused on features and interfaces, this thesis focuses on performance, introducing \glsxtrshort{api} changes only when required by performance considerations. More specifically, this thesis concentrates on scheduling and \glsxtrshort{io}. Prior to this work, the \CFA runtime used a strictly \glsxtrshort{fifo} \gls{rQ}. -This work exclusively concentrates on Linux as it's operating system since the existing \CFA runtime and compiler does not already support other operating systems. Furthermore, as \CFA is yet to be released, supporting version of Linux older that the latest version is not a goal of this work. +This work exclusively concentrates on Linux as it's operating system since the existing \CFA runtime and compiler does not already support other operating systems. Furthermore, as \CFA is yet to be released, supporting version of Linux older than the latest version is not a goal of this work. Index: doc/theses/thierry_delisle_PhD/thesis/text/io.tex =================================================================== --- doc/theses/thierry_delisle_PhD/thesis/text/io.tex (revision d95969a75f9a2102e5866017daba99c54b1bb621) +++ doc/theses/thierry_delisle_PhD/thesis/text/io.tex (revision c292244d4c299c228d7e40cd8a91071f5066c92f) @@ -2,12 +2,27 @@ As mentionned in Section~\ref{prev:io}, User-Level \glsxtrshort{io} requires multiplexing the \glsxtrshort{io} operations of many \glspl{thrd} onto fewer \glspl{proc} using asynchronous \glsxtrshort{io} operations. Various operating systems offer various forms of asynchronous operations and as mentioned in Chapter~\ref{intro}, this work is exclusively focuesd on Linux. -\section{Existing options} -Since \glsxtrshort{io} operations are generally handled by the +\section{Kernel Interface} +Since this work fundamentally depends on operating system support, the first step of any design is to discuss the available interfaces and pick one (or more) as the foundations of the \glsxtrshort{io} subsystem. -\subsection{\lstinline|epoll|, \lstinline|poll| and \lstinline|select|} +\subsection{\lstinline|O_NONBLOCK|} +In Linux, files can be opened with the flag @O_NONBLOCK@~\cite{MAN:open} (or @SO_NONBLOCK@~\cite{MAN:accept}, the equivalent for sockets) to use the file descriptors in ``nonblocking mode''. In this mode, ``Neither the open() nor any subsequent I/O operations on the [opened file descriptor] will cause the calling +process to wait.'' This feature can be used as the foundation for the \glsxtrshort{io} subsystem. However, for the subsystem to be able to block \glspl{thrd} until an operation completes, @O_NONBLOCK@ must be use in conjunction with a system call that monitors when a file descriptor becomes ready, \ie, the next \glsxtrshort{io} operation on it will not cause the process to wait\footnote{In this context, ready means to \emph{some} operation can be performed without blocking. It does not mean that the last operation that return \lstinline|EAGAIN| will succeed on the next try. A file that is ready to read but has only 1 byte available would be an example of this distinction.}. -\subsection{Linux's AIO} +There are three options to monitor file descriptors in Linux\footnote{For simplicity, this section omits to mention \lstinline|pselect| and \lstinline|ppoll|. The difference between these system calls and \lstinline|select| and \lstinline|poll| respectively is not relevant for this discussion.}, @select@~\cite{MAN:select}, @poll@~\cite{MAN:poll} and @epoll@~\cite{MAN:epoll}. All three of these options offer a system call that blocks a \gls{kthrd} until at least one of many file descriptor becomes ready. The group of file descriptors being waited on is often referred to as the \newterm{interest set}. +\paragraph{\lstinline|select|} is the oldest of these options, it takes as an input a contiguous array of bits, where each bits represent a file descriptor of interest. On return, it modifies the set in place to identify which of the file descriptors changed status. This means that calling select in a loop requires re-initializing the array each time and the number of file descriptors supported has a hard limit. Another limit of @select@ is that once the call is started, the interest set can no longer be modified. Monitoring a new file descriptor generally requires aborting any in progress call to @select@\footnote{Starting a new call to \lstinline|select| in this case is possible but requires a distinct kernel thread, and as a result is not a acceptable multiplexing solution when the interest set is large and highly dynamic unless the number of parallel calls to select can be strictly bounded.}. +\paragraph{\lstinline|poll|} is an improvement over select, which removes the hard limit on the number of file descriptors and the need to re-initialize the input on every call. It works using an array of structures as an input rather than an array of bits, thus allowing a more compact input for small interest sets. Like @select@, @poll@ suffers from the limitation that the interest set cannot be changed while the call is blocked. + +\paragraph{\lstinline|epoll|} further improves on these two functions, by allowing the interest set to be dynamically added to and removed from while a \gls{kthrd} is blocked on a call to @epoll@. This is done by creating an \emph{epoll instance} with a persistent intereset set and that is used across multiple calls. This advantage significantly reduces synchronization overhead on the part of the caller (in this case the \glsxtrshort{io} subsystem) since the interest set can be modified when adding or removing file descriptors without having to synchronize with other \glspl{kthrd} potentially calling @epoll@. + +However, all three of these system calls suffer from generality problems to some extent. The man page for @O_NONBLOCK@ mentions that ``[@O_NONBLOCK@] has no effect for regular files and block devices'', which means none of these three system calls are viable multiplexing strategies for these types of \glsxtrshort{io} operations. Furthermore, @epoll@ has been shown to have some problems with pipes and ttys\cit{Peter's examples in some fashion}. Finally, none of these are useful solutions for multiplexing \glsxtrshort{io} operations that do not have a corresponding file descriptor and can be awkward for operations using multiple file descriptors. + +\subsection{The POSIX asynchronous I/O (AIO)} +An alternative to using @O_NONBLOCK@ is to use the AIO interface. Its interface lets programmers enqueue operations to be performed asynchronously by the kernel. Completions of these operations can be communicated in various ways, either by sending a Linux signal, spawning a new \gls{kthrd} or by polling for completion of one or more operation. For the purpose multiplexing operations, spawning a new \gls{kthrd} is counter-productive but a related solution is discussed in Section~\ref{io:morethreads}. Since using interrupts handlers can also lead to fairly complicated interactions between subsystems, I will concentrate on the different polling methods. AIO only supports read and write operations to file descriptors and those do not have the same limitation as @O_NONBLOCK@, \ie, the file descriptors can be regular files and blocked devices. It also supports batching more than one of these operations in a single system call. + +AIO offers two different approach to polling. @aio_error@ can be used as a spinning form of polling, returning @EINPROGRESS@ until the operation is completed, and @aio_suspend@ can be used similarly to @select@, @poll@ or @epoll@, to wait until one or more requests have completed. For the purpose of \glsxtrshort{io} multiplexing, @aio_suspend@ is the intended interface. Even if AIO requests can be submitted concurrently, @aio_suspend@ suffers from the same limitation as @select@ and @poll@, \ie, the interest set cannot be dynamically changed while a call to @aio_suspend@ is in progress. Unlike @select@ and @poll@ however, it also suffers from the limitation that it does not specify which requests have completed, meaning programmers then have to poll each request in the interest set using @aio_error@ to identify which requests have completed. This means that, like @select@ and @poll@ but not @epoll@, the time needed to examine polling results increases based in the total number of requests monitored, not the number of completed requests. + +AIO does not seem to be a particularly popular interface, which I believe is in part due to this less than ideal polling interface. Linus Torvalds talks about this interface as follows : \begin{displayquote} @@ -30,17 +45,60 @@ in ``some kind of arbitrary \textit{queue up asynchronous system call} model''. -This description is actually quite close to the interface of the interface described in the next section. +This description is actually quite close to the interface described in the next section. -\subsection{\texttt{io\_uring}} -A very recent addition to Linux, @io_uring@\cit{io\_uring} is a framework that aims to solve many of the problems listed with the above mentioned solutions. +\subsection{\lstinline|io_uring|} +A very recent addition to Linux, @io_uring@\cite{MAN:io_uring} is a framework that aims to solve many of the problems listed with the above mentioned interfaces. Like AIO, it represents \glsxtrshort{io} operations as entries added on a queue. But like @epoll@, new requests can be submitted while a blocking call waiting for requests to complete is already in progress. The @io_uring@ interface uses two ring buffers (referred to simply as rings) as its core, a submit ring to which programmers push \glsxtrshort{io} requests and a completion buffer which programmers poll for completion. + +One of the big advantages over the interfaces listed above is that it also supports a much wider range of operations. In addition to supporting reads and writes to any file descriptor like AIO, it supports other operations like @open@, @close@, @fsync@, @accept@, @connect@, @send@, @recv@, @splice@, \etc. + +On top of these, @io_uring@ adds many ``bells and whistles'' like avoiding copies between the kernel and user-space with shared memory, allowing different mechanisms to communicate with device drivers and supporting chains of requests, \ie, requests that automatically trigger followup requests on completion. \subsection{Extra Kernel Threads}\label{io:morethreads} -Finally, if the operating system does not offer any satisfying forms of asynchronous \glsxtrshort{io} operations, a solution is to fake it by creating a pool of \glspl{kthrd} and delegating operations to them in order to avoid blocking \glspl{proc}. +Finally, if the operating system does not offer any satisfying forms of asynchronous \glsxtrshort{io} operations, a solution is to fake it by creating a pool of \glspl{kthrd} and delegating operations to them in order to avoid blocking \glspl{proc}. The is a compromise on multiplexing. In the worst case, where all \glspl{thrd} are consistently blocking on \glsxtrshort{io}, it devolves into 1-to-1 threading. However, regardless of the frequency of \glsxtrshort{io} operations, it achieves the fundamental goal of not blocking \glspl{proc} when \glspl{thrd} are ready to run. This approach is used by languages like Go\cit{Go} and frameworks like libuv\cit{libuv}, since it has the advantage that it can easily be used across multiple operating systems. This advantage is especially relevant for languages like Go, which offer an homogenous \glsxtrshort{api} across all platforms. As opposed to C, which has a very limited standard api for \glsxtrshort{io}, \eg, the C standard library has no networking. \subsection{Discussion} +These options effectively fall into two broad camps of solutions, waiting for \glsxtrshort{io} to be ready versus waiting for \glsxtrshort{io} to be completed. All operating systems that support asynchronous \glsxtrshort{io} must offer an interface along one of these lines, but the details can vary drastically. For example, Free BSD offers @kqueue@~\cite{MAN:bsd/kqueue} which behaves similarly to @epoll@ but with some small quality of life improvements, while Windows (Win32)~\cit{https://docs.microsoft.com/en-us/windows/win32/fileio/synchronous-and-asynchronous-i-o} offers ``overlapped I/O'' which handles submissions similarly to @O_NONBLOCK@, with extra flags on the synchronous system call, but waits for completion events, similarly to @io_uring@. +For this project, I have chosen to use @io_uring@, in large parts due to its generality. While @epoll@ has been shown to be a good solution to socket \glsxtrshort{io} (\cite{DBLP:journals/pomacs/KarstenB20}), @io_uring@'s transparent support for files, pipes and more complex operations, like @splice@ and @tee@, make it a better choice as the foundation for a general \glsxtrshort{io} subsystem. \section{Event-Engine} +The event engines reponsibility is to use the kernel interface to multiplex many \glsxtrshort{io} operations onto few \glspl{kthrd}. In concrete terms, this means that \glspl{thrd} enter the engine threw an interface, the event engines then starts the operation and parks the calling \glspl{thrd}, returning control to the \gls{proc}. The parked \glspl{thrd} are then rescheduled by the event engine once the desired operation has completed. + +\subsection{\lstinline|io_uring| in depth} +Before going into details on the design of the event engine, I will present some more details on the usage of @io_uring@ which are important for the design of the engine. + +\begin{figure} + \centering + \input{io_uring.pstex_t} + \caption[Overview of \lstinline|io_uring|]{Overview of \lstinline|io_uring| \smallskip\newline Two ring buffer are used to communicate with the kernel, one for completions~(right) and one for submissions~(left). The completion ring contains entries, \newterm{CQE}s: Completion Queue Entries, that are produced by the kernel when an operation completes and then consumed by the application. On the other hand, the application produces \newterm{SQE}s: Submit Queue Entries, which it appends to the submission ring for the kernel to consume. Unlike the completion ring, the submission ring does not contain the entries directly, it indexes into the SQE array (denoted \emph{S}) instead.} + \label{fig:iouring} +\end{figure} + +Figure~\ref{fig:iouring} shows an overview of @io_uring@. New \glsxtrshort{io} operations are submitted to the kernel following 4 steps which use the components shown in the figure. + +\paragraph{First} an @sqe@ must be allocated from the pre-allocated array (denoted \emph{S} in Figure~\ref{fig:iouring}). This array is create at the same time as the rings, is in kernel-locked memory, which means it is both visible by the kernel and the application, and has a fixed size determined at creation. How these entries are allocated is not important for the functionning of @io_uring@, the only requirement is that no entry is reused before the kernel has consumed it. + +\paragraph{Secondly} the @sqe@ must be filled according to the desired operation. This step is straight forward, the only detail worth mentionning is that @sqe@s have a @user_data@ field that must be filled in order to match submission and completion entries. + +\paragraph{Thirdly} the @sqe@ must be submitted to the submission ring, this requires appending the index of the @sqe@ to the ring following regular ring buffer steps: \lstinline|{ buffer[head] = item; head++ }|. Since the head is visible to the kernel, some memory barriers may be required to prevent the compiler to reorder these operations. Since the submission ring is a regular ring buffer, more than one @sqe@ can be added at once and the head can be updated only after the entire batch has been updated. + +\paragraph{Finally} the kernel must be notified of the change to the ring using the system call @io_uring_enter@. The number of elements appended to the submission ring is passed as a parameter and the number of elements consumed is returned. The @io_uring@ instance can be constructed so that this step is not required, but this requires elevated privilege and early version of @io_uring@ had additionnal restrictions. + +The submission side is simpler, applications call @io_uring_enter@ with the flag @IORING_ENTER_GETEVENTS@ to wait on a desired number of operations to complete. The same call can be used to both submit @sqe@s and wait for operations to complete. When operations do complete the kernel appends a @cqe@ to the completion ring and advances the head of the ring. Each @cqe@ contains the result of the operation as well as a copy of the @user_data@ field of the @sqe@ that triggered the operation. It is not necessary to call @io_uring_enter@ to get new events, the kernel can directly modify the completion ring, the system call is only needed if the application wants to block waiting on operations to complete. + +The @io_uring_enter@ system call is protected by a lock inside the kernel. This means that concurrent call to @io_uring_enter@ using the same instance are possible, but there is can be no performance gained from parallel calls to @io_uring_enter@. It is possible to do the first three submission steps in parallel, however, doing so requires careful synchronization. \section{Interface} +Finally, the last important part of the I/O subsystem is it's interface. There are multiple approaches that can be offered to programmers, each with advantages and disadvantages. The new \glsxtrshort{io} subsystem can replace the C runtime's API or extend it. And in the later case the interface can go from very similar to vastly different. The following sections discuss some useful options using @read@ as an example. The standard Linux interface for C is : + +@ssize_t read(int fd, void *buf, size_t count);@. + +\subsection{Replacement} +Replacing the C \glsxtrshort{api} + +\subsection{Synchronous Extension} + +\subsection{Asynchronous Extension} + +\subsection{Interface directly to \lstinline|io_uring|} Index: doc/theses/thierry_delisle_PhD/thesis/thesis.tex =================================================================== --- doc/theses/thierry_delisle_PhD/thesis/thesis.tex (revision d95969a75f9a2102e5866017daba99c54b1bb621) +++ doc/theses/thierry_delisle_PhD/thesis/thesis.tex (revision c292244d4c299c228d7e40cd8a91071f5066c92f) @@ -242,6 +242,6 @@ \part{Design} \input{text/core.tex} +\input{text/io.tex} \input{text/practice.tex} -\input{text/io.tex} \part{Evaluation} \label{Evaluation}