source: doc/theses/thierry_delisle_PhD/thesis/text/eval_macro.tex @ 94ce03a

\chapter{Macro-Benchmarks}\label{macrobench}
The previous chapter has demonstrated that the scheduler achieves its performance goal in small and controlled scenario.
The next step is then to demonstrate that this stays true in more realistic and complete scenarios.
This chapter presents two flavours of webservers that demonstrate that \CFA performs competitively with production environments.

Webservers where chosen because they offer fairly simple applications that are still useful as standalone products.
Furthermore, webservers are generally amenable to parallelisation since their workloads are mostly homogenous.
They therefore offer a stringent performance benchmark for \CFA.
Indeed existing solutions are likely to have close to optimal performance while the homogeneity of the workloads mean the additional fairness is not needed.
This means that there is very little room to use for the extra cost of fairness.
As such, these experiements should highlight the fairness cost in realistic scenarios.

\section{Memcached}
Memcached~\cite{memcached} is an in memory key-value store that is used in many production environments, \eg \cite{atikoglu2012workload}.
This also server also has the notable added benefit that there exists a full-featured front-end for performance testing called @mutilate@~\cite{GITHUB:mutilate}.
Experimenting on memcached allows for a simple test of the \CFA runtime as a whole, it will exercise the scheduler, the idle-sleep mechanism, as well the \io subsystem for sockets.
Note that this experiment does not exercise the \io subsytem with regards to disk operations.

\subsection{Benchmark Environment}
These experiments are run on a cluster of homogenous Supermicro SYS-6017R-TDF compute nodes with the following characteristics:
The server runs Ubuntu 20.04.3 LTS on top of Linux Kernel 5.11.0-34.
Each node has 2 Intel(R) Xeon(R) CPU E5-2620 v2 running at 2.10GHz.
These CPUs have 6 cores per CPUs and 2 \glspl{hthrd} per core, for a total of 24 \glspl{hthrd}.
The cpus each have 384 KB, 3 MB and 30 MB of L1, L2 and L3 caches respectively.
Each node is connected to the network through a Mellanox 10 Gigabit Ethernet port.
The network route uses 1 Mellanox SX1012 10/40 Gigabit Ethernet cluster switch.

\subsection{Memcached with threads per connection}
Comparing against memcached using a user-level runtime only really make sense if the server actually uses this threading model.
Indeed, evaluating a user-level runtime with 1 \at per \proc is not meaningful since it does not exercise the runtime, it simply adds some overhead to the underlying OS scheduler.

One approach is to use a webserver that uses a thread-per-connection model, where each incoming connection is served by a single \at in a strict 1-to-1 pairing.
This models adds flexibility to the implementation, as the serving logic can now block on user-level primitives without affecting other connections.

Memcached is not built according to a thread-per-connection model, but there exists a port of it that is, which was built for libfibre in \cite{DBLP:journals/pomacs/KarstenB20}.
Therefore this version can both be compared to the original version and to a port to the \CFA runtime.

As such, this memcached experiment compares 3 different varitions of memcached:
\begin{itemize}
 \item \emph{vanilla}: the official release of memcached, version~1.6.9.
 \item \emph{fibre}: a modification of vanilla which uses the thread per connection model on top of the libfibre runtime~\cite{DBLP:journals/pomacs/KarstenB20}.
 \item \emph{cfa}: a modification of the fibre webserver that replaces the libfibre runtime with \CFA.
\end{itemize}

\subsection{Throughput} \label{memcd:tput}
\begin{figure}
        \centering
        \input{result.memcd.rate.qps.pstex_t}
        \caption[Memcached Benchmark: Throughput]{Memcached Benchmark: Throughput\smallskip\newline Desired vs Actual query rate for 15360 connections. Target QPS is the query rate that the clients are attempting to maintain and Actual QPS is the rate at which the server is able to respond.}
        \label{fig:memcd:rate:qps}
\end{figure}
This experiment is done by having the clients establish 15360 total connections, which persist for the duration of the experiments.
The clients then send queries, attempting to follow a desired query rate and the server responds to the desired rate as best they can.
Figure~\ref{fig:memcd:rate:qps} shows the difference between desired rate, ``Target \underline{Q}ueries \underline{P}er \underline{S}econd'', and the actual rate, ``Actual QPS'', for all three webservers.
As with the experiments in the previous chapter, 15 runs for each rate were measured and the graph shows all datapoints.
The solid line represents the median while the dashed and dotted lines represent the maximum and minimum respectively.
For rates below 500K queries per seconds, all three webservers can easily keep up to the desired rate, resulting in all datapoints being perfectly overlapped.
Beyond this limit, individual runs become visible and all three servers begin to distinguish themselves, where vanilla memcached generally achieves better throughput while \CFA and libfibre fight for second place.
Overall however the performance of all three servers is very similar, especially considering that at 500K the server has reached saturation, which is discussed more in the next section.

\subsection{Tail Latency}
\begin{figure}
        \centering
        \input{result.memcd.rate.99th.pstex_t}
        \caption[Memcached Benchmark : 99th Percentile Lantency]{Memcached Benchmark : 99th Percentile Lantency\smallskip\newline 99th Percentile of the response latency as a function of \emph{desired} query rate for 15360 connections. }
        \label{fig:memcd:rate:tail}
\end{figure}
Another important performance metric to look at is \newterm{tail} latency.
Since many web applications rely on a combination of different queries made in parallel, the latency of the slowest response, \ie tail latency, can dictate overall performance.
Figure~\ref{fig:memcd:rate:tail} shows the 99th percentile latency results for the same experiment memcached experiment.
Again, each series is made of 15 runs with the median, maximum and minimum highlighted with lines.
As is expected, the latency starts low and increases as the server gets close to saturation, point at which the latency increses dramatically.
Because of this dramatic increase, the Y axis is presented using log scale.
Note that the figure shows \emph{target} query rate, the actual response rate is given in Figure~\ref{fig:memcd:rate:qps} as this is the same underlying experiment.

For all three servers the saturation point is reached before 500K queries per second, which was when throughput started to change among the webservers.
In this experiement, all three webservers are much more distinguishable than the throughput experiment.
Vanilla achieves low latency mostly across the board followed by libfibre and \CFA.
However, all three webservers achieve micro second latencies and the increases in latency mostly follow eachother.

\subsection{Update rate}
Since Memcached is effectively a simple database, an aspect that can significantly affect performance is wirtes.
The information that is cached by memcached can be written to concurrently with other queries.
I could therefore be interesting to see how this update rate affects performance.
\begin{figure}
        \subfloat[][\CFA: Throughput]{
                \resizebox{0.5\linewidth}{!}{
                        \input{result.memcd.forall.qps.pstex_t}
                }
                \label{fig:memcd:updt:forall:qps}
        }
        \subfloat[][\CFA: Latency]{
                \resizebox{0.5\linewidth}{!}{
                        \input{result.memcd.forall.lat.pstex_t}
                }
                \label{fig:memcd:updt:forall:lat}
        }

        \subfloat[][LibFibre: Throughput]{
                \resizebox{0.5\linewidth}{!}{
                        \input{result.memcd.fibre.qps.pstex_t}
                }
                \label{fig:memcd:updt:fibre:qps}
        }
        \subfloat[][LibFibre: Latency]{
                \resizebox{0.5\linewidth}{!}{
                        \input{result.memcd.fibre.lat.pstex_t}
                }
                \label{fig:memcd:updt:fibre:lat}
        }

        \subfloat[][Vanilla: Throughput]{
                \resizebox{0.5\linewidth}{!}{
                        \input{result.memcd.vanilla.qps.pstex_t}
                }
                \label{fig:memcd:updt:vanilla:qps}
        }
        \subfloat[][Vanilla: Latency]{
                \resizebox{0.5\linewidth}{!}{
                        \input{result.memcd.vanilla.lat.pstex_t}
                }
                \label{fig:memcd:updt:vanilla:lat}
        }
        \caption[Throughput and Latency results at different update rates (percentage of writes).]{Throughput and Latency results at different update rates (percentage of writes).\smallskip\newline Description}
        \label{fig:memcd:updt}
\end{figure}
Figure~\ref{fig:memcd:updt} shows the results for the same experiement as the throughput and latency experiement but with multiple update rate.
Each experiment was repeated with a update percentage of 3\%, 5\%, 10\% and 50\%.
The previous experiements were run with 3\% update rates.
In the end, this experiment mostly demonstrates that the performance of memcached is affected very little by the update rate.
I believe this is because the underlying locking pattern is actually fairly similar.
Indeed, since values can be much bigger than what the server can read atomically, a lock must be acquired while the value is read.
These results suggects that memcached does not use a readers-writer lock to protect each values and instead relies on having a sufficient number of keys to limit the contention.
In the end, this shows yet again that \CFA achieves equivalent performance.


\section{Static Web-Server}
The memcached experiment has two aspects of the \io subsystem it does not exercise, accepting new connections and interacting with disks.
On the other hand, static webservers, servers that offer static webpages, do stress disk \io since they serve files from disk\footnote{Dynamic webservers, which construct pages as they are sent, are not as interesting since the construction of the pages do not exercise the runtime in a meaningfully different way.}.
The static webserver experiments will compare NGINX~\cit{nginx} with a custom webserver developped for this experiment.

\subsection{\CFA webserver}
Unlike the memcached experiment, the webserver experiment relies on a custom designed webserver.
It is a simple thread-per-connection webserver where a fixed number of \ats are created upfront.
Each of the \at calls @accept@, through @io_uring@, on the listening port and handle the incomming connection once accepted.
Most of the implementation is fairly straight forward however the inclusion of file \io introduces a new challenge that had to be hacked around.

Normally, webservers use @sendfile@\cite{MAN:sendfile} to send files over the socket.
@io_uring@ does not support @sendfile@, it supports @splice@\cite{MAN:splice} instead, which is strictly more powerful.
However, because of how linux implements file \io, see Subsection~\ref{ononblock}, @io_uring@'s implementation must delegate calls to splice to worker threads inside the kernel.
As of Linux 5.13, @io_uring@ caps the numer of these worker threads to @RLIMIT_NPROC@ and therefore, when tens of thousands of splice requests are made, it can create tens of thousands of \glspl{kthrd}.
Such a high number of \glspl{kthrd} is more than Linux can handle in this scenario so performance suffers significantly.
For this reason, the \CFA webserver calls @sendfile@ directly.
This approach works up to a certain point, but once the server approaches saturation, it leads to a new problem.

When the saturation point of the server is attained, latency will increase and inevitably some client connections will timeout.
As these clients close there connections, the server must close these sockets without delay so the OS can reclaim the resources used by these connections.
Indeed, until they are closed on the server end, the connection will linger in the CLOSE-WAIT tcp state~\cite{rfc:tcp} and the tcp buffers will be preserved.
However, this poses a problem using blocking @sendfile@ calls.
The calls can block if they do not have suffcient memory, which can be caused by having too many connections in the CLOSE-WAIT state.
Since blocking in calls to @sendfile@ blocks the \proc rather than the \at, this prevents other connections from closing their sockets.
This leads to a vicious cycle where timeouts lead to @sendfile@ calls running out of resources, which lead to more timeouts.

Normally, this is address by marking the sockets as non-blocking and using @epoll@ to wait for sockets to have sufficient resources.
However, since @io_uring@ respects non-blocking semantics marking all sockets as non-blocking effectively circumvents the @io_uring@ subsystem entirely.
For this reason, the \CFA webserver sets and resets the @O_NONBLOCK@ flag before and after any calls to @sendfile@.
Normally @epoll@ would also be used when these calls to @sendfile@ return @EAGAIN@, but since this would not help in the evaluation of the \CFA runtime, the \CFA webserver simply yields and retries in these cases.

It is important to state that in Linux 5.15 @io_uring@ introduces the ability for users to limit the number of worker threads that are created, through the @IORING_REGISTER_IOWQ_MAX_WORKERS@ option.
However, as of writing this document Ubuntu does not have a stable release of Linux 5.15.
There exists versions of the kernel that are currently under testing, but these caused unrelated but nevertheless prohibitive issues in this experiment.
Presumably, the new kernel would remove the need for the hack described above, as it would allow connections in the CLOSE-WAIT state to be closed even while the calls to @splice@/@sendfile@ are underway.
However, since this could not be tested, this is purely a conjecture at this point.

\subsection{Benchmark Environment}
Unlike the memcached experiment, the webserver run on a more heterogenous environment.
The server runs Ubuntu 20.04.4 LTS on top of Linux Kernel 5.13.0-52.
It has an AMD Opteron(tm) Processor 6380 running at 2.50GHz.
These CPUs has only 8 \glspl{hthrd} enabled by grub, which is sufficient to achieve line rate.
This cpus each have 64 KB, 256 KiB and 8 MB of L1, L2 and L3 caches respectively.
The kernel is setup to limit the memory at 25Gb.

The client machines each have two 2.8 GHz Xeon CPUs, and four one-gigabit Ethernet cards.
Each client machine runs two copies of the workload generator.
They run a 2.6.11-1 SMP Linux kernel, which permits each client load-generator to run on a separate CPU.
Since the clients outnumber the server 8-to-1, this is plenty sufficient to generate enough load for the clients not to become the bottleneck.

\todo{switch}

\subsection{Throughput}
To measure the throughput of both webservers, each server is loaded with over 30,000 files making over 4.5 Gigabytes in total.
Each client runs httperf~\cit{httperf} which establishes a connection, does an http request for one or more files, closes the connection and repeats the process.
The connections and requests are made according to a Zipfian distribution~\cite{zipf}.
Throughput is measured by aggregating the results from httperf of all the clients.
\begin{figure}
        \subfloat[][Throughput]{
                \input{result.swbsrv.25gb.pstex_t}
                \label{fig:swbsrv:ops}
        }

        \subfloat[][Rate of Errors]{
                \input{result.swbsrv.25gb.err.pstex_t}
                \label{fig:swbsrv:err}
        }
        \caption[Static Webserver Benchmark : Throughput]{Static Webserver Benchmark : Throughput\smallskip\newline Throughput vs request rate for short lived connections connections.}
        \label{fig:swbsrv}
\end{figure}
Figure~\ref{fig:swbsrv} shows the results comparing \CFA to NGINX in terms of throughput.
These results are fairly straight forward.
Both servers achieve the same throughput until around 57,500 requests per seconds.
Since the clients are asking for the same files, the fact that the throughput matches exactly is expected as long as both servers are able to serve the desired rate.
Once the saturation point is reached, both servers are still very close.
NGINX achieves slightly better throughtput.
However, Figure~\ref{fig:swbsrv:err} shows the rate of errors, a gross approximation of tail latency, where \CFA achives notably fewet errors once the machine reaches saturation.
This suggest that \CFA is slightly more fair and NGINX may sloghtly sacrifice some fairness for improved throughtput.
It demonstrate that the \CFA webserver described above is able to match the performance of NGINX up-to and beyond the saturation point of the machine.

\subsection{Disk Operations}
The throughput was made using a server with 25gb of memory, this was sufficient to hold the entire fileset in addition to all the code and data needed to run the webserver and the reste of the machine.
Previous work like \cit{Cite Ashif's stuff} demonstrate that an interesting follow-up experiment is to rerun the same throughput experiment but allowing significantly less memory on the machine.
If the machine is constrained enough, it will force the OS to evict files from the file cache and cause calls to @sendfile@ to have to read from disk.
However, what these low memory experiments demonstrate is how the memory footprint of the webserver affects the performance.
However, since what I am to evaluate in this thesis is the runtime of \CFA, I diceded to forgo experiments on low memory server.
The implementation of the webserver itself is simply too impactful to be an interesting evaluation of the underlying runtime.