Index: doc/theses/thierry_delisle_PhD/thesis/text/eval_macro.tex =================================================================== --- doc/theses/thierry_delisle_PhD/thesis/text/eval_macro.tex (revision 1b57a38e2e1c892c9a8ff00a1d51fe1750874daf) +++ doc/theses/thierry_delisle_PhD/thesis/text/eval_macro.tex (revision 94ce03afdd53ca64ae840fa7938fcbbdea9e67ef) @@ -8,4 +8,6 @@ They therefore offer a stringent performance benchmark for \CFA. Indeed existing solutions are likely to have close to optimal performance while the homogeneity of the workloads mean the additional fairness is not needed. +This means that there is very little room to use for the extra cost of fairness. +As such, these experiements should highlight the fairness cost in realistic scenarios. \section{Memcached} @@ -13,5 +15,5 @@ This also server also has the notable added benefit that there exists a full-featured front-end for performance testing called @mutilate@~\cite{GITHUB:mutilate}. Experimenting on memcached allows for a simple test of the \CFA runtime as a whole, it will exercise the scheduler, the idle-sleep mechanism, as well the \io subsystem for sockets. -This experiment does not exercise the \io subsytem with regards to disk operations. +Note that this experiment does not exercise the \io subsytem with regards to disk operations. \subsection{Benchmark Environment} @@ -45,14 +47,15 @@ \centering \input{result.memcd.rate.qps.pstex_t} - \caption[Memcached Benchmark: Throughput]{Memcached Benchmark: Throughput\smallskip\newline Desired vs Actual request rate for 15360 connections. Target QPS is the request rate that the clients are attempting to maintain and Actual QPS is the rate at which the server is able to respond.} + \caption[Memcached Benchmark: Throughput]{Memcached Benchmark: Throughput\smallskip\newline Desired vs Actual query rate for 15360 connections. Target QPS is the query rate that the clients are attempting to maintain and Actual QPS is the rate at which the server is able to respond.} \label{fig:memcd:rate:qps} \end{figure} -Figure~\ref{fig:memcd:rate:qps} shows the result for the throughput of all three webservers. This experiment is done by having the clients establish 15360 total connections, which persist for the duration of the experiments. -The clients then send requests, attempting to follow a desired request rate. -The servers respond to the desired rate as best they can and the difference between desired rate, ``Target \underline{Q}ueries \underline{P}er \underline{S}econd'', and the actual rate, ``Actual QPS''. -The results show that \CFA achieves equivalent throughput even when the server starts to reach saturation. -Only then does it start to fall behind slightly. -This is a demonstration of the \CFA runtime achieving its performance goal. +The clients then send queries, attempting to follow a desired query rate and the server responds to the desired rate as best they can. +Figure~\ref{fig:memcd:rate:qps} shows the difference between desired rate, ``Target \underline{Q}ueries \underline{P}er \underline{S}econd'', and the actual rate, ``Actual QPS'', for all three webservers. +As with the experiments in the previous chapter, 15 runs for each rate were measured and the graph shows all datapoints. +The solid line represents the median while the dashed and dotted lines represent the maximum and minimum respectively. +For rates below 500K queries per seconds, all three webservers can easily keep up to the desired rate, resulting in all datapoints being perfectly overlapped. +Beyond this limit, individual runs become visible and all three servers begin to distinguish themselves, where vanilla memcached generally achieves better throughput while \CFA and libfibre fight for second place. +Overall however the performance of all three servers is very similar, especially considering that at 500K the server has reached saturation, which is discussed more in the next section. \subsection{Tail Latency} @@ -60,53 +63,74 @@ \centering \input{result.memcd.rate.99th.pstex_t} - \caption[Memcached Benchmark : 99th Percentile Lantency]{Memcached Benchmark : 99th Percentile Lantency\smallskip\newline 99th Percentile of the response latency as a function of \emph{desired} request rate for 15360 connections. } + \caption[Memcached Benchmark : 99th Percentile Lantency]{Memcached Benchmark : 99th Percentile Lantency\smallskip\newline 99th Percentile of the response latency as a function of \emph{desired} query rate for 15360 connections. } \label{fig:memcd:rate:tail} \end{figure} Another important performance metric to look at is \newterm{tail} latency. -Since many web applications rely on a combination of different requests made in parallel, the latency of the slowest response, \ie tail latency, can dictate overall performance. +Since many web applications rely on a combination of different queries made in parallel, the latency of the slowest response, \ie tail latency, can dictate overall performance. Figure~\ref{fig:memcd:rate:tail} shows the 99th percentile latency results for the same experiment memcached experiment. +Again, each series is made of 15 runs with the median, maximum and minimum highlighted with lines. As is expected, the latency starts low and increases as the server gets close to saturation, point at which the latency increses dramatically. -Note that the figure shows \emph{target} request rate, the actual response rate is given in Figure~\ref{fig:memcd:rate:qps} as this is the same underlying experiment. +Because of this dramatic increase, the Y axis is presented using log scale. +Note that the figure shows \emph{target} query rate, the actual response rate is given in Figure~\ref{fig:memcd:rate:qps} as this is the same underlying experiment. + +For all three servers the saturation point is reached before 500K queries per second, which was when throughput started to change among the webservers. +In this experiement, all three webservers are much more distinguishable than the throughput experiment. +Vanilla achieves low latency mostly across the board followed by libfibre and \CFA. +However, all three webservers achieve micro second latencies and the increases in latency mostly follow eachother. \subsection{Update rate} -\begin{figure} - \centering - \subfloat[][Throughput]{ - \input{result.memcd.forall.qps.pstex_t} - } - - \subfloat[][Latency]{ - \input{result.memcd.forall.lat.pstex_t} - } - \caption[forall Latency results at different update rates]{forall Latency results at different update rates\smallskip\newline Description} - \label{fig:memcd:updt:forall} -\end{figure} - -\begin{figure} - \centering - \subfloat[][Throughput]{ - \input{result.memcd.fibre.qps.pstex_t} - } - - \subfloat[][Latency]{ - \input{result.memcd.fibre.lat.pstex_t} - } - \caption[fibre Latency results at different update rates]{fibre Latency results at different update rates\smallskip\newline Description} - \label{fig:memcd:updt:fibre} -\end{figure} - -\begin{figure} - \centering - \subfloat[][Throughput]{ - \input{result.memcd.vanilla.qps.pstex_t} - } - - \subfloat[][Latency]{ - \input{result.memcd.vanilla.lat.pstex_t} - } - \caption[vanilla Latency results at different update rates]{vanilla Latency results at different update rates\smallskip\newline Description} - \label{fig:memcd:updt:vanilla} -\end{figure} - +Since Memcached is effectively a simple database, an aspect that can significantly affect performance is wirtes. +The information that is cached by memcached can be written to concurrently with other queries. +I could therefore be interesting to see how this update rate affects performance. +\begin{figure} + \subfloat[][\CFA: Throughput]{ + \resizebox{0.5\linewidth}{!}{ + \input{result.memcd.forall.qps.pstex_t} + } + \label{fig:memcd:updt:forall:qps} + } + \subfloat[][\CFA: Latency]{ + \resizebox{0.5\linewidth}{!}{ + \input{result.memcd.forall.lat.pstex_t} + } + \label{fig:memcd:updt:forall:lat} + } + + \subfloat[][LibFibre: Throughput]{ + \resizebox{0.5\linewidth}{!}{ + \input{result.memcd.fibre.qps.pstex_t} + } + \label{fig:memcd:updt:fibre:qps} + } + \subfloat[][LibFibre: Latency]{ + \resizebox{0.5\linewidth}{!}{ + \input{result.memcd.fibre.lat.pstex_t} + } + \label{fig:memcd:updt:fibre:lat} + } + + \subfloat[][Vanilla: Throughput]{ + \resizebox{0.5\linewidth}{!}{ + \input{result.memcd.vanilla.qps.pstex_t} + } + \label{fig:memcd:updt:vanilla:qps} + } + \subfloat[][Vanilla: Latency]{ + \resizebox{0.5\linewidth}{!}{ + \input{result.memcd.vanilla.lat.pstex_t} + } + \label{fig:memcd:updt:vanilla:lat} + } + \caption[Throughput and Latency results at different update rates (percentage of writes).]{Throughput and Latency results at different update rates (percentage of writes).\smallskip\newline Description} + \label{fig:memcd:updt} +\end{figure} +Figure~\ref{fig:memcd:updt} shows the results for the same experiement as the throughput and latency experiement but with multiple update rate. +Each experiment was repeated with a update percentage of 3\%, 5\%, 10\% and 50\%. +The previous experiements were run with 3\% update rates. +In the end, this experiment mostly demonstrates that the performance of memcached is affected very little by the update rate. +I believe this is because the underlying locking pattern is actually fairly similar. +Indeed, since values can be much bigger than what the server can read atomically, a lock must be acquired while the value is read. +These results suggects that memcached does not use a readers-writer lock to protect each values and instead relies on having a sufficient number of keys to limit the contention. +In the end, this shows yet again that \CFA achieves equivalent performance.