source: doc/theses/mubeen_zulfiqar_MMath/benchmarks.tex @ 00675a1

\chapter{Benchmarks}
\label{s:Benchmarks}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Micro Benchmark Suite
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

There are two basic approaches for evaluating computer software: benchmarks and micro-benchmarks.
\begin{description}
\item[Benchmarks]
are a suite of application programs (SPEC CPU/WEB) that are exercised in a common way (inputs) to find differences among underlying software implementations associated with an application (compiler, memory allocator, web server, \etc).
The applications are suppose to represent common execution patterns that need to perform well with respect to an underlying software implementation.
Benchmarks are often criticized for having overlapping patterns, insufficient patterns, or extraneous code that masks patterns.
\item[Micro-Benchmarks]
attempt to extract the common execution patterns associated with an application and run the pattern independently.
This approach removes any masking from extraneous application code, allows execution pattern to be very precise, and provides an opportunity for the execution pattern to have multiple independent tuning adjustments (knobs).
Micro-benchmarks are often criticized for inadequately representing real-world applications.
\end{description}

While some crucial software components have standard benchmarks, no standard benchmark exists for testing and comparing memory allocators.
In the past, an assortment of applications have been used for benchmarking allocators~\cite{Detlefs93,Berger00,Berger01,berger02reconsidering}: P2C, GS, Espresso/Espresso-2, CFRAC/CFRAC-2, GMake, GCC, Perl/Perl-2, Gawk/Gawk-2, XPDF/XPDF-2, ROBOOP, Lindsay.
As well, an assortment of micro-benchmark have been used for benchmarking allocators~\cite{larson99memory,Berger00,streamflow}: threadtest, shbench, Larson, consume, false sharing.
Many of these benchmark applications and micro-benchmarks are old and may not reflect current application allocation patterns.

This thesis designs and examines a new set of micro-benchmarks for memory allocators that test a variety of allocation patterns, each with multiple tuning parameters.
The aim of the micro-benchmark suite is to create a set of programs that can evaluate a memory allocator based on the key performance matrices such as speed, memory overhead, and cache performance.
% These programs can be taken as a standard to benchmark an allocator's basic goals.
These programs give details of an allocator's memory overhead and speed under certain allocation patterns.
The allocation patterns are configurable (adjustment knobs) to observe an allocator's performance across a spectrum of events for a desired allocation pattern, which is seldom possible with benchmark programs.
Each micro-benchmark program has multiple control knobs specified by command-line arguments.

The new micro-benchmark suite measures performance by allocating dynamic objects and measuring specific matrices.
An allocator's speed is benchmarked in different ways, as are issues like false sharing.


\section{Prior Multi-Threaded Micro-Benchmarks}

Modern memory allocators, such as llheap, must handle multi-threaded programs at the KT and UT level.
The following multi-threaded micro-benchmarks are presented to give a sense of prior work~\cite{Berger00} at the KT level.
None of the prior work address multi-threading at the UT level.


\subsection{threadtest}

This benchmark stresses the ability of the allocator to handle different threads allocating and deallocating independently.
There is no interaction among threads, \ie no object sharing.
Each thread repeatedly allocate 100,000 \emph{8-byte} objects then deallocates them in the order they were allocated.
Runtime of the benchmark evaluates its efficiency.


\subsection{shbench}

This benchmark is similar to threadtest but each thread randomly allocate and free a number of \emph{random-sized} objects.
It is a stress test that also uses runtime to determine efficiency of the allocator.


\subsection{Larson}

This benchmark simulates a server environment.
Multiple threads are created where each thread allocates and frees a number of random-sized objects within a size range.
Before the thread terminates, it passes its array of 10,000 objects to a new child thread to continue the process.
The number of thread generations varies depending on the thread speed.
It calculates memory operations per second as an indicator of memory allocator's performance.


\section{New Multi-Threaded Micro-Benchmarks}

The following new benchmarks were created to assess multi-threaded programs at the KT and UT level.
For generating random values, two generators are supported: uniform~\cite{uniformPRNG} and fisher~\cite{fisherPRNG}.


\subsection{Churn Benchmark}
\label{s:ChurnBenchmark}

The churn benchmark measures the runtime speed of an allocator in a multi-threaded scenerio, where each thread extensively allocates and frees dynamic memory.
Only @malloc@ and @free@ are used to eliminate any extra cost, such as @memcpy@ in @calloc@ or @realloc@.
Churn simulates a memory intensive program that can be tuned to create different scenarios.

\VRef[Figure]{fig:ChurnBenchFig} shows the pseudo code for the churn micro-benchmark.
This benchmark creates a buffer with M spots and an allocation in each spot, and then starts K threads.
Each thread picks a random spot in M, frees the object currently at that spot, and allocates a new object for that spot.
Each thread repeats this cycle N times.
The main thread measures the total time taken for the whole benchmark and that time is used to evaluate the memory allocator's performance.

\begin{figure}
\centering
\begin{lstlisting}
Main Thread
        create worker threads
        note time T1
        ...
        note time T2
        churn_speed = (T2 - T1)
Worker Thread
        initialize variables
        ...
        for ( N )
                R = random spot in array
                free R
                allocate new object at R
\end{lstlisting}
%\includegraphics[width=1\textwidth]{figures/bench-churn.eps}
\caption{Churn Benchmark}
\label{fig:ChurnBenchFig}
\end{figure}

The adjustment knobs for churn are:
\begin{description}[itemsep=0pt,parsep=0pt]
\item[thread:]
number of threads (K).
\item[spots:]
number of spots for churn (M).
\item[obj:]
number of objects per thread (N).
\item[max:]
maximum object size.
\item[min:]
minimum object size.
\item[step:]
object size increment.
\item[distro:]
object size distribution
\end{description}


\subsection{Cache Thrash}
\label{sec:benchThrashSec}

The cache-thrash micro-benchmark measures allocator-induced active false-sharing as illustrated in \VRef{s:AllocatorInducedActiveFalseSharing}.
If memory is allocated for multiple threads on the same cache line, this can significantly slow down program performance.
When threads share a cache line, frequent reads/writes to their cache-line object causes cache misses, which cause escalating delays as cache distance increases.

Cache thrash tries to create a scenerio that leads to false sharing, if the underlying memory allocator is allocating dynamic memory to multiple threads on the same cache lines.
Ideally, a memory allocator should distance the dynamic memory region of one thread from another.
Having multiple threads allocating small objects simultaneously can cause a memory allocator to allocate objects on the same cache line, if its not distancing the memory among different threads.

\VRef[Figure]{fig:benchThrashFig} shows the pseudo code for the cache-thrash micro-benchmark.
First, it creates K worker threads.
Each worker thread allocates an object and intensively reads/writes it for M times to possible invalidate cache lines that may interfere with other threads sharing the same cache line.
Each thread repeats this for N times.
The main thread measures the total time taken to for all worker threads to complete.
Worker threads sharing cache lines with each other will take longer.

\begin{figure}
\centering
\input{AllocInducedActiveFalseSharing}
\medskip
\begin{lstlisting}
Main Thread
        create worker threads
        ...
        signal workers to allocate
        ...
        signal workers to free
        ...
        print addresses from each $thread$
Worker Thread$\(_1\)$
        allocate, write, read, free
        warmup memory in chunkc of 16 bytes
        ...
        malloc N objects
        ...
        free objects
        return object address to Main Thread
Worker Thread$\(_2\)$
        // same as Worker Thread$\(_1\)$
\end{lstlisting}
%\input{MemoryOverhead}
%\includegraphics[width=1\textwidth]{figures/bench-cache-thrash.eps}
\caption{Allocator-Induced Active False-Sharing Benchmark}
\label{fig:benchThrashFig}
\end{figure}

The adjustment knobs for cache access scenarios are:
\begin{description}[itemsep=0pt,parsep=0pt]
\item[thread:]
number of threads (K).
\item[iterations:]
iterations of cache benchmark (N).
\item[cacheRW:]
repetitions of reads/writes to object (M).
\item[size:]
object size.
\end{description}


\subsection{Cache Scratch}
\label{s:CacheScratch}

The cache-scratch micro-benchmark measures allocator-induced passive false-sharing as illustrated in \VRef{s:AllocatorInducedPassiveFalseSharing}.
As for cache thrash, if memory is allocated for multiple threads on the same cache line, this can significantly slow down program performance.
In this scenario, the false sharing is being caused by the memory allocator although it is started by the program sharing an object.

% An allocator can unintentionally induce false sharing depending upon its management of the freed objects.
% If thread Thread$_1$ allocates multiple objects together, they may be allocated on the same cache line by the memory allocator.
% If Thread$_1$ passes these object to thread Thread$_2$, then both threads may share the same cache line but this scenerio is not induced by the allocator;
% instead, the program induced this situation.
% Now if Thread$_2$ frees this object and then allocate an object of the same size, the allocator may return the same object, which is on a cache line shared with thread Thread$_1$.

Cache scratch tries to create a scenario that leads to false sharing and should make the memory allocator preserve the program-induced false sharing, if it does not return a freed object to its owner thread and, instead, re-uses it instantly.
An allocator using object ownership, as described in section \VRef{s:Ownership}, is less susceptible to allocator-induced passive false-sharing.
If the object is returned to the thread who owns it, then the thread that gets a new object is less likely to be on the same cache line.

\VRef[Figure]{fig:benchScratchFig} shows the pseudo code for the cache-scratch micro-benchmark.
First, it allocates K dynamic objects together, one for each of the K worker threads, possibly causing memory allocator to allocate these objects on the same cache line.
Then it create K worker threads and passes an object from the K allocated objects to each of the K threads.
Each worker thread frees the object passed by the main thread.
Then, it allocates an object and reads/writes it repetitively for M times possibly causing frequent cache invalidations.
Each worker repeats this N times.

\begin{figure}
\centering
\input{AllocInducedPassiveFalseSharing}
\medskip
\begin{lstlisting}
Main Thread
        malloc N objects $for$ each worker $thread$
        create worker threads and pass N objects to each worker
        ...
        signal workers to allocate
        ...
        signal workers to free
        ...
        print addresses from each $thread$
Worker Thread$\(_1\)$
        allocate, write, read, free
        warmup memory in chunkc of 16 bytes
        ...
        for ( N )
                free an object passed by Main Thread
                malloc new object
        ...
        free objects
        return new object addresses to Main Thread
Worker Thread$\(_2\)$
        // same as Worker Thread$\(_1\)$
\end{lstlisting}
%\includegraphics[width=1\textwidth]{figures/bench-cache-scratch.eps}
\caption{Program-Induced Passive False-Sharing Benchmark}
\label{fig:benchScratchFig}
\end{figure}

Each thread allocating an object after freeing the original object passed by the main thread should cause the memory allocator to return the same object that was initially allocated by the main thread if the allocator did not return the initial object back to its owner (main thread).
Then, intensive read/write on the shared cache line by multiple threads should slow down worker threads due to to high cache invalidations and misses.
Main thread measures the total time taken for all the workers to complete.

Similar to benchmark cache thrash in section \VRef{sec:benchThrashSec}, different cache access scenarios can be created using the following command-line arguments.
\begin{description}[itemsep=0pt,parsep=0pt]
\item[threads:]
number of threads (K).
\item[iterations:]
iterations of cache benchmark (N).
\item[cacheRW:]
repetitions of reads/writes to object (M).
\item[size:]
object size.
\end{description}


\subsection{Speed Micro-Benchmark}
\label{s:SpeedMicroBenchmark}

The speed benchmark measures the runtime speed of individual and sequences of memory allocation routines:
\begin{enumerate}[itemsep=0pt,parsep=0pt]
\item malloc
\item realloc
\item free
\item calloc
\item malloc-free
\item realloc-free
\item calloc-free
\item malloc-realloc
\item calloc-realloc
\item malloc-realloc-free
\item calloc-realloc-free
\item malloc-realloc-free-calloc
\end{enumerate}

\VRef[Figure]{fig:SpeedBenchFig} shows the pseudo code for the speed micro-benchmark.
Each routine in the chain is called for N objects and then those allocated objects are used when calling the next routine in the allocation chain.
This tests the latency of the memory allocator when multiple routines are chained together, \eg the call sequence malloc-realloc-free-calloc gives a complete picture of the major allocation routines when combined together.
For each chain, the time is recorded to visualize performance of a memory allocator against each chain.

\begin{figure}
\centering
\begin{lstlisting}[morekeywords={foreach}]
Main Thread
        create worker threads
        foreach ( allocation chain )
                note time T1
                ...
                note time T2
                chain_speed = (T2 - T1) / number-of-worker-threads * N )
Worker Thread
        initialize variables
        ...
        foreach ( routine in allocation chain )
                call routine N times
\end{lstlisting}
%\includegraphics[width=1\textwidth]{figures/bench-speed.eps}
\caption{Speed Benchmark}
\label{fig:SpeedBenchFig}
\end{figure}

The adjustment knobs for memory usage are:
\begin{description}[itemsep=0pt,parsep=0pt]
\item[max:]
maximum object size.
\item[min:]
minimum object size.
\item[step:]
object size increment.
\item[distro:]
object size distribution.
\item[objects:]
number of objects per thread.
\item[workers:]
number of worker threads.
\end{description}


\subsection{Memory Micro-Benchmark}
\label{s:MemoryMicroBenchmark}

The memory micro-benchmark measures the memory overhead of an allocator.
It allocates a number of dynamic objects and reads @/proc/self/proc/maps@ to get the total memory requested by the allocator from the OS.
It calculates the memory overhead by computing the difference between the memory the allocator requests from the OS and the memory that the program allocates.
This micro-benchmark is like Larson and stresses the ability of an allocator to deal with object sharing.

\VRef[Figure]{fig:MemoryBenchFig} shows the pseudo code for the memory micro-benchmark.
It creates a producer-consumer scenario with K producer threads and each producer has M consumer threads.
A producer has a separate buffer for each consumer and allocates N objects of random sizes following a settable distribution for each consumer.
A consumer frees these objects.
After every memory operation, program memory usage is recorded throughout the runtime.
This data is used to visualize the memory usage and consumption for the program.

\begin{figure}
\centering
\begin{lstlisting}
Main Thread
        print memory snapshot
        create producer threads
Producer Thread (K)
        set free start
        create consumer threads
        for ( N )
                allocate memory
                print memory snapshot
Consumer Thread (M)
        wait while ( allocations < free start )
        for ( N )
                free memory
                print memory snapshot
\end{lstlisting}
%\includegraphics[width=1\textwidth]{figures/bench-memory.eps}
\caption{Memory Footprint Micro-Benchmark}
\label{fig:MemoryBenchFig}
\end{figure}

The global adjustment knobs for this micro-benchmark are:
\begin{description}[itemsep=0pt,parsep=0pt]
\item[producer (K):]
sets the number of producer threads.
\item[consumer (M):]
sets number of consumers threads for each producer.
\item[round:]
sets production and consumption round size.
\end{description}

The adjustment knobs for object allocation are:
\begin{description}[itemsep=0pt,parsep=0pt]
\item[max:]
maximum object size.
\item[min:]
minimum object size.
\item[step:]
object size increment.
\item[distro:]
object size distribution.
\item[objects (N):]
number of objects per thread.
\end{description}