source: doc/theses/mubeen_zulfiqar_MMath/benchmarks.tex @ 5b643ea

Last change on this file since 5b643ea was fb6691a, checked in by Peter A. Buhr <pabuhr@…>, 2 years ago

final proofread of Mubeen's MMath thesis

  • Property mode set to 100644
File size: 16.8 KB
Line 
1\chapter{Benchmarks}
2\label{s:Benchmarks}
3
4%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
5%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
6%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Micro Benchmark Suite
7%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
8%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
9
10There are two basic approaches for evaluating computer software: benchmarks and micro-benchmarks.
11\begin{description}
12\item[Benchmarks]
13are a suite of application programs (SPEC CPU/WEB) that are exercised in a common way (inputs) to find differences among underlying software implementations associated with an application (compiler, memory allocator, web server, \etc).
14The applications are supposed to represent common execution patterns that need to perform well with respect to an underlying software implementation.
15Benchmarks are often criticized for having overlapping patterns, insufficient patterns, or extraneous code that masks patterns.
16\item[Micro-Benchmarks]
17attempt to extract the common execution patterns associated with an application and run the pattern independently.
18This approach removes any masking from extraneous application code, allows execution pattern to be very precise, and provides an opportunity for the execution pattern to have multiple independent tuning adjustments (knobs).
19Micro-benchmarks are often criticized for inadequately representing real-world applications.
20\end{description}
21
22While some crucial software components have standard benchmarks, no standard benchmark exists for testing and comparing memory allocators.
23In the past, an assortment of applications have been used for benchmarking allocators~\cite{Detlefs93,Berger00,Berger01,berger02reconsidering}: P2C, GS, Espresso/Espresso-2, CFRAC/CFRAC-2, GMake, GCC, Perl/Perl-2, Gawk/Gawk-2, XPDF/XPDF-2, ROBOOP, Lindsay.
24As well, an assortment of micro-benchmark have been used for benchmarking allocators~\cite{larson99memory,Berger00,streamflow}: threadtest, shbench, Larson, consume, false sharing.
25Many of these benchmark applications and micro-benchmarks are old and may not reflect current application allocation patterns.
26
27This thesis designs and examines a new set of micro-benchmarks for memory allocators that test a variety of allocation patterns, each with multiple tuning parameters.
28The aim of the micro-benchmark suite is to create a set of programs that can evaluate a memory allocator based on the key performance metrics such as speed, memory overhead, and cache performance.
29% These programs can be taken as a standard to benchmark an allocator's basic goals.
30These programs give details of an allocator's memory overhead and speed under certain allocation patterns.
31The allocation patterns are configurable (adjustment knobs) to observe an allocator's performance across a spectrum allocation patterns, which is seldom possible with benchmark programs.
32Each micro-benchmark program has multiple control knobs specified by command-line arguments.
33
34The new micro-benchmark suite measures performance by allocating dynamic objects and measuring specific metrics.
35An allocator's speed is benchmarked in different ways, as are issues like false sharing.
36
37
38\section{Prior Multi-Threaded Micro-Benchmarks}
39
40Modern memory allocators, such as llheap, must handle multi-threaded programs at the KT and UT level.
41The following multi-threaded micro-benchmarks are presented to give a sense of prior work~\cite{Berger00} at the KT level.
42None of the prior work addresses multi-threading at the UT level.
43
44
45\subsection{threadtest}
46
47This benchmark stresses the ability of the allocator to handle different threads allocating and deallocating independently.
48There is no interaction among threads, \ie no object sharing.
49Each thread repeatedly allocates 100,000 \emph{8-byte} objects then deallocates them in the order they were allocated.
50The execution time of the benchmark evaluates its efficiency.
51
52
53\subsection{shbench}
54
55This benchmark is similar to threadtest but each thread randomly allocate and free a number of \emph{random-sized} objects.
56It is a stress test that also uses runtime to determine efficiency of the allocator.
57
58
59\subsection{Larson}
60
61This benchmark simulates a server environment.
62Multiple threads are created where each thread allocates and frees a number of random-sized objects within a size range.
63Before the thread terminates, it passes its array of 10,000 objects to a new child thread to continue the process.
64The number of thread generations varies depending on the thread speed.
65It calculates memory operations per second as an indicator of the memory allocator's performance.
66
67
68\section{New Multi-Threaded Micro-Benchmarks}
69
70The following new benchmarks were created to assess multi-threaded programs at the KT and UT level.
71For generating random values, two generators are supported: uniform~\cite{uniformPRNG} and fisher~\cite{fisherPRNG}.
72
73
74\subsection{Churn Benchmark}
75\label{s:ChurnBenchmark}
76
77The churn benchmark measures the runtime speed of an allocator in a multi-threaded scenario, where each thread extensively allocates and frees dynamic memory.
78Only @malloc@ and @free@ are used to eliminate any extra cost, such as @memcpy@ in @calloc@ or @realloc@.
79Churn simulates a memory intensive program and can be tuned to create different scenarios.
80
81\VRef[Figure]{fig:ChurnBenchFig} shows the pseudo code for the churn micro-benchmark.
82This benchmark creates a buffer with M spots and an allocation in each spot, and then starts K threads.
83Each thread picks a random spot in M, frees the object currently at that spot, and allocates a new object for that spot.
84Each thread repeats this cycle N times.
85The main thread measures the total time taken for the whole benchmark and that time is used to evaluate the memory allocator's performance.
86
87\begin{figure}
88\centering
89\begin{lstlisting}
90Main Thread
91        create worker threads
92        note time T1
93        ...
94        note time T2
95        churn_speed = (T2 - T1)
96Worker Thread
97        initialize variables
98        ...
99        for ( N )
100                R = random spot in array
101                free R
102                allocate new object at R
103\end{lstlisting}
104%\includegraphics[width=1\textwidth]{figures/bench-churn.eps}
105\caption{Churn Benchmark}
106\label{fig:ChurnBenchFig}
107\end{figure}
108
109The adjustment knobs for churn are:
110\begin{description}[itemsep=0pt,parsep=0pt]
111\item[thread:]
112number of threads (K).
113\item[spots:]
114number of spots for churn (M).
115\item[obj:]
116number of objects per thread (N).
117\item[max:]
118maximum object size.
119\item[min:]
120minimum object size.
121\item[step:]
122object size increment.
123\item[distro:]
124object size distribution
125\end{description}
126
127
128\subsection{Cache Thrash}
129\label{sec:benchThrashSec}
130
131The cache-thrash micro-benchmark measures allocator-induced active false-sharing as illustrated in \VRef{s:AllocatorInducedActiveFalseSharing}.
132If memory is allocated for multiple threads on the same cache line, this can significantly slow down program performance.
133When threads share a cache line, frequent reads/writes to their cache-line object causes cache misses, which cause escalating delays as cache distance increases.
134
135Cache thrash tries to create a scenario that leads to false sharing, if the underlying memory allocator is allocating dynamic memory to multiple threads on the same cache lines.
136Ideally, a memory allocator should distance the dynamic memory region of one thread from another.
137Having multiple threads allocating small objects simultaneously can cause a memory allocator to allocate objects on the same cache line, if its not distancing the memory among different threads.
138
139\VRef[Figure]{fig:benchThrashFig} shows the pseudo code for the cache-thrash micro-benchmark.
140First, it creates K worker threads.
141Each worker thread allocates an object and intensively reads/writes it for M times to possible invalidate cache lines that may interfere with other threads sharing the same cache line.
142Each thread repeats this for N times.
143The main thread measures the total time taken for all worker threads to complete.
144Worker threads sharing cache lines with each other are expected to take longer.
145
146\begin{figure}
147\centering
148\input{AllocInducedActiveFalseSharing}
149\medskip
150\begin{lstlisting}
151Main Thread
152        create worker threads
153        ...
154        signal workers to allocate
155        ...
156        signal workers to free
157        ...
158Worker Thread$\(_1\)$
159        warm up memory in chunks of 16 bytes
160        ...
161        For N
162                malloc an object
163                read/write the object M times
164                free the object
165        ...
166Worker Thread$\(_2\)$
167        // same as Worker Thread$\(_1\)$
168\end{lstlisting}
169%\input{MemoryOverhead}
170%\includegraphics[width=1\textwidth]{figures/bench-cache-thrash.eps}
171\caption{Allocator-Induced Active False-Sharing Benchmark}
172\label{fig:benchThrashFig}
173\end{figure}
174
175The adjustment knobs for cache access scenarios are:
176\begin{description}[itemsep=0pt,parsep=0pt]
177\item[thread:]
178number of threads (K).
179\item[iterations:]
180iterations of cache benchmark (N).
181\item[cacheRW:]
182repetitions of reads/writes to object (M).
183\item[size:]
184object size.
185\end{description}
186
187
188\subsection{Cache Scratch}
189\label{s:CacheScratch}
190
191The cache-scratch micro-benchmark measures allocator-induced passive false-sharing as illustrated in \VRef{s:AllocatorInducedPassiveFalseSharing}.
192As with cache thrash, if memory is allocated for multiple threads on the same cache line, this can significantly slow down program performance.
193In this scenario, the false sharing is being caused by the memory allocator although it is started by the program sharing an object.
194
195% An allocator can unintentionally induce false sharing depending upon its management of the freed objects.
196% If thread Thread$_1$ allocates multiple objects together, they may be allocated on the same cache line by the memory allocator.
197% If Thread$_1$ passes these object to thread Thread$_2$, then both threads may share the same cache line but this scenerio is not induced by the allocator;
198% instead, the program induced this situation.
199% Now if Thread$_2$ frees this object and then allocate an object of the same size, the allocator may return the same object, which is on a cache line shared with thread Thread$_1$.
200
201Cache scratch tries to create a scenario that leads to false sharing and should make the memory allocator preserve the program-induced false sharing, if it does not return a freed object to its owner thread and, instead, re-uses it instantly.
202An allocator using object ownership, as described in section \VRef{s:Ownership}, is less susceptible to allocator-induced passive false-sharing.
203If the object is returned to the thread that owns it, then the new object that the thread gets is less likely to be on the same cache line.
204
205\VRef[Figure]{fig:benchScratchFig} shows the pseudo code for the cache-scratch micro-benchmark.
206First, it allocates K dynamic objects together, one for each of the K worker threads, possibly causing memory allocator to allocate these objects on the same cache line.
207Then it create K worker threads and passes an object from the K allocated objects to each of the K threads.
208Each worker thread frees the object passed by the main thread.
209Then, it allocates an object and reads/writes it repetitively for M times possibly causing frequent cache invalidations.
210Each worker repeats this N times.
211
212\begin{figure}
213\centering
214\input{AllocInducedPassiveFalseSharing}
215\medskip
216\begin{lstlisting}
217Main Thread
218        malloc N objects $for$ each worker $thread$
219        create worker threads and pass N objects to each worker
220        ...
221        signal workers to allocate
222        ...
223        signal workers to free
224        ...
225Worker Thread$\(_1\)$
226        warmup memory in chunks of 16 bytes
227        ...
228        free the object passed by the Main Thread
229        For N
230                malloc new object
231                read/write the object M times
232                free the object
233        ...
234Worker Thread$\(_2\)$
235        // same as Worker Thread$\(_1\)$
236\end{lstlisting}
237%\includegraphics[width=1\textwidth]{figures/bench-cache-scratch.eps}
238\caption{Program-Induced Passive False-Sharing Benchmark}
239\label{fig:benchScratchFig}
240\end{figure}
241
242Each thread allocating an object after freeing the original object passed by the main thread should cause the memory allocator to return the same object that was initially allocated by the main thread if the allocator did not return the initial object back to its owner (main thread).
243Then, intensive read/write on the shared cache line by multiple threads should slow down worker threads due to to high cache invalidations and misses.
244Main thread measures the total time taken for all the workers to complete.
245
246Similar to benchmark cache thrash in section \VRef{sec:benchThrashSec}, different cache access scenarios can be created using the following command-line arguments.
247\begin{description}[topsep=0pt,itemsep=0pt,parsep=0pt]
248\item[threads:]
249number of threads (K).
250\item[iterations:]
251iterations of cache benchmark (N).
252\item[cacheRW:]
253repetitions of reads/writes to object (M).
254\item[size:]
255object size.
256\end{description}
257
258
259\subsection{Speed Micro-Benchmark}
260\label{s:SpeedMicroBenchmark}
261\vspace*{-4pt}
262
263The speed benchmark measures the runtime speed of individual and sequences of memory allocation routines:
264\begin{enumerate}[topsep=-5pt,itemsep=0pt,parsep=0pt]
265\item malloc
266\item realloc
267\item free
268\item calloc
269\item malloc-free
270\item realloc-free
271\item calloc-free
272\item malloc-realloc
273\item calloc-realloc
274\item malloc-realloc-free
275\item calloc-realloc-free
276\item malloc-realloc-free-calloc
277\end{enumerate}
278
279\VRef[Figure]{fig:SpeedBenchFig} shows the pseudo code for the speed micro-benchmark.
280Each routine in the chain is called for N objects and then those allocated objects are used when calling the next routine in the allocation chain.
281This tests the latency of the memory allocator when multiple routines are chained together, \eg the call sequence malloc-realloc-free-calloc gives a complete picture of the major allocation routines when combined together.
282For each chain, the time is recorded to visualize performance of a memory allocator against each chain.
283
284\begin{figure}
285\centering
286\begin{lstlisting}[morekeywords={foreach}]
287Main Thread
288        create worker threads
289        foreach ( allocation chain )
290                note time T1
291                ...
292                note time T2
293                chain_speed = (T2 - T1) / number-of-worker-threads * N )
294Worker Thread
295        initialize variables
296        ...
297        foreach ( routine in allocation chain )
298                call routine N times
299\end{lstlisting}
300%\includegraphics[width=1\textwidth]{figures/bench-speed.eps}
301\caption{Speed Benchmark}
302\label{fig:SpeedBenchFig}
303\end{figure}
304
305The adjustment knobs for memory usage are:
306\begin{description}[itemsep=0pt,parsep=0pt]
307\item[max:]
308maximum object size.
309\item[min:]
310minimum object size.
311\item[step:]
312object size increment.
313\item[distro:]
314object size distribution.
315\item[objects:]
316number of objects per thread.
317\item[workers:]
318number of worker threads.
319\end{description}
320
321
322\subsection{Memory Micro-Benchmark}
323\label{s:MemoryMicroBenchmark}
324
325The memory micro-benchmark measures the memory overhead of an allocator.
326It allocates a number of dynamic objects and reads @/proc/self/proc/maps@ to get the total memory requested by the allocator from the OS.
327It calculates the memory overhead by computing the difference between the memory the allocator requests from the OS and the memory that the program allocates.
328This micro-benchmark is like Larson and stresses the ability of an allocator to deal with object sharing.
329
330\VRef[Figure]{fig:MemoryBenchFig} shows the pseudo code for the memory micro-benchmark.
331It creates a producer-consumer scenario with K producer threads and each producer has M consumer threads.
332A producer has a separate buffer for each consumer and allocates N objects of random sizes following a configurable distribution for each consumer.
333A consumer frees these objects.
334After every memory operation, program memory usage is recorded throughout the runtime.
335This data is used to visualize the memory usage and consumption for the program.
336
337\begin{figure}
338\centering
339\begin{lstlisting}
340Main Thread
341        print memory snapshot
342        create producer threads
343Producer Thread (K)
344        set free start
345        create consumer threads
346        for ( N )
347                allocate memory
348                print memory snapshot
349Consumer Thread (M)
350        wait while ( allocations < free start )
351        for ( N )
352                free memory
353                print memory snapshot
354\end{lstlisting}
355%\includegraphics[width=1\textwidth]{figures/bench-memory.eps}
356\caption{Memory Footprint Micro-Benchmark}
357\label{fig:MemoryBenchFig}
358\end{figure}
359
360The global adjustment knobs for this micro-benchmark are:
361\begin{description}[itemsep=0pt,parsep=0pt]
362\item[producer (K):]
363sets the number of producer threads.
364\item[consumer (M):]
365sets number of consumers threads for each producer.
366\item[round:]
367sets production and consumption round size.
368\end{description}
369
370The adjustment knobs for object allocation are:
371\begin{description}[itemsep=0pt,parsep=0pt]
372\item[max:]
373maximum object size.
374\item[min:]
375minimum object size.
376\item[step:]
377object size increment.
378\item[distro:]
379object size distribution.
380\item[objects (N):]
381number of objects per thread.
382\end{description}
Note: See TracBrowser for help on using the repository browser.