source: doc/theses/mubeen_zulfiqar_MMath/benchmarks.tex @ 29d8c02

pthread-emulation
Last change on this file since 29d8c02 was 29d8c02, checked in by m3zulfiq <m3zulfiq@…>, 3 months ago

made corrections in thesis as per feedback from the readers

  • Property mode set to 100644
File size: 16.8 KB
Line 
1\chapter{Benchmarks}
2\label{s:Benchmarks}
3
4%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
5%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
6%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Micro Benchmark Suite
7%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
8%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
9
10There are two basic approaches for evaluating computer software: benchmarks and micro-benchmarks.
11\begin{description}
12\item[Benchmarks]
13are a suite of application programs (SPEC CPU/WEB) that are exercised in a common way (inputs) to find differences among underlying software implementations associated with an application (compiler, memory allocator, web server, \etc).
14The applications are supposed to represent common execution patterns that need to perform well with respect to an underlying software implementation.
15Benchmarks are often criticized for having overlapping patterns, insufficient patterns, or extraneous code that masks patterns.
16\item[Micro-Benchmarks]
17attempt to extract the common execution patterns associated with an application and run the pattern independently.
18This approach removes any masking from extraneous application code, allows execution pattern to be very precise, and provides an opportunity for the execution pattern to have multiple independent tuning adjustments (knobs).
19Micro-benchmarks are often criticized for inadequately representing real-world applications.
20\end{description}
21
22While some crucial software components have standard benchmarks, no standard benchmark exists for testing and comparing memory allocators.
23In the past, an assortment of applications have been used for benchmarking allocators~\cite{Detlefs93,Berger00,Berger01,berger02reconsidering}: P2C, GS, Espresso/Espresso-2, CFRAC/CFRAC-2, GMake, GCC, Perl/Perl-2, Gawk/Gawk-2, XPDF/XPDF-2, ROBOOP, Lindsay.
24As well, an assortment of micro-benchmark have been used for benchmarking allocators~\cite{larson99memory,Berger00,streamflow}: threadtest, shbench, Larson, consume, false sharing.
25Many of these benchmark applications and micro-benchmarks are old and may not reflect current application allocation patterns.
26
27This thesis designs and examines a new set of micro-benchmarks for memory allocators that test a variety of allocation patterns, each with multiple tuning parameters.
28The aim of the micro-benchmark suite is to create a set of programs that can evaluate a memory allocator based on the key performance metrics such as speed, memory overhead, and cache performance.
29% These programs can be taken as a standard to benchmark an allocator's basic goals.
30These programs give details of an allocator's memory overhead and speed under certain allocation patterns.
31The allocation patterns are configurable (adjustment knobs) to observe an allocator's performance across a spectrum allocation patterns, which is seldom possible with benchmark programs.
32Each micro-benchmark program has multiple control knobs specified by command-line arguments.
33
34The new micro-benchmark suite measures performance by allocating dynamic objects and measuring specific metrics.
35An allocator's speed is benchmarked in different ways, as are issues like false sharing.
36
37
38\section{Prior Multi-Threaded Micro-Benchmarks}
39
40Modern memory allocators, such as llheap, must handle multi-threaded programs at the KT and UT level.
41The following multi-threaded micro-benchmarks are presented to give a sense of prior work~\cite{Berger00} at the KT level.
42None of the prior work addresses multi-threading at the UT level.
43
44
45\subsection{threadtest}
46
47This benchmark stresses the ability of the allocator to handle different threads allocating and deallocating independently.
48There is no interaction among threads, \ie no object sharing.
49Each thread repeatedly allocates 100,000 \emph{8-byte} objects then deallocates them in the order they were allocated.
50\PAB{Execution time of the benchmark evaluates its efficiency.}
51
52
53\subsection{shbench}
54
55This benchmark is similar to threadtest but each thread randomly allocate and free a number of \emph{random-sized} objects.
56It is a stress test that also uses runtime to determine efficiency of the allocator.
57
58
59\subsection{Larson}
60
61This benchmark simulates a server environment.
62Multiple threads are created where each thread allocates and frees a number of random-sized objects within a size range.
63Before the thread terminates, it passes its array of 10,000 objects to a new child thread to continue the process.
64The number of thread generations varies depending on the thread speed.
65It calculates memory operations per second as an indicator of the memory allocator's performance.
66
67
68\section{New Multi-Threaded Micro-Benchmarks}
69
70The following new benchmarks were created to assess multi-threaded programs at the KT and UT level.
71For generating random values, two generators are supported: uniform~\cite{uniformPRNG} and fisher~\cite{fisherPRNG}.
72
73
74\subsection{Churn Benchmark}
75\label{s:ChurnBenchmark}
76
77The churn benchmark measures the runtime speed of an allocator in a multi-threaded scenerio, where each thread extensively allocates and frees dynamic memory.
78Only @malloc@ and @free@ are used to eliminate any extra cost, such as @memcpy@ in @calloc@ or @realloc@.
79Churn simulates a memory intensive program and can be tuned to create different scenarios.
80
81\VRef[Figure]{fig:ChurnBenchFig} shows the pseudo code for the churn micro-benchmark.
82This benchmark creates a buffer with M spots and an allocation in each spot, and then starts K threads.
83Each thread picks a random spot in M, frees the object currently at that spot, and allocates a new object for that spot.
84Each thread repeats this cycle N times.
85The main thread measures the total time taken for the whole benchmark and that time is used to evaluate the memory allocator's performance.
86
87\begin{figure}
88\centering
89\begin{lstlisting}
90Main Thread
91        create worker threads
92        note time T1
93        ...
94        note time T2
95        churn_speed = (T2 - T1)
96Worker Thread
97        initialize variables
98        ...
99        for ( N )
100                R = random spot in array
101                free R
102                allocate new object at R
103\end{lstlisting}
104%\includegraphics[width=1\textwidth]{figures/bench-churn.eps}
105\caption{Churn Benchmark}
106\label{fig:ChurnBenchFig}
107\end{figure}
108
109The adjustment knobs for churn are:
110\begin{description}[itemsep=0pt,parsep=0pt]
111\item[thread:]
112number of threads (K).
113\item[spots:]
114number of spots for churn (M).
115\item[obj:]
116number of objects per thread (N).
117\item[max:]
118maximum object size.
119\item[min:]
120minimum object size.
121\item[step:]
122object size increment.
123\item[distro:]
124object size distribution
125\end{description}
126
127
128\subsection{Cache Thrash}
129\label{sec:benchThrashSec}
130
131The cache-thrash micro-benchmark measures allocator-induced active false-sharing as illustrated in \VRef{s:AllocatorInducedActiveFalseSharing}.
132If memory is allocated for multiple threads on the same cache line, this can significantly slow down program performance.
133When threads share a cache line, frequent reads/writes to their cache-line object causes cache misses, which cause escalating delays as cache distance increases.
134
135Cache thrash tries to create a scenerio that leads to false sharing, if the underlying memory allocator is allocating dynamic memory to multiple threads on the same cache lines.
136Ideally, a memory allocator should distance the dynamic memory region of one thread from another.
137Having multiple threads allocating small objects simultaneously can cause a memory allocator to allocate objects on the same cache line, if its not distancing the memory among different threads.
138
139\VRef[Figure]{fig:benchThrashFig} shows the pseudo code for the cache-thrash micro-benchmark.
140First, it creates K worker threads.
141Each worker thread allocates an object and intensively reads/writes it for M times to possible invalidate cache lines that may interfere with other threads sharing the same cache line.
142Each thread repeats this for N times.
143The main thread measures the total time taken for all worker threads to complete.
144Worker threads sharing cache lines with each other are expected to take longer.
145
146\begin{figure}
147\centering
148\input{AllocInducedActiveFalseSharing}
149\medskip
150\begin{lstlisting}
151Main Thread
152        create worker threads
153        ...
154        signal workers to allocate
155        ...
156        signal workers to free
157        ...
158Worker Thread$\(_1\)$
159        warm up memory in chunks of 16 bytes
160        ...
161        For N
162                malloc an object
163                read/write the object M times
164                free the object
165        ...
166Worker Thread$\(_2\)$
167        // same as Worker Thread$\(_1\)$
168\end{lstlisting}
169%\input{MemoryOverhead}
170%\includegraphics[width=1\textwidth]{figures/bench-cache-thrash.eps}
171\caption{Allocator-Induced Active False-Sharing Benchmark}
172\label{fig:benchThrashFig}
173\end{figure}
174
175The adjustment knobs for cache access scenarios are:
176\begin{description}[itemsep=0pt,parsep=0pt]
177\item[thread:]
178number of threads (K).
179\item[iterations:]
180iterations of cache benchmark (N).
181\item[cacheRW:]
182repetitions of reads/writes to object (M).
183\item[size:]
184object size.
185\end{description}
186
187
188\subsection{Cache Scratch}
189\label{s:CacheScratch}
190
191The cache-scratch micro-benchmark measures allocator-induced passive false-sharing as illustrated in \VRef{s:AllocatorInducedPassiveFalseSharing}.
192As with cache thrash, if memory is allocated for multiple threads on the same cache line, this can significantly slow down program performance.
193In this scenario, the false sharing is being caused by the memory allocator although it is started by the program sharing an object.
194
195% An allocator can unintentionally induce false sharing depending upon its management of the freed objects.
196% If thread Thread$_1$ allocates multiple objects together, they may be allocated on the same cache line by the memory allocator.
197% If Thread$_1$ passes these object to thread Thread$_2$, then both threads may share the same cache line but this scenerio is not induced by the allocator;
198% instead, the program induced this situation.
199% Now if Thread$_2$ frees this object and then allocate an object of the same size, the allocator may return the same object, which is on a cache line shared with thread Thread$_1$.
200
201Cache scratch tries to create a scenario that leads to false sharing and should make the memory allocator preserve the program-induced false sharing, if it does not return a freed object to its owner thread and, instead, re-uses it instantly.
202An allocator using object ownership, as described in section \VRef{s:Ownership}, is less susceptible to allocator-induced passive false-sharing.
203\PAB{If the object is returned to the thread that owns it, then the new object that the thread gets is less likely to be on the same cache line.}
204
205\VRef[Figure]{fig:benchScratchFig} shows the pseudo code for the cache-scratch micro-benchmark.
206First, it allocates K dynamic objects together, one for each of the K worker threads, possibly causing memory allocator to allocate these objects on the same cache line.
207Then it create K worker threads and passes an object from the K allocated objects to each of the K threads.
208Each worker thread frees the object passed by the main thread.
209Then, it allocates an object and reads/writes it repetitively for M times possibly causing frequent cache invalidations.
210Each worker repeats this N times.
211
212\begin{figure}
213\centering
214\input{AllocInducedPassiveFalseSharing}
215\medskip
216\begin{lstlisting}
217Main Thread
218        malloc N objects $for$ each worker $thread$
219        create worker threads and pass N objects to each worker
220        ...
221        signal workers to allocate
222        ...
223        signal workers to free
224        ...
225Worker Thread$\(_1\)$
226        warmup memory in chunks of 16 bytes
227        ...
228        free the object passed by the Main Thread
229        For N
230                malloc new object
231                read/write the object M times
232                free the object
233        ...
234Worker Thread$\(_2\)$
235        // same as Worker Thread$\(_1\)$
236\end{lstlisting}
237%\includegraphics[width=1\textwidth]{figures/bench-cache-scratch.eps}
238\caption{Program-Induced Passive False-Sharing Benchmark}
239\label{fig:benchScratchFig}
240\end{figure}
241
242Each thread allocating an object after freeing the original object passed by the main thread should cause the memory allocator to return the same object that was initially allocated by the main thread if the allocator did not return the initial object back to its owner (main thread).
243Then, intensive read/write on the shared cache line by multiple threads should slow down worker threads due to to high cache invalidations and misses.
244Main thread measures the total time taken for all the workers to complete.
245
246Similar to benchmark cache thrash in section \VRef{sec:benchThrashSec}, different cache access scenarios can be created using the following command-line arguments.
247\begin{description}[itemsep=0pt,parsep=0pt]
248\item[threads:]
249number of threads (K).
250\item[iterations:]
251iterations of cache benchmark (N).
252\item[cacheRW:]
253repetitions of reads/writes to object (M).
254\item[size:]
255object size.
256\end{description}
257
258
259\subsection{Speed Micro-Benchmark}
260\label{s:SpeedMicroBenchmark}
261
262The speed benchmark measures the runtime speed of individual and sequences of memory allocation routines:
263\begin{enumerate}[itemsep=0pt,parsep=0pt]
264\item malloc
265\item realloc
266\item free
267\item calloc
268\item malloc-free
269\item realloc-free
270\item calloc-free
271\item malloc-realloc
272\item calloc-realloc
273\item malloc-realloc-free
274\item calloc-realloc-free
275\item malloc-realloc-free-calloc
276\end{enumerate}
277
278\VRef[Figure]{fig:SpeedBenchFig} shows the pseudo code for the speed micro-benchmark.
279Each routine in the chain is called for N objects and then those allocated objects are used when calling the next routine in the allocation chain.
280This tests the latency of the memory allocator when multiple routines are chained together, \eg the call sequence malloc-realloc-free-calloc gives a complete picture of the major allocation routines when combined together.
281For each chain, the time is recorded to visualize performance of a memory allocator against each chain.
282
283\begin{figure}
284\centering
285\begin{lstlisting}[morekeywords={foreach}]
286Main Thread
287        create worker threads
288        foreach ( allocation chain )
289                note time T1
290                ...
291                note time T2
292                chain_speed = (T2 - T1) / number-of-worker-threads * N )
293Worker Thread
294        initialize variables
295        ...
296        foreach ( routine in allocation chain )
297                call routine N times
298\end{lstlisting}
299%\includegraphics[width=1\textwidth]{figures/bench-speed.eps}
300\caption{Speed Benchmark}
301\label{fig:SpeedBenchFig}
302\end{figure}
303
304The adjustment knobs for memory usage are:
305\begin{description}[itemsep=0pt,parsep=0pt]
306\item[max:]
307maximum object size.
308\item[min:]
309minimum object size.
310\item[step:]
311object size increment.
312\item[distro:]
313object size distribution.
314\item[objects:]
315number of objects per thread.
316\item[workers:]
317number of worker threads.
318\end{description}
319
320
321\subsection{Memory Micro-Benchmark}
322\label{s:MemoryMicroBenchmark}
323
324The memory micro-benchmark measures the memory overhead of an allocator.
325It allocates a number of dynamic objects and reads @/proc/self/proc/maps@ to get the total memory requested by the allocator from the OS.
326It calculates the memory overhead by computing the difference between the memory the allocator requests from the OS and the memory that the program allocates.
327This micro-benchmark is like Larson and stresses the ability of an allocator to deal with object sharing.
328
329\VRef[Figure]{fig:MemoryBenchFig} shows the pseudo code for the memory micro-benchmark.
330It creates a producer-consumer scenario with K producer threads and each producer has M consumer threads.
331A producer has a separate buffer for each consumer and allocates N objects of random sizes following a configurable distribution for each consumer.
332A consumer frees these objects.
333After every memory operation, program memory usage is recorded throughout the runtime.
334This data is used to visualize the memory usage and consumption for the program.
335
336\begin{figure}
337\centering
338\begin{lstlisting}
339Main Thread
340        print memory snapshot
341        create producer threads
342Producer Thread (K)
343        set free start
344        create consumer threads
345        for ( N )
346                allocate memory
347                print memory snapshot
348Consumer Thread (M)
349        wait while ( allocations < free start )
350        for ( N )
351                free memory
352                print memory snapshot
353\end{lstlisting}
354%\includegraphics[width=1\textwidth]{figures/bench-memory.eps}
355\caption{Memory Footprint Micro-Benchmark}
356\label{fig:MemoryBenchFig}
357\end{figure}
358
359The global adjustment knobs for this micro-benchmark are:
360\begin{description}[itemsep=0pt,parsep=0pt]
361\item[producer (K):]
362sets the number of producer threads.
363\item[consumer (M):]
364sets number of consumers threads for each producer.
365\item[round:]
366sets production and consumption round size.
367\end{description}
368
369The adjustment knobs for object allocation are:
370\begin{description}[itemsep=0pt,parsep=0pt]
371\item[max:]
372maximum object size.
373\item[min:]
374minimum object size.
375\item[step:]
376object size increment.
377\item[distro:]
378object size distribution.
379\item[objects (N):]
380number of objects per thread.
381\end{description}
Note: See TracBrowser for help on using the repository browser.