# source:doc/proposals/concurrency/concurrency.tex@694ee7d

aaron-thesisarm-ehcleanup-dtorsdeferred_resndemanglerjacob/cs343-translationjenkins-sandboxnew-astnew-ast-unique-exprnew-envno_listpersistent-indexerresolv-newwith_gc
Last change on this file since 694ee7d was 694ee7d, checked in by Thierry Delisle <tdelisle@…>, 5 years ago

initial commit of concurrency proposal

• Property mode set to 100644
File size: 25.5 KB
Line
1% requires tex packages: texlive-base texlive-latex-base tex-common texlive-humanities texlive-latex-extra texlive-fonts-recommended
2
3% inline code �...� (copyright symbol) emacs: C-q M-)
4% red highlighting �...� (registered trademark symbol) emacs: C-q M-.
5% blue highlighting �...� (sharp s symbol) emacs: C-q M-_
6% green highlighting �...� (cent symbol) emacs: C-q M-"
7% LaTex escape �...� (section symbol) emacs: C-q M-'
8% keyword escape �...� (pilcrow symbol) emacs: C-q M-^
9% math escape $...$ (dollar symbol)
10
11\documentclass[twoside,11pt]{article}
12
13%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
14
15% Latex packages used in the document.
16\usepackage[T1]{fontenc}                                % allow Latin1 (extended ASCII) characters
17\usepackage{textcomp}
18\usepackage[latin1]{inputenc}
19\usepackage{fullpage,times,comment}
20\usepackage{epic,eepic}
21\usepackage{upquote}                                                                    % switch curled `'" to straight
22\usepackage{calc}
23\usepackage{xspace}
24\usepackage{graphicx}
25\usepackage{varioref}                                                                   % extended references
26\usepackage{listings}                                                                   % format program code
27\usepackage[flushmargin]{footmisc}                                              % support label/reference in footnote
28\usepackage{latexsym}                                   % \Box glyph
29\usepackage{mathptmx}                                   % better math font with "times"
30\usepackage[usenames]{color}
31\usepackage[pagewise]{lineno}
32\renewcommand{\linenumberfont}{\scriptsize\sffamily}
33\input{common}                                          % bespoke macros used in the document
35\usepackage{breakurl}
36\renewcommand{\UrlFont}{\small\sf}
37
38\setlength{\topmargin}{-0.45in}                                                 % move running title into header
40
41%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
42
43% Names used in the document.
44
45\newcommand{\Version}{1.0.0}
46\newcommand{\CS}{C\raisebox{-0.9ex}{\large$^\sharp$}\xspace}
47
48\newcommand{\Textbf}[2][red]{{\color{#1}{\textbf{#2}}}}
49\newcommand{\Emph}[2][red]{{\color{#1}\textbf{\emph{#2}}}}
50\newcommand{\R}[1]{\Textbf{#1}}
51\newcommand{\B}[1]{{\Textbf[blue]{#1}}}
52\newcommand{\G}[1]{{\Textbf[OliveGreen]{#1}}}
53\newcommand{\uC}{$\mu$\CC}
54\newcommand{\cit}{\textsuperscript{[Citation Needed]}\xspace}
55
56
57\newsavebox{\LstBox}
58
59%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
60
61\setcounter{secnumdepth}{3}                             % number subsubsections
63\makeindex
64
65%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
66
67\begin{document}
68% \linenumbers
69
70\title{Concurrency in \CFA}
71\author{Thierry Delisle \\
72Dept. of Computer Science, University of Waterloo, \\ Waterloo, Ontario, Canada
73}
74
75\maketitle
76\section{Introduction}
77This proposal provides a minimal core concurrency API that is both simple, efficient and can be reused to build "higher level" features. The simplest possible core is a thread and a lock but this low level approach is hard to master. An easier approach for users is be to support higher level construct as the basis of the concurrency in \CFA.
78Indeed, for higly productive parallel programming high-level approaches are much more popular. Examples are task based parallelism, message passing, implicit threading.
79
80There are actually to problems that need to be solved in the design of the concurrency for a language. Which concurrency tools are available to the users and which parallelism tools are available. While these two concepts are often seen together, they are in fact distinct concepts that require different sorts of tools. Concurrency tools need to handle mutual exclusion and synchronization while parallelism tools are more about performance, cost and ressource utilisation.
81
82\section{Concurrency}
83Several tool can be used to solve concurrency challenges. Since these challenges always appear with the use of mutable shared state, some languages and libraries simply disallow mutable shared states completely (Erlang, Haskel, Akka (Scala))\cit. In the paradigms, interaction between concurrent objects rely on message passing or other paradigms that often closely relate to networking concepts. However, in imperative or OO languages these approaches entail a clear distinction between concurrent and non concurrent paradigms. Which in turns mean that programmers need to learn two sets of designs patterns in order to be effective at their jobs. Approaches based on shared memory are more closely related to non-concurrent paradigms since they often rely
84
85Finally, an approach that is gaining in popularity is transactionnal memory\cit. However, the performance and feature set is currently too restrictive to be possible to add such a paradigm to a language like C or \CC\cit.
86
87\section{Monitors}
88A monitor is a set of routines that ensure mutual exclusion when accessing shared state. This concept is generally associated with Object-Oriented Languages like Java\cit or \uC\cit but does not strictly require OOP semantics. The only requirements is to be able to declare a handle to a shared object and a set of routines that act on it :
89\begin{lstlisting}
90        typedef \*some monitor type*\ monitor;
91        int f(monitor& m);
92
93        int main() {
94                monitor m;
95                f(m);
96        }
97\end{lstlisting}
98
99\subsection{Call semantics}
100The above example of monitors already displays some of their intrinsic caracteristics. Indeed, it is necessary to use pass-by-reference over pass-by-value for monitor routines. This semantics is important because since at their core, monitors are simply implicit mutual exclusion objects (locks) and copying semantics of these is ill defined. Therefore, monitors are implicitly non-copyable.
101
102Another aspect to consider is when a monitor acquires its mutual exclusion. Indeed, a monitor may need to be passed to helper routines that do not acquire the monitor mutual exclusion on entry. Examples of this can be both external helper routines (\texttt{swap}, \texttt{sort}, etc.) or internal helper routines like the following example :
103
104\begin{lstlisting}
105
106\end{lstlisting}
107
108Having both \texttt{mutex} and \texttt{nomutex} keywords could be argued to be redundant based on the meaning of a routine having neither of these keywords. If there were a meaning to routine \texttt{h} then one could argue that it should be to default to \texttt{mutex} to be safe by default. On the other hand, making one of these keywords mandatory would provide the same semantics but without the ambiguity of supporting routine \texttt{h}. Mandatory keywords would also have the added benefice of being more clearly self-documented. In any case, the option of having routine \texttt{h} mean \texttt{nomutex} should be rejected since it is unsafe by default and may easily cause subtle errors.
109
110Furthermore, it is important to establish when mutex/nomutex may be used depending on type parameters.
111\begin{lstlisting}
112        int f01(monitor& mutex m);
113        int f02(const monitor& mutex m);
114        int f03(monitor* mutex m);
115        int f04(monitor* mutex * m);
116        int f05(monitor** mutex m);
117        int f06(monitor[10] mutex m);
118        int f07(monitor[] mutex m);
119        int f08(vector(monitor)& mutex m);
120        int f09(list(monitor)& mutex m);
121        int f10([monitor*, int]& mutex m);
122        int f11(graph(monitor*)& mutex m);
123\end{lstlisting}
124
125For the first routines it seems to make sense to support the mutex keyword for such small variations. The difference between pointers and reference (\texttt{f01} vs \texttt{f03}) or const and non-const (\texttt{f01} vs \texttt{f02}) has no significance to mutual exclusion. It may not always make sense to acquire the monitor when extra dereferences (\texttt{f04}, \texttt{f05}) are added but it is still technically feasible and the present of the explicit mutex keywork does make it very clear of the user's intentions. Passing in a known-sized array(\texttt{f06}) is also technically feasible but is close to the limits. Indeed, the size of the array is not actually enforced by the compiler and if replaced by a variable-sized array (\texttt{f07}) or a higher-level container (\texttt{f08}, \texttt{f09}) it becomes much more complex to properly acquire all the locks needed for such a complex critical section. This implicit acquisition also poses the question of what qualifies as a container. If the mutex keyword is supported on monitors stored inside of other types it can quickly become complex and unclear which monitor should be acquired and when. The extreme example of this is \texttt{f11} which takes a possibly cyclic graph of pointers to monitors. With such a routine signature the intuition of which monitors will be acquired on entry is lost. Where to draw the lines is up for debate but it seems reasonnable to consider \texttt{f03} as accepted and \texttt{f06} as rejected.
126
127\subsection{Data semantics}
128Once the call semantics are established, the next step is to establish data semantics. Indeed, until now a monitor is used simply as a generic handle but in most cases monitors contian shared data. This data should be intrinsic to the monitor declaration to prevent any accidental use of data without its appripriate protection. For example :
129\begin{lstlisting}
130        mutex struct counter_t {
131                int value;
132        };
133
134        void ?{}(counter_t& mutex this) {
135                this.cnt = 0;
136        }
137
138        int ++?(counter_t& mutex this) {
139                return ++this->value;
140        }
141
142        void ?{}(int* this, counter_t& mutex cnt) {
143                *this = (int)cnt;
144        }
145\end{lstlisting}
146\begin{tabular}{ c c }
148\begin{lstlisting}
149        void main(counter_t& mutex c) {
150                for(;;) {
151                        int count = c;
152                        sout | count | endl;
153                }
154        }
155\end{lstlisting}&\begin{lstlisting}
156        void main(counter_t& mutex c) {
157                for(;;) {
158                        ++c;
159                }
160        }
161
162\end{lstlisting}
163\end{tabular}
164\\
165
166
167This simple counter monitor offers an example of monitor usage. Notice how the counter is used without any explicit synchronisation and yet is perfectly safe reglardless of how many threads use it simultaneously. \\
168
169These simple mutual exclusion semantics also naturally expand to multi-monitor calls.
170\begin{lstlisting}
171        int f(MonitorA& mutex a, MonitorB& mutex b);
172
173        MonitorA a;
174        MonitorB b;
175        f(a,b);
176\end{lstlisting}
177This code acquires both locks before entering the critical section. In practice, writing multi-locking routines that can lead to deadlocks can be very tricky. Having language level support for such feature is therefore a significant asset for \CFA. However, as the this proposal shows, this does have significant repercussions relating to scheduling. The ability to acquire multiple monitors at the same time does incur a significant pitfall even without looking into scheduling. For example :
178\begin{lstlisting}
179        void foo(A& mutex a, B& mutex a) {
180                //...
181        }
182
183        void bar(A& mutex a, B& mutex a)
184                //...
185                foo(a, b);
186                //...
187        }
188\end{lstlisting}
189
190
191% Here, there is a language design choice that has to be made. It is impossible to protect the user from both barging and deadlocks and therefore this code has the potential to deadlock if some other threads try to acquire the locks in a different order (keep in mind that the lock ordering may be invisible or non-deterministic). The alternative is to allow the algorithm to release the lock on monitor \texttt{a}. This would effectively prevent the deadlock but could also mean that mutual exclusion may be dropped in the midle of routine \texttt{bar}.
192%
193% Indeed, there are two options for acquiring multiple locks while preventing deadlocks. The first option is to prescribe some arbitrary order of locking. If used consistently in the application this solution is both deadlock-free and barging-free. However, it also relies on the user to consistently follow the ordering when manually specifying the order. If the lock ordering is based on lock creation order or heap address ordering, it may be impossible for users to statically predict the correct lock acquiring order which means that deadlocks are a very real possibility. On the other hand, if the locking algorithm tries to dynamically find the correct lock ordering then it must release all locks after each wrong ordering attempts. This does not cause any significant issue in the context where a users tries to acquire multiple locks at once since the thread is not already in a critical section. However, if the thread was already holding a lock then releasing all locks on failed attempts may mean violating the mutual exclusion of the critical section. Notice that this is only an issue when nested mutex routines are used, in any other case monitors will behave consistently between both algorithms. Since releasing a lock in the middle of a critical section effectively violates mutual exclusion, it seems reasonnable to reject algorithms that dynamically guess the order of lock acquiring since users need to be very comfortable with multi-lock semantics before they can expect nested monitor calls to end-up releasing locks.
194
195
196\subsubsection{Internal scheduling}
197Monitors should also be able to do some sort of synchronization to be able to somewhat schedule what threads access it. Internal scheduling is one of the simple examples of such a feature. It allows users to declare condition variables and wait for them to be signaled. Here is a simple example of such a technique :
198
199\begin{lstlisting}
200        mutex struct A {
201                condition e;
202        }
203
204        void foo(A& mutex a) {
205                //...
206                wait(a.e);
207                //...
208        }
209
210        void bar(A& mutex a) {
211                signal(a.e);
212        }
213\end{lstlisting}
214
215Here routine \texttt{foo} will wait on the \texttt{signal} from \texttt{bar} before making further progress, effectively ensuring a basic ordering. However, nothing prevents users from miss-using this syntax and therefore some additionnal protection would be useful. For example, if \texttt{bar} was rewritten as follows:
216
217\begin{tabular}{ c c }
219\begin{lstlisting}
220        void foo(monitor& mutex m) {
221                //...
222                wait(m.e);
223                //...
224        }
225
226        foo(a);
227\end{lstlisting}&\begin{lstlisting}
228        void bar(monitor& mutex b, condition& e) {
229                signal(e);
230        }
231
232
233
234        bar(b, a.e);
235\end{lstlisting}
236\end{tabular}
237\\
238
239In this example, thread 2 tries to \texttt{signal} a condition variable for which it did not acquire the lock. There are at least two solutions to this problem. Either the wait routine tries to reacquire every needed lock upon exit or the signaller must implicitly transfer lock ownership to the signalled task. The first case can be easily implemented by hand and does not prevent any barging and therefore the second approach is preferred. This effectively means that condition variables need to be both aware of the locks used by the waiting task and the signaller. However, before we look at what this lock awareness means we need to look at another example to properly grasp the problem.
240
241\begin{tabular}{ c c }
243\begin{lstlisting}
244void foo(monitor& mutex m) {
245        //...
246        wait(m.e);
247        //...
248}
249
250foo(a);
251\end{lstlisting}&\begin{lstlisting}
252void bar(monitor& mutex a, monitor& mutex b) {
253        signal(a.e);
254}
255
256
257
258bar(a, b);
259\end{lstlisting}
260\end{tabular}
261\\
262
263Here, the issue is that even if thread 2 does hold the proper lock, it also holds an extra lock that must be delt with. The proposed solution is to make 2 changes to the condition variable declaration. First, the condition variable should be constructed with a reference to the monitor it syncrhonizes :
264
265\begin{lstlisting}
266        mutex struct A {
267                condition e;
268        }
269
270        void ?{}(A& this) {
271                &e{this};
272        }
273
274        void foo(A& mutex a) {
275                //...
276                wait(a.e);
277                //...
278        }
279
280        void bar(A& mutex a) {
281                signal(a.e);
282        }
283\end{lstlisting}
284
285By explicitly tying a condition variable to a particular monitor it is possible for the run-time to know which monitor needs to be signaled. This also enables run-time check to make sure that the proper context is acquired before trying to \texttt{signal} a condition variable. In this case, run time checks are probably sufficient since \texttt{signal} should be used inside a critical section and even though multi-threading applications are often non-deterministic, the inside of critical sections should be relatively reliable. This implementation of the condition variable object also means that the context of the dual monitor routine, the routine will hold-on to the monitor that is not referenced by the condition variable, i.e. :
286
287\begin{tabular}{ c c }
289\begin{lstlisting}
290void foo(monitor& mutex a,
291         monitor& mutex b) {
292        //...
293        wait(a.e); //releases a, holds b
294        //...
295}
296
297foo(a, b);
298\end{lstlisting}&\begin{lstlisting}
299void bar(monitor& mutex a) {
300        signal(a.e);
301}
302
303
304
305
306bar(a);
307\end{lstlisting}
308\end{tabular}
309\\
310
311The second aspect to this solution is the support for multi-monitor condition variables :
312\begin{lstlisting}
313monitor m1;
314monitor m2;
315condition2 e = {m1, m2};
316\end{lstlisting}
317\begin{tabular}{ c c }
319\begin{lstlisting}
320void foo(monitor& mutex a, monitor& mutex b) {
321        //...
322        wait(e); //releases a & b
323        //...
324}
325
326foo(a, b);
327\end{lstlisting}&\begin{lstlisting}
328void bar(monitor& mutex a, monitor& mutex b) {
329        signal(e);
330}
331
332
333
334bar(a, b);
335\end{lstlisting}
336\end{tabular}
337\\
338
339Notice here that the type used for the condition variable (\texttt{condition2}) explicitly states the number of monitors that will be synchronized at compile time. The risk with this condition variable semantics is that the user must be in a context where all monitors were properly acquired before waiting/signalling. This can be enforced by run-time checks but would be very difficult to statically enforce. An option that can be used to alleviate this risk is to have the signal routine acquire the monitors that were used to brand the condition variable. This guarantees that the proper locks will be transferred to the signaled but does inherit the risks the come with acquiring multiple locks some of the locks were already acquired.
340This would lead to an API similar to this :
341\begin{lstlisting}
342        //default ctor which brands the condition variable on construction
343        void ?{}(condition* this, monitor& brand);
344        void ^?{}(condition* this);
345
346        //copying an condition variable is forbidden
347        void ?{}(condition* this, const condition& other) = delete;
348        void ?=?(condition* this, const condition& other) = delete;
349
350        //releases branded locks and waits for signal
351        void wait(condition* this);
352
353        //acquires branded locks and transfers them to the signalled task
354        //(upon exit for signal and dirrectly for signalBlock)
355        void signal(condition* this);
356        void signalBlock(condition* this);
357\end{lstlisting}
358
359\subsection{External scheduling}
360As one might expect, the alternative to Internal scheduling is to use external scheduling instead. The goal of external scheduling is to be able to have the same scheduling power as internal scheduling without the requirement that any thread can acquire the monitor lock. This method is somewhat more robust to deadlocks since one of the threads keeps a relatively tight control on scheduling. External scheduling can generally be done either in terms of control flow (see \uC) or in terms of data (see Go). Of course, both of these paradigms have their own strenghts and weaknesses but for this project control flow semantics where chosen to stay consistent with the reset of the languages semantics. Two challenges specific to \CFA arise when trying to add external scheduling which is loose object definitions and multi-monitor routines.
361
362\subsubsection{Loose object definitions}
363In \uC monitor definitions include an exhaustive list of monitor operations :
364\begin{lstlisting}
365        _Monitor blarg {
366        public:
367                void f() { _Accept(g); }
368                void g();
369        private:
370        }
371\end{lstlisting}
372
373Since \CFA is not an object oriented it becomes much more difficult to implement but also much less clear for the user :
374
375\begin{lstlisting}
376        mutex struct A {};
377
378        void f(A& mutex a) { accept(g); }
379        void g(A& mutex a);
380\end{lstlisting}
381
382While this is the direct translation of the \uC code, at the time of compiling routine \texttt{f} the \CFA does not already have a declaration of \texttt{g} while the \uC compiler does. This means that either the compiler has to dynamically find which routines are "acceptable" or the language needs a way of statically listing "acceptable" routines. Since \CFA has no existing concept that resemble dynamic routine definitions or pattern matching, the static approach seems the more consistent with the current language paradigms. This approach leads to the \uC example being translated to :
383\begin{lstlisting}
384        accept( void g(mutex struct A& mutex a) )
385        mutex struct A {};
386
387        void f(A& mutex a) { accept(g); }
388        void g(A& mutex a);
389\end{lstlisting}
390
391This syntax is the most consistent with the language since it somewhat mimics the \texttt{forall} declarations. However, the fact that it comes before the struct declaration does means the type needs to be forward declared (done inline in the example). Here are a few alternatives to this syntax : \\
392\begin{tabular}[t]{l l}
393Alternative 1 & Alternative 2 \\
394\begin{lstlisting}
395mutex struct A
396accept( void g(A& mutex a) )
397{};
398\end{lstlisting}&\begin{lstlisting}
399mutex struct A {}
400accept( void g(A& mutex a) );
401
402\end{lstlisting} \\
403Alternative 3 & Alternative 4 \\
404\begin{lstlisting}
405mutex struct A {
406        accept( void g(A& mutex a) )
407};
408
409\end{lstlisting}&\begin{lstlisting}
410mutex struct A {
411        accept :
412                void g(A& mutex a) );
413};
414\end{lstlisting}
415\end{tabular}
416
417
418An other aspect to consider is what happens if multiple overloads of the same routine are used. For the time being it is assumed that multiple overloads of the same routine should be scheduled regardless of the overload used. However, this could easily be extended in the future.
419
420\subsubsection{Multi-monitor scheduling}
421
422External scheduling, like internal scheduling, becomes orders of magnitude more complex when we start introducing multi-monitor syntax. Even in the simplest possible case some new semantics need to be established :
423\begin{lstlisting}
424        accept( void f(mutex struct A& mutex this))
425        mutex struct A {};
426
427        mutex struct B {};
428
429        void g(A& mutex a, B& mutex b) {
430                accept(f); //ambiguous, which monitor
431        }
432\end{lstlisting}
433
434The obvious solution is to specify the correct monitor as follows :
435
436\begin{lstlisting}
437        accept( void f(mutex struct A& mutex this))
438        mutex struct A {};
439
440        mutex struct B {};
441
442        void g(A& mutex a, B& mutex b) {
443                accept( f, b );
444        }
445\end{lstlisting}
446
447This is unambiguous. The both locks will be acquired and kept, when routine \texttt{f} is called the lock for monitor \texttt{a} will be temporarily transferred from \texttt{g} to \texttt{f} (while \texttt{g} still holds lock \texttt{b}). This behavior can be extended to multi-monitor accept statment as follows.
448
449\begin{lstlisting}
450        accept( void f(mutex struct A& mutex, mutex struct A& mutex))
451        mutex struct A {};
452
453        mutex struct B {};
454
455        void g(A& mutex a, B& mutex b) {
456                accept( f, b, a );
457        }
458\end{lstlisting}
459
460Note that the set of monitors passed to the \texttt{accept} statement must be entirely contained in the set of monitor already acquired in the routine. \texttt{accept} used in any other context is Undefined Behaviour.
461
462\subsection{Implementation Details}
463\subsubsection{Interaction with polymorphism}
464At first glance, interaction between monitors and \CFA's concept of polymorphism seem complexe to support. However, it can be reasoned that entry-point locking can solve most of the issues that could be present with polymorphism.
465
466First of all, interaction between \texttt{otype} polymorphism and monitors is impossible since monitors do not support copying. Therefore the main question is how to support \texttt{dtype} polymorphism. We must remember that monitors' main purpose is to ensure mutual exclusion when accessing shared data. This implies that mutual exclusion is only required for routines that do in fact access shared data. However, since \texttt{dtype} polymorphism always handle incomplete types (by definition) no \texttt{dtype} polymorphic routine can access shared data since the data would require knowledge about the type. Therefore the only concern when combining \texttt{dtype} polymorphism and monitors is to protect access to routines. With callsite-locking, this would require significant amount of work since any \texttt{dtype} routine could have to obtain some lock before calling a routine. However, with entry-point-locking calling a monitor routine becomes exactly the same as calling it from anywhere else.
467
468\subsubsection{External scheduling queues}
469To support multi-monitor external scheduling means that some kind of entry-queues must be used that is aware of both monitors. However, acceptable routines must be aware of the entry queues which means they most be stored inside at least one of the monitors that will be acquired. This in turn adds the requirement a systematic algorithm of disambiguating which queue is relavant regardless of user ordering. The proposed algorithm is to fall back on monitors lock ordering and specify that the monitor that is acquired first is the lock with the relevant entry queue. This assumes that the lock acquiring order is static for the lifetime of all concerned objects gut that is a reasonnable contraint. This algorithm choice has two consequences, the ofthe highest priority monitor is no longer a true FIFO queue and the queue of the lowest priority monitor is both required and probably unused. The queue can no longer be a FIFO queue because instead of simply containing the waiting threads in order arrival, they also contain the second mutex. Therefore, another thread with the same highest priority monitor but a different lowest priority monitor may arrive first but enter the critical section after a thread with the correct pairing. Secondly, since it may not be known at compile time which monitor will be the lowest priority monitor, every monitor needs to have the correct queues even though it is probably that half the multi-monitor queues will go unused for the entire duration of the program.
470
471\section{Parrallelism}
472
474
475
476\section{Naming}
477
478\section{Future work}
479
480\section*{Acknowledgements}
481
482
483
484\bibliographystyle{plain}
485\bibliography{citations}
486
487
488\end{document}
Note: See TracBrowser for help on using the repository browser.