Context Navigation

← Previous Change
Next Change →

Changeset 41b8ea4 for doc/papers

Timestamp:

Oct 7, 2020, 5:10:45 PM (5 years ago)

Author:

Fangren Yu <f37yu@…>

Branches:

ADT, arm-eh, ast-experimental, enum, forall-pointer-decay, jacob/cs343-translation, master, new-ast-unique-expr, pthread-emulation, qualifiedEnum

Children:

490fb92e, 69c5c00

Parents:

2fb35df (diff), 848439f (diff)
Note: this is a merge changeset, the changes displayed below correspond to the merge itself.
Use the (diff) links above to see all the changes relative to each parent.

Message:

Merge branch 'master' of plg.uwaterloo.ca:software/cfa/cfa-cc into master

Location:

doc/papers/concurrency

Files:

: 1 added
: 3 edited

Paper.tex (modified) (17 diffs)
annex/local.bib (modified) (1 diff)
mail2 (modified) (1 diff)
response3 (added)

Legend:

: Unmodified
: Added
: Removed

doc/papers/concurrency/Paper.tex

-              r2fb35df
+              r41b8ea4
 {}
 \lstnewenvironment{C++}[1][]                            % use C++ style
 {\lstset{language=C++,moredelim=**[is][\protect\color{red}]{`}{`},#1}\lstset{#1}}
+{\lstset{language=C++,moredelim=**[is][\protect\color{red}]{`}{`}}\lstset{#1}}
 {}
 \lstnewenvironment{uC++}[1][]
 {\lstset{language=uC++,moredelim=**[is][\protect\color{red}]{`}{`},#1}\lstset{#1}}
+{\lstset{language=uC++,moredelim=**[is][\protect\color{red}]{`}{`}}\lstset{#1}}
 {}
 \lstnewenvironment{Go}[1][]
 {\lstset{language=Golang,moredelim=**[is][\protect\color{red}]{`}{`},#1}\lstset{#1}}
+{\lstset{language=Golang,moredelim=**[is][\protect\color{red}]{`}{`}}\lstset{#1}}
 {}
 \lstnewenvironment{python}[1][]
 {\lstset{language=python,moredelim=**[is][\protect\color{red}]{`}{`},#1}\lstset{#1}}
+{\lstset{language=python,moredelim=**[is][\protect\color{red}]{`}{`}}\lstset{#1}}
 {}
 \lstnewenvironment{java}[1][]
 {\lstset{language=java,moredelim=**[is][\protect\color{red}]{`}{`},#1}\lstset{#1}}
+{\lstset{language=java,moredelim=**[is][\protect\color{red}]{`}{`}}\lstset{#1}}
 {}
 …
 \begin{document}
 \linenumbers                            % comment out to turn off line numbering
+%\linenumbers                           % comment out to turn off line numbering
 \maketitle
 …
 \label{s:RuntimeStructureCluster}
 A \newterm{cluster} is a collection of user and kernel threads, where the kernel threads run the user threads from the cluster's ready queue, and the operating system runs the kernel threads on the processors from its ready queue.
+A \newterm{cluster} is a collection of user and kernel threads, where the kernel threads run the user threads from the cluster's ready queue, and the operating system runs the kernel threads on the processors from its ready queue~\cite{Buhr90a}.
 The term \newterm{virtual processor} is introduced as a synonym for kernel thread to disambiguate between user and kernel thread.
 From the language perspective, a virtual processor is an actual processor (core).
 …
 \end{cfa}
 where CPU time in nanoseconds is from the appropriate language clock.
+Each benchmark is performed @N@ times, where @N@ is selected so the benchmark runs in the range of 2--20 seconds for the specific programming language.
+Each benchmark is performed @N@ times, where @N@ is selected so the benchmark runs in the range of 2--20 seconds for the specific programming language;
+each @N@ appears after the experiment name in the following tables.
 The total time is divided by @N@ to obtain the average time for a benchmark.
 Each benchmark experiment is run 13 times and the average appears in the table.
+For languages with a runtime JIT (Java, Node.js, Python), a single half-hour long experiment is run to check stability;
+all long-experiment results are statistically equivalent, \ie median/average/standard-deviation correlate with the short-experiment results, indicating the short experiments reached a steady state.
 All omitted tests for other languages are functionally identical to the \CFA tests and available online~\cite{CforallConcurrentBenchmarks}.
-% tar --exclude-ignore=exclude -cvhf benchmark.tar benchmark
-% cp -p benchmark.tar /u/cforall/public_html/doc/concurrent_benchmark.tar
 \paragraph{Creation}
 …
 \begin{multicols}{2}
+\lstset{language=CFA,moredelim=**[is][\color{red}]{@}{@},deletedelim=**[is][]{`}{`}}
+\begin{cfa}
+@coroutine@ MyCoroutine {};
+\begin{cfa}[xleftmargin=0pt]
+`coroutine` MyCoroutine {};
 void ?{}( MyCoroutine & this ) {
 #ifdef EAGER
 …
 void main( MyCoroutine & ) {}
 int main() {
         BENCH( for ( N ) { @MyCoroutine c;@ } )
+        BENCH( for ( N ) { `MyCoroutine c;` } )
         sout | result;
+}
 …
 \begin{tabular}[t]{@{}r*{3}{D{.}{.}{5.2}}@{}}
 \multicolumn{1}{@{}c}{} & \multicolumn{1}{c}{Median} & \multicolumn{1}{c}{Average} & \multicolumn{1}{c@{}}{Std Dev} \\
 \CFA generator                  & 0.6           & 0.6           & 0.0           \\
 \CFA coroutine lazy             & 13.4          & 13.1          & 0.5           \\
 \CFA coroutine eager    & 144.7         & 143.9         & 1.5           \\
 \CFA thread                             & 466.4         & 468.0         & 11.3          \\
 \uC coroutine                   & 155.6         & 155.7         & 1.7           \\
 \uC thread                              & 523.4         & 523.9         & 7.7           \\
 Python generator                & 123.2         & 124.3         & 4.1           \\
 Node.js generator               & 33.4          & 33.5          & 0.3           \\
 Goroutine thread                & 751.0         & 750.5         & 3.1           \\
 Rust tokio thread               & 1860.0        & 1881.1        & 37.6          \\
 Rust thread                             & 53801.0       & 53896.8       & 274.9         \\
 Java thread (   10 000)         & 119256.0      & 119679.2      & 2244.0        \\
 Java thread (1 000 000)         & 123100.0      & 123052.5      & 751.6         \\
 Pthreads thread                 & 31465.5       & 31419.5       & 140.4
+\multicolumn{1}{@{}r}{N\hspace*{10pt}} & \multicolumn{1}{c}{Median} & \multicolumn{1}{c}{Average} & \multicolumn{1}{c@{}}{Std Dev} \\
+\CFA generator (1B)                     & 0.6           & 0.6           & 0.0           \\
+\CFA coroutine lazy     (100M)  & 13.4          & 13.1          & 0.5           \\
+\CFA coroutine eager (10M)      & 144.7         & 143.9         & 1.5           \\
+\CFA thread (10M)                       & 466.4         & 468.0         & 11.3          \\
+\uC coroutine (10M)                     & 155.6         & 155.7         & 1.7           \\
+\uC thread (10M)                        & 523.4         & 523.9         & 7.7           \\
+Python generator (10M)          & 123.2         & 124.3         & 4.1           \\
+Node.js generator (10M)         & 33.4          & 33.5          & 0.3           \\
+Goroutine thread (10M)          & 751.0         & 750.5         & 3.1           \\
+Rust tokio thread (10M)         & 1860.0        & 1881.1        & 37.6          \\
+Rust thread     (250K)                  & 53801.0       & 53896.8       & 274.9         \\
+Java thread (250K)                      & 119256.0      & 119679.2      & 2244.0        \\
+% Java thread (1 000 000)               & 123100.0      & 123052.5      & 751.6         \\
+Pthreads thread (250K)          & 31465.5       & 31419.5       & 140.4
 \end{tabular}
 \end{multicols}
 …
 Internal scheduling is measured using a cycle of two threads signalling and waiting.
 Figure~\ref{f:schedint} shows the code for \CFA, with results in Table~\ref{t:schedint}.
+Note, the incremental cost of bulk acquire for \CFA, which is largely a fixed cost for small numbers of mutex objects.
+Java scheduling is significantly greater because the benchmark explicitly creates multiple threads in order to prevent the JIT from making the program sequential, \ie removing all locking.
+Note, the \CFA incremental cost for bulk acquire is a fixed cost for small numbers of mutex objects.
+User-level threading has one kernel thread, eliminating contention between the threads (direct handoff of the kernel thread).
+Kernel-level threading has two kernel threads allowing some contention.
 \begin{multicols}{2}
 \lstset{language=CFA,moredelim=**[is][\color{red}]{@}{@},deletedelim=**[is][]{`}{`}}
 \begin{cfa}
+\setlength{\tabcolsep}{3pt}
+\begin{cfa}[xleftmargin=0pt]
 volatile int go = 0;
+@condition c;@
 @monitor@ M {} m1/*, m2, m3, m4*/;
 void call( M & @mutex p1/*, p2, p3, p4*/@ ) {
         @signal( c );@
+}
 void wait( M & @mutex p1/*, p2, p3, p4*/@ ) {
+`condition c;`
+`monitor` M {} m1/*, m2, m3, m4*/;
+void call( M & `mutex p1/*, p2, p3, p4*/` ) {
+        `signal( c );`
+}
+void wait( M & `mutex p1/*, p2, p3, p4*/` ) {
         go = 1; // continue other thread
         for ( N ) { @wait( c );@ } );
+        for ( N ) { `wait( c );` } );
+}
 thread T {};
 …
 \begin{tabular}{@{}r*{3}{D{.}{.}{5.2}}@{}}
 \multicolumn{1}{@{}c}{} & \multicolumn{1}{c}{Median} & \multicolumn{1}{c}{Average} & \multicolumn{1}{c@{}}{Std Dev} \\
 \CFA @signal@, 1 monitor        & 364.4         & 364.2         & 4.4           \\
 \CFA @signal@, 2 monitor        & 484.4         & 483.9         & 8.8           \\
 \CFA @signal@, 4 monitor        & 709.1         & 707.7         & 15.0          \\
 \uC @signal@ monitor            & 328.3         & 327.4         & 2.4           \\
 Rust cond. variable                     & 7514.0        & 7437.4        & 397.2         \\
 Java @notify@ monitor (  1 000 000)             & 8717.0        & 8774.1        & 471.8         \\
 Java @notify@ monitor (100 000 000)             & 8634.0        & 8683.5        & 330.5         \\
 Pthreads cond. variable         & 5553.7        & 5576.1        & 345.6
+\multicolumn{1}{@{}r}{N\hspace*{10pt}} & \multicolumn{1}{c}{Median} & \multicolumn{1}{c}{Average} & \multicolumn{1}{c@{}}{Std Dev} \\
+\CFA @signal@, 1 monitor (10M)  & 364.4         & 364.2         & 4.4           \\
+\CFA @signal@, 2 monitor (10M)  & 484.4         & 483.9         & 8.8           \\
+\CFA @signal@, 4 monitor (10M)  & 709.1         & 707.7         & 15.0          \\
+\uC @signal@ monitor (10M)              & 328.3         & 327.4         & 2.4           \\
+Rust cond. variable     (1M)            & 7514.0        & 7437.4        & 397.2         \\
+Java @notify@ monitor (1M)              & 8717.0        & 8774.1        & 471.8         \\
+% Java @notify@ monitor (100 000 000)           & 8634.0        & 8683.5        & 330.5         \\
+Pthreads cond. variable (1M)    & 5553.7        & 5576.1        & 345.6
 \end{tabular}
 \end{multicols}
 …
 External scheduling is measured using a cycle of two threads calling and accepting the call using the @waitfor@ statement.
 Figure~\ref{f:schedext} shows the code for \CFA with results in Table~\ref{t:schedext}.
 Note, the incremental cost of bulk acquire for \CFA, which is largely a fixed cost for small numbers of mutex objects.
+Note, the \CFA incremental cost for bulk acquire is a fixed cost for small numbers of mutex objects.
 \begin{multicols}{2}
 \lstset{language=CFA,moredelim=**[is][\color{red}]{@}{@},deletedelim=**[is][]{`}{`}}
+\setlength{\tabcolsep}{5pt}
 \vspace*{-16pt}
 \begin{cfa}
 @monitor@ M {} m1/*, m2, m3, m4*/;
 void call( M & @mutex p1/*, p2, p3, p4*/@ ) {}
 void wait( M & @mutex p1/*, p2, p3, p4*/@ ) {
         for ( N ) { @waitfor( call : p1/*, p2, p3, p4*/ );@ }
+\begin{cfa}[xleftmargin=0pt]
+`monitor` M {} m1/*, m2, m3, m4*/;
+void call( M & `mutex p1/*, p2, p3, p4*/` ) {}
+void wait( M & `mutex p1/*, p2, p3, p4*/` ) {
+        for ( N ) { `waitfor( call : p1/*, p2, p3, p4*/ );` }
+}
 thread T {};
 …
 \columnbreak
 \vspace*{-16pt}
+\vspace*{-18pt}
 \captionof{table}{External-scheduling comparison (nanoseconds)}
 \label{t:schedext}
 \begin{tabular}{@{}r*{3}{D{.}{.}{3.2}}@{}}
 \multicolumn{1}{@{}c}{} & \multicolumn{1}{c}{Median} &\multicolumn{1}{c}{Average} & \multicolumn{1}{c@{}}{Std Dev} \\
 \CFA @waitfor@, 1 monitor       & 367.1 & 365.3 & 5.0   \\
 \CFA @waitfor@, 2 monitor       & 463.0 & 464.6 & 7.1   \\
 \CFA @waitfor@, 4 monitor       & 689.6 & 696.2 & 21.5  \\
 \uC \lstinline[language=uC++]|_Accept| monitor  & 328.2 & 329.1 & 3.4   \\
 Go \lstinline[language=Golang]|select| channel  & 365.0 & 365.5 & 1.2
+\multicolumn{1}{@{}r}{N\hspace*{10pt}} & \multicolumn{1}{c}{Median} &\multicolumn{1}{c}{Average} & \multicolumn{1}{c@{}}{Std Dev} \\
+\CFA @waitfor@, 1 monitor (10M) & 367.1 & 365.3 & 5.0   \\
+\CFA @waitfor@, 2 monitor (10M) & 463.0 & 464.6 & 7.1   \\
+\CFA @waitfor@, 4 monitor (10M) & 689.6 & 696.2 & 21.5  \\
+\uC \lstinline[language=uC++]|_Accept| monitor (10M)    & 328.2 & 329.1 & 3.4   \\
+Go \lstinline[language=Golang]|select| channel (10M)    & 365.0 & 365.5 & 1.2
 \end{tabular}
 \end{multicols}
 …
 \begin{multicols}{2}
 \lstset{language=CFA,moredelim=**[is][\color{red}]{@}{@},deletedelim=**[is][]{`}{`}}
 \begin{cfa}
 @monitor@ M {} m1/*, m2, m3, m4*/;
 call( M & @mutex p1/*, p2, p3, p4*/@ ) {}
+\setlength{\tabcolsep}{3pt}
+\begin{cfa}[xleftmargin=0pt]
+`monitor` M {} m1/*, m2, m3, m4*/;
+call( M & `mutex p1/*, p2, p3, p4*/` ) {}
 int main() {
         BENCH( for( N ) call( m1/*, m2, m3, m4*/ ); )
 …
 \label{t:mutex}
 \begin{tabular}{@{}r*{3}{D{.}{.}{3.2}}@{}}
 \multicolumn{1}{@{}c}{} & \multicolumn{1}{c}{Median} &\multicolumn{1}{c}{Average} & \multicolumn{1}{c@{}}{Std Dev} \\
 test-and-test-set lock                  & 19.1  & 18.9  & 0.4   \\
 \CFA @mutex@ function, 1 arg.   & 48.3  & 47.8  & 0.9   \\
 \CFA @mutex@ function, 2 arg.   & 86.7  & 87.6  & 1.9   \\
 \CFA @mutex@ function, 4 arg.   & 173.4 & 169.4 & 5.9   \\
 \uC @monitor@ member rtn.               & 54.8  & 54.8  & 0.1   \\
 Goroutine mutex lock                    & 34.0  & 34.0  & 0.0   \\
 Rust mutex lock                                 & 33.0  & 33.2  & 0.8   \\
 Java synchronized method (   100 000 000)               & 31.0  & 30.9  & 0.5   \\
 Java synchronized method (10 000 000 000)               & 31.0 & 30.2 & 0.9 \\
 Pthreads mutex Lock                             & 31.0  & 31.1  & 0.4
+\multicolumn{1}{@{}r}{N\hspace*{10pt}} & \multicolumn{1}{c}{Median} &\multicolumn{1}{c}{Average} & \multicolumn{1}{c@{}}{Std Dev} \\
+test-and-test-set lock (50M)            & 19.1  & 18.9  & 0.4   \\
+\CFA @mutex@ function, 1 arg. (50M)     & 48.3  & 47.8  & 0.9   \\
+\CFA @mutex@ function, 2 arg. (50M)     & 86.7  & 87.6  & 1.9   \\
+\CFA @mutex@ function, 4 arg. (50M)     & 173.4 & 169.4 & 5.9   \\
+\uC @monitor@ member rtn. (50M)         & 54.8  & 54.8  & 0.1   \\
+Goroutine mutex lock (50M)                      & 34.0  & 34.0  & 0.0   \\
+Rust mutex lock (50M)                           & 33.0  & 33.2  & 0.8   \\
+Java synchronized method (50M)          & 31.0  & 30.9  & 0.5   \\
+% Java synchronized method (10 000 000 000)             & 31.0 & 30.2 & 0.9 \\
+Pthreads mutex Lock (50M)                       & 31.0  & 31.1  & 0.4
 \end{tabular}
 \end{multicols}
 …
 \begin{multicols}{2}
+\lstset{language=CFA,moredelim=**[is][\color{red}]{@}{@},deletedelim=**[is][]{`}{`}}
+\begin{cfa}[aboveskip=0pt,belowskip=0pt]
+@coroutine@ C {};
+void main( C & ) { for () { @suspend;@ } }
+\begin{cfa}[xleftmargin=0pt]
+`coroutine` C {};
+void main( C & ) { for () { `suspend;` } }
 int main() { // coroutine test
         C c;
         BENCH( for ( N ) { @resume( c );@ } )
+        BENCH( for ( N ) { `resume( c );` } )
         sout | result;
+}
 int main() { // thread test
         BENCH( for ( N ) { @yield();@ } )
+        BENCH( for ( N ) { `yield();` } )
         sout | result;
+}
 …
 \label{t:ctx-switch}
 \begin{tabular}{@{}r*{3}{D{.}{.}{3.2}}@{}}
 \multicolumn{1}{@{}c}{} & \multicolumn{1}{c}{Median} &\multicolumn{1}{c}{Average} & \multicolumn{1}{c@{}}{Std Dev} \\
 C function                      & 1.8           & 1.8           & 0.0   \\
 \CFA generator          & 1.8           & 2.0           & 0.3   \\
 \CFA coroutine          & 32.5          & 32.9          & 0.8   \\
 \CFA thread                     & 93.8          & 93.6          & 2.2   \\
 \uC coroutine           & 50.3          & 50.3          & 0.2   \\
 \uC thread                      & 97.3          & 97.4          & 1.0   \\
 Python generator        & 40.9          & 41.3          & 1.5   \\
 Node.js await           & 1852.2        & 1854.7        & 16.4  \\
 Node.js generator       & 33.3          & 33.4          & 0.3   \\
 Goroutine thread        & 143.0         & 143.3         & 1.1   \\
 Rust async await        & 32.0          & 32.0          & 0.0   \\
 Rust tokio thread       & 143.0         & 143.0         & 1.7   \\
 Rust thread                     & 332.0         & 331.4         & 2.4   \\
 Java thread     (      100 000)         & 405.0         & 415.0         & 17.6  \\
 Java thread (  100 000 000)                     & 413.0 & 414.2 & 6.2 \\
 Java thread (5 000 000 000)                     & 415.0 & 415.2 & 6.1 \\
 Pthreads thread         & 334.3         & 335.2         & 3.9
+\multicolumn{1}{@{}r}{N\hspace*{10pt}} & \multicolumn{1}{c}{Median} &\multicolumn{1}{c}{Average} & \multicolumn{1}{c@{}}{Std Dev} \\
+C function (10B)                        & 1.8           & 1.8           & 0.0   \\
+\CFA generator (5B)                     & 1.8           & 2.0           & 0.3   \\
+\CFA coroutine (100M)           & 32.5          & 32.9          & 0.8   \\
+\CFA thread (100M)                      & 93.8          & 93.6          & 2.2   \\
+\uC coroutine (100M)            & 50.3          & 50.3          & 0.2   \\
+\uC thread (100M)                       & 97.3          & 97.4          & 1.0   \\
+Python generator (100M)         & 40.9          & 41.3          & 1.5   \\
+Node.js await (5M)                      & 1852.2        & 1854.7        & 16.4  \\
+Node.js generator (100M)        & 33.3          & 33.4          & 0.3   \\
+Goroutine thread (100M)         & 143.0         & 143.3         & 1.1   \\
+Rust async await (100M)         & 32.0          & 32.0          & 0.0   \\
+Rust tokio thread (100M)        & 143.0         & 143.0         & 1.7   \\
+Rust thread (25M)                       & 332.0         & 331.4         & 2.4   \\
+Java thread (100M)                      & 405.0         & 415.0         & 17.6  \\
+% Java thread (  100 000 000)                   & 413.0 & 414.2 & 6.2 \\
+% Java thread (5 000 000 000)                   & 415.0 & 415.2 & 6.1 \\
+Pthreads thread (25M)           & 334.3         & 335.2         & 3.9
 \end{tabular}
 \end{multicols}
 …
 Languages using 1:1 threading based on pthreads can at best meet or exceed, due to language overhead, the pthread results.
 Note, pthreads has a fast zero-contention mutex lock checked in user space.
+Languages with M:N threading have better performance than 1:1 because there is no operating-system interactions.
+Languages with M:N threading have better performance than 1:1 because there is no operating-system interactions (context-switching or locking).
+As well, for locking experiments, M:N threading has less contention if only one kernel thread is used.
 Languages with stackful coroutines have higher cost than stackless coroutines because of stack allocation and context switching;
 however, stackful \uC and \CFA coroutines have approximately the same performance as stackless Python and Node.js generators.
 The \CFA stackless generator is approximately 25 times faster for suspend/resume and 200 times faster for creation than stackless Python and Node.js generators.
+The Node.js context-switch is costly when asynchronous await must enter the event engine because a promise is not fulfilled.
+Finally, the benchmark results correlate across programming languages with and without JIT, indicating the JIT has completed any runtime optimizations.
 …
 The authors recognize the design assistance of Aaron Moss, Rob Schluntz, Andrew Beach, and Michael Brooks; David Dice for commenting and helping with the Java benchmarks; and Gregor Richards for helping with the Node.js benchmarks.
 This research is funded by a grant from Waterloo-Huawei (\url{http://www.huawei.com}) Joint Innovation Lab. %, and Peter Buhr is partially funded by the Natural Sciences and Engineering Research Council of Canada.
+This research is funded by the NSERC/Waterloo-Huawei (\url{http://www.huawei.com}) Joint Innovation Lab. %, and Peter Buhr is partially funded by the Natural Sciences and Engineering Research Council of Canada.
 {%

doc/papers/concurrency/annex/local.bib

r2fb35df	r41b8ea4
59	59	@manual{Cpp-Transactions,
60	60	keywords = {C++, Transactional Memory},
61		title = {Tech~~nical Specification~~ for C++ Extensions for Transactional Memory},
	61	title = {Tech. Spec. for C++ Extensions for Transactional Memory},
62	62	organization= {International Standard ISO/IEC TS 19841:2015 },
63	63	publisher = {American National Standards Institute},

doc/papers/concurrency/mail2

-              r2fb35df
+              r41b8ea4
 Software: Practice and Experience Editorial Office
+Date: Wed, 2 Sep 2020 20:55:34 +0000
+From: Richard Jones <onbehalfof@manuscriptcentral.com>
+Reply-To: R.E.Jones@kent.ac.uk
+To: tdelisle@uwaterloo.ca, pabuhr@uwaterloo.ca
+Subject: Software: Practice and Experience - Decision on Manuscript ID
+ SPE-19-0219.R2
+-Sep-2020
+Dear Dr Buhr,
+Many thanks for submitting SPE-19-0219.R2 entitled "Advanced Control-flow and Concurrency in Cforall" to Software: Practice and Experience. The paper has now been reviewed and the comments of the referees are included at the bottom of this letter. I apologise for the length of time it has taken to get these.
+Both reviewers consider this paper to be close to acceptance. However, before I can accept this paper, I would like you address the comments of Reviewer 2, particularly with regard to the description of the adaptation Java harness to deal with warmup. I would expect to see a convincing argument that the computation has reached a steady state. I would also like you to provide the values for N for each benchmark run. This should be very straightforward for you to do. There are a couple of papers on steady state that you may wish to consult (though I am certainly not pushing my own work).
+) Barrett, Edd; Bolz-Tereick, Carl Friedrich; Killick, Rebecca; Mount, Sarah and Tratt, Laurence. Virtual Machine Warmup Blows Hot and Cold. OOPSLA 2017. https://doi.org/10.1145/3133876
+Virtual Machines (VMs) with Just-In-Time (JIT) compilers are traditionally thought to execute programs in two phases: the initial warmup phase determines which parts of a program would most benefit from dynamic compilation, before JIT compiling those parts into machine code; subsequently the program is said to be at a steady state of peak performance. Measurement methodologies almost always discard data collected during the warmup phase such that reported measurements focus entirely on peak performance. We introduce a fully automated statistical approach, based on changepoint analysis, which allows us to determine if a program has reached a steady state and, if so, whether that represents peak performance or not. Using this, we show that even when run in the most controlled of circumstances, small, deterministic, widely studied microbenchmarks often fail to reach a steady state of peak performance on a variety of common VMs. Repeating our experiment on 3 different machines, we found that at most 43.5% of pairs consistently reach a steady state of peak performance.
+) Kalibera, Tomas and Jones, Richard. Rigorous Benchmarking in Reasonable Time. ISMM  2013. https://doi.org/10.1145/2555670.2464160
+Experimental evaluation is key to systems research. Because modern systems are complex and non-deterministic, good experimental methodology demands that researchers account for uncertainty. To obtain valid results, they are expected to run many iterations of benchmarks, invoke virtual machines (VMs) several times, or even rebuild VM or benchmark binaries more than once. All this repetition costs time to complete experiments. Currently, many evaluations give up on sufficient repetition or rigorous statistical methods, or even run benchmarks only in training sizes. The results reported often lack proper variation estimates and, when a small difference between two systems is reported, some are simply unreliable.In contrast, we provide a statistically rigorous methodology for repetition and summarising results that makes efficient use of experimentation time. Time efficiency comes from two key observations. First, a given benchmark on a given platform is typically prone to much less non-determinism than the common worst-case of published corner-case studies. Second, repetition is most needed where most uncertainty arises (whether between builds, between executions or between iterations). We capture experimentation cost with a novel mathematical model, which we use to identify the number of repetitions at each level of an experiment necessary and sufficient to obtain a given level of precision.We present our methodology as a cookbook that guides researchers on the number of repetitions they should run to obtain reliable results. We also show how to present results with an effect size confidence interval. As an example, we show how to use our methodology to conduct throughput experiments with the DaCapo and SPEC CPU benchmarks on three recent platforms.
+You have 42 days from the date of this email to submit your revision. If you are unable to complete the revision within this time, please contact me to request a short extension.
+You can upload your revised manuscript and submit it through your Author Center. Log into https://mc.manuscriptcentral.com/spe and enter your Author Center, where you will find your manuscript title listed under "Manuscripts with Decisions".
+When submitting your revised manuscript, you will be able to respond to the comments made by the referee(s) in the space provided.  You can use this space to document any changes you make to the original manuscript.
+If you would like help with English language editing, or other article preparation support, Wiley Editing Services offers expert help with English Language Editing, as well as translation, manuscript formatting, and figure formatting at www.wileyauthors.com/eeo/preparation. You can also check out our resources for Preparing Your Article for general guidance about writing and preparing your manuscript at www.wileyauthors.com/eeo/prepresources.
+Once again, thank you for submitting your manuscript to Software: Practice and Experience. I look forward to receiving your revision.
+Sincerely,
+Richard
+Prof. Richard Jones
+Editor, Software: Practice and Experience
+R.E.Jones@kent.ac.uk
+Referee(s)' Comments to Author:
+Reviewing: 1
+Comments to the Author
+Overall, I felt that this draft was an improvement on previous drafts and I don't have further changes to request.
+I appreciated the new language to clarify the relationship of external and internal scheduling, for example, as well as the new measurements of Rust tokio. Also, while I still believe that the choice between thread/generator/coroutine and so forth could be made crisper and clearer, the current draft of Section 2 did seem adequate to me in terms of specifying the considerations that users would have to take into account to make the choice.
+Reviewing: 2
+Comments to the Author
+First: let me apologise for the delay on this review. I'll blame the global pandemic combined with my institution's senior management's counterproductive decisions for taking up most of my time and all of my energy.
+At this point, reading the responses, I think we've been around the course enough times that further iteration is unlikely to really improve the paper any further, so I'm happy to recommend acceptance.    My main comments are that there were some good points in the responses to *all* the reviews and I strongly encourage the authors to incorporate those discursive responses into the final paper so they may benefit readers as well as reviewers.   I agree with the recommendations of reviewer #2 that the paper could usefully be split in to two, which I think I made to a previous revision, but I'm happy to leave that decision to the Editor.
+Finally, the paper needs to describe how the Java harness was adapted to deal with warmup; why the computation has warmed up and reached a steady state - similarly for js and Python. The tables should also give the "N" chosen for each benchmark run.
+minor points
+* don't start sentences with "However"
+* most downloaded isn't an "Award"
+Date: Thu, 1 Oct 2020 05:34:29 +0000
+From: Richard Jones <onbehalfof@manuscriptcentral.com>
+Reply-To: R.E.Jones@kent.ac.uk
+To: pabuhr@uwaterloo.ca
+Subject: Revision reminder - SPE-19-0219.R2
+-Oct-2020
+Dear Dr Buhr
+SPE-19-0219.R2
+This is a reminder that your opportunity to revise and re-submit your manuscript will expire 14 days from now. If you require more time please contact me directly and I may grant an extension to this deadline, otherwise the option to submit a revision online, will not be available.
+If your article is of potential interest to the general public, (which means it must be timely, groundbreaking, interesting and impact on everyday society) then please e-mail ejp@wiley.co.uk explaining the public interest side of the research. Wiley will then investigate the potential for undertaking a global press campaign on the article.
+I look forward to receiving your revision.
+Sincerely,
+Prof. Richard Jones
+Editor, Software: Practice and Experience
+https://mc.manuscriptcentral.com/spe
+Date: Tue, 6 Oct 2020 15:29:41 +0000
+From: Mayank Roy Chowdhury <onbehalfof@manuscriptcentral.com>
+Reply-To: speoffice@wiley.com
+To: tdelisle@uwaterloo.ca, pabuhr@uwaterloo.ca
+Subject: SPE-19-0219.R3 successfully submitted
+-Oct-2020
+Dear Dr Buhr,
+Your manuscript entitled "Advanced Control-flow and Concurrency in Cforall" has been successfully submitted online and is presently being given full consideration for publication in Software: Practice and Experience.
+Your manuscript number is SPE-19-0219.R3.  Please mention this number in all future correspondence regarding this submission.
+You can view the status of your manuscript at any time by checking your Author Center after logging into https://mc.manuscriptcentral.com/spe.  If you have difficulty using this site, please click the 'Get Help Now' link at the top right corner of the site.
+Thank you for submitting your manuscript to Software: Practice and Experience.
+Sincerely,
+Software: Practice and Experience Editorial Office

Note: See TracChangeset for help on using the changeset viewer.