source: doc/papers/concurrency/response @ 1967109

    A revised version of your manuscript that takes into account the comments
    of the referees will be reconsidered for publication.

We have attempted to address all the referee's comments in the revised version
of the paper, with notes below for each comment.

=============================================================================

    Reviewing: 1

    As far as I can tell, the article contains three main ideas: an
    asynchronous execution / threading model; a model for monitors to provide
    mutual exclusion; and an implementation.  The first two ideas are drawn
    together in Table 1: unfortunately this is on page 25 of 30 pages of
    text. Implementation choices and descriptions are scattered throughout the
    paper - and the sectioning of the paper seems almost arbitrary.

Fixed, Table 1 is moved to the start and explained in detail.

    The article is about its contributions.  Simply adding feature X to
    language Y isn't by itself a contribution, (when feature X isn't already a
    contribution).

C++ (Y) added object-oriented programming (X) to C, where OO programming (X)
was not a contribution.

    For example: why support two kinds of generators as well as user-level
    threads?  Why support both low and high level synchronization constructs?

Fixed, as part of discussing Table 1.

    Similarly I would have found the article easier to follow if it was written
    top down, presenting the design principles, present the space of language
    features, justify chosen language features (and rationale) and those
    excluded, and then present implementation, and performance.

Fixed, the paper is now restructured in this form.

    Then the writing of the article is often hard to follow, to say the
    least. Two examples: section 3 "stateful functions" - I've some idea
    what that is (a function with Algol's "own" or C's "static" variables?
    but in fact the paper has a rather more specific idea than that.

Fixed, at the start of this section.

    The top of page 3 throws a whole lot of definitions at the reader
    "generator" "coroutine" "stackful" "stackless" "symmetric" "asymmetric"
    without every stopping to define each one

Hopefully fixed by moving Table 1 forward.

    --- but then in footnote "C" takes the time to explain what C's "main"
    function is? I cannot imagine a reader of this paper who doesn't know what
    "main" is in C; especially if they understand the other concepts already
    presented in the paper.

Fixed by shortening.

    The start of section 3 then does the same
    thing: putting up a whole lot of definitions, making distinctions and
    comparisons, even talking about some runtime details, but the critical
    definition of a monitor doesn't appear until three pages later, at the
    start of section 5 on p15, lines 29-34 are a good, clear, description
    of what a monitor actually is.  That needs to come first, rather than
    being buried again after two sections of comparisons, discussions,
    implementations, and options that are ungrounded because they haven't
    told the reader what they are actually talking about.  First tell the
    reader what something is, then how they might use it (as programmers:
    what are the rules and restrictions) and only then start comparison
    with other things, other approaches, other languages, or
    implementations.

Hopefully fixed by moving Table 1 forward.

    The description of the implementation is similarly lost in the trees
    without ever really seeing the wood. Figure 19 is crucial here, but
    it's pretty much at the end of the paper, and comments about
    implementations are threaded throughout the paper without the context
    (fig 19) to understand what's going on.

We have to agree to disagree on the location of Fig 19. Early discussion about
implementation for the various control structures are specific to that feature.
Fig 19 shows the global runtime structure, which manages only the threading
aspect of the control structures and their global organization.

    The protocol for performance testing may just about suffice for C (although
    is N constantly ten million, or does it vary for each benchmark)

Fixed, the paper states N varies per language/benchmark so the benchmark runs
long enough to get a good average per operation.

    but such evaluation isn't appropriate for garbage-collected or JITTed
    languages like Java or Go.

Please explain. All the actions in the benchmarks occur independently of the
storage-management scheme, e.g., acquiring a lock is an aspect of execution not
storage. In fact, garbage-collected or JITTed languages cheat on benchmarks and
we had to take great care to prevent cheating and measure the actual operation.

    p1 only a subset of C-forall extensions?

Fixed, removed.

    p1 "has features often associated with object-oriented programming
    languages, such as constructors, destructors, virtuals and simple
    inheritance."  There's no need to quibble about this. Once a language has
    inheritance, it's hard to claim it's not object-oriented.

We have to agree to disagree. Object languages are defined by the notion of
nested functions in a aggregate structure with a special receiver parameter
"this", not by inheritance.  Inheritance is a polymorphic mechanism, e.g,
Plan-9 C has simple inheritance but is not object-oriented. Because Cforall
does not have a specific receiver, it is possible to have multiple function
parameters as receivers, which introduces new concepts like bulk acquire for
monitors.

    p2 barging? signals-as-hints?

Added a footnote for barging. We feel these terms are well known in the
concurrency literature, especially in pthreads and Java, and both terms have
citations with extensive explanations and further citations.

    p3 start your discussion of generations with a simple example of a
    C-forall generator.  Fig 1(b) might do: but put it inline instead of
    the python example - and explain the key rules and restrictions on the
    construct.  Then don't even start to compare with coroutines until
    you've presented, described and explained your coroutines...
    p3 I'd probably leave out the various "C" versions unless there are
    key points to make you can't make in C-forall. All the alternatives
    are just confusing.

Hopefully fixed as this block of text has been rewritten.

    p4 but what's that "with" in Fig 1(B)

Footnote D explains the semantic of "with", which is like unqualified access
for the receiver to the fields of a class from member routines, i.e., no
"this->".

    p5 start with the high level features of C-forall generators...

Hopefully fixed by moving Table 1 forward.

    p5 why is the paper explaining networking protocols?

Fixed, added discussion on this point.

    p7 lines 1-9 (transforming generator to coroutine - why would I do any of
    this? Why would I want one instead of the other (do not use "stack" in your
    answer!)

As stated on line 1 because state declarations from the generator type can be
moved out of the coroutine type into the coroutine main

    p10 last para "A coroutine must retain its last resumer to suspend back
    because the resumer is on a different stack. These reverse pointers allow
    suspend to cycle backwards, " I've no idea what is going on here?  why
    should I care?  Shouldn't I just be using threads instead?  why not?

Hopefully fixed by moving Table 1 forward.

    p16 for the same reasons - what reasons?

Hopefully fixed by moving Table 1 forward.

    p17 if the multiple-monitor entry procedure really is novel, write a paper
    about that, and only about that.

We do not believe this is a practical suggestion.

    p23 "Loose Object Definitions" - no idea what that means.  in that
    section: you can't leave out JS-style dynamic properties.  Even in
    OOLs that (one way or another) allow separate definitions of methods
    (like Objective-C, Swift, Ruby, C#) at any time a runtime class has a
    fixed definition.  Quite why the detail about bit mask implementation
    is here anyway, I've no idea.

Fixed by rewriting the section.

    p25 this cluster isn't a CLU cluster then?

No. A CLU cluster is like a class in an object-oriented programming language.
A CFA cluster is a runtime organizational mechanism.

    * conclusion should conclude the paper, not the related.

We do not understand this comment.

=============================================================================

    Reviewing: 2

    There is much description of the system and its details, but nothing about
    (non-artificial) uses of it. Although the microbenchmark data is
    encouraging, arguably not enough practical experience with the system has
    been reported here to say much about either its usability advantages or its
    performance.

We have a Catch-22 problem. Without publicity, there is no user community;
without a user community, there are no publications for publicity.

    p2: lines 4--9 are a little sloppy. It is not the languages but their
    popular implementations which "adopt" the 1:1 kernel threading model.

Fixed.

    line 10: "medium work" -- "medium-sized work"?

Fixed.

    line 18: "is all sequential to the compiler" -- not true in modern
    compilers, and in 2004 H-J Boehm wrote a tech report describing exactly why
    ("Threads cannot be implemented as a library", HP Labs).

We will have to disagree on this point. First, I am aware of Hans's 2004 paper
because in that paper Hans cites my seminal work on this topic from 1995, which
we cite in this paper.  Second, while modern memory-models have been added to
languages like Java/C/C++ and new languages usually start with a memory model,
it is still the programmer's responsibility to use them for racy code. Only
when the programing language provides race-free constructs is the language
aware of the concurrency; otherwise the code is sequential. Hans's paper "You
Don't Know Jack About Shared Variables or Memory Models" talks about these
issues, and is also cited in the paper.

    line 20: "knows the optimization boundaries" -- I found this vague. What's
    an example?

Fixed.

    line 31: this paragraph has made a lot of claims. Perhaps forward-reference
    to the parts of the paper that discuss each one.

Fixed by adding a road-map paragraph at the end of the introduction.

    line 33: "so the reader can judge if" -- this reads rather
    passive-aggressively. Perhaps better: "... to support our argument that..."

Fixed.

    line 41: "a dynamic partitioning mechanism" -- I couldn't tell what this
    meant

Fixed.

    p3. Presenting concept of a "stateful function" as a new language feature
    seems odd. In C, functions often have local state thanks to static local
    variables (or globals, indeed). Of course, that has several
    limitations. Can you perhaps present your contributions by enumerating
    these limitations? See also my suggestion below about a possible framing
    centred on a strawman.

Fixed, at the start of this section.

    line 2: "an old idea that is new again" -- this is too oblique

Fixed, removed.

    lines 2--15: I found this to be a word/concept soup. Stacks, closures,
    generators, stackless stackful, coroutine, symmetric, asymmetric,
    resume/suspend versus resume/resume... there needs to be a more gradual and
    structured way to introduce all this, and ideally one that minimises
    redundancy. Maybe present it as a series of "definitions" each with its own
    heading, e.g. "A closure is stackless if its local state has statically
    known fixed size"; "A generator simply means a stackless closure." And so
    on. Perhaps also strongly introduce the word "activate" as a direct
    contrast with resume and suspend. These are just a flavour of the sort of
    changes that might make this paragraph into something readable.

    Continuing the thought: I found it confusing that by these definitions, a
    stackful closure is not a stack, even though logically the stack *is* a
    kind of closure (it is a representation of the current thread's
    continuation).

Fixed. Rewrote paragraph and moved Table 1 forward.

    lines 24--27: without explaining what the boost functor types mean, I don't
    think the point here comes across.

Replaced with uC++ example because boost appears to have dropped symmetric
coroutines.

    line 34: "semantically coupled" -- I wasn't sure what this meant

Fixed.

    p4: the point of Figure 1 (C) was not immediately clear. It seem to be
    showing how one might "compile down" Figure 1 (B). Or is that Figure 1 (A)?

Fixed. Rewrote sentence.

    It's right that the incidental language features of the system are not
    front-and-centre, but I'd appreciate some brief glossing of non-C languages
    features as they appear. Examples are the square bracket notation, the pipe
    notation and the constructor syntax. These explanations could go in the
    caption of the figure which first uses them, perhaps. Overall I found the
    figure captions to be terse, and a missed opportunity to explain clearly
    what was going on.

Fixed, added descriptive footnote about Cforall. We prefer to put text in the
body of the paper and keep captions short.

    p5 line 23: "This restriction is removed..." -- give us some up-front
    summary of your contributions and the elements of the language design that
    will be talked about, so that this isn't an aside. This will reduce the
    "twisty passages" feeling that characterises much of the paper.

Fixed, remove parenthesis.

    line 40: "a killer asymmetric generator" -- this is stylistically odd, and
    the sentence about failures doesn't convincingly argue that C\/ will help
    with them. Have you any experience writing device drivers using C\/? Or any
    argument that the kinds of failures can be traced to the "stack-ripping"
    style that one is forced to use without coroutines ?

Fixed, added new paragraph.

    Also, a typo on line
    41: "device drives". And saying "Windows/Linux" is sloppy... what does the
    cited paper actually say?

Fixed.

    p6 lines 13--23: this paragraph is difficult to understand. It seems to be
    talking about a control-flow pattern roughly equivalent to tail recursion.
    What is the high-level point, other than that this is possible?

Fixed, rewrote start of the paragraph.

    line 34: "which they call coroutines" -- a better way to make this point is
    presumably that the C++20 proposal only provides a specialised kind of
    coroutine, namely generators, despite its use of the more general word.

Fixed.

    line 47: "... due to dynamic stack allocation, execution..." -- this
    sentence doesn't scan. I suggest adding "and for" in the relevant places
    where currently there are only commas.

Fixed.

    p8 / Figure 5 (B) -- the GNU C extension of unary "&&" needs to be
    explained.

Fixed, added explanation at first usage in Figure 1(C) and reference.

    The whole figure needs a better explanation, in fact.

Fixed, rewrote start of the paragraph.

    p9, lines 1--10: I wasn't sure this stepping-through really added much
    value. What are the truly important points to note about this code?

Fixed, shortened and merged with previous paragraph.

    p10: similarly, lines 3--27 again are somewhere between tedious and
    confusing. I'm sure the motivation and details of "starter semantics" can
    both be stated much more pithily.

Fixed, shortened these paragraphs.

    line 32: "a self-resume does not overwrite the last resumer" -- is this a
    hack or a defensible principled decision?

Fixed, removed but it is a defensible principled decision.

    p11: "a common source of errors" -- among beginners or among production
    code? Presumably the former.

Forgetting is not specific to beginners.

    line 23: "with builtin and library" -- not sure what this means

Fixed.

    lines 31--36: these can be much briefer. The only important point here
    seems to be that coroutines cannot be copied.

Fixed, shortened.

    p12: line 1: what is a "task"? Does it matter?

Fixed, "task" has been changed to "thread" throughout the paper.

     line 7: calling it "heap stack" seems to be a recipe for
     confusion. "Stack-and-heap" might be better, and contrast with
     "stack-and-VLS" perhaps. When "VLS" is glossed, suggest actually expanding
     its initials: say "length" not "size".

Fixed, make correction and rewrote some of the text.

     line 21: are you saying "cooperative threading" is the same as
     "non-preemptive scheduling", or that one is a special case (kind) of the
     other? Both are defensible, but be clear.

Fixed, clarified the definitions.

    line 27: "mutual exclusion and synchronization" -- the former is a kind of
    the latter, so I suggest "and other forms of synchronization".

We have to agree to disagree. Included a citation that explains the
differences.

    line 30: "can either be a stackless or stackful" -- stray "a", but also,
    this seems to be switching from generic/background terminology to
    C\/-specific terminology.

Fixed, but the terms stackless or stackful are not specific to Cforall; they
are well known in the literature.

    An expositional idea occurs: start the paper with a strawman naive/limited
    realisation of coroutines -- say, Simon Tatham's popular "Coroutines in C"
    web page -- and identify point by point what the limitations are and how
    C\/ overcomes them. Currently the presentation is often flat (lacking
    motivating contrasts) and backwards (stating solutions before
    problems). The foregoing approach might fix both of these.

We prefer the current structure of our paper and believe the paper does
explain basic coding limitations and how they are overcome in using high-level
control-floe mechanisms.

    page 13: line 23: it seems a distraction to mention the Python feature
    here.

Why? It is the first location in the paper where dynamic allocation and
initialization are mentioned.

    p14 line 5: it seems odd to describe these as "stateless" just because they
    lack shared mutable state. It means the code itself is even more
    stateful. Maybe the "stack ripping" argument could usefully be given here.

Fixed, changed "stateless" to "non-shared".

    line 16: "too restrictive" -- would be good to have a reference to justify
    this, or at least give a sense of what the state-of-the-art performance in
    transactional memory systems is (both software and hardware)

Fixed, added 2 citations.

    line 22: "simulate monitors" -- what about just *implementing* monitors?
    isn't that what these systems do? or is the point more about refining them
    somehow into something more specialised?

Fixed, changed "simulate monitors" to "manually implement a monitor".

    p15: sections 4.1 and 4.2 seem adrift and misplaced. Split them into basic
    parts (which go earlier) and more advanced parts (e.g. barging, which can
    be explained later).

Fixed, removed them by shortening and merging with previous section.

    line 31: "acquire/release" -- misses an opportunity to contrast the
    monitor's "enter/exit" abstraction with the less structured acquire/release
    of locks.

Fixed, added "by call/return" in sentence.

    p16 line 12: the "implicit" versus "explicit" point is unclear. Is it
    perhaps about the contract between an opt-in *discipline* and a
    language-enforced *guarantee*?

Fixed.

    line 28: no need to spend ages dithering about which one is default and
    which one is the explicit qualifier. Tell us what you decided, briefly
    justify it, and move on.

Fixed, shortened paragraph.

    p17: Figure 11: since the main point seems to be to highlight bulk acquire,
    include a comment which identifies the line where this is happening.

Fixed.

    line 2: "impossible to statically..." -- or dynamically. Doing it
    dynamically would be perfectly acceptable (locking is a dynamic operation
    after all)

Fixed, clarified the "statically" applied to the unknown-sized pointer types.

    "guarantees acquisition order is consistent" -- assuming it's done in a
    single bulk acquire.

Fixed.

    p18: section 5.3: the text here is a mess. The explanations of "internal"
    versus "external" scheduling are unclear, and "signals as hints" is not
    explained. "... can cause thread starvation" -- means including a while
    loop, or not doing so? "There are three signalling mechanisms.." but the
    text does not follow that by telling us what they are. My own scribbled
    attempt at unpicking the internal/external thing: "threads already in the
    monitor, albeit waiting, have priority over those trying to enter".

Fixed, rewrote and shortened paragraphs.

    p19: line 3: "empty condition" -- explain that condition variables don't
    store anything. So being "empty" means that the queue of waiting threads
    (threads waiting to be signalled that the condition has become true) is
    empty.

Fixed, changed condition variable to condition queue throughout the paper.

    line 6: "... can be transformed into external scheduling..." -- OK, but
    give some motivation.

The paper states that it removes the condition queues and signal/wait. Changed
"transform" to "simplified".

    p20: line 6: "mechnaism"

Fixed.

    lines 16--20: this is dense and can probably only be made clear with an
    example
    
Fixed, rewrote and added example.

    p21 line 21: clarify that nested monitor deadlock was describe earlier (in
    5.2). (Is the repetition necessary?)

Fixed, put in a forward reference, and the point bears repeating because
releasing a subset of acquired monitors in unique to Cforall concurrency.

    line 27: "locks, and by extension monitors" -- this is true but the "by
    extension" argument is faulty. It is perfectly possible to use locks as a
    primitive and build a compositional mechanism out of them,
    e.g. transactions.

True, but that is not what we said. Locks are not composable, monitors are
built using locks not transactions, so by extension monitors are not composable.

    p22 line 2: should say "restructured"

Fixed.

    line 33: "Implementing a fast subset check..." -- make clear that the
    following section explains how to do this. Restructuring the sections
    themselves could do this, or noting in the text.

Fixed, added a forward reference to the following sections.

    p23: line 3: "dynamic member adding, eg, JavaScript" -- needs to say "as
    permitted in JavaScript", and "dynamically adding members" is stylistically
    better

Fixed.

    p23: line 18: "urgent stack" -- back-reference to where this was explained
    before

Fixed.

    p24 line 7: I did not understand what was more "direct" about "direct
    communication". Also, what is a "passive monitor" -- just a monitor, given
    that monitors are passive by design?

The back half of line 7 defines "direct". For example, Go, Java, pthread
threads cannot directly call/communicate with one another, where they can in
Ada, uC++, and Cforall threads. Figure 18 show this exact difference.

A monitor object is *passive* because it does not have a thread, while a Go,
Java, Cforall "thread" object is *active* because it has a thread.

    line 14 / section 5.9: this table was useful and it (or something like it)
    could be used much earlier on to set the structure of the rest of the
    paper.

Fixed, Table 1 is moved to the start and explained in detail.

    The explanation at present is too brief, e.g. I did not really understand
    the point about cases 7 and 8. Table 1: what does "No / Yes" mean?

Fixed, expanded the explanation.

    p25 line 2: instead of casually dropping in a terse explanation for the
    newly introduced term "virtual processor", introduce it
    properly. Presumably the point is to give a less ambiguous meaning to
    "thread" by reserving it only for C\/'s green threads.

Fixed.

    p26 line 15: "transforms user threads into fibres" -- a reference is needed
    to explain what "fibres" means... guessing it's in the sense of Adya et al.

Fixed. In a prior correct, the term fibre from Adya is defined.

    line 20: "Microsoft runtime" -- means Windows?

Fixed.

    lines 21--26: don't say "interrupt" to mean "signal", especially not
    without clear introduction. You can use "POSIX signal" to disambiguate from
    condition variables' "signal".

We have to agree to disagree on this terminology. Interrupt is the action of
stopping the CPU while a signal is a specific kind of interrupt. The two terms
seem to be well understood in the literature.

    p27 line 3: "frequency is usually long" -- that's a "time period" or
    "interval", not a frequency

Fixed.

    line 5: the lengthy quotation is not really necessary; just paraphrase the
    first sentence and move on.

Fixed.

    line 20: "to verify the implementation" -- I don't think that means what is
    intended

Fixed, changed "verify" to "test".

    Tables in section 7 -- too many significant figures. How many overall runs
    are described? What is N in each case?

Fixed. As stated, N=31.

    p29 line 2: "to eliminate this cost" -- arguably confusing since nowadays
    on commodity CPUs most of the benefits of inlining are not to do with call
    overheads, but from later optimizations enabled as a consequence of the
    inlining

Fixed.

    line 41: "a hierarchy" -- are they a hierarchy? If so, this could be
    explained earlier. Also, to say these make up "an integrated set... of
    control-flow features" verges on the tautologous.

Fixed, rewrote sentence.

    p30 line 15: "a common case being web servers and XaaS" -- that's two cases

Fixed.

============================================================================

    Reviewing: 3

    * Expand on the motivations for including both generator and coroutines, vs
      trying to build one atop the other

Fixed, Table 1 is moved to the start and explained in detail.

    * Expand on the motivations for having both symmetric and asymmetric
      coroutines?

A coroutine is not marked as symmetric or asymmetric, it is a coroutine.
Symmetric or asymmetric is a stylistic use of a coroutine. By analogy, a
function is not marked as recursive or non-recursive. Recursion is a style of
programming with a function. So there is no notion of motivation for having
both symmetric and asymmetric as they follow from how a programmer uses suspend
and resume.

    * Comparison to async-await model adopted by other languages

Fixed, added a new section on this topic.

    * Consider performance comparisons against node.js and Rust frameworks

Fixed.

    * Discuss performance of monitors vs finer-grained memory models and atomic
      operations found in other languages

The paper never suggested high-level concurrency constructs can or should
replace race programming or hardware atomics. The paper suggests programmers
use high-level constructs when and where is it feasible because they are easy
and safer to use. The monitor example of an atomic counter is just that, an
example, not the way it should be done if maximal performance is required.  We
have tried to make this point clear in the paper.

    * Why both internal/external scheduling for synchronization?

Some additional motivation has been added.

    * Generators are not exposed as a "function" that returns a generator
      object, but rather as a kind of struct, with communication happening via
      mutable state instead of "return values".

Yes, Cforall uses an object-style of coroutine, which allows multiple interface
functions that pass and return values through a structure. This approach allows
a generator function to have different kinds of return values and different
kinds of parameters to produce those values. Our generators can provide this
capability via multiple interface functions to the generator/coroutine state,
which is discussed on page 5, lines 13-21.

      That is, the generator must be manually resumed and (if I understood) it
      is expected to store values that can then later be read (perhaps via
      methods), instead of having a `yield <Expr>` statement that yields up a
      value explicitly.

All generators are manually resumed, e.g., Python/nodejs use "next" to resume a
generator. Yes, yield <Expr> has a single interface with one input/return type,
versus the Cforall approach allowing arbitrary number of interfaces of
arbitrary types.

    * Both "symmetric" and "asymmetric" generators are supported, instead of
      only asymmetric.

Yes, because they support different functionality as discussed in Chris
Marlin's seminal work and both forms are implemented in Simula67. We did not
invent symmetric and asymmetric generators/coroutines, we took them from the
literature.

    * Coroutines (multi-frame generators) are an explicit mechanism.

    In most other languages, coroutines are rather built by layering
    single-frame generators atop one another (e.g., using a mechanism like
    async-await),

We disagree. Node.js has async-await but has a separate coroutine feature.
While there are claims that coroutines can be built from async-await and/or
continuations, in actuality they cannot.

    and symmetric coroutines are basically not supported. I'd like to see a bit
    more justification for Cforall including all the above mechanisms -- it
    seemed like symmetric coroutines were a useful building block for some of
    the user-space threading and custom scheduler mechanisms that were briefly
    mentioned later in the paper.

Hopefully fixed by moving Table 1 forward.

    In the discussion of coroutines, I would have expected a bit more of a
    comparison to the async-await mechanism offered in other languages.

We added a new section at the start to point out there is no comparison between
coroutines and async-await.

    Certainly the semantics of async-await in JavaScript implies
    significantly more overhead (because each async fn is a distinct heap
    object). [Rust's approach avoids this overhead][zc], however, and might be
    worthy of a comparison (see the Performance section).

We could not get Rust async-await to work, and when reading the description of
rust async-await, it appears to be Java-style executors with futures (possibly
fast futures).

    There are several sections in the paper that compare against atomics -- for
    example, on page 15, the paper shows a simple monitor that encapsulates an
    integer and compares that to C++ atomics. Later, the paper compares the
    simplicity of monitors against the `volatile` quantifier from Java. The
    conclusion in section 8 also revisits this point.
    While I agree that monitors are simpler, they are obviously also
    significantly different from a performance perspective -- the paper doesn't
    seem to address this at all. It's plausible that (e.g.) the `Aint` monitor
    type described in the paper can be compiled and mapped to the specialized
    instructions offered by hardware, but I didn't see any mention of how this
    would be done.

Fixed, see response above.

    There is also no mention of the more nuanced memory ordering
    relations offered by C++11 and how one might achieve similar performance
    characteristics in Cforall (perhaps the answer is that one simply doesn't
    need to; I think that's defensible, but worth stating explicitly).

Cforall is built on C, and therefore has full access to all the gcc atomics,
and automatically gets any gcc updates.  Furthermore, section 6.9 states that
Cforall provides the full panoply of low-level locks, as does Java, Go, C++,
for performance programming.

    Cforall includes both internal and external scheduling; I found the
    explanation for the external scheduling mechanism to be lacking in
    justification. Why include both mechanisms when most languages seem to make
    do with only internal scheduling? It would be useful to show some scenarios
    where external scheduling is truly more powerful.

Fixed. Pointed out external scheduling is simpler as part of rewriting in that
section, and added additional examples.

    I would have liked to see some more discussion of external scheduling and
    how it interacts with software engineering best practices. It seems
    somewhat similar to AOP in certain regards. It seems to add a bit of "extra
    semantics" to monitor methods, in that any method may now also become a
    kind of synchronization point.

Fixed somewhat. Pointed out that external scheduling has been around for a long
time (40 years) in Ada, so there is a body of the software-engineering
experience using it. As well, I have been teaching it for 30 years in the
concurrency course at Waterloo. We don't know what software engineering best
practices you imagine it interacting with. Yes, monitor functions are
synchronization points with external scheduling.

    The "open-ended" nature of this feels like it could easily lead to subtle
    bugs, particularly when code refactoring occurs (which may e.g. split an
    existing method into two).

Any time a public interface is refactored, it invalids existing calls, so there
is always an issue. For mutex routines and external scheduling, the waitfor
statements may have to be updated, but that update is part of the refactoring.

    This seems particularly true if external scheduling can occur across
    compilation units -- the paper suggested that this is true, but I wasn't
    entirely clear.

Every aspect of Cforall allows separate compilation. The function prototypes
necessary for separate compilation provide all the information necessary to
compile any aspect of a program.

    I would have also appreciated a few more details on how external scheduling
    is implemented. It seems to me that there must be some sort of "hooks" on
    mutex methods so that they can detect whether some other function is
    waiting on them and awaken those blocked threads. I'm not sure how such
    hooks are inserted, particularly across compilation units.

Hooks are inserted by the Cforall translator, in the same way that Java
inserted hooks into a "synchronized" member of a monitor. As for Java, as long
as the type information is consistent across compilation units, the correct
code is inserted.

    The material in Section 5.6 didn't quite clarify the matter for me. For
    example, it left me somewhat confused about whether the `f` and `g`
    functions declared were meant to be local to a translation unit, or shared
    with other unit.

There are no restrictions with respect to static or external mutex functions.
Cforall is C. Any form of access or separate compilation in C applies to
Cforall. As in C, function prototypes carry all necessary information to
compile the code.

    To start, I did not realize that the `mutex_opt` notation was a keyword, I
    thought it was a type annotation. I think this could be called out more
    explicitly.

Fixed, indicated "mutex" is a C-style parameter-only declaration type-qualifier.

    Later, in section 5.2, the paper discusses `nomutex` annotations, which
    initially threw me, as they had not been introduced (now I realize that
    this paragraph is there to justify why there is no such keyword). The
    paragraph might be rearranged to make that clearer, perhaps by leading with
    the choice that Cforall made.

Fixed, rewrote paragraph removing nomutex.

    On page 17, the paper states that "acquiring multiple monitors is safe from
    deadlock", but this could be stated a bit more precisely: acquiring
    multiple monitors in a bulk-acquire is safe from deadlock (deadlock can
    still result from nested acquires).

Fixed.

    On page 18, the paper states that wait states do not have to be enclosed in
    loops, as there is no concern of barging. This seems true but there are
    also other reasons to use loops (e.g., if there are multiple reasons to
    notify on the same condition). Thus the statement initially surprised me,
    as barging is only one of many reasons that I typically employ loops around
    waits.

Fixed. Rewrote the sentence. Note, for all non-barging cases where you employ a
loop around a wait, the unblocking task must change state before blocking
again.  In the barging case, the unblocking thread blocks again without
changing state.

    I did not understand the diagram in Figure 12 for some time. Initially, I
    thought that it was generic to all monitors, and I could not understand the
    state space. It was only later that I realized it was specific to your
    example. Updating the caption from "Monitor scheduling to "Monitor
    scheduling in the example from Fig 13" might have helped me quite a bit.

Fixed, updated text to clarify. Did not change the caption because the
signal_block does not apply to Figure 13.

    I spent quite some time reading the boy/girl dating example (\*) and I
    admit I found it somewhat confusing. For example, I couldn't tell whether
    there were supposed to be many "girl" threads executing at once, or if
    there was only supposed to be one girl and one boy thread executing in a
    loop.

The paper states:

  The dating service matches girl and boy threads with matching compatibility
  codes so they can exchange phone numbers.

so there are many girl/boy threads. There is nothing preventing an individual
girl/boy from arranging multiple dates.

    Are the girl/boy threads supposed to invoke the girl/boy methods or vice
    versa?

As long as the girls/boys are consistent in the calls, it does not matter. The
goal is to find a partner and exchange phone numbers.

    Surely there is some easier way to set this up?

There are some other solutions using monitors but they all have a similar
structure.

    The paper offered a number of comparisons to Go, C#, Scala, and so forth,
    but seems to have overlooked another recent language, Rust. In many ways,
    Rust seems to be closest in philosophy to Cforall, so it seems like an odd
    omission. I already mentioned above that Rust is in the process of shipping
    [async-await syntax][aa], which is definitely an alternative to the
    generator/coroutine approach in Cforall (though one with clear pros/cons).

We cannot get rust async-await example programs to compile nor does the select!
macro compile.

  @plg2[1]% rustc --version
  rustc 1.40.0 (73528e339 2019-12-16)
  
  @plg2[2]% cat future.rs 
  use futures::executor::block_on;
  
  async fn hello_world() {
      println!("hello, world!");
  }
  
  fn main() {
      let future = hello_world(); // Nothing is printed
      block_on(future); // `future` is run and "hello, world!" is printed
  }
  
  @plg2[3]% rustc -C opt-level=3 future.rs
  error[E0670]: `async fn` is not permitted in the 2015 edition
   --> future.rs:3:1
    |
  3 | async fn hello_world() {
    | ^^^^^
  
  error[E0433]: failed to resolve: maybe a missing crate `futures`?
   --> future.rs:1:5
    |
  1 | use futures::executor::block_on;
    |     ^^^^^^^ maybe a missing crate `futures`?
  
  error[E0425]: cannot find function `block_on` in this scope
   --> future.rs:9:5
    |
  9 |     block_on(future); // `future` is run and "hello, world!" is printed
    |     ^^^^^^^^ not found in this scope
  
  error: aborting due to 3 previous errors
  
  Some errors have detailed explanations: E0425, E0433, E0670.
  For more information about an error, try `rustc --explain E0425`.


    In the performance section in particular, you might consider comparing
    against some of the Rust web servers and threading systems.

This paper is not about building web-servers. Nor are web-servers a reasonable
benchmark for language concurrency. Web-servers are a benchmark for
non-blocking I/O library efficiency accessed in the underlying operating
system. Our prior work on web-server performance:

@inproceedings{Pariag07,
    author      = {David Pariag and Tim Brecht and Ashif Harji and Peter Buhr and Amol Shukla},
    title       = {Comparing the Performance of Web Server Architectures},
    booktitle   = {Proceedings of the 2007 Eurosys conference},
    month       = mar,
    year        = 2007,
    pages       = {231--243},
}

@inproceedings{Harji12,
    author      = {Ashif S. Harji and Peter A. Buhr and Tim Brecht},
    title       = {Comparing High-Performance Multi-core Web-Server Architectures},
    booktitle   = {Proceedings of the 5th Annual International Systems and Storage Conference},
    series      = {SYSTOR '12},
    publisher   = {ACM},
    address     = {New York, NY, USA},
    location    = {Haifa, Israel},
    month       = jun,
    year        = 2012,
    articleno   = 1,
    pages       = {1:1--1:12},
}

shows the steps to build a high-performance web-server, which are largely
independent of the server architecture and programing language.

    It would seem worth trying to compare their "context switching" costs as
    well -- I believe both actix and tokio have a notion of threads that could
    be readily compared.

Again, context-switching speed is largely irrelevant because the amount of code
to process an http request is large enough to push any concurrency costs into
the background.

    Another addition that might be worth considering is to compare against
    node.js promises, although I think the comparison to process creation is
    not as clean.

Done.