Reviewing: 1 I still have a couple of issues --- perhaps the largest is that it's still not clear at this point in the paper what some of these options are, or crucially how they would be used. I don't know if it's possible to give high-level examples or use cases to be clear about these up front - or if that would duplicate too much information from later in the paper - either way expanding out the discussion - even if just two a couple of sentences for each row - would help me more. Section 2.1 is changed to address this suggestion. * 1st para section 2 begs the question: why not support each dimension independently, and let the programmer or library designer combine features? As seen in Table 1, not all of the combinations work, and having programmers directly use these low-level mechanisms is error prone. Accessing these fundamental mechanisms through higher-level constructs has always been the purpose of a programming language. * Why must there "be language mechanisms to create, block/unblock, and join with a thread"? There aren't in Smalltalk (although there are in the runtime). Especially given in Cforall those mechanisms are *implicit* on thread creation and destruction? Fixed, changed sentence to: A programmer needs mechanisms to create, block and unblock, and join with a thread, even if these basic mechanisms are supplied indirectly through high-level features. * "Case 1 is a function that borrows storage for its state (stack frame/activation) and a thread from its invoker" this much makes perfect sense to me, but I don't understand how a non-stateful, non-threaded function can then retain "this state across callees, ie, function local-variables are retained on the stack across calls." how can it retain function-local values *across calls* when it doesn't have any functional-local state? In the following example: void foo() { // local variables and code } void bar() { // local variables foo(); } bar is the caller and foo is the callee. bar borrows the program stack and thread to make the call to foo. When foo, the callee, is executing, bar's local variables (state) is retained on the *borrowed* stack across the call. (Note, I added *borrowed* to that sentence in the paper to help clarify.) Furthermore, foo's local variables are also retain on the borrowed stack. When foo and bar return, all of their local state is gone (not retained). This behaviour is standard call/return semantics in an imperative language. I'm not sure if I see two separate cases here - roughly equivalent to C functions without static storage, and then C functions *with* static storage. Yes, but there is only one instance of the static storage across all activations of the C function. For generators and coroutines, each instance has its own state, like an object in an OO language. I assumed that was the distinction between cases 1 & 3; but perhaps the actual distinction is that 3 has a suspend/resume point, and so the "state" in figure 1 is this component of execution state (viz figs 1 & 2), not the state representing the cross-call variables? So case 3 is like an object with the added ability to retain where it was last executing. When a generator is resumed, the generator object (structure instance) is passed as an explicit reference, and within this object is the restart location in the generator's "main". When the generator main executes, it uses the borrowed stack for its local variables and any functions it calls, just like an object member borrows the stack for its local variables but also has an implicit receiver to the object state. A generator can have static storage, too, which is a single instance across all generator instances of that type, as for static storage in an object type. All the kinds of storage are at play with semantics that is the same as in other languages. > but such evaluation isn't appropriate for garbage-collected or JITTed > languages like Java or Go. For JITTed languages in particular, reporting peak performance needs to "warm up" the JIT with a number of iterators before beginning measurement. Actually for JIT's its even worse: see Edd Barrett et al OOPSLA 2017. Of our testing languages, only Java and Node.js are JITTED. To ensure the Java test-programs correctly measured the specific feature, we consulted with Dave Dice at Oracle who works directly on the development of the Oracle JVM Just-in-Time Compiler. We modified our test programs based on his advise, and he validated our programs as correctly measuring the specified language feature. Dave's help is recognized in the Acknowledgment section. Similarly, we verified our Node.js programs with Gregor Richards, an expert in just-in-time compilation for dynamic typing. Hence, we have taken into account all issues related to performing benchmarks in JITTED languages. Also, all the benchmark programs are publicly available for independent verification. * footnote A - I've looked at various other papers & the website to try to understand how "object-oriented" Cforall is - I'm still not sure. This footnote says Cforall has "virtuals" - presumably virtual functions, i.e. dynamic dispatch - and inheritance: that really is OO as far as I (and most OO people) are concerned. For example Haskell doesn't have inheritance, so it's not OO; while CLOS (the Common Lisp *Object* System) or things like Cecil and Dylan are considered OO even though they have "multiple function parameters as receivers", lack "lexical binding between a structure and set of functions", and don't have explicit receiver invocation syntax. Python has receiver syntax, but unlike Java or Smalltalk or C++, method declarations still need to have an explicit "self" receiver parameter. Seems to me that Go, for example, is more-or-less OO with interfaces, methods, and dynamic dispatch (yes also and an explicit receiver syntax but that's not determinative); while Rust lacks dynamic dispatch built-in. C is not OO as a language, but as you say given it supports function pointers with structures, it does support an OO programming style. This is why I again recommend just not buying into this fight: not making any claims about whether Cforall is OO or is not - because as I see it, the rest of the paper doesn't depend on whether Cforall is OO or not. That said: this is just a recommendation, and I won't quibble over this any further. We believe it is important to identify Cforall as a non-OO language because it heavily influences the syntax and semantics used to build its concurrency. Since many aspects of Cforall are not OO, the rest of the paper *does* depend on Cforall being identified as non-OO, otherwise readers would have significantly different expectations for the design. We did not mean to suggest that languages that support function pointers with structures support an OO style. We revised the footnote to avoid this interpretation. Finally, Golang does not identify itself as an OO language. https://golang.org/doc/faq#Is_Go_an_object-oriented_language * is a "monitor function" the same as a "mutex function"? if so the paper should pick one term; if not, make the distinction clear. Fixed. Picked "mutex". Changed the language and all places in the paper. * "As stated on line 1 because state declarations from the generator type can be moved out of the coroutine type into the coroutine main" OK sure, but again: *why* would a programmer want to do that? (Other than, I guess, to show the difference between coroutines & generators?) Perhaps another way to put this is that the first para of 3.2 gives the disadvantages of coroutines vs-a-vs generators, briefly describes the extended semantics, but never actually says why a programmer may want those extended semantics, or how they would benefit. I don't mean to belabour the point, but (generalist?) readers like me would generally benefit from those kinds of discussions about each feature throughout the paper: why might a programmer want to use them? On page 8, it states: Having to manually create the generator closure by moving local-state variables into the generator type is an additional programmer burden (removed by the coroutine in Section 3.2). ... also these variables can now be refactored into helper function, where the helper function suspends at arbitrary call depth. So imagine a coroutine helper function that is only called occasionally within the coroutine but it has a large array that is retained across suspends within the helper function. For a generator, this large array has to be declared in the generator type enlarging each generator instance even through the array is only used occasionally. Whereas, the coroutine only needs the array allocated when needed. Now a coroutine has a stack which occupies storage, but the maximum stack size only needs to be the call chain allocating the most storage, whereas the generator has a maximum size of all variable that could be created. > p17 if the multiple-monitor entry procedure really is novel, write a paper > about that, and only about that. > We do not believe this is a practical suggestion. * I'm honestly not trying to be snide here: I'm not an expert on monitor or concurrent implementations. Brinch Hansen's original monitors were single acquire; this draft does not cite any other previous work that I could see. I'm not suggesting that the brief mention of this mechanism necessarily be removed from this paper, but if this is novel (and a clear advance over a classical OO monitor a-la Java which only acquires the distinguished receiver) then that would be worth another paper in itself. First, to explain multiple-monitor entry in Cforall as a separate paper would require significant background on Cforall concurrency, which means repeating large sections of this paper. Second, it feels like a paper just on multiple-monitor entry would be a little thin, even if the capability is novel to the best of our knowledge. Third, we feel multiple-monitor entry springs naturally from the overarching tone in the paper that Cforall is a non-OO programming language, allowing multiple-mutex receivers. My typo: the paper's conclusion should come at the end, after the future work section. Combined into a Conclusions and Future Work section. Reviewing: 2 on the Boehm paper and whether code is "all sequential to the compiler": I now understand the authors' position better and suspect we are in violent agreement, except for whether it's appropriate to use the rather breezy phrase "all sequential to the compiler". It would be straightforward to clarify that code not using the atomics features is optimized *as if* it were sequential, i.e. on the assumption of a lack of data races. Fixed, "as inline and library code is compiled as sequential without any explicit concurrent directive." on the distinction between "mutual exclusion" and "synchronization": the added citation does help, in that it makes a coherent case for the definition the authors prefer. However, the text could usefully clarify that this is a matter of definition not of fact, given especially that in my assessment the authors' preferred definition is not the most common one. (Although the mention of Hoare's apparent use of this definition is one data point, countervailing ones are found in many contemporaneous or later papers, e.g. Habermann's 1972 "Synchronization of Communicating Processes" (CACM 15(3)), Reed & Kanodia's 1979 "Synchronization with eventcounts and sequencers" (CACM (22(2)) and so on.) Fixed, "We contend these two properties are independent, ...". With respect to the two papers, Habermann fundamentally agrees with our definitions, where the term mutual exclusion is the same but Habermann uses the term "communication" for our synchronization. However, the term "communication" is rarely used to mean synchronization. The fact that Habermann collectively calls these two mechanisms synchronization is the confusion. Reed & Kanodia state: By mutual exclusion we mean any mechanism that forces the time ordering of execution of pieces of code, called critical regions, in a system of concurrent processes to be a total ordering. But there is no timing order for a critical region (which I assume means critical section); threads can arrive at any time in any order, where the mutual exclusion for the critical section ensures one thread executes at a time. Interestingly, Reed & Kanodia's critical region is Habermann's communication not critical section. These papers only buttress our contention about the confusion of these terms in the literature. section 2 (an expanded version of what was previously section 5.9) lacks examples and is generally obscure and allusory ("the most advanced feature" -- name it! "in triplets" -- there is only one triplet!; Fixed. These independent properties can be used to compose different language features, forming a compositional hierarchy, where the combination of all three is the most advanced feature, called a thread/task/process. While it is possible for a language to only provide threads for composing programs~\cite{Hermes90}, this unnecessarily complicates and makes inefficient solutions to certain classes of problems. what are "execution locations"? "initialize" and "de-initialize" Fixed through an example at the end of the sentence. \item[\newterm{execution state}:] is the state information needed by a control-flow feature to initialize and manage both compute data and execution location(s), and de-initialize. For example, calling a function initializes a stack frame including contained objects with constructors, manages local data in blocks and return locations during calls, and de-initializes the frame by running any object destructors and management operations. what? "borrowed from the invoker" is a concept in need of explaining or at least a fully explained example -- in what sense does a plain function borrow" its stack frame? A function has no storage except for static storage; it gets its storage from the program stack. For a function to run it must borrow storage from somewhere. When the function returns, the borrowed storage is returned. Do you have a more appropriate word? "computation only" as opposed to what? This is a term in concurrency for operations that compute without blocking, i.e., the operation starts with everything it needs to compute its result and runs to completion, blocking only when it is done and returns its result. Computation only occurs in "embarrassingly parallel" problems. in 2.2, in what way is a "request" fundamental to "synchronization"? I assume you are referring to the last bullet. Synchronization must be able to control the service order of requests including prioritizing selection from different kinds of outstanding requests, and postponing a request for an unspecified time while continuing to accept new requests. Habermann states on page 173 Looking at deposit we see that sender and receiver should be synchronized with respect to buffer overflow: deposit of another message must be delayed if there is no empty message frame, and such delay should be removed when the first empty frame becomes available. This is programmed as synchronization(frame) : deposit is preceded by wait(frame), accept is followed by signal( frame), and the constant C[frame] is set equal to "bufsize." Here synchronization is controlling the service order of requests: when the buffer is full, requests for insert (sender) are postponed until a request to remove (receiver) occurs. Without the ability to control service order among requests, the producer/consumer problem with a bounded buffer cannot be solved. Hence, this capability is fundamental. and the "implicitly" versus "explicitly" point needs stating as elsewhere, with a concrete example e.g. Java built-in mutexes versus java.util.concurrent). Fixed. MES must be available implicitly in language constructs, \eg Java built-in monitors, as well as explicitly for specialized requirements, \eg @java.util.concurrent@, because requiring programmers to build MES using low-level locks often leads to incorrect programs. section 6: 6.2 omits the most important facts in preference for otherwise inscrutable detail: "identify the kind of parameter" (first say *that there are* kinds of parameter, and what "kinds" means!); "mutex parameters are documentation" is misleading (they are also semantically significant!) and fails to say *what* they mean; the most important thing is surely that 'mutex' is a language feature for performing lock/unlock operations at function entry/exit. So say it! These sections have been rewritten address the comments. The meanings of examples f3 and f4 remain unclear. Rewrote the paragraph. Meanwhile in 6.3, "urgent" is not introduced (we are supposed to infer its meaning from Figure 12, Defined Hoare's urgent list at the start of the paragraph. but that Figure is incomprehensible to me), and we are told of "external scheduling"'s long history in Ada but not clearly what it actually means; We do not know how to address your comment because the description is clear to us and other non-reviewers who have read the paper. I had forgotten an important citation to prior work on this topic, which is now referenced: Buhr Peter A., Fortier Michel, Coffin Michael H.. Monitor Classification. ACM Computing Surveys. 1995; 27(1):63-107. This citation is a 45 page paper expanding on the topic of internal scheduling. I also added a citation to Hoare's monitor paper where signal_block is defined: When a process signals a condition on which another process is waiting, the signalling process must wait until the resumed process permits it to proceed. External scheduling from Ada was subsequently added to uC++ and Cforall. Furthermore, Figure 20 shows a direct comparison of CFA procedure-call external-scheduling with Go channel external-scheduling. So external scheduling exists in languages beyond Ada. 6.4's description of "waitfor" tells us it is different from an if-else chain but tries to use two *different* inputs to tell us that the behavior is different; tell us an instance where *the same* values of C1 and C2 give different behavior (I even wrote out a truth table and still don't see the semantic difference) Again, it is unclear what is the problem. For the if-statement, if C1 is true, it only waits for a call to mem1, even if C2 is true and there is an outstanding call to mem2. For the waitfor, if both C1 and C2 are true and there is a call to mem2 but not mem1, it accepts the call to mem2, which cannot happen with the if-statement. So for all true when clauses, any outstanding call is immediately accepted. If there are no outstanding calls, the waitfor blocks until the next call to any of the true when clauses occurs. I added the following sentence to further clarify. Hence, the @waitfor@ has parallel semantics, accepting any true @when@ clause. Note, the same parallel semantics exists for the Go select statement with respect to waiting for a set of channels to receive data. While Go select does not have a when clause, it would be trivial to add it, making the select more expressive. The authors frequently use bracketed phrases, and sometimes slashes "/", in ways that are confusing and/or detrimental to readability. Page 13 line 2's "forward (backward)" is one particularly egregious example. In general I would recommend the the authors try to limit their use of parentheses and slashes as a means of forcing a clearer wording to emerge. Many of the slashes and parentheticals have been removed. Some are retained to express joined concepts: call/return, suspend/resume, resume/resume, I/O. Also, the use of "eg." is often cursory and does not explain the examples given, which are frequently a one- or two-word phrase of unclear referent. A few of these are fixed. Considering the revision more broadly, none of the more extensive or creative rewrites I suggested in my previous review have been attempted, nor any equivalent efforts to improve its readability. If you reread the previous response, we addressed all of your suggestions except An expositional idea occurs: start the paper with a strawman naive/limited realisation of coroutines -- say, Simon Tatham's popular "Coroutines in C" web page -- and identify point by point what the limitations are and how C\/ overcomes them. Currently the presentation is often flat (lacking motivating contrasts) and backwards (stating solutions before problems). The foregoing approach might fix both of these. We prefer the current structure of our paper and believe the paper does explain basic coding limitations and how they are overcome in using high-level control-floe mechanisms. and we have addressed readability issues in this version. The hoisting of the former section 5.9 is a good idea, but the newly added material accompanying it (around Table 1) suffers fresh deficiencies in clarity. Overall the paper is longer than before, even though (as my previous review stated), I believe a shorter paper is required in order to serve the likely purpose of publication. (Indeed, the authors' letter implies that a key goal of publication is to build community and gain external users.) This comment is the referee's opinion, which we do not agree with it. Our previous 35 page SP&E paper on Cforall: Moss Aaron, Schluntz Robert, Buhr Peter A.. Cforall: Adding Modern Programming Language Features to C. Softw. Pract. Exper. 2018; 48(12):2111-2146. has a similar structure and style to this paper, and it received an award from John Wiley & Sons: Software: Practice & Experience for articles published between January 2017 and December 2018, "most downloads in the 12 months following online publication showing the work generated immediate impact and visibility, contributing significantly to advancement in the field". So we have demonstrated an ability to build community and gain external users. Given this trajectory, I no longer see a path to an acceptable revision of the present submission. Instead I suggest the authors consider splitting the paper in two: one half about coroutines and stack management, the other about mutexes, monitors and the runtime. (A briefer presentation of the runtime may be helpful in the first paper also, and a brief recap of the generator and coroutine support is obviously needed in the second too.) Again we disagree with the referee's suggestion to vastly restructure the paper. What advantage is there is presenting exactly the same material across two papers, which will likely end up longer than a single paper? The current paper has a clear theme that fundamental execution properties generate a set of basic language mechanisms, and we then proceed to show how these mechanisms can be designed into the programing language Cforall. I do not buy the authors' defense of the limited practical experience or "non-micro" benchmarking presented. Yes, gaining external users is hard and I am sympathetic on that point. But building something at least *somewhat* substantial with your own system should be within reach, and without it the "practice and experience" aspects of the work have not been explored. Clearly C\/ is the product of a lot of work over an extended period, so it is a surprise that no such experience is readily available for inclusion. Understood. There are no agreed-upon concurrency benchmarks, which is why micro-benchmarks are often used. Currently, the entire Cforall runtime is written in Cforall (10,000+ lines of code (LOC)). This runtime is designed to be thread safe, automatically detects the use of concurrency features at link time, and bootstraps into a threaded runtime almost immediately at program startup so threads can be declared as global variables and may run to completion before the program main starts. The concurrent core of the runtime is 3,500 LOC and bootstraps from low-level atomic primitives into Cforall locks and high-level features. In other words, the concurrent core uses itself as quickly as possible to start using high-level concurrency. There are 12,000+ LOC in the Cforall tests-suite used to verify language features, which are run nightly. Of theses, there are 2000+ LOC running standard concurrent tests, such as aggressively testing each language feature, and classical examples such as bounded buffer, dating service, matrix summation, quickSort, binary insertion sort, etc. More experience will be available soon, based on ongoing work in the "future works" section. Specifically, non-blocking I/O is working with the new Linux io_uring interface and a new high-performance ready-queue is under construction to take into account this change. With non-blocking I/O, it will be possible to write applications like high-performance web servers, as is now done in Rust and Go. Also, completed is Java-style executors for work-based concurrent programming and futures. Under construction is a high-performance actor system. It does not seem right to state that a stack is essential to Von Neumann architectures -- since the earliest Von Neumann machines (and indeed early Fortran) did not use one. Reference Manual Fortran II for the IBM 704 Data Processing System, 1958 IBM, page 2 https://archive.computerhistory.org/resources/text/Fortran/102653989.05.01.acc.pdf Since a subprogram may call for other subprograms to any desired depth, a particular CALL statement may be defined by a pyramid of multi-level subprograms. I think we may be differing on the meaning of stack. You may be imagining a modern stack that grows and shrinks dynamically. Whereas early Fortran preallocated a stack frame for each function, like Python allocates a frame for a generator. Within each preallocated Fortran function is a frame for local variables and a pointer to store the return value for a call. The Fortran call/return mechanism uses these frames to build a traditional call stack linked by the return pointer. The only restriction is that a function stack frame can only be used once, implying no direct or indirect recursion. Hence, without a stack mechanism, there can be no call/return to "any desired depth", where the maximum desired depth is limited by the number of functions. So call/return requires some form of a stack, virtually all programming language have call/return, past and present, and these languages run on Von Neumann machines that do not distinguish between program and memory space, have mutable state, and the concept of a pointer to data or code. To elaborate on something another reviewer commented on: it is a surprise to find a "Future work" section *after* the "Conclusion" section. A "Conclusions and future work" section often works well. Done. Reviewing: 3 but it remains really difficult to have a good sense of which idea I should use and when. This applies in different ways to different features from the language: * coroutines/generators/threads: here there is some discussion, but it can be improved. * internal/external scheduling: I didn't find any direct comparison between these features, except by way of example. See changes below. I would have preferred something more like a table or a few paragraphs highlighting the key reasons one would pick one construct or the other. Section 2.1 is changed to address this suggestion. The discussion of clusters and pre-emption in particular feels quite rushed. We believe a brief introduction to the Cforall runtime structure is important because clustering within a user-level versus distributed system is unusual, Furthermore, the explanation of preemption is important because several new languages, like Go and Rust tokio, are not preemptive. Rust threads are preemptive only because it uses kernel threads, which UNIX preempts. * Recommend to shorten the comparison on coroutine/generator/threads in Section 2 to a paragraph with a few examples, or possibly a table explaining the trade-offs between the constructs Not done, see below. * Recommend to clarify the relationship between internal/external scheduling -- is one more general but more error-prone or low-level? Done, see below. There is obviously a lot of overlap between these features, and in particular between coroutines and generators. As noted in the previous review, many languages have chosen to offer *only* generators, and to build coroutines by stacks of generators invoking one another. Coroutines built from stacks of generators have problems, such as no symmetric control-flow. Furthermore, stacks of generators have a problem with the following programming pattern. logger is a library function called from a function or a coroutine, where the doit for the coroutine suspends. With stacks of generators, there has to be a function and generator version of logger to support these two scenarios. If logger is a library function, it may be impossible to create the generator logger because the logger function is opaque. #include #include forall( otype T | { void doit( T ); } ) void logger( T & t ) { doit( t ); } coroutine C {}; void main( C & c ) with( c ) { void doit( C & c ) { suspend; } logger( c ); } void mem( C & c ) { resume( c ); } int main() { C c; mem( c ); mem( c ); struct S {}; S s; void doit( S & s ) {} logger( s ); } Additional text has been added to the start of Section 3.2 addressing this comment. In fact, the end of Section 2.1 (on page 5) contains a particular paragraph that embodies this "top down" approach. It starts, "programmers can now answer three basic questions", and thus gives some practical advice for which construct you should use and when. I think giving some examples of specific applications that this paragraph, combined with some examples of cases where each construct was needed, would be a better approach. I don't think this comparison needs to be very long. It seems clear enough that one would * prefer generators for simple computations that yield up many values, * prefer coroutines for more complex processes that have significant internal structure, We do not believe the number of values yielded is an important factor is choosing between a generator or coroutine; either form can receive (input) or return (output) millions of values. As well, simple versus complex computations is also not an absolute criterion for selection as both forms can be simple or complex. As stated in the paper, a coroutine brings generality to the implementation because of the addition stack, whereas generators have restrictions on standard software-engineering practices: variable placement, no helper functions without creating an explicit generator stack, and no symmetric control-flow. It is true that simple generators probably suffer less from restrictions. Also, simplicity means the amount of work done is small so the cost of the suspend/resume becomes a factor, where experiments show suspend/resume for a generator is 10 times faster than for a coroutines. * prefer threads for cases where parallel execution is desired or needed. Agreed, but this description does not mention mutual exclusion and synchronization, which is essential in any meaningful concurrent program. Our point here is to illustrate that a casual "top down" explanation is insufficient to explain the complexity of the underlying execution properties. We presented some rule-of-thumbs at the end of Section 2 but programmers must understand all the underlying mechanisms and their interactions to exploit the execution properties to their fullest, and to understand when a programming language does or does not provide a desired mechanism. I did appreciate the comparison in Section 2.3 between async-await in JS/Java and generators/coroutines. I agree with its premise that those mechanisms are a poor replacement for generators (and, indeed, JS has a distinct generator mechanism, for example, in part for this reason). I believe I may have asked for this in a previous review, but having read it, I wonder if it is really necessary, since those mechanisms are so different in purpose. Given that you asked about this before, I believe other readers might also ask the same question because async-await is very popular. So I think this section does help to position the work in the paper among other work, and hence, it is appropriate to keep it in the paper. I find the motivation for supporting both internal and external scheduling to be fairly implicit. After several reads through the section, I came to the conclusion that internal scheduling is more expressive than external scheduling, but sometimes less convenient or clear. Is this correct? If not, it'd be useful to clarify where external scheduling is more expressive. I would find it very interesting to try and capture some of the properties that make internal vs external scheduling the better choice. For example, it seems to me that external scheduling works well if there are only a few "key" operations, but that internal scheduling might be better otherwise, simply because it would be useful to have the ability to name a signal that can be referenced by many methods. To address this point, the last paragraph on page 22 (now page 23) has been augmented to the following: Given external and internal scheduling, what guidelines can a programmer use to select between them? In general, external scheduling is easier to understand and code because only the next logical action (mutex function(s)) is stated, and the monitor implicitly handles all the details. Therefore, there are no condition variables, and hence, no wait and signal, which reduces coding complexity and synchronization errors. If external scheduling is simpler than internal, why not use it all the time? Unfortunately, external scheduling cannot be used if: scheduling depends on parameter value(s) or scheduling must block across an unknown series of calls on a condition variable (\ie internal scheduling). For example, the dating service cannot be written using external scheduling. First, scheduling requires knowledge of calling parameters to make matching decisions and parameters of calling threads are unavailable within the monitor. Specifically, a thread within the monitor cannot examine the @ccode@ of threads waiting on the calling queue to determine if there is a matching partner. (Similarly, if the bounded buffer or readers/writer are restructured with a single interface function with a parameter denoting producer/consumer or reader/write, they cannot be solved with external scheduling.) Second, a scheduling decision may be delayed across an unknown number of calls when there is no immediate match so the thread in the monitor must block on a condition. Specifically, if a thread determines there is no opposite calling thread with the same @ccode@, it must wait an unknown period until a matching thread arrives. For complex synchronization, both external and internal scheduling can be used to take advantage of best of properties of each. Consider the bounded buffer from Figure 13: if it had multiple methods for removing elements, and not just `remove`, then the `waitfor(remove)` call in `insert` might not be sufficient. Section 6.4 Extended waitfor shows the waitfor is very powerful and can handle your request: waitfor( remove : buffer ); or waitfor( remove2 : buffer ); and its shorthand form (not shown in the paper) waitfor( remove, remove2 : buffer ); A call to either of these remove functions satisfies the waitfor (exact selection details are discussed in Section 6.4). The same is true, I think, of the `signal_block` function, which I have not encountered before; In Tony Hoare's seminal paper on Monitors "Monitors: An Operating System Structuring Concept", it states on page 551: When a process signals a condition on which another process is waiting, the signalling process must wait until the resumed process permits it to proceed. We therefore introduce for each monitor a second semaphore "urgent" (initialized to 0), on which signalling processes suspend themselves by the operation P(urgent). Hence, the original definition of signal is in fact signal_block, i.e., the signaller blocks. Later implementations of monitor switched to signaller nonblocking because most signals occur before returns, which allows the signaller to continue execution, exit the monitor, and run concurrently with the signalled thread that restarts in the monitor. When the signaller is not going to exit immediately, signal_block is appropriate. it seems like its behavior can be modeled with multiple condition variables, but that's clearly more complex. Yes. Buhr, Fortier and Coffin show in Monitor Classification, ACM Computing Surveys, 27(1):63-107, that all extant monitors with different signalling semantics can be transformed into each other. However, some transformations are complex and runtime expensive. One question I had about `signal_block`: what happens if one signals but no other thread is waiting? Does it block until some other thread waits? Or is that user error? On page 20, it states: Signalling is unconditional because signalling an empty condition queue does nothing. To the best of our knowledge, all monitors have the same semantics for signalling an empty condition queue, regardless of the kind of signal, i.e., signal or signal_block. I believe that one difference between the Go program and the Cforall equivalent is that the Goroutine has an associated queue, so that multiple messages could be enqueued, whereas the Cforall equivalent is effectively a "bounded buffer" of length 1. Is that correct? Actually, the buffer length is 0 for the Cforall call and the Go unbuffered send so both are synchronous communication. I think this should be stated explicitly. (Presumably, one could modify the Cforall program to include an explicit vector of queued messages if desired, but you would also be reimplementing the channel abstraction.) Fixed, by adding the following sentences: The different between call and channel send occurs for buffered channels making the send asynchronous. In \CFA, asynchronous call and multiple buffers is provided using an administrator and worker threads~\cite{Gentleman81} and/or futures (not discussed). Also, in Figure 20, I believe that there is a missing `mutex` keyword. Fixed. I was glad to see that the paper acknowledged that Cforall still had low-level atomic operations, even if their use is discouraged in favor of higher-level alternatives. There was never an attempt to not acknowledged that Cforall had low-level atomic operations. The original version of the paper stated: 6.6 Low-level Locks For completeness and efficiency, Cforall provides a standard set of low-level locks: recursive mutex, condition, semaphore, barrier, etc., and atomic instructions: fetchAssign, fetchAdd, testSet, compareSet, etc. and that section is still in the paper. In fact, we use these low-level mechanisms to build all of the high-level concurrency constructs in Cforall. However, I still feel that the conclusion overstates the value of the contribution here when it says that "Cforall high-level race-free monitors and threads provide the core mechanisms for mutual exclusion and synchronization, without the need for volatile and atomics". I feel confident that Java programmers, for example, would be advised to stick with synchronized methods whenever possible, and it seems to me that they offer similar advantages -- but they sometimes wind up using volatiles for performance reasons. I think we are agreeing violently. 99.9% of Java/Cforall/Go/Rust concurrent programs can achieve very good performance without volatile or atomics because the runtime system has already used these low-level capabilities to build an efficient set of high-level concurrency constructs. 0.1% of the time programmers need to build their own locks and synchronization mechanisms. This need also occurs for storage management. Both of these mechanisms are allowed in Cforall but are fraught with danger and should be discouraged. Specially, it takes a 7th dan Black Belt programmer to understand fencing for a WSO memory model, such as on the ARM. It doesn't help that the C++ atomics are baroque and incomprehensible. I'm sure Hans Boehm, Doug Lea, Dave Dice and me would agree that 99% of hand-crafted locks created by programmers are broken and/or non-portable. I was also confused by the term "race-free" in that sentence. In particular, I don't think that Cforall has any mechanisms for preventing *data races*, and it clearly doesn't prevent "race conditions" (which would bar all sorts of useful programs). I suppose that "race free" here might be referring to the improvements such as removing barging behavior. We use the term "race free" to mean the same as Boehm/Adve's "data-race freedom" in Boehm Hans-J., Adve Sarita V., You Don't Know Jack About Shared Variables or Memory Models. Communications ACM. 2012; 55(2):48-54. https://queue.acm.org/detail.cfm?id=2088916 which is cited in the paper. Furthermore, we never said that Cforall has mechanisms for preventing *all* data races. We said Cforall high-level race-free monitors and threads[, when used with mutex access function] (added to the paper), implies no data races within these constructs, unless a programmer directly publishes shared state. This approach is exactly what Boehm/Adve advocate for the vast majority of concurrent programming. It would perhaps be more interesting to see a comparison built using [tokio] or [async-std], two of the more prominent user-space threading libraries that build on Rust's async-await feature (which operates quite differently than Javascript's async-await, in that it doesn't cause every aync function call to schedule a distinct task). Done. Several figures used the `with` keyword. I deduced that `with(foo)` permits one to write `bar` instead of `foo.bar`. It seems worth introducing. Apologies if this is stated in the paper, if so I missed it. Page 6, footnote F states: The Cforall "with" clause opens an aggregate scope making its fields directly accessible, like Pascal "with", but using parallel semantics; multiple aggregates may be opened. On page 20, section 6.3, "external scheduling and vice versus" should be "external scheduling and vice versa". Fixed. On page 5, section 2.3, the paper states "we content" but it should be "we contend". Fixed. Page 1. I don't believe that it s fair to imply that Scala is "research vehicle" as it is used by major players, Twitter being the most prominent example. Fixed. Removed "research". Page 15. Must Cforall threads start after construction (e.g. see your example on page 15, line 21)? Yes. Our experience in Java, uC++ and Cforall is that 95% of the time programmers want threads to start immediately. (Most Java programs have no code between a thread declaration and the call to start the thread.) Therefore, this semantic should be the default because (see page 13): Alternatives, such as explicitly starting threads as in Java, are repetitive and forgetting to call start is a common source of errors. To handle the other 5% of the cases, there is a trivial Cforall pattern providing Java-style start/join. The additional cost for this pattern is small in comparison to the work performed between the start and join. thread T {}; void start( T & mutex ) {} // any function name void join( T & mutex ) {} // any function name void main( T & t ) { sout | "start"; waitfor( start : t ); // wait to be started sout | "restart"; // perform work waitfor( join : t ); // wait to be joined sout | "join"; } int main() { T t[3]; // threads start and delay sout | "need to start"; for ( i; 3 ) start( t[i] ); sout | "need to join"; for ( i; 3 ) join( t[i] ); sout | "threads stopped"; } // threads deleted from stack $ a.out need to start start start start need to join restart restart restart threads stopped join join join Page 18, line 17: is using Fixed.