Changes in / [d2cdd4f:0a061c0]

Files:

• benchmark/Makefile.am (modified) (3 diffs)
• doc/theses/andrew_beach_MMath/performance.tex (modified) (4 diffs)

benchmark/Makefile.am

@@ -74 +74 @@
         echo "int main() { return 0; }" > ${@}
 
+.SILENT:                # do not print recipe
 .NOTPARALLEL:
 .PHONY: jenkins cleancsv
@@ -115 +116 @@
 
 %.run : %$(EXEEXT) ${REPEAT}
-        @rm -f .result.log
-        @echo "------------------------------------------------------"
-        @echo $<
-        @${REPEAT} ${repeats} -- ./a.out\
+        rm -f .result.log
+        echo "------------------------------------------------------"
+        echo $<
+        ${REPEAT} ${repeats} -- ./a.out\
                 $(if ${$(subst -,_,$(basename $@))_DURATION},\
                         ${$(subst -,_,$(basename $@))_DURATION},\
                         ${$(firstword $(subst -, ,$(basename $@)))_DURATION}) | tee -a .result.log
-        @${STATS} .result.log
-        @echo "------------------------------------------------------"
-        @rm -f a.out .result.log *.class
+        ${STATS} .result.log
+        echo "------------------------------------------------------"
+        rm -f a.out .result.log *.class
 
 #       ${REPEAT} ${repeats} -- /usr/bin/time -f "%Uu %Ss %Er %Mkb" ./a.out
@@ -149 +150 @@
 jenkins$(EXEEXT): cleancsv
 @DOifskipcompile@
-        @+make --silent compile.csv
-        @-+make --silent compile.diff.csv
+        +make compile.csv
+        -+make compile.diff.csv
 @DOendif@
-        @+make --silent ctxswitch.csv
-        @-+make --silent ctxswitch.diff.csv
-        @+make --silent mutex.csv
-        @-+make --silent mutex.diff.csv
-        @+make --silent schedint.csv
-        @-+make --silent schedint.diff.csv
+        +make ctxswitch.csv
+        -+make ctxswitch.diff.csv
+        +make mutex.csv
+        -+make mutex.diff.csv
+        +make schedint.csv
+        -+make schedint.diff.csv
 @DOifskipcompile@
         cat compile.csv

doc/theses/andrew_beach_MMath/performance.tex

@@ -42 +42 @@
 repeatedly. This is to avoids start-up or tear-down time from
 affecting the timing results.
-Tests ran their main loop a million times.
+Most test were run 1 000 000 (a million) times.
 The Java versions of the test also run this loop an extra 1000 times before
 beginning to time the results to ``warm-up" the JVM.
@@ -130 +130 @@
 
 \section{Results}
-Each test was run eleven times. The top three and bottom three results were
-discarded and the remaining five values are averaged.
+Each test is was run five times, the best and worst result were discarded and
+the remaining values were averaged.
 
 In cases where a feature is not supported by a language the test is skipped
@@ -138 +138 @@
 was put into the termination column.
 
-% Raw Data:
-% run-algol-a.sat
-% ---------------
-% Raise Empty   &  82687046678 &  291616256 &   3252824847 & 15422937623 & 14736271114 \\
-% Raise D'tor   & 219933199603 &  297897792 & 223602799362 &         N/A &         N/A \\
-% Raise Finally & 219703078448 &  298391745 &          N/A &         ... & 18923060958 \\
-% Raise Other   & 296744104920 & 2854342084 & 112981255103 & 15475924808 & 21293137454 \\
-% Cross Handler &      9256648 &   13518430 &       769328 &     3486252 &    31790804 \\
-% Cross Finally &       769319 &        N/A &          N/A &     2272831 &    37491962 \\
-% Match All     &   3654278402 &   47518560 &   3218907794 &  1296748192 &   624071886 \\
-% Match None    &   4788861754 &   58418952 &   9458936430 &  1318065020 &   625200906 \\
-%
-% run-algol-thr-c
-% ---------------
-% Raise Empty   &   3757606400 &   36472972 &   3257803337 & 15439375452 & 14717808642 \\
-% Raise D'tor   &  64546302019 &  102148375 & 223648121635 &         N/A &         N/A \\
-% Raise Finally &  64671359172 &  103285005 &          N/A & 15442729458 & 18927008844 \\
-% Raise Other   & 294143497130 & 2630130385 & 112969055576 & 15448220154 & 21279953424 \\
-% Cross Handler &      9646462 &   11955668 &       769328 &     3453707 &    31864074 \\
-% Cross Finally &       773412 &        N/A &          N/A &     2253825 &    37266476 \\
-% Match All     &   3719462155 &   43294042 &   3223004977 &  1286054154 &   623887874 \\
-% Match None    &   4971630929 &   55311709 &   9481225467 &  1310251289 &   623752624 \\
 \begin{tabular}{|l|c c c c c|}
 \hline
@@ -174 +152 @@
 \hline
 \end{tabular}
-
-% run-plg7a-a.sat
-% ---------------
-% Raise Empty   &  57169011329 &  296612564 &   2788557155 & 17511466039 & 23324548496 \\
-% Raise D'tor   & 150599858014 &  318443709 & 149651693682 &         N/A &         N/A \\
-% Raise Finally & 148223145000 &  373325807 &          N/A &         ... & 29074552998 \\
-% Raise Other   & 189463708732 & 3017109322 &  85819281694 & 17584295487 & 32602686679 \\
-% Cross Handler &      8001654 &   13584858 &      1555995 &     6626775 &    41927358 \\
-% Cross Finally &      1002473 &        N/A &          N/A &     4554344 &    51114381 \\
-% Match All     &   3162460860 &   37315018 &   2649464591 &  1523205769 &   742374509 \\
-% Match None    &   4054773797 &   47052659 &   7759229131 &  1555373654 &   744656403 \\
-%
-% run-plg7a-thr-a
-% ---------------
-% Raise Empty   &   3604235388 &   29829965 &   2786931833 & 17576506385 & 23352975105 \\
-% Raise D'tor   &  46552380948 &  178709605 & 149834207219 &         N/A &         N/A \\
-% Raise Finally &  46265157775 &  177906320 &          N/A & 17493045092 & 29170962959 \\
-% Raise Other   & 195659245764 & 2376968982 &  86070431924 & 17552979675 & 32501882918 \\
-% Cross Handler &    397031776 &   12503552 &      1451225 &     6658628 &    42304965 \\
-% Cross Finally &      1136746 &        N/A &          N/A &     4468799 &    46155817 \\
-% Match All     &   3189512499 &   39124453 &   2667795989 &  1525889031 &   733785613 \\
-% Match None    &   4094675477 &   48749857 &   7850618572 &  1566713577 &   733478963 \\
-
-% PLG7A (in seconds)
-\begin{tabular}{|l|c c c c c|}
-\hline
-              & \CFA (Terminate) & \CFA (Resume) & \Cpp & Java & Python \\
-\hline
-% Raise Empty   & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\
-% Raise D'tor   & 0.0 & 0.0 & 0.0 & N/A & N/A \\
-% Raise Finally & 0.0 & 0.0 & N/A & 0.0 & 0.0 \\
-% Raise Other   & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\
-% Cross Handler & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\
-% Cross Finally & 0.0 & N/A & N/A & 0.0 & 0.0 \\
-% Match All     & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\
-% Match None    & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\
-Raise Empty   & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\
-Raise D'tor   & 0.0 & 0.0 & 0.0 & N/A & N/A \\
-Raise Finally & 0.0 & 0.0 & N/A & 0.0 & 0.0 \\
-Raise Other   & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\
-Cross Handler & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\
-Cross Finally & 0.0 & N/A & N/A & 0.0 & 0.0 \\
-Match All     & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\
-Match None    & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\
-\hline
-\end{tabular}
-
-One result that is not directly related to \CFA but is important to keep in
-mind is that in exceptions the standard intuitions about which languages
-should go faster often do not hold. There are cases where Python out-preforms
-\Cpp and Java. The most likely explination is that, since exceptions are
-rarely considered to be the common case, the more optimized langages have 
-optimized at their expence. In addition languages with high level            
-repersentations have a much easier time scanning the stack as there is less
-to decode.
-
-This means that while \CFA does not actually keep up with Python in every
-case it is no worse than roughly half the speed of \Cpp. This is good
-enough for the prototyping purposes of the project.
-
-One difference not shown is that optimizations in \CFA is very fragile.
-The \CFA compiler uses gcc as part of its complation process and the version
-of gcc could change the speed of some of the benchmarks by 10 times or more.
-Similar changes to g++ for the \Cpp benchmarks had no significant changes.
-Because of the connection between gcc and g++; this suggests it is not the
-optimizations that are changing but how the optimizer is detecting if the
-optimizations can be applied. So the optimizations are always applied in
-g++, but only newer versions of gcc can detect that they can be applied in
-the more complex \CFA code.
-
-Resumption exception handling is also incredibly fast. Often an order of
-magnitude or two better than the best termination speed.
-There is a simple explination for this; traversing a linked list is much   
-faster than examining and unwinding the stack. When resumption does not do as
-well its when more try statements are used per raise. Updating the interal
-linked list is not very expencive but it does add up.
-
-The relative speed of the Match All and Match None tests (within each
-language) can also show the effectiveness conditional matching as compared
-to catch and rethrow.
-\begin{itemize}[nosep]
-\item
-Java and Python get similar values in both tests.
-Between the interperated code, a higher level repersentation of the call
-stack and exception reuse it it is possible the cost for a second
-throw can be folded into the first.
-% Is this due to optimization?
-\item
-Both types of \CFA are slighly slower if there is not a match.
-For termination this likely comes from unwinding a bit more stack through
-libunwind instead of executing the code normally.
-For resumption there is extra work in traversing more of the list and running
-more checks for a matching exceptions.
-% Resumption is a bit high for that but this is my best theory.
-\item
-Then there is \Cpp, which takes 2--3 times longer to catch and rethrow vs.
-just the catch. This is very high, but it does have to repeat the same
-process of unwinding the stack and may have to parse the LSDA of the function
-with the catch and rethrow twice, once before the catch and once after the
-rethrow.
-% I spent a long time thinking of what could push it over twice, this is all
-% I have to explain it.
-\end{itemize}
-The difference in relative performance does show that there are savings to
-be made by performing the check without catching the exception.