Index: doc/theses/andrew_beach_MMath/performance.tex =================================================================== --- doc/theses/andrew_beach_MMath/performance.tex (revision 1e567ab33db4d4af31f49c41af45949205368ff5) +++ doc/theses/andrew_beach_MMath/performance.tex (revision 0b67a196067335695e22840084565d940d44c522) @@ -42,5 +42,5 @@ repeatedly. This is to avoids start-up or tear-down time from affecting the timing results. -Most test were run 1 000 000 (a million) times. +Tests ran their main loop a million times. The Java versions of the test also run this loop an extra 1000 times before beginning to time the results to ``warm-up" the JVM. @@ -130,6 +130,6 @@ \section{Results} -Each test is was run five times, the best and worst result were discarded and -the remaining values were averaged. +Each test was run eleven times. The top three and bottom three results were +discarded and the remaining five values are averaged. In cases where a feature is not supported by a language the test is skipped @@ -138,4 +138,26 @@ was put into the termination column. +% Raw Data: +% run-algol-a.sat +% --------------- +% Raise Empty & 82687046678 & 291616256 & 3252824847 & 15422937623 & 14736271114 \\ +% Raise D'tor & 219933199603 & 297897792 & 223602799362 & N/A & N/A \\ +% Raise Finally & 219703078448 & 298391745 & N/A & ... & 18923060958 \\ +% Raise Other & 296744104920 & 2854342084 & 112981255103 & 15475924808 & 21293137454 \\ +% Cross Handler & 9256648 & 13518430 & 769328 & 3486252 & 31790804 \\ +% Cross Finally & 769319 & N/A & N/A & 2272831 & 37491962 \\ +% Match All & 3654278402 & 47518560 & 3218907794 & 1296748192 & 624071886 \\ +% Match None & 4788861754 & 58418952 & 9458936430 & 1318065020 & 625200906 \\ +% +% run-algol-thr-c +% --------------- +% Raise Empty & 3757606400 & 36472972 & 3257803337 & 15439375452 & 14717808642 \\ +% Raise D'tor & 64546302019 & 102148375 & 223648121635 & N/A & N/A \\ +% Raise Finally & 64671359172 & 103285005 & N/A & 15442729458 & 18927008844 \\ +% Raise Other & 294143497130 & 2630130385 & 112969055576 & 15448220154 & 21279953424 \\ +% Cross Handler & 9646462 & 11955668 & 769328 & 3453707 & 31864074 \\ +% Cross Finally & 773412 & N/A & N/A & 2253825 & 37266476 \\ +% Match All & 3719462155 & 43294042 & 3223004977 & 1286054154 & 623887874 \\ +% Match None & 4971630929 & 55311709 & 9481225467 & 1310251289 & 623752624 \\ \begin{tabular}{|l|c c c c c|} \hline @@ -152,2 +174,107 @@ \hline \end{tabular} + +% run-plg7a-a.sat +% --------------- +% Raise Empty & 57169011329 & 296612564 & 2788557155 & 17511466039 & 23324548496 \\ +% Raise D'tor & 150599858014 & 318443709 & 149651693682 & N/A & N/A \\ +% Raise Finally & 148223145000 & 373325807 & N/A & ... & 29074552998 \\ +% Raise Other & 189463708732 & 3017109322 & 85819281694 & 17584295487 & 32602686679 \\ +% Cross Handler & 8001654 & 13584858 & 1555995 & 6626775 & 41927358 \\ +% Cross Finally & 1002473 & N/A & N/A & 4554344 & 51114381 \\ +% Match All & 3162460860 & 37315018 & 2649464591 & 1523205769 & 742374509 \\ +% Match None & 4054773797 & 47052659 & 7759229131 & 1555373654 & 744656403 \\ +% +% run-plg7a-thr-a +% --------------- +% Raise Empty & 3604235388 & 29829965 & 2786931833 & 17576506385 & 23352975105 \\ +% Raise D'tor & 46552380948 & 178709605 & 149834207219 & N/A & N/A \\ +% Raise Finally & 46265157775 & 177906320 & N/A & 17493045092 & 29170962959 \\ +% Raise Other & 195659245764 & 2376968982 & 86070431924 & 17552979675 & 32501882918 \\ +% Cross Handler & 397031776 & 12503552 & 1451225 & 6658628 & 42304965 \\ +% Cross Finally & 1136746 & N/A & N/A & 4468799 & 46155817 \\ +% Match All & 3189512499 & 39124453 & 2667795989 & 1525889031 & 733785613 \\ +% Match None & 4094675477 & 48749857 & 7850618572 & 1566713577 & 733478963 \\ + +% PLG7A (in seconds) +\begin{tabular}{|l|c c c c c|} +\hline + & \CFA (Terminate) & \CFA (Resume) & \Cpp & Java & Python \\ +\hline +% Raise Empty & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ +% Raise D'tor & 0.0 & 0.0 & 0.0 & N/A & N/A \\ +% Raise Finally & 0.0 & 0.0 & N/A & 0.0 & 0.0 \\ +% Raise Other & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ +% Cross Handler & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ +% Cross Finally & 0.0 & N/A & N/A & 0.0 & 0.0 \\ +% Match All & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ +% Match None & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ +Raise Empty & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ +Raise D'tor & 0.0 & 0.0 & 0.0 & N/A & N/A \\ +Raise Finally & 0.0 & 0.0 & N/A & 0.0 & 0.0 \\ +Raise Other & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ +Cross Handler & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ +Cross Finally & 0.0 & N/A & N/A & 0.0 & 0.0 \\ +Match All & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ +Match None & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ +\hline +\end{tabular} + +One result that is not directly related to \CFA but is important to keep in +mind is that in exceptions the standard intuitions about which languages +should go faster often do not hold. There are cases where Python out-preforms +\Cpp and Java. The most likely explination is that, since exceptions are +rarely considered to be the common case, the more optimized langages have +optimized at their expence. In addition languages with high level +repersentations have a much easier time scanning the stack as there is less +to decode. + +This means that while \CFA does not actually keep up with Python in every +case it is no worse than roughly half the speed of \Cpp. This is good +enough for the prototyping purposes of the project. + +One difference not shown is that optimizations in \CFA is very fragile. +The \CFA compiler uses gcc as part of its complation process and the version +of gcc could change the speed of some of the benchmarks by 10 times or more. +Similar changes to g++ for the \Cpp benchmarks had no significant changes. +Because of the connection between gcc and g++; this suggests it is not the +optimizations that are changing but how the optimizer is detecting if the +optimizations can be applied. So the optimizations are always applied in +g++, but only newer versions of gcc can detect that they can be applied in +the more complex \CFA code. + +Resumption exception handling is also incredibly fast. Often an order of +magnitude or two better than the best termination speed. +There is a simple explination for this; traversing a linked list is much +faster than examining and unwinding the stack. When resumption does not do as +well its when more try statements are used per raise. Updating the interal +linked list is not very expencive but it does add up. + +The relative speed of the Match All and Match None tests (within each +language) can also show the effectiveness conditional matching as compared +to catch and rethrow. +\begin{itemize}[nosep] +\item +Java and Python get similar values in both tests. +Between the interperated code, a higher level repersentation of the call +stack and exception reuse it it is possible the cost for a second +throw can be folded into the first. +% Is this due to optimization? +\item +Both types of \CFA are slighly slower if there is not a match. +For termination this likely comes from unwinding a bit more stack through +libunwind instead of executing the code normally. +For resumption there is extra work in traversing more of the list and running +more checks for a matching exceptions. +% Resumption is a bit high for that but this is my best theory. +\item +Then there is \Cpp, which takes 2--3 times longer to catch and rethrow vs. +just the catch. This is very high, but it does have to repeat the same +process of unwinding the stack and may have to parse the LSDA of the function +with the catch and rethrow twice, once before the catch and once after the +rethrow. +% I spent a long time thinking of what could push it over twice, this is all +% I have to explain it. +\end{itemize} +The difference in relative performance does show that there are savings to +be made by performing the check without catching the exception.