# Changeset 0b67a19 for doc/theses/andrew_beach_MMath/performance.tex

Ignore:
Timestamp:
Aug 5, 2021, 10:52:59 AM (18 months ago)
Branches:
enum, forall-pointer-decay, jacob/cs343-translation, master, new-ast-unique-expr, pthread-emulation, qualifiedEnum
Children:
b2525d9
Parents:
199894e
Message:

Andrew MMath: First draft of the performance results.

File:
1 edited

Unmodified
Added
Removed
• ## doc/theses/andrew_beach_MMath/performance.tex

 r199894e repeatedly. This is to avoids start-up or tear-down time from affecting the timing results. Most test were run 1 000 000 (a million) times. Tests ran their main loop a million times. The Java versions of the test also run this loop an extra 1000 times before beginning to time the results to warm-up" the JVM. \section{Results} Each test is was run five times, the best and worst result were discarded and the remaining values were averaged. Each test was run eleven times. The top three and bottom three results were discarded and the remaining five values are averaged. In cases where a feature is not supported by a language the test is skipped was put into the termination column. % Raw Data: % run-algol-a.sat % --------------- % Raise Empty   &  82687046678 &  291616256 &   3252824847 & 15422937623 & 14736271114 \\ % Raise D'tor   & 219933199603 &  297897792 & 223602799362 &         N/A &         N/A \\ % Raise Finally & 219703078448 &  298391745 &          N/A &         ... & 18923060958 \\ % Raise Other   & 296744104920 & 2854342084 & 112981255103 & 15475924808 & 21293137454 \\ % Cross Handler &      9256648 &   13518430 &       769328 &     3486252 &    31790804 \\ % Cross Finally &       769319 &        N/A &          N/A &     2272831 &    37491962 \\ % Match All     &   3654278402 &   47518560 &   3218907794 &  1296748192 &   624071886 \\ % Match None    &   4788861754 &   58418952 &   9458936430 &  1318065020 &   625200906 \\ % % run-algol-thr-c % --------------- % Raise Empty   &   3757606400 &   36472972 &   3257803337 & 15439375452 & 14717808642 \\ % Raise D'tor   &  64546302019 &  102148375 & 223648121635 &         N/A &         N/A \\ % Raise Finally &  64671359172 &  103285005 &          N/A & 15442729458 & 18927008844 \\ % Raise Other   & 294143497130 & 2630130385 & 112969055576 & 15448220154 & 21279953424 \\ % Cross Handler &      9646462 &   11955668 &       769328 &     3453707 &    31864074 \\ % Cross Finally &       773412 &        N/A &          N/A &     2253825 &    37266476 \\ % Match All     &   3719462155 &   43294042 &   3223004977 &  1286054154 &   623887874 \\ % Match None    &   4971630929 &   55311709 &   9481225467 &  1310251289 &   623752624 \\ \begin{tabular}{|l|c c c c c|} \hline \hline \end{tabular} % run-plg7a-a.sat % --------------- % Raise Empty   &  57169011329 &  296612564 &   2788557155 & 17511466039 & 23324548496 \\ % Raise D'tor   & 150599858014 &  318443709 & 149651693682 &         N/A &         N/A \\ % Raise Finally & 148223145000 &  373325807 &          N/A &         ... & 29074552998 \\ % Raise Other   & 189463708732 & 3017109322 &  85819281694 & 17584295487 & 32602686679 \\ % Cross Handler &      8001654 &   13584858 &      1555995 &     6626775 &    41927358 \\ % Cross Finally &      1002473 &        N/A &          N/A &     4554344 &    51114381 \\ % Match All     &   3162460860 &   37315018 &   2649464591 &  1523205769 &   742374509 \\ % Match None    &   4054773797 &   47052659 &   7759229131 &  1555373654 &   744656403 \\ % % run-plg7a-thr-a % --------------- % Raise Empty   &   3604235388 &   29829965 &   2786931833 & 17576506385 & 23352975105 \\ % Raise D'tor   &  46552380948 &  178709605 & 149834207219 &         N/A &         N/A \\ % Raise Finally &  46265157775 &  177906320 &          N/A & 17493045092 & 29170962959 \\ % Raise Other   & 195659245764 & 2376968982 &  86070431924 & 17552979675 & 32501882918 \\ % Cross Handler &    397031776 &   12503552 &      1451225 &     6658628 &    42304965 \\ % Cross Finally &      1136746 &        N/A &          N/A &     4468799 &    46155817 \\ % Match All     &   3189512499 &   39124453 &   2667795989 &  1525889031 &   733785613 \\ % Match None    &   4094675477 &   48749857 &   7850618572 &  1566713577 &   733478963 \\ % PLG7A (in seconds) \begin{tabular}{|l|c c c c c|} \hline & \CFA (Terminate) & \CFA (Resume) & \Cpp & Java & Python \\ \hline % Raise Empty   & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ % Raise D'tor   & 0.0 & 0.0 & 0.0 & N/A & N/A \\ % Raise Finally & 0.0 & 0.0 & N/A & 0.0 & 0.0 \\ % Raise Other   & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ % Cross Handler & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ % Cross Finally & 0.0 & N/A & N/A & 0.0 & 0.0 \\ % Match All     & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ % Match None    & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ Raise Empty   & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ Raise D'tor   & 0.0 & 0.0 & 0.0 & N/A & N/A \\ Raise Finally & 0.0 & 0.0 & N/A & 0.0 & 0.0 \\ Raise Other   & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ Cross Handler & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ Cross Finally & 0.0 & N/A & N/A & 0.0 & 0.0 \\ Match All     & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ Match None    & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ \hline \end{tabular} One result that is not directly related to \CFA but is important to keep in mind is that in exceptions the standard intuitions about which languages should go faster often do not hold. There are cases where Python out-preforms \Cpp and Java. The most likely explination is that, since exceptions are rarely considered to be the common case, the more optimized langages have optimized at their expence. In addition languages with high level repersentations have a much easier time scanning the stack as there is less to decode. This means that while \CFA does not actually keep up with Python in every case it is no worse than roughly half the speed of \Cpp. This is good enough for the prototyping purposes of the project. One difference not shown is that optimizations in \CFA is very fragile. The \CFA compiler uses gcc as part of its complation process and the version of gcc could change the speed of some of the benchmarks by 10 times or more. Similar changes to g++ for the \Cpp benchmarks had no significant changes. Because of the connection between gcc and g++; this suggests it is not the optimizations that are changing but how the optimizer is detecting if the optimizations can be applied. So the optimizations are always applied in g++, but only newer versions of gcc can detect that they can be applied in the more complex \CFA code. Resumption exception handling is also incredibly fast. Often an order of magnitude or two better than the best termination speed. There is a simple explination for this; traversing a linked list is much faster than examining and unwinding the stack. When resumption does not do as well its when more try statements are used per raise. Updating the interal linked list is not very expencive but it does add up. The relative speed of the Match All and Match None tests (within each language) can also show the effectiveness conditional matching as compared to catch and rethrow. \begin{itemize}[nosep] \item Java and Python get similar values in both tests. Between the interperated code, a higher level repersentation of the call stack and exception reuse it it is possible the cost for a second throw can be folded into the first. % Is this due to optimization? \item Both types of \CFA are slighly slower if there is not a match. For termination this likely comes from unwinding a bit more stack through libunwind instead of executing the code normally. For resumption there is extra work in traversing more of the list and running more checks for a matching exceptions. % Resumption is a bit high for that but this is my best theory. \item Then there is \Cpp, which takes 2--3 times longer to catch and rethrow vs. just the catch. This is very high, but it does have to repeat the same process of unwinding the stack and may have to parse the LSDA of the function with the catch and rethrow twice, once before the catch and once after the rethrow. % I spent a long time thinking of what could push it over twice, this is all % I have to explain it. \end{itemize} The difference in relative performance does show that there are savings to be made by performing the check without catching the exception.
Note: See TracChangeset for help on using the changeset viewer.