# Changeset 060b12d for doc/theses/aaron_moss_PhD/phd/experiments.tex

Ignore:
Timestamp:
Feb 15, 2019, 9:00:06 PM (3 years ago)
Branches:
aaron-thesis, arm-eh, cleanup-dtors, jacob/cs343-translation, jenkins-sandbox, master, new-ast, new-ast-unique-expr
Children:
44eeddd, a6f991f
Parents:
4c41b17
Message:

Add scatter plots describing instance difficulty

File:
1 edited

### Legend:

Unmodified
 r4c41b17 Since the top centile of expression resolution instances requires approximately two-thirds of the resolver's time, optimizing the resolver for specific hard problem instances has proven to be an effective technique for reducing overall runtime. \TODO{Discuss metrics of difficulty.} % TODO: look at overloads % TODO: histogram of hard cases The data below indicates that number of assertions necessary to resolve has the greatest effect on runtime, as seen in Figure~\ref{per-prob-assns-fig}. However, since the number of assertions required is only known once resolution is finished, the most-promising pre-resolution metric of difficulty is the nesting depth of the expression; as seen in Figure~\ref{per-prob-depth-fig}, expressions of depth $> 10$ in this dataset are uniformly difficult. Figure~\ref{per-prob-subs-fig} presents a similar pattern for number of subexpressions, though given that the expensive tail of problem instances occurs at approximately twice the depth values, it is reasonable to believe that the difficult expressions in question are deeply-nested invocations of binary functions rather than wider but shallowly-nested expressions. % TODO statistics to tease out difficulty? Is ANOVA the right keyword? % TODO maybe metrics to sum number of poly-overloads invoked \begin{figure} \centering \input{per-prob-assns} \caption[Top-level expression resolution time by number of assertions resolved.]{Top-level expression resolution time by number of assertions resolved. Note log scales on both axes.} \label{per-prob-assns-fig} \end{figure} \begin{figure} \centering \input{per-prob-depth} \caption[Top-level expression resolution time by maximum nesting depth of expression.]{Top-level expression resolution time by maximum nesting depth of expression. Note log scales on both axes.} \label{per-prob-depth-fig} \end{figure} \begin{figure} \centering \input{per-prob-subs} \caption[Top-level expression resolution time by number of subexpressions.]{Top-level expression resolution time by number of subexpressions. Note log scales on both axes.} \label{per-prob-subs-fig} \end{figure} \section{\CFA{} Results}