\chapter{Experiments}
\label{expr-chap}

I have implemented a prototype system to test the practical effectiveness of the various algorithms described in Chapters~\ref{resolution-chap} and~\ref{env-chap}. 
This prototype system essentially just implements the expression resolution pass of \CFACC{}, with a simplified version of the \CFA{} type system and a parser to read in problem instances. 
The prototype system allows for quicker iteration on algorithms due to its simpler language model and lack of a requirement to generate runnable code, yet captures enough of the nuances of \CFA{} to have some predictive power for the runtime performance of algorithmic variants in \CFACC{} itself. 

There are three sources of problem instances for the resolver prototype. 
The first is small, hand-written tests designed to test the expressive power and correctness of the prototype. 
These tests are valuable for regression testing, but not time-consuming enough to be useful performance tests. 
The second sort of problem instances are procedurally generated according to a set of parameters (distributions of polymorphic versus monomorphic functions, number of function arguments, number of types, \etc{}); the procedural problem generator can be used to explore the behaviour of an algorithm with respect to certain sorts of problem instances by varying the input parameters. 
I have implemented a flagged \CFACC{} pass which outputs information which can be used to initialize the procedural generator's parameters to realistic values. 
The final sort of problem instances are derived from actual \CFA{} code. 
The prototype has a rich enough representation of \CFA{} that actual instances of expression resolution can be expressed with good fidelity, and I have implemented a compiler pass for \CFACC{} which can generate instances from \CFA{} code. 
Since at this juncture all development in \CFA{} is done by our research team, I have tested the prototype system on all \CFA{} code currently extant, primarily the standard library and compiler test suite.

\section{Resolver Prototype Features}

The resolver prototype can express most of the \CFA{} features described in Chapter~\ref{cfa-chap}. 
It supports both monomorphic and polymorphic functions, with type assertions for polymorphic functions. 
Traits are not explicitly represented, but \CFACC{} inlines traits before the resolver pass, so this is a faithful representation of the existing compiler problem. 
The prototype system supports variable declarations as well as function declarations, and has a lexical-scoping scheme which obeys \CFA{}-like overloading and overriding rules. 

The type system of the resolver prototype also captures the key aspects of the \CFA{} type system. 
\emph{Concrete types} represent the built-in arithmetic types of \CFA{}, along with the implicit conversions between them. 
Each concrete type is represented by an integer ID, and the conversion cost from $x$ to $y$ is $|y-x|$, a safe conversion if $y > x$, or an unsafe conversion if $y < x$. 
This is markedly simpler than the graph of conversion costs in \CFA{}, but captures the essentials of the design well. 
For simplicity, !zero_t! and !one_t!, the types of !0! and !1!, are represented by the type corresponding to !int!.
\emph{Named types} are analogues to \CFA{} aggregates, such as structs and unions; aggregate fields are encoded as unary functions from the struct type to the field type, named based on the field name. 
Named types also support type parameters, and as such can represent generic types as well. 
Generic named types are used to represent the built-in parameterized types of \CFA{} as well; !T*! is encoded as \texttt{\#\$ptr<T>}. 
\CFA{} arrays are also represented as pointers, to simulate array-to-pointer decay, while top-level reference types are replaced by their referent to simulate the variety of reference conversions. 
Function types have first-class representation in the prototype as the type of both function declarations and function pointers, though the function type in the prototype system loses information about type assertions, so polymorphic function pointers cannot be expressed.
Void and tuple types are also supported in the prototype, to express the multiple-return-value functions in \CFA{}, though varargs functions and !ttype! tuple-typed type variables are absent from the prototype system.
The prototype system also does not represent type qualifiers (\eg{} !const!, !volatile!), so all such qualifiers are stripped during conversion to the prototype system. 

The resolver prototype supports three sorts of expressions in its input language. 
The simplest are \emph{value expressions}, which are expressions declared to be a certain type; these implement literal expressions in \CFA{}, and, already being typed, are passed through the resolver unchanged.
The second sort, \emph{name expressions}, represent a variable expression in \CFA{}; these contain the name of a variable or function, and are matched to an appropriate declaration overloading that name by the resolver. 
The third input expression, the \emph{function expression}, represents a call to a function, with a name and zero or more argument subexpressions. 
As is usual in \CFA{}, operators are represented as function calls; however, as mentioned above, the prototype system represents field access expressions !a.f! as function expressions as well. 

The main area for future expansion in the design of the resolver prototype is conversions. 
Cast expressions are implemented in the output language of the resolver, but cannot be expressed in the input. 
The only implicit conversions supported are between the arithmetic-like concrete types, which captures most, but not all, of \CFA{}'s built-in implicit conversions\footnote{Notable absences include \lstinline{void*} to other pointer types, or \lstinline{0} to pointer types.}. 
Future work should include a way to express implicit (and possibly explicit) conversions in the input language, with an investigation of the most efficient way to handle implicit conversions, and potentially design for user-defined conversions.

\section{Resolver Prototype Design}

As discussed above, the resolver prototype works over a simplified version of the \CFA{} type system, for speed of development. 
The build system for the resolver prototype uses a number of conditional compilation flags to switch between algorithm variants while retaining maximal shared code. 
A different executable name is also generated for each algorithmic variant so that distinct variants can be more easily tested against each other.

The primary architectural difference between the resolver prototype and \CFACC{} is that the prototype system uses a simple mark-and-sweep garbage collector for memory management, while \CFACC{} takes a manual memory management approach. 
This decision was made for the purpose of faster development iteration, but has proved to be a significant performance benefit as well. 
\CFACC{} frequently needs to make deep clones of significant object graphs to ensure memory ownership (followed by eventual deletion of these clones), an unnecessarily time-consuming process. 
The prototype, on the other hand, only needs to clone modified nodes, and can share identical subsets of the object graph. 
The key design decision enabling this is that all subnodes are held by !const! pointer, and thus cannot be mutated once they have been stored in a parent node.
With minimal programming discipline, it can thus be ensured that any expression is either mutable or shared, but never both; the Dotty research compiler for Scala takes a similar architectural approach\cit{}. % this citation would be "personal correspondence"
The tree mutator abstraction is designed to take advantage of this, only creating new nodes if a node must actually be mutated.
I attempted to port this garbage collector to \CFACC{}, but without success. 
The GC could be used for memory management with few changes to the codebase, but without a substantial re-write to enforce the same ``!const! children'' discipline \CFACC{} could not take advantage of the potential to share sub-objects; without sharing of sub-objects the GC variant of \CFACC{} must do all the same allocations and deletions and garbage-collector overhead degraded performance unacceptably (though it did fix some known memory leaks intoduced by failures of the existing manual memory management scheme).

Another minor architectural difference between \CFACC{} and the prototype system is that \CFACC{} makes extensive use of the pointer-chasing !std::list!, !std::set!, and !std::map! data structures, while the prototype uses the array-based !std::vector! and the hash-based !unordered_! variants of !set! and !map! instead. \TODO{investigate performance difference by testing a resolver prototype variant with List etc. redefined}

The final difference between \CFACC{} and the resolver prototype is that, as an experiment in language usability, the prototype performs resolution-based rather than unification-based assertion satisfaction, as discussed in Section~\ref{resn-conclusion-sec}. 
This enables coding patterns not available in \CFACC{}, \eg{} a more flexible approach to type assertion satisfaction and better handling of functions returning polymorphic type variables that do not exist in the parameter list. 
\TODO{test performance; shouldn't be too hard to change \texttt{resolveAssertions} to use unification}

\section{Experimental Results}

% use Jenkins daily build logs to rebuild speedup graph with more data

% look back at Resolution Algorithms section for threads to tie up "does the algorithm look like this?"