AUTOTUNING SYSTEM

VERSION 0.16

The full developer documentation is available at: doc/dev.pdf
The user documentation, which explains the system is: doc/user.pdf


FILES

0.14
|-- autotune                The tuner. (link to tuner/tune.py)
|
|-- doc                     Documentation for the tuner
|   |-- dev.pdf (.tex)      Developer Documentation
|   |-- tutorial.pdf (.tex) A Beginner's Tutorial
|   `-- user.pdf (.tex)     User Documentation
|
|-- examples                Example programs to demonstrate the system's use.
|   |-- hello               Simple test case, compiles a 'hello world' program.
|   |-- laplace3d           A CUDA test case.
|   |-- looping             Simple test case where parameters control the 
|   |                         number of loop iterations.
|   |-- maths               Simple test case where parameters are summed using 
|   |                         'expr'. Uses a custom figure of merit.
|   |-- matlab              A MATLAB program to determine the optimum level 
|   |                         of 'strip-mining' vectorisation.
|   `-- matrix              Blocked matrix multiplication test case.
|       |-- comparison      Compares the blocked and naive implementations.
|       |-- modified        The modified, tunable, version, after the changes 
|       |                     from the tutorial.
|       `-- original        The original version, not ready for tuning. 
|                             Used by the tutorial.
|
|-- README_Dev              Breif intro for developers.
|-- README_User             Breif intro for users.
|
|-- tuner                   The source code for the tuner itself.
|   |
|   |-- evaluator.py        Defines how tests are evaluated, handling all 
|   |                         compilation and command execution. This is 
|   |                         encapsulated in the Evaluator class.
|   |
|   |-- helpers.py          Provides several small helper functions.
|   |
|   |-- logging.py          Allows output of the testing logs as a CSV file.
|   |
|   |-- optimisation_bf.py  A brute force optimiser, OptimisationBF, 
|   |                         implementing the same methods as Optimisation.
|   |
|   |-- optimisation.py     The optimisation algorithm. Defines the 
|   |                         Optimisation class.
|   |
|   |-- output.py           Controls where output from the tuner is sent.
|   |                         e.g. to the screen or a script file.
|   |
|   |-- test_evaluations.py This defines generateEvalFunc, which is a function 
|   |                         taking a VarTree and retunrning an evaluate() 
|   |                         function which exhibits the exact variable 
|   |                         independence described by the VarTree. This is 
|   |                         used for testing. Also defines the FuncEvaluator 
|   |                         class, implementing the same interface as 
|   |                         Evaluator, but using an evaluation function 
|   |                         instead of running commands.
|   |
|   |-- testing.py          Checks Optimisation against OptimisationBF for 
|   |                         different inputs.
|   |
|   |-- tune_conf.py        Reads settings from the config file.
|   |
|   |-- tune.py             The main script, sets up and runs the optimisation.
|   |
|   |-- vartree_parser.py   Defines a vartree parser, generated by the wisent 
|   |                         parser-generator. Used by vartree.py to convert 
|   |                         token strings to parse trees.
|   |
|   `-- vartree.py          Defines the VarTree class and uses the parser to 
|                             convert strings to instances of VarTree.
|
`-- utilities               Utilities to analyse tuning results.
    |-- common.py           Some helper functions for reading in the CSV file.
    |-- csv_plot.m          A matlab script to draw plots.
    |-- output_gnuplot.py   Outputs a 'gnuplot' script for plotting graphs.
    |-- output_screen.py    Displays a graph on screen with 'matplotlib'.
    `-- parameter_importance.py  Compares how much effect each parameter has on 
                            the score of the tests.










CHANGES (0.16)

Added the utility exhaustive_results.py, which reconstructs a CSV file of all 
  brute-force tests from the normal tuner's CSV file.

Began work towards refactoring Evaluate to separate the evaluation strategy 
  (e.g. sequential, parallel, interleaved, ...) from the mechanincs of running 
  and scoring tests.




CHANGES (0.15)

Added a check for KeyboardInterrupt (Ctrl+C) to tune.py while the tuning is 
  running. The tuner will now exit gracefully if this exception is thrown, and 
  will write a partial log file. Added a note about this to the user docs.

Added the -s or --stddev flag to output_screen, allowing the standard deviation 
  from the mean to be added to the plot. Currently, this is centred on the 
  mean (NOT typically the overall score), not sure if this is the most 
  desirable behaviour? Updated the user docs and README_Utilities.

Added the 'script' option to the configuration file, to log output. This 
  involved creating the output.py module, to control where output is sent, 
  and modifying the output parts of evaluator.py.

Added documentation of the new 'script' option.

Added the utility parameter_importance.py which performs a basic analysis of 
  the effect each parameter has on the score of a test.

Added the 'importance' option, which will cause extra tests to be run 
  (if needed) to gather more useful parameter importance data than the main CSV 
  file. This is saved as a separate file (possibly re-using some results) which 
  can be used with the new utility.



CHANGES (0.14)

New directory structure:
0.14
|-- doc
|   `-- [ALL DOCUMENTATION]
|-- examples
|   `-- [ALL EXAMPLE PROGRAMS, in separate subfolders]
|-- tuner
|   `-- [THE CODE]
`-- utilities
    `-- [RESULTS ANALYSIS, graph plotters, etc.]

Added first utility: output_gnuplot.py, which converts a csv file generated by 
  the system into a gnuplot source file to plot a graph. Made some small 
  improvements to the graphs over the 'integrated' version.

Began work on a second utility: output_screen.py, which uses matplotlib to draw 
  the graphs on screen. (which can also be saved)

Removed gnplot output from the main tuner, because this is now handled by the 
  output_gnuplot utility, which converts the CSV file into a gnuplot file.

Reorganised the configuration file format, to hopefully make things more 
  logically laid out. Updated the examples.

Wrote most of the user documentation and a tutorial, some developer docs.

Checked which version of python is required (2.5), and added a check for 
  this to the main script.

Added a function to Optimisation (and OptimisationBF) which calculates the 
  number of tests which will be required during tuning. This is displayed 
  before tuning begins.

Added the -r or --reference flag to output_gnuplot and output_screen to allow 
  plotting of a reference line to compare the tuner's results to some reference 
  measurement.


CHANGES (0.13)

Restructured the optimisation and logging to make everything more flexible.
  Add the Evaluator class, in evaluator.py 
    (replaces the old evaluate() functions)
  Remove the Logging class, logs are kept by Evaluator. Log output (e.g. to 
    CSV) is now handled by seperate functions. These are in logging.py and were
    reworked to reduce redundancy.
  Added the FuncEvaluator class to test_evaluations.py, for "dry run" testing.

Added helpers.py which provides several small helper functions used in various 
parts of the code.

Removed various problems with failed tests and other 'special cases'.


CHANGES (0.12)

Added gnuplot output, which creates a *.plt (and corresponding *.dat) file 
to display a graph of the testing process.

Began (slightly) restructuring the code to be more flexible.


CHANGES (0.11) (version used in project report)

Modified the testing code in testing.py and so the two tests are the same.

Generally clean up all code.

Changed all references to varlist to vartree, a much better name.


CHANGES (0.10)

Updated the parser to allow vartrees to omit the braces for simple, flat lists.

Added standard deviation and coefficient of variation to the stats which are 
reported for the testing.

N.B. This is done by keeping a running sum (for mean) and running sum of 
squares (for std dev.), which will be slightly less accurate for large numbers 
of tests.

Added a check for invalid numbers of test repettitions.

Added a check for repeated variable names in the VarTree

Added logging feature so that a testing run can be saved out to a CSV file.

Added an option to the conf file allowing the user to choose how multiple tests 
are aggregated into the overall score. Previously it was an average, now they 
can choose between min/med/max/avg.





