Experimental design and computer performance analysis

Abstract

While queueing theory provides substantial insight into the solution of many computer performance prediction problems, there remain situations for which queueing theory is inadequate, e.g., predicting the effect of a change in the memory size of a multiprogramming system; or determining the impact of a change to an operating system tuning parameter. Experimental design provides an effective methodology for addressing these kinds of problems. This tutorial introduces experimental design as a tool for the computer performance analyst. The experimental design methodology covers both the actual design and planning of an experiment, as well as the analysis of its results. After some background discussion of the methodology and its terminology we proceed with a discussion of design issues that confront a computer performance experimenter. The different types of variability that are present in an experiment are discussed, together with techniques for their control or elimination. Next we review the analysis of experiments, stressing the importance of assumption validation. For example, the analysis associated with experimental design (e.g., the analysis of variance, or ANOVA) is straightforward and available in many statistical packages. However, the validity of this analysis depends on several key assumptions that are rarely examined. Indeed, it is the nature of computer performance data that it frequently violates these assumptions! We describe and illustrate graphical techniques for assessing compliance with these assumptions, as well as approaches for ensuring experimental data is appropriate for this analysis.