Performance Analysis is a specialist discipline involving systematic observations to enhance performance and improve decision-making, primarily delivered through the provision of objective statistical (Data Analysis) and visual feedback (Video Analysis).
Performance Analysis Process
Three basic steps in the performance analysis process:
- Data Collection: Data collection is the process by which data about program performance are obtained from an executing program. Data are normally collected in a file, either during or after execution, although in some situations it may be presented to the user in real-time. Three basic data collection techniques can be distinguished:
- Profiles record the amount of time spent in different parts of a program. This information, though minimal, is often invaluable for highlighting performance problems. Profiles typically are gathered automatically.
- Counters record either frequencies of events or cumulative times. The insertion of counters may require some programmer intervention.
- Event traces record each occurrence of various specified events, thus typically producing a large amount of data. Traces can be produced either automatically or with programmer intervention.
- Data Transformation: The raw data produced by profiles, counters, or traces are rarely in the form required to answer performance questions. Hence, data transformations are applied, often with the goal of reducing total data volume. Transformations can be used to determine mean values or other higher-order statistics or to extract profile and counter data from traces. For example, a profile recording the time spent in each subroutine on each processor might be transformed to determine the mean time spent in each subroutine on each processor and the standard deviation from this mean. Similarly, a trace can be processed to produce a histogram giving the distribution of message sizes. Each of the various performance tools described in subsequent sections incorporates some set of built-in transformations; more specialized transformations can also be coded by the programmer.
- Data Visualization: Parallel performance data are inherently multidimensional, consisting of execution times, communication costs, and so on, for multiple program components, on different processors, and for different problem sizes. Although data reduction techniques can be used in some situations to compress performance data to scalar values, it is often necessary to be able to explore the raw multidimensional data. As is well-known in computational science and engineering, this process can benefit enormously from the use of data visualization techniques. Both conventional and more specialized display techniques can be applied to performance data.
A wide variety of data collection, transformation, and visualization tools are available. When selecting a tool for a particular task, the following issues should be considered:
- Accuracy. In general, performance data obtained using sampling techniques are less accurate than data obtained by using counters or timers. In the case of timers, the accuracy of the clock must be taken into account.
- Simplicity. The best tools in many circumstances are those that collect data automatically, with little or no programmer intervention, and that provide convenient analysis capabilities.
- Flexibility. A flexible tool can be extended easily to collect additional performance data or to provide different views of the same data. Flexibility and simplicity are often opposing requirements.
- Intrusiveness. Unless a computer provides hardware support, performance data collection inevitably introduces some overhead. We need to be aware of this overhead and account for it when analyzing data.
- Abstraction. A good performance tool allows data to be examined at a level of abstraction appropriate for the programming model of the parallel program. For example, when analyzing an execution trace from a message-passing program, we probably wish to see individual messages, particularly if they can be related to send and receive statements in the source program. However, this presentation is probably not appropriate when studying a data-parallel program, even if compilation generates a message-passing program. Instead, we would like to see communication costs related to data-parallel program statements.
There are numerous ways to make use of performance analysis results and to trigger corresponding actions. In the simplest case, the results are incorporated into management reports and lay the foundation for further decisions. This is especially true for the long-term case. For the short-term case, the results of the performance analysis could automatically trigger actions within the system or could be fed into the dashboards of control centers or information systems on the shop floor to trigger immediate action by the operational personnel.
Performance-Based Earned Value® (PBEVSM)
Performance-Risk-Valuation Investment Technology (PRVit)
Performance Measurement Baseline (PMB)
Performance Measurement Tools
Performance Reference Model (PRM)