Prueba de rendimiento (informática)

En informática , un benchmark es el acto de ejecutar un programa informático , un conjunto de programas u otras operaciones, con el fin de evaluar el rendimiento relativo de un objeto, normalmente mediante la ejecución de una serie de pruebas y ensayos estándar sobre él. ^{[ 1 ]}

El término benchmark también se utiliza comúnmente para referirse a los propios programas de benchmarking diseñados con gran detalle.

Las pruebas de rendimiento suelen asociarse con la evaluación de las características de rendimiento del hardware informático , por ejemplo, el rendimiento de las operaciones de coma flotante de una CPU , pero existen circunstancias en las que la técnica también es aplicable al software . Por ejemplo, las pruebas de rendimiento de software se realizan con compiladores o sistemas de gestión de bases de datos (DBMS).

Los benchmarks proporcionan un método para comparar el rendimiento de varios subsistemas en diferentes arquitecturas de chips/sistemas . La evaluación comparativa como parte de la integración continua se denomina evaluación comparativa continua. ^{[ 2 ]}

Con el avance de la arquitectura informática , se hizo más difícil comparar el rendimiento de distintos sistemas informáticos simplemente observando sus especificaciones. Por ello, se desarrollaron pruebas que permitían comparar diferentes arquitecturas. Por ejemplo, los procesadores Pentium 4 generalmente funcionaban a una frecuencia de reloj más alta que los procesadores Athlon XP o PowerPC , lo que no se traducía necesariamente en una mayor potencia de cálculo; un procesador con una frecuencia de reloj más baja podía tener un rendimiento igual o incluso superior al de un procesador que funcionara a una frecuencia más alta. Véase BogoMips y el mito del megahercio .

Las pruebas de rendimiento están diseñadas para simular un tipo específico de carga de trabajo en un componente o sistema. Las pruebas sintéticas lo logran mediante programas creados específicamente para imponer dicha carga al componente. Las pruebas de rendimiento de aplicaciones ejecutan programas reales en el sistema. Si bien las pruebas de rendimiento de aplicaciones suelen ofrecer una medida mucho más precisa del rendimiento real de un sistema determinado, las pruebas sintéticas son útiles para probar componentes individuales, como un disco duro o un dispositivo de red.

Benchmarks are particularly important in CPU design, giving processor architects the ability to measure and make tradeoffs in microarchitectural decisions. For example, if a benchmark extracts the key algorithms of an application, it will contain the performance-sensitive aspects of that application. Running this much smaller snippet on a cycle-accurate simulator can give clues on how to improve performance.

Since around 1995, the SPEC collection of benchmarks have become widely used.

Computer companies are known to tune their systems to improve performance on benchmark tests that are not representative of real world usage. Benchmarks have been improved to more closely mimic real world usage, so that any tuning carries over to application performance.

Software vendors also use benchmarks in their marketing, such as the "benchmark wars" between rival relational database makers in the 1980s and 1990s. Companies commonly report only those benchmarks (or aspects of benchmarks) that show their products in the best light. They also have been known to mis-represent the significance of benchmarks, again to show their products in the best possible light.^[3][4]

When performance is critical, the only benchmark that matters is the target environment's application suite.

Features of benchmarking software may include recording/exporting the course of performance to a spreadsheet file, visualization such as drawing line graphs or color-coded tiles, and pausing the process to be able to resume without having to start over. Software can have additional features specific to its purpose, for example, disk benchmarking software may be able to optionally start measuring the disk speed within a specified range of the disk rather than the full disk, measure random access reading speed and latency, have a "quick scan" feature which measures the speed through samples of specified intervals and sizes, and allow specifying a data block size, meaning the number of requested bytes per read request.^[5]

Benchmarking is not easy and often involves several iterative rounds in order to arrive at predictable, useful conclusions. Interpretation of benchmarking data is also extraordinarily difficult. Here is a partial list of common challenges:

• Vendors tend to tune their products specifically for industry-standard benchmarks. Norton SysInfo (SI) is particularly easy to tune for, since it mainly biased toward the speed of multiple operations. Use extreme caution in interpreting such results.
• Some vendors have been accused of "cheating" at benchmarks — designing their systems such that they give much higher benchmark numbers, but are not as effective at the actual likely workload.^[6]
• Many benchmarks focus entirely on the speed of computational performance, neglecting other important features of a computer system, such as:
  • Qualities of service, aside from raw performance. Examples of unmeasured qualities of service include security, availability, reliability, execution integrity, serviceability, scalability (especially the ability to quickly and nondisruptively add or reallocate capacity), etc. There are often real trade-offs between and among these qualities of service, and all are important in business computing. Transaction Processing Performance Council Benchmark specifications partially address these concerns by specifying ACID property tests, database scalability rules, and service level requirements.
  • In general, benchmarks do not measure Total cost of ownership. Transaction Processing Performance Council Benchmark specifications partially address this concern by specifying that a price/performance metric must be reported in addition to a raw performance metric, using a simplified TCO formula. However, the costs are necessarily only partial, and vendors have been known to price specifically (and only) for the benchmark, designing a highly specific "benchmark special" configuration with an artificially low price. Even a tiny deviation from the benchmark package results in a much higher price in real world experience.
  • Facilities burden (space, power, and cooling). When more power is used, a portable system will have a shorter battery life and require recharging more often. A server that consumes more power and/or space may not be able to fit within existing data center resource constraints, including cooling limitations. There are real trade-offs as most semiconductors require more power to switch faster. See also performance per watt.
  • In some embedded systems, where memory is a significant cost, better code density can significantly reduce costs.
• Vendor benchmarks tend to ignore requirements for development, test, and disaster recovery computing capacity. Vendors only like to report what might be narrowly required for production capacity in order to make their initial acquisition price seem as low as possible.
• Benchmarks are having trouble adapting to widely distributed servers, particularly those with extra sensitivity to network topologies. The emergence of grid computing, in particular, complicates benchmarking since some workloads are "grid friendly", while others are not.
• Users can have very different perceptions of performance than benchmarks may suggest. In particular, users appreciate predictability — servers that always meet or exceed service level agreements. Benchmarks tend to emphasize mean scores (IT perspective), rather than maximum worst-case response times (real-time computing perspective), or low standard deviations (user perspective).
• Many server architectures degrade dramatically at high (near 100%) levels of usage — "fall off a cliff" — and benchmarks should (but often do not) take that factor into account. Vendors, in particular, tend to publish server benchmarks at continuous at about 80% usage — an unrealistic situation — and do not document what happens to the overall system when demand spikes beyond that level.
• Many benchmarks focus on one application, or even one application tier, to the exclusion of other applications. Most data centers are now implementing virtualization extensively for a variety of reasons, and benchmarking is still catching up to that reality where multiple applications and application tiers are concurrently running on consolidated servers.
• There are few (if any) high quality benchmarks that help measure the performance of batch computing, especially high volume concurrent batch and online computing. Batch computing tends to be much more focused on the predictability of completing long-running tasks correctly before deadlines, such as end of month or end of fiscal year. Many important core business processes are batch-oriented and probably always will be, such as billing.
• Benchmarking institutions often disregard or do not follow basic scientific method. This includes, but is not limited to: small sample size, lack of variable control, and the limited repeatability of results.^[7]

There are seven vital characteristics for benchmarks.^[8] These key properties are:

1. Relevance: Benchmarks should measure relatively vital features.
2. Representativeness: Benchmark performance metrics should be broadly accepted by industry and academia.
3. Equity: All systems should be fairly compared.
4. Repeatability: Benchmark results can be verified.
5. Cost-effectiveness: Benchmark tests are economical.
6. Scalability: Benchmark tests should work across systems possessing a range of resources from low to high.
7. Transparency: Benchmark metrics should be easy to understand.

1. Real program
   • word processing software
   • tool software of CAD
   • user's application software (i.e.: MIS)
   • Video games
   • Compilers building a large project, for example Chromium browser or Linux kernel
2. Component Benchmark / Microbenchmark
   • core routine consists of a relatively small and specific piece of code.
   • medir el rendimiento de los componentes básicos de una computadora ^{[ 9 ]}
   • Puede utilizarse para la detección automática de parámetros de hardware del ordenador, como el número de registros, el tamaño de la caché , la latencia de la memoria , etc.
3. Núcleo
   • contiene códigos clave
   • normalmente abstraído del programa real
   • núcleo popular: bucle de Livermore
   • Prueba de rendimiento de Linpack (contiene una subrutina básica de álgebra lineal escrita en lenguaje FORTRAN).
   • Los resultados se representan en Mflop/s.
4. Referencia sintética
   • Procedimiento para programar pruebas de rendimiento sintéticas:
     • Obtener estadísticas de todo tipo de operaciones de muchos programas de aplicación.
     • obtener proporción de cada operación
     • Escribe un programa basado en la proporción anterior.
   • Los tipos de índices de referencia sintéticos son:
     • Piedra de afilar
     • Dhrystone
   • Estos fueron los primeros benchmarks informáticos estándar de la industria para uso general. No necesariamente obtienen puntuaciones altas en las computadoras modernas con arquitectura de procesamiento en paralelo.
5. pruebas de rendimiento de E/S
6. Pruebas de rendimiento de bases de datos
   • Medir el rendimiento y los tiempos de respuesta de los sistemas de gestión de bases de datos (DBMS).
7. Pruebas comparativas paralelas
   • utilizado en máquinas con múltiples núcleos y/o procesadores, o sistemas que constan de varias máquinas

• Benchmarking (business perspective)
• Figure of merit
• Lossless compression benchmarks
• Performance Counter Monitor
• Test suite – a collection of test cases intended to show that a software program has some specified set of behaviors

• Gray, Jim, ed. (1993). The Benchmark Handbook for Database and Transaction Systems . Morgan Kaufmann Series in Data Management Systems (2.ª  ed.). Morgan Kaufmann Publishers, Inc. ISBN 1-55860-292-5.
• Scalzo, Bert; Kline, Kevin; Fernandez, Claudia; Burleson, Donald K.; Ault, Mike (2007). Métodos prácticos para la evaluación comparativa de bases de datos en Oracle y SQL Server . Rampant TechPress. ISBN 978-0-9776715-3-3.
• Nambiar, Raghunath; Poess, Meikel, eds. (2009). Evaluación del desempeño y benchmarking . Springer. ISBN 978-3-642-10423-7.

• Lewis, Byron C.; Crews, Albert E. (1985). "La evolución del benchmarking como técnica de evaluación del rendimiento informático" . MIS Quarterly . 9 (1): 7– 16. doi : 10.2307/249270 . ISSN 0276-7783 . JSTOR 249270 .  Las fechas: 1962-1976