User Tools

Site Tools


benchmark_guidelines

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
benchmark_guidelines [2009/03/29 15:50]
andrewz highlight
benchmark_guidelines [2009/03/29 15:52] (current)
andrewz +boxplot
Line 25: Line 25:
 If available, **use older hardware**---older,​ but not ancient. ​ The benchmark will take longer, so it's easier to measure small differences. ​ Also, most people don't have new hardware, so new hardware is not representative. If available, **use older hardware**---older,​ but not ancient. ​ The benchmark will take longer, so it's easier to measure small differences. ​ Also, most people don't have new hardware, so new hardware is not representative.
  
-Before publishing, **evaluate the [[http://​en.wikipedia.org/​wiki/​Variance|variance]]**. ​ If the variance is high, identify the cause, fix it, and test again.+Before publishing, **evaluate the [[http://​en.wikipedia.org/​wiki/​Variance|variance]]**. ​ If the variance is high, identify the cause, fix it, and test again.   
 + 
 +**Display a [[http://​en.wikipedia.org/​wiki/​Boxplot|boxplot]]** to illustrate variance.
  
 The only way to measure accurately cold start times is use the real thing: a reboot. ​  ​Simulating a cold start by  flushing the disk cache does not represent a real reboot. ​ There are many factors including operating system disk cache and physical disk cache. ​ Even if you could flush all the caches, the process of starting up the operating system loads certain data into the cache (such as common libraries). ​ **Just reboot**. The only way to measure accurately cold start times is use the real thing: a reboot. ​  ​Simulating a cold start by  flushing the disk cache does not represent a real reboot. ​ There are many factors including operating system disk cache and physical disk cache. ​ Even if you could flush all the caches, the process of starting up the operating system loads certain data into the cache (such as common libraries). ​ **Just reboot**.
benchmark_guidelines.txt · Last modified: 2009/03/29 15:52 by andrewz