In this 5 part Benchmarking Series we have discussed:
- The Need for Benchmarking
- The Elements of a Benchmark
- The Applicability/Usage of a Benchmark
- Benchmark Sizing Guide
We're now going to discuss the Benchmarking Process itself.
"How do you measure up?"
END GOAL
The start of any benchmark needs to be a clear understanding by everyone involved as to what the end goal is. This ties very closely to what was discussed in " The Need for Benchmarking " post in this series. Some examples of possible end goals:
- Can my system handle x users and still perform as expected?
- How will my system perform when upgrading my database platform?
- Does changing/upgrading my application code affect the performance of my system?
ASSESSMENT / REVIEW
This is where you take all the elements of your benchmark (as discussed in " The Elements of a Benchmark " post in this series) and execute your tests to understand how your environment will perform in relation to the question(s) being asked in your "end goal". Outputs of this exercise can be (as described in " The Applicability/Usage of a Benchmark " post in this series):
- Software Bottlenecks
- Hardware Bottlenecks
-
A
Sizing Guide
to compare with information on
- TPS (Transactions Per Second)
- Transaction Time
- Transaction Time vs Number of users
- Performance Information
ACTION PLAN
Now that we understand our "end goal" and we've done our "assessment/review" if we haven't achieved success we need to develop and implement an action plan to address the items found in the assessment. Items that should be discussed in this action plan can be:
- Category (Hardware, Software, Application, Database, etc)
- Risk Level
- Impact Level
- Description
- Recommended Resolution
At this point (once the Action Plan has been implemented) you will go back to the Assessment/Review phase and gather new benchmark results. This cycle will continue until you are happy with your results and have been able to answer the question(s) from your End Goal.
WHITE PAPER
The final part of the Benchmarking Process (depending on what your End Goal was) might be to create a whitepaper to summarize and report the findings. Information that might be included:
- Executive Summary - Why was this Benchmark done?
- What Benchmark tests were performed?
- What hardware was used?
- What software was used?
- Summary results
- Best Practices
"You don't know where you are going till you know where you've been"
Benchmarks should play an important role in your enviroment. Whether you are using them for release management, determining scalability, or troubleshooting it's important to have hard facts about how your environment performs under normal conditions. Having the right hardware, software, and repeatable tests in place will help you identify what bottlenecks you might have (and under what conditions). This information allows you to make changes and have a good understanding as to how those changes will affect the overall performance of your environment which helps ensure the overall stability of the system.
_____________________________________________________________________________________________