Business Scenario-Based Performance Diagnostics Procedures
Internal
Assumptions
Load Generator
We need a load generator that is capable of the following:
- Generate load on behalf of multiple application users, concurrently, in such a way that each user should be capable of performing an interactive session setup, interact repeatedly within the application by executing a set of arbitrary business scenarios in a loop, and then end the session.
- Insert arbitrary data in the requests it had previously recorded and it is replaying as part of the load tests. At minimum, the load generator must be able to insert arbitrary constant strings as custom header values in specific requests in the run sequence. More sophisticated analysis require additional capabilities, such as inserting unique identifiers per request, iteration, inserting request sequence numbers, etc..
NeoLoad is one of products.
Load Test Environment
Business Scenario Recording
This step consists in the recording of the business scenario whose performance will be measured under load. This step is highly dependent on the tool used to record the interaction with the application and it usually consists in intercepting the traffic with a proxy. The load generation tool records the HTTP request details, stores them and used the stored data to replay the application traffic.
Business Scenario "Annotation"
Mark the Beginning and the End of the Interesting Scenarios
The scenarios to be measured must be "annotated" by configuring the load generator to send specific HTTP headers with the first and the last request of the scenario.
Business-Scenario-Start-Marker
The first request of the scenario must include a custom HTTP header named Business-Scenario-Start-Marker. The value of the header must be a string designating the scenario type, for example "Read bank account value". The load test will consist in sending multiple scenarios of the same type, in a loop, and after data processing we will be able to tell whether the specific scenario is within the required performance limits. Shorter values (i.e. "TypeA", "TypeB", etc.) are preferred, because ultimately it translates in sending less bytes over the network, though the amount of traffic generated because of the marker headers is most likely going to be insignificant relative to the amount of application traffic.
Business-Scenario-Stop-Marker
The last request of the scenario must include a custom HTTP header named Business-Scenario-Stop-Marker. The value of the header must be the same string used when marking the start of the scenario with Business-Scenario-Start-Marker.
If the end of the scenario is not explicitly marked, the data processing utilities will assume a scenario ends when the next Business-Scenario-Start-Marker header carrying the same scenario type string is encountered. Assuming each iteration contains just one scenario of a specific type, this default behavior will produce in most cases invalid data, effectively equating a scenario with the whole iteration.
Target Environment Preparation
Target environment preparation operations are usually one-time operations that must be executed every time a target test environment is built. The configuration, tools and utilities produced at this step will be repeatedly used for each load test executed within the environment.
Component Life Cycle
In order to eliminate artifacts introduced by the state modified as result of previous load tests, the environment should be capable to stop and re-start all its active elements (load balancers, application servers, databases, caches) between load test runs.
In case the tests are executed in an environment managed by em, the best way of restarting environment is to restart the environment's VMs. This way all services are restarted and initialized. Alternatively, the VM can be restarted manually.
If restarting the VMs is not practical, at least the component application servers must be restarted.
Instrumentation
As an one-time operation, the active elements of the environment must be instrumented to collect performance data. Various metric sources and data collection strategies can be used:
Log Management
In most cases, various logs are directly used as event stream sources. Even if that is not the case, they contain useful context data, so they must be collected and archived after each load run. Automation of this process is part of the test environment preparation.
Load Test Execution
This step will be executed repeatedly, for different versions or configurations that need to be performance tested.
Restart the Environment
All components of the test environment should be restarted prior to a load test run for reasons explained in the "Component Life Cycle" section.
pt restart
The command will stop the environment components, clean logs and temporary data directories, where necessary, and restart the environment components, leaving the environment ready for testing.
Apply the Load
Collect and Archive The Raw Data
pt collect
The command will collect all relevant configuration, logs and test-related data and archive them in the data repository, using a unified format known to all data processing and report generation utilities.