Introduction
To understand the resources and vCores/cores size that you will need for each project/use case you have 2 paths:
– Reliable: Create a performance benchmarking/test, execute load tests monitoring the resources, start at light load then increase. All the methodology is explained in this document
– Estimated: Pre-Sales sizing procedure (based on relative complexity and fixed parameters). Please contact MuleSoft Customer Success or Pre-Sales representative for the account
Concepts
Types of tests
Load Testing
Load testing is performed to determine a system’s behaviour under both normal and anticipated peak load conditions.
Stress Testing
Testing activity that determines the robustness of software by testing beyond the limits of normal operation, often to a breaking point, in order to observe the results.
Soak Testing
Is a type of performance test that verifies a system’s stability and performance characteristics over an extended period of time. It is typical in this type of performance test to maintain a certain level of user concurrency for an extended period of time.
Metrics/Values
Concurrency
Refers to the ability of an application to be executed out-of-order or in partial order, without affecting the final result. Concurrency is typically a stated requirement – eg. “application must handle up to 10,000 concurrent users with a 90th percentile round-trip time of 100ms maximum”. The goal of testing is to measure what concurrency a given system will support; this means that concurrency is a variable parameter.
Throughput
Refers to the number of transactions or requests that can be made in a given period of time. It is a useful measurement to check the load capacity of the server.
Methodology
Define the Scope
- The final goal of the test (e.g. check if the system remains stable during xx hours of load)
- Systems/Nodes that are going to be included in the test
- Representative environment, should have the same resources and specs of the production environment
- Application(s), resources and data to be used
- Define the “redline” % of system resources in production – e.g. “do not allow CPU% to exceed 70%”
- Always monitor the system resources under every load test, at a minimum capture these metrics
- CPU%, CPU load
- mem%
- disk%, IOPS
- network%, network socket
- GC log
- Considerations and limitations
Define the monitoring tools to use
- Define the tools that you are going to use to monitor the systems (e.g CloudHub Dashboard, Analytics dashboard, Database Analytics, jConsole, etc)
Prepare your test plan based on the defined scope
Create a representative workload (to begin, make this static – repeatability is everything)
- Request methods (GET, PUT, POST)
- API(s) that are part of the scenario.
- Payload size
- Set this as your new baseline for capacity planning
- Use this baseline to mathematically find out the metrics you are looking for
- Start at a light load, if resources are plentiful, increase the load
- If TPS increases past the value of the previous test, latency stays relatively flat and no resources are close to “redline”, increase the TPS
- If TPS does NOT increase and none of the resources get close to full capacity, it means there is likely thread contention – profile the application to determine where thread-contention is occurring
- If TPS does NOT increase and latency starts to increase linearly, and one of the resources gets close to full capacity, the system is saturated. Reduce load slightly and rerun the load test, set this as the baseline
Ensure differentiation between performance of: proxy, back-end, or proxy+back-end
This applies specifically to use cases where proxies to legacy back-ends are being designed and deployed. It is important to plan ahead for ability to test performance of only the proxy – to prevent possibly poor-performance of legacy back-ends from tainting accurate impressions of performance. While overall business-performance is most important, nothing is helped if the API Gateway is blamed for performance for slowness.
It is important to establish the baseline performance of proxy-only, however artificial that is in a business-sense. Comparison of overall system performance to proxy-only performance gives a valuable baseline and will aid in diagnosis and rectification of performance problems.
Run Load tests to find the baseline
- Run the load test until the resources get close to the “redline”
Run Soak tests
- Define a representative unit of time to see the stability of the system (e.g. 4-8 hrs)
- Run the test based on the baseline load
- Store the results/metrics
Build a report
- Include the defined scope, configurations, monitoring tools and test plan
- Include the base line
- Include the results
- Include conclusions based on the results