Performance Testing Web Service Using SoapUI
Performance testing is a critical step in the life cycle of any project. This step checks if the application is capable of handling expected load within desired response time limits in given environment resources. Web services are not free from the performance requirements. Generally, web services involve remote access by many clients, hence web services need to have high performance. Due to remote access involvement, testing a web service is not as easy as testing a standalone programs or web based applications. There is need of special tools to test web services. When it comes to performance testing of web services, some factors add to complexity of performance testing. This article takes a closer look at these factors affecting PT and explains how to use SoapUI to overcome the complexities while performance testing web services. It also elaborates the process of performance testing with example.
What is SoapUI?
SoapUI is web service testing tool. It is available in two flavors – free for all and SoapUI Pro (not free). The Pro version provides additional features. SoapUI also provides plug-ins so that we can connect it to standard IDEs e.g. eclipse, intellij, netbeans. We can use SoapUI for functional testing as well as performance testing. It also supports soap request based testing, Groovy script based testing and many other test features.
This tool gives elaborate quantitative test results which can be plotted to see different graphs. Below is a sample screen of load testing using SoapUI. Further details of this tool usage can be found on SoapUI website ( http://www.soapui.org/ ).
Example Service
Let us take an example of a web service doing primary operations related to application users. This can be a central web service catering to user administration needs of a pool of applications in an enterprise. It offers following services to the callers.
- authenticateUser – taking user credentials input and returning whether user is authenticated or not.
- authorizeUser – taking user details input and returning set of user permissions.
- searchUser – exact match and wildcard searches for user.
- modifyUser – allowing updating user attributes.
We are going to performance test above services to see if those meet the non functional requirements.
Non-Functional-Requirements (NFRs)
Gathering and validating non functional requirements is a very important step. There are multiple important dimensions of this activity. Requirements finalized in this stage are going to be the performance goals for the authors of our service. First important thing in finalizing the requirement is validating them. It must be checked if it is possible to meet the requirements in given technology and environment scenario. It may require experience of the technology or a small proof of concept (POC) to get feel of the technology pack. But it is very important to check feasibility of achieving the performance requirements before actually accepting those and committing on them.
Next is collecting all requirements. Quantification of following parameters is necessary in case of individual services or all services taken together. Historical business data proves helpful in this case. You can also extrapolate based on the data available, and come out with numbers against each service in our example. Extra care must be taken in case of business critical or complex scenarios. In our example, authentication and authorization are going to be highly important scenarios, because if these two take more time, then users may not be able to use pool of applications. Similar logic should be applied to any system to identify critical flows.
Following data should be available for all the services under test. Sometimes we can take freedom of having this data for critical flows only. But it is advisable to have this data for all flows under test. Let us understand each of the following data headers.
- Average Load: When the service is running in normal scenario, then how many requests it should process in given time interval. In above example, we may have around 10 – 15 users using authentication method per 5 minutes. This means 15/300 = 0.05 requests/seconds
- Peak Load: This is the special case, when all of sudden the services are flooded with requests. E.g. online shop sites are flooded by huge number of requests during festival seasons. In our case, during morning 15 minutes approximately 500 user login. This means 500/(15×60) =0.50 requests/second.
- Load Mix: Load mix considers the load of all services running during average load scenario and peak load scenario. It may be the case that only few services are heavily loaded while others are having average load. We need to consider the combinations of service loads here. In our case, during peak load 500 users may be using authentication method, but only 100 users will be going for authorization method, while 10 user each for search user and modify user method. All such combinations will lead to different test scenarios.
- Response Time: Response time can be defined for all methods in one go, or it can be defined for each service separately depending on the heaviness/complexity of the operations involved. In our case the requirement is plain – response time should be less than 1 second for all the methods.
Here is summary NFRs for our services.
| Requirement | Value |
| Response Time for Any Service | < 1 second |
| Authentication Service Peak Load | 500 users/15 minutes |
| Authentication Service Average Load | 15 users / 5 minutes |
| Authorization Service Load | Approximately 25% less than Authentication Service |
| Search Service Load | 10% of Authentication Service |
| Modify Service Load | 10% of Authentication Service |
| Application Server Machine CPU | < 60% |
| Database Server Machine CPU | < 60% |
What is Under Test?
Identify the services to be performance tested. These can be the business critical services identified during NFR gathering. Performance test strategy should be devised focusing on these critical flows. Testing other flows should be planned considering availability of time (and practically money).
We test all services from our example.
Performance Test Strategy
It is time to put NFRs, critical flows to be tested and our intelligence in work. In this step, we define scenarios to expose service bottlenecks with minimum testing effort. Though there are standard ways of doing tests, but little strategic testing can reduce performance testing effort considerably. It is also true that performance testing is an iterative process. Let us see what are general strategies used in performance testing.
- Load Variation Scenarios: Gradually vary user load on each service individually. Consider average load as well as peak load while identifying load intervals. Usually 5 variations are enough to reveal service behavioral trend. In our case we have peak load of 500 users in 15 minutes. Around 30 users per minutes. So let us have load boundaries of 0 – 50 concurrent users/threads. (Going forward we will use users and threads interchangeably when we speak about load.)
- Load Mix Scenarios: Combination of different services load is to be defined here. This is based on the load mix identified during NFR gathering. Considering average load and peak load, define different combinations of services loads. E.g. 50 threads for authentication, 30 for authorization, 5 for search and 5 for modify user. Bottom line is – we should be as close as possible to the real life system.
- Data Driven Load Variation: In this case, we consider functional aspect to vary the load. Search user service above allows wild card search, this means it will return far more data as compared to exact search. Similarly, in case of authorizeUser service, there will be huge amount of permission data for few users. Such load variation depending on functional data scenarios needs to be combined while defining strategy. Some of the applications may have database queries involved in backend processing. Most of the database systems cache the queries to give better performance. If we have only one data set, then there will be misleading results due to DB caching. Create different sets of data to counter this problem also.
- Stress Test: We need to determine at which point our services stop serving or starts giving response beyond acceptable limit. This point can also be the load point when service is no more reliable. Stress test should be used to gradually increase the stress until system breaks. In our services, we increase load from 1 thread to 500 threads in 900 seconds duration.
- Endurance Test: This is primarily used to identify memory leaks. We leave system running for a day or two with average load. If there are memory leaks then there should be performance degradation with respect to time.
- SoapUI Provided Strategies: All above strategies can be easily implemented using SoapUI. In addition to this, SoapUI provides additional strategies like burst, where we can simulate sudden increase in peak load. Such load pulse can be created using this strategy.
- Test Duration: Decide on the duration of test. If you are running endurance test separately, then duration can start from 10 minutes as well. For above example we run the tests for 10 minutes at least.
In brief, the test scenarios for authentication service from our example are as follows.
| Strategy | Details |
| Single Data Input Load Variation | Single set of user credentials,
Concurrent User Load Variation 10-20-30-40-50 Duration: 600 seconds |
| Data Load Mix Load Variation | Multiple sets of user credentials,
Concurrent User Load Variation 10-20-30-40-50 Duration: 600 seconds |
| Endurance Test | Multiple set of user credentials,
Concurrent user load – 10 Duration: 24 hours |
| Stress Test | Multiple set of user credentials,
Concurrent users – 1 to 500 added gradually (Thread strategy in SoapUI) Duration: 900 seconds |
Observations
While running performance test, SoapUI provides most of the data that tells about the health of services. But, we also need to monitor health of the machine hosting application and also health of other resources supporting main application. These can be down line systems, database, etc. Let us see what all data we are going to collect and how SoapUI helps us in it.
- Test Results on SoapUI: SoapUI provides response time, data exchange details, test counts and tests run per second. Save this data directly in ‘csv’ format.
- Test Graphs from SoapUI: SoapUI also provides graphical details of the test run. Save this data, it helps to identify average response for 90% of the tests run.
- Hardware Resources: In addition to the data provided by SoapUI, we must monitor health of the boxes running application. Mostly applications are running on Unix or Linux. Commands like “Sar -u 1 1000 > usage.log” can generate a file containing CPU usage every second for 1000 seconds. System and user CPU are of our use in this data.
- Database Monitoring: If the services involve database operations, then monitor database health also. DB reads, IO, and health of the machine holding database provide vital information.
Example SoapUI Test Results:
Execution
Using SoapUI, we can directly execute almost all scenarios. For the mixed load scenario, where we have multiple services running simultaneously, open load tests for all of them and start the tests. SoapUI does allow us to run multiple load test simultaneously.
Here is the data we obtained after running our tests.
| Threads | Min Time (ms) | Avg Time (ms) | Max Time (ms) | Tests/Sec | DB System CPU | Application System CPU |
| 10 | 10 | 25 | 300 | 50 | 30% | 91% |
| 20 | 21 | 35 | 410 | 53 | 32% | 93% |
| 30 | 47 | 53 | 600 | 55 | 33% | 94% |
| 40 | 59 | 71 | 930 | 56 | 34% | 95% |
| 50 | 76 | 92 | 1180 | 57 | 35% | 96% |
Analysis
Once we run tests as per above strategies, it is going to result in huge amount of data. We should use this data to identify problems and bottlenecks. Here are few methods of identifying problems.
- Number Based Analysis: Sometimes we can get obvious conclusions by simply looking at the data. E.g. response time is beyond our acceptable limit. Also observe the maximum CPU usage. Following data shows that the authenticateUser service usage system CPU beyond acceptable limit. CPU beyond 60% is generally something of concern. Maximum system CPU is 96% which is not at all acceptable. We need to tune this service.
- Graphical Analysis: This technique usually makes our life very easy. Plot consolidated test results to observe different strategy results. You can see if there is any pattern in the results. E.g. increase in load, should not increase the response time and other machine usage parameters, if there is any abnormality then investigate the cause behind it.
- Mapping Results to Strategy: While defining strategies, we have to consider few impacting parameters – increase in load, mixing load, data dependent load variation, etc. Plot graphs to see effect of these parameters. Also check if the response is within acceptable limits.
- Network Latency: Sometimes we have SoapUI installed on a machine which observes considerable delay factor in response time data. Try printing execution time on server side to determine the network latency factor. Subtract it to determine actual time.
Improving Performance
We have identified performance problem through above analysis, now it is time to dig out the cause and fix it. You can use technology specific techniques to reach out to actual bottleneck. There are many profiling tools available in Java as well as other technologies. Primarily we will adapt to two solutions. First is fixing the application and second is fixing the environment running the application. Depending on symptoms, we will have to choose appropriate solution.
In case of authentication method, we made a fix to reduce the system resource usage. This is how it impacted.
Here is the data we obtained after running our tests again.
| Threads | Min Time (ms) | Avg Time (ms) | Max Time (ms) | Tests/Sec | DB System CPU | Application System CPU |
| 10 | 10 | 22 | 234 | 81 | 30% | 25% |
| 20 | 21 | 31 | 367 | 103 | 32% | 29% |
| 30 | 33 | 43 | 435 | 135 | 33% | 30% |
| 40 | 39 | 52 | 545 | 165 | 34% | 33% |
| 50 | 42 | 64 | 648 | 210 | 35% | 37% |
Retests
Once we have fixes deployed, then it is time to go for second round of performance test. As mentioned earlier, performance test is an iterative process. We identify first set of problems, fix those and move to next level, which may be desired performance or finding out another bottleneck. Following steps need to be followed in second and subsequent cycles.
- Identify the impacted tests and test scenarios
- See if any additional test scenarios are required
- Execute the tests
- Analyze results and find out if it is acceptable performance
- Repeat the cycle
Benchmarking
Performance test should fix the performance benchmarks for services under different governing parameters. Load, response time, cpu usage etc. metrics should be available for ready reference. In addition to this, we must publish the break points for the services under given environment parameters. It definitely helps to check alignment of services against business growth.
If you enjoyed this post, please consider to leave a comment or subscribe to the feed and get future articles delivered to your feed reader.
Nice article to get a clear understanding of what performance tests are actually supposed to do…
Your explanation is good
How can perform performance testing in SOAP UI with options
Means take one example and run using SOAP UI