Are the Qualities of Service Fit for Use?

Are the Qualities of Service Fit for Use? As discussed in Chapter 3, qualities of service (QoS) are captured in the work item database. Tests specialize according to the different qualities of service. For example, load tests, configuration tests, security tests, and usability tests are all radically different. Load Testing Load testing aims to answer two primary questions: Does the software respond appropriately under expected load conditions? To answer this, you compose performance tests that combine reasonable scenario tests, data, and workloads. Under what stress conditions does the software stop responding well? For this, you take the same scenarios and data and crank up the workload progressively, watching the corresponding effect on performance and system indicators. All the automated tests managed by VSTSweb tests, unit tests, and any additional test types you createcan be used for load testing (see Figures 7.7 through 7.10). With VSTS, you can model the workload to represent a realistic mix of users, each running different tests. Finally, VSTS automatically collects diagnostic data from the servers under test to highlight problems for you. Figure 7.7. In VSTS, a load test is a container for any arbitrary set of tests with workload settings. First, you choose how to ramp the load. Often you want to observe the system with gradually increasing user load so that you can spot any "hockey stick" effect in the response time as the user load increases. [View full size image] Figure 7.8. Next, you choose the tests (unit, web, or other) and the percentage of load to create from each of the atomic tests. [View full size image] Figure 7.9. The next steps are to choose the browser and network mixes that best reflect your end-user population. [View full size image] Figure 7.10. Load tests can generate huge amounts of data from the servers under test, and it's often hard to know what's relevant. VSTS simplifies this by asking you to choose only which services to watch on which machines and automating the rest of the decisions. [View full size image] Understanding the Output While a load test runs, and after it completes, you need to look at two levels of data (see Figure 7.11). Average response time shows you the end-to-end response time for a page to finish loading, exactly as a user would experience it. That's straightforward, and you can assess whether the range is within acceptable limits. At the same time, while the test runs, all the relevant performance data is collected from the chosen servers, and these counters give you clues as to where the bottlenecks are in the running system. Figure 7.11. This graph shows two kinds of data together. Average Response Time is the page load time as a user would experience it. Requests/Sec is a measurement of the server under test, indicating a cause of the slowdown. Note additionally the warning and error icons that flag problems among the tree of counters in the upper left. Some of these may lead you to configuration problems that can be tuned in the server settings; others may point to application errors that need to be fixed in code. [View full size image] Diagnosing When a load test points to a likely application performance problem, the developer of the suspect code is usually the best person to diagnose the problem. As a tester, you can attach the test result to a bug directly to forward it to an appropriate teammate, and when your teammate opens the bug, the same graphs will be available for viewing. He or she can then use the Performance Wizard to instrument the application and rerun the test that you ran, as shown in Figure 7.12. Figure 7.12. In addition to the information offered by the perfmon counters, you can rerun the test with profiling (or attach the test result to a bug and have a colleague open it and rerun with profiling). This takes you from the system view to the code view of the application and lets you drill into the specific methods and call sequences that may be involved during the slowdown. [View full size image] The profiling report can rank the actual suspect functions and lead you straight to the code that may need optimizing. This sequence of load testing to profiling is a very efficient way to determine how to tune an application. You can use it in any iteration as soon as enough of the system is available to drive under load. Security Testing Security testing is a specialized type of negative testing. In security testing, you are trying to prove that the software under test is vulnerable to attack in ways that it should not be. The essence of security testing is to use a fault model, based on vulnerabilities observed on other systems, and a series of attacks to exploit the vulnerabilities. There are many published attack patterns that can identify the vast majority of vulnerabilities.[7] Many companies provide penetration testing services, and many community tools are available to facilitate security testing. You can drive the tools from VSTS test suites, but they are not delivered as part of the VSTS product itself. Testing for Security Violations The only checks for security in VSTS are in the code analysis described in the previous chapter, but partners such as SPIDynamics offer solutions that plug into the test framework. See http://msdn.microsoft.com/vstudio/teamsystem/downloads/partners/default.aspx for a current list. Usability Testing In Chapter 3, I discussed usability testing as part of the requirements process, so I won't repeat it here. It is equally relevant, of course, throughout the lifecycle.