Tool 20: Testing

[ Team LiB ]

Tool 20: Testing Imagine a complex assembly machine that puts together videocassettes. It has 15 assembly steps, each followed by a test step: position, test, position, test.… At one step, a robot puts a part in place; at the next step a sensor checks to be sure that the part is there. As you move down the manufacturing line, you find that every time something is done, a test follows to be sure it was done correctly. These tests assure that there are no missing or misaligned parts and that all the parts fit together as the product is being assembled. When a videocassette passes these tests, you know it was put together the way it was designed. During manufacturing, a representative sample of videocassettes are pulled off the line and put into a test bed to be sure that they function smoothly and play back high quality video in a variety of machines currently in use by consumers. These tests show that the videocassettes will work correctly when they are actually used. In software development, we also test that design intent is achieved and that the system does what customers want it to do. When developers write code, there should be a test to be sure that each feature works as intended and that all of the pieces work together. These tests have been categorized as unit tests, system tests, and integration tests. As we move from programming one module at a time to programming entire capabilities and features, the distinction between unit, system, and integration tests has less meaning. A better name for these tests might be developer tests, because their purpose is to assure that code does what the developer intended it to do. Tests to be sure that the system does what customers want have been called acceptance tests, but this term has traditionally been used to refer to tests that run at the end of development. A better name for tests that make sure that a system does what customers intend is customer tests, since their purpose is to assure that the system will do what customers expect it to do. Customer tests are run throughout the development, not just at the end. Tests play several pivotal roles during the software development process. First, tests unambiguously communicate how things are supposed to work. Second, they provide feedback on whether the system actually works the way it is supposed to work. Third, tests provide the scaffolding that allows developers to make changes throughout the development process, making tools such as last responsible moment, set-based development, and refactoring useful in practice. When development is done, the test suite provides an accurate representation of how the system was actually built. Finally, by developing and maintaining test suites for all systems in production, making changes to production systems that interact with each other can be done safely by running a full suite of tests for all related applications. Communication When a product is released to manufacturing, tests are released along with it to tell the manufacturing organization exactly what constitutes an acceptable product. In the same way, developer tests convey exactly how the system is supposed to work internally, while customer tests convey by example exactly what customers need an application to do. Customer tests can be a viable replacement for, or supplement to, most requirements documents. Suppose a developer has a conversation with a customer about details of a feature. The conversation should not be considered complete until it is expressed as a customer test. Whether the test is written by a customer representative, tester, or developer, it is a precise description of how the feature should work for the customer. Now imagine a quick design session among developers determining how the feature will be implemented. The implementation is not complete until the design details are exercised by developer tests. By documenting the design in tests, developers can write code with a clear understanding of exactly what it is supposed to do. This is a good way to refine thinking and help developers write code with conceptual integrity. There are alternatives to writing tests as a communication device prior to coding, but there is no alternative to writing tests to demonstrate whether the system does what it is supposed to do. So, you may as well get double duty out of tests by using them to document what the system is supposed to do, just as manufacturing often uses tests to convey product specifications. Feedback When a developer writes code, she or he should get immediate feedback about whether the code works as intended. In other words, there should be a test for each mechanism the developer implements. In fact, developers will find a way to test their code as soon as it is written anyway; this is how code is developed. Why not capture that test and use it? You are going to test the system anyway, so you may as well capitalize on the fact that development is a cycle of experiments with a successful test at the end of each cycle. The reason you are developing software in short iterations is so that you can provide feedback about how the system works to the customers or customer representatives and get their input on how to proceed. In order to get that feedback, you need to show them what the software developed during the iteration does for them. In other words, you need a set of demos or scripts that demonstrate the developed functionality. These need to be understood by customers well enough to be sure that everything they care about is successfully implemented in the iteration. So, why not have testers on the team to write customer tests during each iteration? Since you are going to demonstrate features at the end of the iteration anyway, you may as well capture the demonstration in tests. As long as you've got them, developer and customer tests should be automated as much as possible[24] and run as part of the daily build. If the tests are not automated or if they take too much time, they won't be run often enough. Big batches of changes will be made before testing, which will make failure much more likely, and it will be much more difficult to tell which change caused the tests to fail. [24] For a discussion on deciding when to automate tests, see Marick, "When Should a Test Be Automated?" Scaffolding Scaffolding is a supporting framework that allows workers to do things that would otherwise be dangerous. If you develop software in iterations, delay decisions until the last responsible moment, and use set-based development and refactoring, you are going to be making serious changes to code once it has been written. This is dangerous, as we all know, because changes tend to have unintended consequences. Any nontrivial system requires that hundreds of thousands of details must all be correct at the same time. Many of these details interact with each other in ways far too complex to anticipate. The larger the code base, the more devious the interactions might be. To make changes safely, there must be a way to immediately find and fix unintended consequences. The most effective way to facilitate change is to have an automated test suite that tests the mechanisms the developers intend to implement and the behavior the customers need to have. A test suite will find unintended consequences right away, and if it is good, it will also pinpoint the cause of the problem. In this sense, automated test suites are scaffolding that provides safety and access to the builders of the software system as they complete the construction of a partially built edifice. You can't effectively use the other tools in this chapter without this scaffolding. It may seem like writing tests slows down development; in fact, testing does not cost, it pays, both during development and over the system's lifecycle. When you think about it, the tests are there for you to find, formalize, and automate, because developers somehow check their work as they code, and ways are found to demonstrate to customers how the system works at the end of iterations. The thing you need to do is capture those tests, make sure they are correct and complete, put them under version control, automate them, consider them as part of the released product, and continue to use and improve them. You might end up with as many lines of test code as of product code, but the benefit will far outweigh the cost. As-Built The Importance of As-Built Drawings At the end of July 2002, news stories told of the dramatic rescue of nine men from the flooded Quecreek coalmine near Somerset, Pennsylvania. The men were trapped deep in the mine after they drilled into an abandoned mine full of water. Drawings of the abandoned mine had not been updated to show a large cavern that was dug just before the old mine was closed. An updated map was subsequently found, but it was not available to the miners as they worked. So they drilled into the water-filled cavern, not knowing it was there. Mining is not the only industry with inaccurate maps. It's a safe bet that your city does not have accurate as-built maps of its underground infrastructure. For that matter, few buildings have accurate as-built or as-maintained drawings. It doesn't come as a surprise that it is difficult, if not impossible, to maintain accurate as-built documentation of software. Heroic attempts are made to do this with safety-critical software, but as the coalmine accident shows, there can always be lapses. However, if a system has a comprehensive test suite that contains both developer tests and customer tests, those tests will in fact be an accurate as-built reflection of the system. If the tests are clear and well organized, they are an invaluable resource for understanding how the system works from a developer's and a customer's point of view. The other thing a test suite does is give an indication of the health of the as-built system. Defect counts, types, and trends are a very good indication of whether a system is converging, when a product is ready to ship or deploy, and how robust the system is. The bottom line is, you should have complete, automated (as far as practical) suites of developer and customer tests. They should be subject to the same discipline in design, semantics, versioning, builds, synchronization, and refactoring as the system itself. If there doesn't seem to be enough time, the first thing to do is reallocate the effort used in requirements documentation to writing customer tests. Require developers to write and automate their own developer tests, while providing training and coaching in test development and automation. You will get more payoffs from an effective test program than from most other investments you might make. Maintenance The software industry needs to find a way to make software easy to change after it is running in production, since well over half of development occurs after initial release. Furthermore, making changes to production software has to be economical�that is, changes must be made relatively quickly and at a reasonable cost. There are many ways to make software more changeable�layering, clumping, and hiding potential variability; components; use of commercial software; and so on. It is also a good idea to have the development team retain responsibility for application maintenance to preserve domain learning. All of these techniques are important, but we must add one other mechanism that to the list: maintaining a set of comprehensive tests throughout the lifecycle of the system. If a scaffolding of tests was built during development, all you have to do is re-erect the scaffolding and proceed with the changes. Then, the system can be safely repaired and refactored throughout its useful life. Scaffolding is as useful for maintenance as it was for the original construction. Let's say you have a complex system with many applications using common services�a common database, middleware, or hardware, for example. You know enough about complex systems by now to suspect that a change in any one application could have an adverse reaction on an unrelated application. Since you don't have a reliable set of as-built documentation, you have to figure out for yourself how all the systems actually work before you can safely change any of them. No wonder maintenance is so difficult. What you need is the test suite for each application, developed as scaffolding for change during development. These tests, assuming you keep them healthy, constitute an accurate set of as-built documentation for all the applications in your environment. If each application has an up-to-date test suite to prove its integrity, you can test the entire environment before a change is released.

[ Team LiB ]