Implementing Automated Software Testing: How to Save Time and Lower Costs While Raising Quality by Elfriede DustinImplementing Automated Software Testing: How to Save Time and Lower Costs While Raising Quality by Elfriede Dustin

Implementing Automated Software Testing: How to Save Time and Lower Costs While Raising Quality

byElfriede Dustin, Thom Garrett, Bernie Gauf

Paperback | March 4, 2009

Pricing and Purchase Info

$57.58 online 
$57.99 list price
Earn 288 plum® points

Prices and offers may vary in store


In stock online

Ships free on orders over $25

Not available in stores


“This book fills a huge gap in our knowledge of software testing. It does an excellent job describing how test automation differs from other test activities, and clearly lays out what kind of skills and knowledge are needed to automate tests. The book is essential reading for students of testing and a bible for practitioners.”
—Jeff Offutt, Professor of Software Engineering, George Mason University

“This new book naturally expands upon its predecessor, Automated Software Testing, and is the perfect reference for software practitioners applying automated software testing to their development efforts. Mandatory reading for software testing professionals!”
—Jeff Rashka, PMP, Coauthor of Automated Software Testing and Quality Web Systems

Testing accounts for an increasingly large percentage of the time and cost of new software development. Using automated software testing (AST), developers and software testers can optimize the software testing lifecycle and thus reduce cost. As technologies and development grow increasingly complex, AST becomes even more indispensable.  This book builds on some of the proven practices and the automated testing lifecycle methodology (ATLM) described in Automated Software Testing and provides a renewed practical, start-to-finish guide to implementing AST successfully.

In Implementing Automated Software Testing, three leading experts explain AST in detail, systematically reviewing its components, capabilities, and limitations. Drawing on their experience deploying AST in both defense and commercial industry, they walk you through the entire implementation process—identifying best practices, crucial success factors, and key pitfalls along with solutions for avoiding them. You will learn how to:
  •     Make a realistic business case for AST, and use it to drive your initiative
  •     Clarify your testing requirements and develop an automation strategy that reflects them
  •     Build efficient test environments and choose the right automation tools and techniques for your environment
  •     Use proven metrics to continuously track your progress and adjust accordingly
Whether you’re a test professional, QA specialist, project manager, or developer, this book can help you bring unprecedented efficiency to testing—and then use AST to improve your entire development lifecycle.

Elfriede Dustin, Thom Garrett, and Bernie Gauf work together at Innovative Defense Technologies (, which specializes in the design, development, and implementation of automated software testing solutions. Elfriede Dustin has authored multiple software testing books and articles based on her many years of actual hands-on ...
Title:Implementing Automated Software Testing: How to Save Time and Lower Costs While Raising QualityFormat:PaperbackDimensions:368 pages, 8.9 × 7.3 × 1 inPublished:March 4, 2009Publisher:Pearson EducationLanguage:English

The following ISBNs are associated with this title:

ISBN - 10:0321580516

ISBN - 13:9780321580511


Read from the Book

Preface Is your test automation strategy a losing proposition? Are you soured on the notion of automated software testing based on less than adequate past results? Are your test automation silver bullets missing their mark? Are you disappointed in your test automators? We at IDT1 have identified a boilerplate solution, strategies, and ideas, all provided in this book, that can help increase the chances of your automated testing success. Given the arsenal of system and application software testing strategies, techniques, and solutions, automated software testing is one of the most effective practices that if implemented correctly can help increase testing efficiencies and ultimately reduce the testing cost while contributing to increased systems and software quality in terms of faster, broader, and more efficient defect detection. This book is a guide that can help organizations implement successful automated software testing programs and efforts. The book does not provide gimmicks or magical solutions, as none exist, but it provides experience-based discussions and recommendations. It includes a thorough dissection of automation issues, such as in Part I of the book, where we describe what automated software testing is and is not; why a business case is required for successful automation, including step-by-step instructions for developing one; why to automate and when. Then we summarize why automation often fails and the pitfalls and blunders that can be prevented; we describe the tools that are available to help implement successful automation efforts, with a focus on open-source testing tools. In Part II of the book we present six keys to successfully implementing automated software testing. These are Key 1: Know Your Requirements Key 2: Develop the Automated Test Strategy Key 3: Test the Automated Software Test Framework (ASTF) Key 4: Continuously Track Progress—and Adjust Accordingly Key 5: Implement AST Processes Key 6: Put the Right People on the Project—Know the Skill Sets Required IDT conducted two separate surveys related to automated software testing with approximately 700 total responses from test professionals all over the world, across organizations that were diverse in size and in what they do. The survey showed two very consistent themes: About 70% of survey respondents said they believe automation is high-payoff, but they are generally not sure why to automate and how automation applies to their project. Half of the survey respondents also said they felt they lacked the experience, time, or budgets to implement automation. Most seem to agree: Automated software testing is useful, and an increasing need for it exists. However, the lack of experience seems to be the reason why automation is not implemented more often with a higher success rate. Finding people with the skills for the project is therefore important; a summary of skills required is provided in Chapter 10. For more details on the outcome of this survey, see Chapter 4. Material Coverage and Book Organization Part I: What Is Automated Software Testing and Why Should We Automate? Chapter 1, What Is Effective Automated Software Testing (AST)?, describes what automated software testing is. The definition of automated software testing we use throughout this book is the “application and implementation of software technology throughout the entire software testing lifecycle (STL) with the goal to improve STL efficiencies and effectiveness.” In Chapter 2, Why Automate?, we address this question that is asked so often. Here we discuss the challenges of software testing today and how the time and cost of software testing can be reduced. Reasons for why to automate, laying the foundation to help build the business case discussed step by step in Chapter 3, are presented here. In Chapter 3, The Business Case, we define a step-by-step approach to defining the business case, which will cover the business need, the reasons for an automated software testing project, the business benefits (tangible and intangible), an analysis of the expected costs and timescales, an investment appraisal, and return on investment (ROI). Chapter 4, Why Automated Software Testing Fails and Pitfalls to Avoid, clarifies some of the myths and realities surrounding automated software testing. The goal is for companies and organizations to review the lessons described here and not to repeat them during their automated software testing implementations. Part II: How to Automate: Top Six Keys for Automation Payoff Once management has been convinced by the business case that was laid out in Part I of this book and understands the pitfalls to avoid and the realities of automated testing, the next step is to determine how to automate. Part II of the book addresses how to successfully implement the various automated software testing tasks. We have determined that successful automated software testing can be achieved by implementing six top keys, described next. Chapter 5, Key 1: Know Your Requirements, covers the importance of understanding the requirements before developing an automated testing strategy. Here we discuss approaches to determining the problem we are trying to solve along with how to gather information when requirements are not available. Chapter 6, Key 2: Develop the Automated Test Strategy, discusses developing an automated testing approach in detailed steps, including test environment considerations, configuration management for automated test scripts, and related artifacts, among others. Here we also discuss what to consider when deciding what to automate and the importance of choosing the right tool, whether open-source, vendor-provided, or in-house-developed. Chapter 7, Key 3: Test the Automated Software Test Framework (ASTF), covers the importance of understanding testing techniques and documenting test cases as part of automated testing. Automators often forget that documentation is still a vital part of the automated test program. The test case documentation serves as the blueprint for the automated software testing efforts. This chapter describes the importance of tracing test cases back to requirements; the content of the test cases, such as needing to include inputs and expected results; and how documented test cases become the basis for developing and implementing the automated tests. Chapter 8, Key 4: Continuously Track Progress—and Adjust Accordingly, addresses the importance of tracking the goal that was set at the outset of the automation program. For example, during the discussion of business case development in Chapter 3 we explain the need for defining goals; in this chapter we discuss how peer reviews, inspections, and various automation and testing metrics can help measure and track progress against those goals. Chapter 9, Key 5: Implement AST Processes, points out the need for a lightweight process. Some automated testing scripts can be implemented successfully without much process in place, but in order to effectively implement a large automated testing program a lightweight adaptable process should be in place. This chapter discusses a summary of this process, linking back to the details in various chapters. Chapter 10, Key 6: Put the Right People on the Project—Know the Skill Sets Required, clarifies the skill sets needed for developing automated software testing, for instance, a skill set similar to that of the software development team, which includes requirements analysis, design, software development, and testing. Key 6 points out that although knowledge of testing techniques and analytical skills is important, effective automated software testing implementation requires software development skills. The skills described here parallel the automated testing process described in Chapter 9. Audience The target audience of this book is software test professionals such as test managers, leads, and practitioners. It is also geared toward all quality assurance professionals, QA leads, and practitioners. Project managers and software developers looking to improve the effectiveness and quality of their software delivery will also benefit from this book. 1 © Copyright Pearson Education. All rights reserved.

Table of Contents

Foreword by Admiral Edmund P. Giambastiani, Jr.     xv
Foreword by Dr. William Nylin, Jr.       xvii
Preface      xix
Acknowledgments      xxiii
About the Authors      xxv

Part I: What Is Automated Software Testing and Why Should We Automate?      1
Chapter 1: What Is Effective Automated Software Testing (AST)?      3

1.1 Automated Software Testing Definition 4
1.2 Automated Software Testing Recipes 5
1.3 Advances in AST Technologies 8
1.4 Automating Various Software Testing Types 11
1.5 Providing AST-Based Production Support 16
1.6 Automating Standards Assessments 18
Summary 20
Notes 21

Chapter 2: Why Automate?      23
2.1 The Challenges of Testing Software Today 24
2.2 Reducing the Time and Cost of Software Testing 26
2.3 Impacting Software Quality 38
2.4 Improvements to Your Software Test Program 42
Summary 49
Notes 50

Chapter 3: The Business Case      51
3.1 Definition of the Business Case 51
3.2 Identifying the Business Needs 53
3.3 Justifying Automation in Terms of Cost and Benefits 55
3.4 Risks 65
3.5 Other Considerations 67
Summary 68
Notes 68

Chapter 4: Why Automated Software Testing Fails and Pitfalls to Avoid      69
4.1 R&D Does Not Generally Focus on Automated or Manual Testing Efforts 71
4.2 AST Myths and Realities 74
4.3 Lack of Software Development Considerations for AST 83
4.4 The Forest for the Trees—Not Knowing Which Tool to Pick 91
4.5 Lack of Automation Standards across Tool Vendors 94
4.6 Lack of Business Case 97
Summary 97
Notes 97

Part II: How to Automate: Top Six Keys for Automation Payoff      99
Chapter 5: Key 1: Know Your Requirements      101

5.1 Understand the AST Support Requirements 102
5.2 Additional Information in Support of AST Requirements 114
5.3 When Information Is Not Available 116
5.4 Start Implementing Your Requirements Traceability Matrix (RTM) 124
Summary 128
Notes 128

Chapter 6: Key 2: Develop the Automated Test Strategy      129
6.1 The AST Strategy Document 131
6.2 Scope and Automated Test Objectives 132
6.3 Identify the Approach 139
6.4 Automated Software Test Framework (ASTF) 146
6.5 AST Environment/Configuration 150
6.6 Automating the RTM 159
6.7 Automated Defect Tracking 164
Summary 164
Notes 165

Chapter 7: Key 3: Test the Automated Software Test Framework (ASTF)      167
7.1 Verify That the ASTF Meets Specified Requirements and That Features Behave As Expected 169
7.2 Peer-Review All ASTF-Related Artifacts, Including Design, Development, and Test Cases 170
7.3 Verify Requirements and Coverage 182
7.4 Hold a Customer Review 183
Summary 184
Notes 184

Chapter 8: Key 4: Continuously Track Progress—and Adjust Accordingly      187
8.1 AST Program Tracking and Defect Prevention 188
8.2 AST Metrics 192
8.3 Root Cause Analysis 205
Summary 206
Notes 207

Chapter 9: Key 5: Implement AST Processes      209
9.1 AST Phases and Milestones 211
9.2 AST Phase 1: Requirements Gathering—Analyze Automated Testing Needs 212
9.3 AST Phase 2: Test Case Design and Development 215
9.4 AST Phase 3: Automated Software Testing Framework (ASTF) and Test Script Development 216
9.5 AST Phase 4: Automated Test Execution and Results Reporting 217
9.6 AST Phase 5: Program Review and Assessment 218
9.7 Virtual Quality Gates 219
9.8 Process Measurement 220
Summary 221
Notes 222

Chapter 10: Key 6: Put the Right People on the Project—Know the Skill Sets Required      223
10.1 Program Management 232
10.2 Systems Engineering 236
10.3 Software Development 239
10.4 Configuration Management 242
10.5 Quality Assurance 244
10.6 Subject Matter Experts (SMEs) 246
Summary 247
Notes 248

Appendices      249
Appendix A: Process Checklist      251

A.1 AST Phase 1: Requirements Gathering—Analyze Automated Testing Needs 252
A.2 AST Phase 2: Test Case Design and Development 253
A.3 AST Phase 3: Automated Software Testing Framework (ASTF) and Test Script Development 254
A.4 AST Phase 4: Automated Test Execution and Results Reporting 255
A.5 AST Phase 5: Program Review and Assessment 256

Appendix B: AST Applied to Various Testing Types      257
B.1 Security Testing 257
B.2 Soak Testing 261
B.3 Concurrency Testing 263
B.4 Performance Testing 265
B.5 Code Coverage Testing 269
B.6 Unit Testing 271
Notes 274

Appendix C: The Right Tool for the Job      275
C.1 Requirements Management (RM) 276
C.2 Unit Testing Frameworks—Example Evaluation 281
C.3 Configuration Management—Example Evaluation 284
C.4 Defect Tracking—Example Evaluation 292
C.5 Security Testing 299
C.6 Automated Software Testing Framework (ASTF)—Example Evaluation 306
C.7 Other STL Support Tools 316
Notes 319

Appendix D: Case Study: An Automated Software Testing Framework (ASTF) Example      321
D.1 Key Design Features 323
D.2 Test Manager 325
D.3 More on Automated Test Case and Test Code Generation 326
D.4 Results Reporting 328
D.5 Automated Defect Reporting 328
Notes 329

Contributing Authors      331

Index      333