How Google Tests Software

Foreword by Alberto Savoia xiii Foreword by Patrick Copeland xvii Preface xxiii Chapter 1: Introduction to Google Software Testing 1 Quality?Test 5 Roles 6 Organizational Structure 8 Crawl, Walk, Run 10 Types of Tests 12 Chapter 2: The Software Engineer in Test 15 The Life of an SET 17 Development and Test Workflow 17 Who Are These SETs Anyway? 22 The Early Phase of a Project 22 Team Structure 24 Design Docs 25 Interfaces and Protocols 27 Automation Planning 28 Testability 29 SET Workflow: An Example 32 Test Execution 40 Test Size Definitions 41 Use of Test Sizes in Shared Infrastructure 44 Benefits of Test Sizes 46 Test Runtime Requirements 48 Case 1: Change in Common Library 52 Test Certified 54 An Interview with the Founders of the Test Certified Program 57 Interviewing SETs 62 An Interview with Tool Developer Ted Mao 68 An Interview with Web Driver Creator Simon Stewart 70 Chapter 3: The Test Engineer 75 A User-Facing Test Role 75 The Life of a TE 76 Test Planning 79 Risk 97 Life of a Test Case 108 Life of a Bug 113 Recruiting TEs 127 Test Leadership at Google 134 Maintenance Mode Testing 137 Quality Bots Experiment 141 BITE Experiment 153 Google Test Analytics 163 Free Testing Workflow 169 External Vendors 173 An Interview with Google Docs TE Lindsay Webster 175 An Interview with YouTube TE Apple Chow 181 Chapter 4: The Test Engineering Manager 187 The Life of a TEM 187 Getting Projects and People 189 Impact 191 An Interview with Gmail TEM Ankit Mehta 193 An Interview with Android TEM Hung Dang 198 An Interview with Chrome TEM Joel Hynoski 202 The Test Engineering Director 206 An Interview with Search and Geo Test Director Shelton Mar 207 An Interview with Engineering Tools Director Ashish Kumar 211 An Interview with Google India Test Director Sujay Sahni 214 An Interview with Engineering Manager Brad Green 219 An Interview with James Whittaker 222 Chapter 5: Improving How Google Tests Software 229 Fatal Flaws in Google's Process 229 The Future of the SET 231

James Whittaker is an engineering director at Google and has been responsible for testing Chrome, maps, and Google web apps. He used to work for Microsoft and was a professor before that. James is one of the best-known names in testing the world over. Jason Arbon is a test engineer at Google and has been responsible for testing Google Desktop, Chrome, and Chrome OS. He also served as development lead for an array of open-source test tools and personalization experiments. He worked at Microsoft prior to joining Google. Jeff Carollo is a software engineer in test at Google and has been responsible for testing Google Voice, Toolbar, Chrome, and Chrome OS. He has consulted with dozens of internal Google development teams helping them improve initial code quality. He converted to a software engineer in 2010 and leads development of Google+ APIs. He also worked at Microsoft prior to joining Google."