Official Google Cloud Certified Professional Data Engineer Study Guide

Questions 86 Chapter 4 Designing a Data Processing Solution 89 Designing Infrastructure 90 Choosing Infrastructure 90 Availability, Reliability, and Scalability of Infrastructure 93 Hybrid Cloud and Edge Computing 96 Designing for Distributed Processing 98 Distributed Processing: Messaging 98 Distributed Processing: Services 101 Migrating a Data Warehouse 102 Assessing the Current State of a Data Warehouse 102 Designing the Future State of a Data Warehouse 103 Migrating Data, Jobs, and Access Controls 104 Validating the Data Warehouse 105 Exam Essentials 105 Review Questions 107 Chapter 5 Building and Operationalizing Processing Infrastructure 111 Provisioning and Adjusting Processing Resources 112 Provisioning and Adjusting Compute Engine 113 Provisioning and Adjusting Kubernetes Engine 118 Provisioning and Adjusting Cloud Bigtable 124 Provisioning and Adjusting Cloud Dataproc 127 Configuring Managed Serverless Processing Services 129 Monitoring Processing Resources 130 Stackdriver Monitoring 130 Stackdriver Logging 130 Stackdriver Trace 131 Exam Essentials 132 Review Questions 134 Chapter 6 Designing for Security and Compliance 139 Identity and Access Management with Cloud IAM 140 Predefined Roles 141 Custom Roles 143 Using Roles with Service Accounts 145 Access Control with Policies 146 Using IAM with Storage and Processing Services 148 Cloud Storage and IAM 148 Cloud Bigtable and IAM 149 BigQuery and IAM 149 Cloud Dataflow and IAM 150 Data Security 151 Encryption 151 Key Management 153 Ensuring Privacy with the Data Loss Prevention API 154 Detecting Sensitive Data 154 Running Data Loss Prevention Jobs 155 Inspection Best Practices 156 Legal Compliance 156 Health Insurance Portability and Accountability Act (HIPAA) 156 Children's Online Privacy Protection Act 157 FedRAMP 158 General Data Protection Regulation 158 Exam Essentials 158 Review Questions 161 Chapter 7 Designing Databases for Reliability, Scalability, and Availability 165 Designing Cloud Bigtable Databases for Scalability and Reliability 166 Data Modeling with Cloud Bigtable 166 Designing Row-keys 168 Designing for Time Series 170 Use Replication for Availability and Scalability 171 Designing Cloud Spanner Databases for Scalability and Reliability 172 Relational Database Features 173 Interleaved Tables 174 Primary Keys and Hotspots 174 Database Splits 175 Secondary Indexes 176 Query Best Practices 177 Designing BigQuery Databases for Data Warehousing 179 Schema Design for Data Warehousing 179 Clustered and Partitioned Tables 181 Querying Data in BigQuery 182 External Data Access 183 BigQuery ML 185 Exam Essentials 185 Review Questions 188 Chapter 8 Understanding Data Operations for Flexibility and Portability 191 Cataloging and Discovery with Data Catalog 192 Searching in Data Catalog 193 Tagging in Data Catalog 194 Data Preprocessing with Dataprep 195 Cleansing Data 196 Discovering Data 196 Enriching Data 197 Importing and Exporting Data 197 Structuring and Validating Data 198 Visualizing with Data Studio 198 Connecting to Data Sources 198 Visualizing Data 200 Sharing Data 200 Exploring Data with Cloud Datalab 200 Jupyter Notebooks 201 Managing Cloud Datalab Instances 201 Adding Libraries to Cloud Datalab Instances 202 Orchestrating Workflows with Cloud Composer 202 Airflow Environments 203 Creating DAGs 203 Airflow Logs 204 Exam Essentials 204 Review Questions 206 Chapter 9 Deploying Machine Learning Pipelines 209 Structure of ML Pipelines 210 Data Ingestion 211 Data Preparation 212 Data Segregation 215 Model Training 217 Model Evaluation 218 Model Deployment 220 Model Monitoring 221 GCP Options for Deploying Machine Learning Pipeline 221 Cloud AutoML 221 BigQuery ML 223 Kubeflow 223 Spark Machine Learning 224 Exam Essentials 225 Review Questions 227 Chapter 10 Choosing Training and Serving Infrastructure 231 Hardware Accelerators 232 Graphics Processing Units 232 Tensor Processing Units 233 Choosing Between CPUs, GPUs, and TPUs 233 Distributed and Single Machine Infrastructure 234 Single Machine Model Training 234 Distributed Model Training 235 Serving Models 236 Edge Computing with GCP 237 Edge Computing Overview 237 Edge Computing Components and Processes 239 Edge TPU 240 Cloud IoT 240 Exam Essentials 241 Review Questions 244 Chapter 11 Measuring, Monitoring, and Troubleshooting Machine Learning Models 247 Three Types of Machine Learning Algorithms 248 Supervised Learning 248 Unsupervised Learning 253 Anomaly Detection 254 Reinforcement Learning 254 Deep Learning 255 Engineering Machine Learning Models 257 Model Training and Evaluation 257 Operationalizing ML Models 262 Common Sources of Error in Machine Learning Models 263 Data Quality 264 Unbalanced Training Sets 264 Types of Bias 264 Exam Essentials 265 Review Questions 267 Chapter 12 Leveraging Prebuilt Models as a Service 269 Sight 270 Vision AI 270 Video AI 272 Conversation 274 Dialogflow 274 Cloud Text-to-Speech API 275 Cloud Speech-to-Text API 275 Language 276 Translation 276 Natural Language 277 Structured Data 278 Recommendations AI API 278 Cloud Inference API 280 Exam Essentials 280 Review Questions 282 Appendix Answers to Review Questions 285 Chapter 1: Selecting Appropriate Storage Technologies 286 Chapter 2: Building and Operationalizing Storage Systems 288 Chapter 3: Designing Data Pipelines 290 Chapter 4: Designing a Data Processing Solution 291 Chapter 5: Building and Operationalizing Processing Infrastructure 293 Chapter 6: Designing for Security and Compliance 295 Chapter 7: Designing Databases for Reliability, Scalability, and Availability 296 Chapter 8: Understanding Data Operations for Flexibility and Portability 298 Chapter 9: Deploying Machine Learning Pipelines 299 Chapter 10: Choosing Training and Serving Infrastructure 301 Chapter 11: Measuring, Monitoring, and Troubleshooting Machine Learning Models 303 Chapter 12: Leveraging Prebuilt Models as a Service 304 Index 307