GCP_Data_Engineer_Day 4

A large enterprise using GCP has recently acquired a startup that has an IoT platform. The acquiring company wants to migrate the IoT platform from an on-premises data center to GCP and wants to use Google Cloud managed services whenever possible. What GCP service would you recommend for ingesting IoT data?

• Cloud Storage

• Cloud SQL

• Cloud Pub/Sub

• BigQuery streaming inserts

You are using Cloud Pub/Sub to buffer records from an application that generates a stream of data based on user interactions with a website. The messages are read by another service that transforms the data and sends it to a machine learning model that will use it for training. A developer has just released some new code, and you notice that messages are sent repeatedly at 10-minute intervals. What might be the cause of this problem?

• The new code release changed the subscription ID

• The new code release changed the topic ID.

• The new code disabled acknowledgments from the consumer

• The new code changed the subscription from pull to push.

It is considered a good practice to make your processing logic idempotent when consuming messages from a Cloud Pub/Sub topic. Why is that?

• Messages may be delivered multiple times

• Messages may be received out of order.

• Messages may be delivered out of order

• A consumer service may need to wait extended periods of time between the delivery of messages

Your department is migrating some stream processing to GCP and keeping some on premises. You are tasked with designing a way to share data from on-premises pipelines that use Kafka with GPC data pipelines that use Cloud Pub/Sub. How would you do that?

• Use CloudPubSubConnector and Kafka Connect

• Stream data to a Cloud Storage bucket and read from there

• Write a service to read from Kafka and write to Cloud Pub/Sub

• Use Cloud Pub/Sub Import Service

A team of data warehouse developers is migrating a set of legacy Python scripts that have been used to transform data as part of an ETL process. They would like to use a service that allows them to use Python and requires minimal administration and operations support. Which GCP service would you recommend?

• Cloud Dataproc

• Cloud Dataflow

• Cloud Spanner

• Cloud Dataprep

 

A group of IoT sensors is sending streaming data to a Cloud Pub/Sub topic. A Cloud Dataflow service pulls messages from the topic and reorders the messages sorted by event time. A message is expected from each sensor every minute. If a message is not received from a sensor, the stream processing application should use the average of the values in the last four messages. What kind of window would you use to implement the missing data logic?

• Sliding window

• Tumbling window

• Extrapolation window

• Crossover window

You are designing a data pipeline to populate a sales data mart. The sponsor of the project has had quality control problems in the past and has defined a set of rules for filtering out bad data before it gets into the data mart. At what stage of the data pipeline would you implement those rules?

• Ingestion

• Storage

• Transformation

• Analysis

A team of developers wants to create standardized patterns for processing IoT data. Several teams will use these patterns. The developers would like to support collaboration and facilitate the use of patterns for building streaming data pipelines. What component should they use?

• Cloud Dataflow Python Scripts

• Cloud Dataproc PySpark jobs

• Cloud Dataflow templates

• Cloud Dataproc templates

You are deploying 10,000 new Internet of Things devices to collect temperature data in your warehouses globally. You need to process, store and analyze these very large datasets in real time. What should you do?

• Send the data to Google Cloud Datastore and then export to BigQuery.

• . Send the data to Google Cloud Pub/Sub, stream Cloud Pub/Sub to Google Cloud Dataflow, and store the data in Google BigQuery.

• Send the data to Cloud Storage and then spin up an Apache Hadoop cluster as needed in Google Cloud Dataproc whenever analysis is required

• Export logs in batch to Google Cloud Storage and then spin up a Google Cloud SQL instance, import the data from Cloud Storage, and run an analysis as needed.

By default, which of the following windowing behavior does Dataflow apply to bounded data sets?

• Windows at every 100 MB of data

• Single, Global Window

• Windows at every 1 minute

• Windows at every 10 minutes