Font size
WorksheetsStreaming Data Challenges
Total questions: 26
Worksheet time: 13mins
What is Dataflow used for?
Only for batch data processing
Only for streaming data processing
For both batch and streaming data processing
For data storage
What type of collections can be processed with Dataflow?
Bounded collections only
Unbounded collections only
Both bounded and unbounded collections
Neither bounded nor unbounded collections
In a streaming pipeline, how does Dataflow process incoming messages?
In batches
In real time
After storing them in a database
At the end of the day
What are some of the challenges associated with processing streaming data? (Select all that apply)
Scalability
Fault tolerance
Model being used (streaming or repeated batch)
High cost of processing
Why is timing or latency a challenge in streaming data processing?
Because messages are always sent in order
Because network delays or faulty sensors can prevent messages from being sent on time
Because messages are processed in batches
Because Dataflow does not support real-time processing
How does Dataflow handle aggregation in streaming scenarios?
By using a single formula for all incoming data
By dividing time into windows and aggregating within those windows
By ignoring time and processing data as it comes
By storing data and processing it later
What feature does Dataflow provide to handle message timestamps?
It ignores message timestamps
It uses a default date timestamp (DTS) for all messages
It allows modifying timestamps based on some property of the data
It does not support timestamp modifications
What happens if Pub/Sub IO is configured to use custom message IDs in Dataflow?
Messages with duplicate IDs are processed multiple times
Messages with duplicate IDs are discarded if they have been seen in the last 10 minutes
All messages are processed regardless of their IDs
Messages are stored for future reference
Which of the following is true about the default date timestamp (DTS) in Dataflow?
It represents the time the sensor data was captured
It represents the time of entry to the system
It is used to discard old messages
It cannot be modified
What does the PTransform do in the context of modifying timestamps?
It extracts the date timestamp from the data portion of the element
It adds a default date timestamp to the element
It deletes the date timestamp from the element
It prevents the modification of the date timestamp
What kind of service does Dataflow provide?
On-premise service
Serverless service
Manual service
Desktop service
How does Dataflow handle different steps in a streaming pipeline?
Processes them in batches
Processes them in real time
Skips intermediate steps
Processes them at the end of the day
What is the benefit of dividing time into windows for streaming data?
It makes the data processing slower
It helps in aggregating data over a manageable period
It reduces the amount of data to be processed
It discards old data
What happens when a sensor's message is delayed due to network issues?
The message is ignored
The message is processed immediately
There may be a need to modify the timestamp
The message is stored indefinitely
What does the timestamp in a Pub/Sub message represent?
The time the message was read by the sensor
The time the message was received by the system
The time the message was processed
The time the message was deleted
How does Dataflow manage message ordering and latency issues?
By ignoring the timestamps
By modifying the timestamps based on data properties
By deleting old messages
By processing messages in reverse order
What is the purpose of using custom message IDs in Pub/Sub IO with Dataflow?
To prioritize messages
To de-duplicate messages
To encrypt messages
To categorize messages
How often does Dataflow check for duplicate messages based on custom IDs?
Every minute
Every 5 minutes
Every 10 minutes
Every hour
What does Dataflow use to automatically manage windowing, time, and aggregation?
Manual scripts
Default configurations
Built-in features
External plugins
In Dataflow, what is a PTransform used for in the context of timestamps?
To add a new timestamp
To extract and modify the timestamp from the data element
To delete the timestamp
To encrypt the timestamp
What advantage does Dataflow offer in terms of pipeline scalability?
It requires manual intervention for scaling
It automatically scales to handle increased data volumes
It does not support scaling
It can only handle small data volumes
How does Dataflow ensure fault tolerance in streaming data processing?
By ignoring failed messages
By using built-in features to handle failures and continue processing
By requiring manual restart of the pipeline
By storing all messages and processing them later
What challenge is specifically mentioned in relation to aggregating data in streaming scenarios?
High computational cost
Difficulty in maintaining windows and roll threads
Lack of support for aggregation
Limited storage capacity
What type of timestamp is added by default when a message enters the Dataflow system?
Sensor read timestamp
System entry timestamp (DTS)
Custom message timestamp
Network latency timestamp
How does Dataflow handle messages with significant latency between sensor read time and message send time?
It ignores these messages
It automatically adjusts timestamps based on latency
It requires manual adjustment of timestamps
It stores these messages for later processing
What element is applied by Pub/Sub to messages that allows Dataflow to perform aggregation of different time windows?
Message Groups
Triggers
Timestamps
Watermarks
