WorksheetsBig Data 8: Spark Streaming
Total questions: 146
Worksheet time: 1hrs 13mins
Name
Class
Date
1.
In the Spark ecosystem diagram, which component is described as enabling structured data processing via a unified API for queries on existing Hadoop deployments?
a)
Spark SQL
b)
Spark Streaming
c)
MLlib
d)
GraphX
e)
SparkR
2.
In the ecosystem diagram, what is the role of the Spark Core Engine layer relative to Spark SQL, Streaming, MLlib, GraphX, and SparkR?
a)
It provides the core utilities and architecture that the other components build on
b)
It is a connector layer that only ingests data from Twitter
c)
It is a UI-only module for monitoring jobs
d)
It is an optional library used only for graph computation
e)
It is a sink-specific module used only for writing to HDFS
3.
Which statement best matches the definition of discretized stream processing in Spark Streaming?
a)
A streaming computation is executed as a series of very small, deterministic batch jobs
b)
A streaming computation is executed as a single long-running non-deterministic job
c)
A streaming computation requires manual partition replication for every transformation
d)
A streaming computation is executed only when a user queries an interactive UI
e)
A streaming computation bypasses RDD operations to avoid scheduling overhead
4.
Under discretized stream processing, how does Spark represent each batch of the live stream for processing?
a)
As RDDs processed with standard RDD operations
b)
As a single mutable in-memory table shared across batches
c)
As an actor message queue that cannot be recomputed
d)
As a persistent file block that must be read from disk for every step
e)
As a pre-aggregated result table that cannot be transformed
5.
A Spark Streaming job uses very small batches. Which pair of values is consistent with the slide's claim about achievable batch size and typical latency?
a)
Batch size as low as 0.5 second with latency around 1 second
b)
Batch size as low as 5 seconds with latency around 50 milliseconds
c)
Batch size as low as 1 minute with latency around 1 second
d)
Batch size as low as 0.1 second with latency around 10 seconds
e)
Batch size as low as 10 seconds with latency around 0.1 second
6.
What system-level advantage is highlighted as a consequence of Spark Streaming's discretized (batch-based) approach?
a)
Potential to combine batch processing and streaming processing in the same system
b)
Eliminates the need for any output operations
c)
Guarantees exactly-once delivery to any sink without sink design considerations
d)
Prevents recomputation by disabling lineage tracking
e)
Requires all sources to be TCP sockets
7.
In the Twitter example, what does a DStream represent in Spark Streaming?
a)
A sequence of RDDs representing a stream of data
b)
A single RDD that is continuously mutated over time
c)
A table that only supports SQL queries
d)
A sink configuration for writing results to HDFS
e)
A scheduler component that controls checkpoint intervals
8.
In the Twitter example diagram, how is each micro-batch of the tweets stream stored for processing?
a)
In memory as an RDD that is immutable and distributed
b)
On disk as a single local file per executor
c)
Only in the driver process as a mutable collection
d)
Inside a Kafka topic partition that Spark cannot cache
e)
As a broadcast variable replicated to all nodes
9.
In the hashtags example, which operation is explicitly described as a DStream transformation that creates a new DStream from an existing one?
a)
flatMap
b)
saveAsHadoopFiles
c)
foreach
d)
checkpoint
e)
startStream
10.
According to the hashtags transformation diagram, what happens at the RDD level when a DStream transformation like flatMap is applied?
a)
New RDDs are created for every batch
b)
The original RDD is mutated in place across batches
c)
Only the first batch creates an RDD; later batches reuse it
d)
RDDs are created only when an action is called on the driver
e)
RDDs are replaced by a single global table shared by all batches
11.
In the Twitter hashtags pipeline, which operation is classified as an output operation that pushes data to external storage?
a)
saveAsHadoopFiles
b)
flatMap
c)
window
d)
countByValue
e)
reduceByKey
12.
When using saveAsHadoopFiles in the example, what is the unit of data written to HDFS?
a)
The result of each batch is saved
b)
Only the final batch after streaming stops is saved
c)
Only batches that contain errors are saved
d)
A single combined file for all batches is saved continuously
e)
Only the driver-side collected output is saved
13.
In the Java version of the example, how is the transformation logic (e.g., for flatMap) provided?
a)
As a Function object passed to the transformation
b)
As a SQL string executed on Spark SQL
c)
As a broadcast variable updated every batch
d)
As an HDFS configuration file read at runtime
e)
As a checkpointed RDD written by the sink
14.
Which pairing matches the slide's terminology for Scala vs Java streaming types?
a)
Scala uses DStream, Java uses JavaDStream
b)
Scala uses JavaDStream, Java uses DStream
c)
Scala uses DataFrame, Java uses RDD
d)
Scala uses GraphX, Java uses MLlib
e)
Scala uses SparkR, Java uses Spark SQL
15.
Which mechanism is described as allowing lost partitions to be recomputed after a worker failure in Spark Streaming?
a)
RDDs remember the sequence of operations that created them from fault-tolerant input data
b)
Executors permanently store all intermediate results on local disk
c)
Transformations are executed twice and compared for correctness
d)
The driver rewrites all results from scratch without using input data
e)
Kafka guarantees that no partitions can ever be lost
16.
The slide states that Spark Streaming is fault-tolerant because batches of input data are replicated where?
a)
In the memory of multiple worker nodes
b)
Only on the driver node
c)
Only in HDFS before processing begins
d)
Only in a single executor chosen as leader
e)
Only in the Spark UI event log
17.
Which item is categorized on the slide as a stateful operation (not a standard per-batch RDD operation)?
a)
countByValueAndWindow
b)
map
c)
join
d)
reduce
e)
countByValue
18.
Which output operation is described as enabling arbitrary processing on each batch of results?
a)
foreach
b)
window
c)
reduceByKey
d)
countByValue
e)
flatMap
19.
In the hashtag counting example, what does tagCounts = hashTags.countByValue() compute at each batch?
a)
Counts of each distinct hashtag value in that batch
b)
A global count accumulated across all past batches without windows
c)
Counts of each distinct tweet status object
d)
Only the top-1 hashtag per batch
e)
A running average of counts across batches
20.
In the pipeline tweets -> hashTags -> tagCounts, on which DStream is countByValue applied?
a)
The hashTags DStream
b)
The tweets DStream before flatMap
c)
A sink DStream created by saveAsHadoopFiles
d)
A windowed DStream created by reduceByKeyAndWindow
e)
A broadcasted DStream replicated to executors
21.
In the expression hashTags.window(Minutes(10), Seconds(1)).countByValue(), what does Minutes(10) represent?
a)
The window length over which data is considered
b)
The trigger interval for checking new input
c)
The checkpoint interval for state recovery
d)
The maximum allowed lateness for event time
e)
The time to persist each RDD in memory
22.
In the same expression, what does Seconds(1) represent?
a)
The sliding interval (how often the window moves)
b)
The window length (how much history is included)
c)
The maximum network delay tolerated by the source
d)
The time to write results to HDFS
e)
The duration of each RDD partition
23.
The window + countByValue pipeline is illustrated over time (t-1, t, t+1...). What is being counted at time t when using a sliding window?
a)
All data that falls within the current window, not just the newest batch
b)
Only the newest batch at time t
c)
Only batches that have not yet been checkpointed
d)
Only records that arrive exactly at the trigger boundary
e)
Only records that were seen in the previous window
24.
Based on the slide, which ordering correctly describes the computation for windowed counting?
a)
Create a windowed DStream, then apply countByValue to the data in each window
b)
Apply countByValue per batch, then retroactively build a window over counts
c)
Persist the DStream, then apply window and skip counting
d)
Broadcast the DStream, then reduce over executors only
e)
Write to HDFS first, then window over files
25.
Which operation is presented as a smarter window-based alternative that avoids recomputing counts from scratch for each window?
a)
countByValueAndWindow
b)
countByValue
c)
saveAsHadoopFiles
d)
join
e)
foreach
26.
In the slide's incremental window counting approach, what two updates are performed when the window advances by one batch?
a)
Add counts from the new batch in the window and subtract counts from the batch that fell out of the window
b)
Add counts from all historical batches and reset counts for the newest batch
c)
Subtract counts from the new batch and add counts from the dropped batch
d)
Only add counts from the new batch; never remove old contributions
e)
Only subtract old counts; never incorporate new data
27.
The slide states that incremental computation generalizes beyond counting. What extra requirement is needed to support this for many reduce operations?
a)
An inverse reduce function to remove the contribution of data that leaves the window
b)
A larger batch size to reduce scheduler overhead
c)
A sink that supports exactly-once writes without retries
d)
A SQL engine to translate RDD operations into queries
e)
A broadcast variable to store the entire window state
28.
Which expression matches the slide's example of implementing counting via a windowed reduce that adds and subtracts?
a)
reduceByKeyAndWindow(_ + _, _ - _, Minutes(1), ...)
b)
countByValueAndWindow(Minutes(1), Seconds(1))
c)
window(Minutes(1), Seconds(1)).countByValue()
d)
reduceByKey(_ + _).persist()
e)
foreachRDD(rdd => rdd.count())
29.
What capability of DStreams is emphasized on the caching/persistence slide?
a)
They can be cached or persisted for reuse in subsequent transformations
b)
They automatically write every batch to HDFS by default
c)
They must be recomputed from scratch for every downstream action
d)
They are mutable, so caching is unnecessary
e)
They can only be stored on the driver
30.
Which method is explicitly named for enabling caching/persistence of a DStream?
a)
persist()
b)
withWatermark()
c)
startStream()
d)
saveAsHadoopFiles()
e)
countByValue()
31.
According to the slide, accumulator updates must be performed using operations with which algebraic properties?
a)
Associative and commutative
b)
Idempotent and monotonic
c)
Injective and surjective
d)
Deterministic and reversible
e)
Serializable and checkpointed
32.
What is highlighted as a practical benefit of accumulators during execution?
a)
They can be viewed in the UI to help understand how computation stages run
b)
They replace the need for output operations
c)
They allow arbitrary per-record side effects without constraints
d)
They store the full dataset on every executor
e)
They guarantee window aggregations are exact without recomputation
33.
What is the primary semantic of a broadcast variable as described on the slide?
a)
A read-only variable available on each compute node for sharing data across nodes
b)
A per-batch mutable counter shared by all tasks
c)
A write-only sink for sending output to external storage
d)
A stream source that emits data from Kafka
e)
A checkpoint file that stores RDD lineage
34.
Why does Spark use specialized algorithms for broadcast variables according to the slide?
a)
To reduce communication cost when distributing the read-only value
b)
To increase the number of partitions by duplicating input batches
c)
To convert DStreams into RDDs without translation
d)
To ensure the value can be updated independently on each executor
e)
To guarantee sub-second latency regardless of batch size
35.
Which throughput and scale claim is stated on the performance slide?
a)
Can process about 6 GB/sec (about 60 million records/sec) on 100 nodes at sub-second latency
b)
Can process about 600 MB/sec on 10 nodes at minute-level latency
c)
Can process about 6 TB/sec on 1000 nodes at millisecond latency
d)
Can process about 60 GB/sec on 10 nodes at sub-second latency
e)
Can process about 6 GB/sec on a single node at sub-second latency
36.
In the cited test setup on the performance slide, which environment is described?
a)
100 streams of data on 100 EC2 instances with 4 cores each
b)
10 streams of data on 10 EC2 instances with 16 cores each
c)
A single stream on 100 EC2 instances with 1 core each
d)
100 streams on 10 on-prem servers with 64 cores each
e)
10 streams on 100 EC2 instances with 2 cores each
37.
Which ordering of throughput per node is consistent with the slide's comparison among Spark Streaming, Storm, and Apache S4?
a)
Spark Streaming highest, Storm next, Apache S4 lowest
b)
Storm highest, Spark Streaming next, Apache S4 lowest
c)
Apache S4 highest, Spark Streaming next, Storm lowest
d)
Storm highest, Apache S4 next, Spark Streaming lowest
e)
All three have approximately equal throughput per node
38.
The slide reports approximate records/second/node values. Which triple matches those figures?
a)
Spark Streaming ~670k, Storm ~115k, Apache S4 ~7.5k
b)
Spark Streaming ~115k, Storm ~670k, Apache S4 ~7.5k
c)
Spark Streaming ~7.5k, Storm ~115k, Apache S4 ~670k
d)
Spark Streaming ~670k, Storm ~7.5k, Apache S4 ~115k
e)
Spark Streaming ~75k, Storm ~115k, Apache S4 ~670k
39.
What recovery time is highlighted for Spark Streaming when dealing with faults or stragglers in the fast fault recovery slide?
a)
Within about 1 second
b)
Within about 30 seconds
c)
Within about 5 minutes
d)
Only after the job is restarted manually
e)
Only after a full recomputation from the original dataset on disk
40.
The fast fault recovery figure is labeled as a sliding WordCount example. Which checkpoint interval is shown in the caption?
a)
30 seconds
b)
1 second
c)
10 minutes
d)
5 minutes
e)
1 hour
41.
In the real application example, what is Spark Streaming used for at Conviva according to the slide?
a)
Real-time monitoring of video metadata
b)
Batch-only training of machine learning models
c)
Offline graph computation for social network analysis
d)
Interactive SQL queries over static parquet tables
e)
Compiling R scripts to JVM bytecode
42.
Which set of claims is listed for the Conviva application on the slide?
a)
Achieved about 1-2 second latency, processed millions of video sessions, and scaled linearly with cluster size
b)
Achieved millisecond latency, processed thousands of sessions, and scaled sublinearly
c)
Achieved 1-2 minute latency, processed billions of sessions, and did not scale with cluster size
d)
Achieved 10 second latency, processed only metadata samples, and required single-node execution
e)
Achieved constant throughput independent of cluster size, but only with batch processing
43.
Immediately after the DStreams section, which streaming paradigm is introduced as the next topic in the slide deck?
a)
Structured streaming
b)
GraphX streaming
c)
SparkR streaming
d)
MLlib online learning
e)
Spark SQL-only batch processing
44.
Which pain point is explicitly tied to the DStream API exposing batch time?
a)
It is hard to incorporate event-time processing
b)
It prevents using any RDD operations
c)
It forces all sinks to be HDFS
d)
It eliminates the need for failure handling
e)
It makes window operations impossible
45.
Which item is listed as a pain point related to combining different processing styles?
a)
Interoperate streaming with batch and interactive workloads
b)
Replicate every RDD partition to all nodes
c)
Disable lineage to reduce overhead
d)
Avoid using any window operations
e)
Require TCP sockets as the only input source
46.
Which pain point focuses on correctness guarantees at the boundary between streaming computation and external systems?
a)
Requires carefully constructing sinks that handle failures correctly
b)
Requires translating all Scala code into Java
c)
Requires converting all RDDs into GraphX graphs
d)
Requires reducing batch duration to 0.1 seconds
e)
Requires using only stateless transformations
47.
In the structured streaming new model, how is input data conceptually represented?
a)
As an append-only table
b)
As a mutable global variable updated in place
c)
As a set of checkpoint files that must be rewritten every trigger
d)
As a per-node local file that Spark does not track
e)
As a fixed-size bounded table that is rewritten each minute
48.
In the new model, what does the term trigger describe?
a)
How frequently to check the input for new data
b)
How long to keep a broadcast variable in memory
c)
How to serialize RDD lineage for recomputation
d)
How to assign partition keys for Kafka
e)
How to define the schema of a DStream
49.
In the new model, what does the slide say about the result of the query over time?
a)
The final operated table is updated every trigger interval
b)
The result table is computed only once and never changes
c)
The result is emitted only when the job terminates
d)
The result is stored only as an RDD lineage graph
e)
The result is always a single row per trigger
50.
What does complete output mode do according to the slide?
a)
Writes the full result table every time
b)
Writes only rows that changed since the previous trigger
c)
Writes only new rows and never updates existing rows
d)
Writes only intermediate state and never outputs results
e)
Writes only late data beyond the watermark
51.
Which description matches delta output mode in the new model?
a)
Write only the rows that changed in the result from the previous batch
b)
Write the full result table every trigger
c)
Write only new input rows before any computation
d)
Write only data that arrived after the watermark
e)
Write only the schema and not the data
52.
What caveat about output modes is explicitly stated on the slide?
a)
Not all output modes are feasible with all queries
b)
All output modes are always feasible for any query
c)
Delta output is required for all window queries
d)
Append output cannot be used with append-only input
e)
Complete output is incompatible with triggers
53.
In the slide's quick example model, which query is being run over the unbounded input table?
a)
A word count query
b)
A graph shortest-path query
c)
A join between two bounded tables
d)
A model training query
e)
A broadcast update query
54.
In the same example, which output behavior is shown under complete mode?
a)
Print all the counts to the console each trigger
b)
Print only the newly changed counts to the console
c)
Append only new input rows to the console without aggregation
d)
Write only watermark metadata to the console
e)
Write only the schema of the result to the console
55.
The event-time windowed aggregation example is labeled with window and slide settings. Which combination matches the slide?
a)
10 minute windows sliding every 5 minutes
b)
5 minute windows sliding every 10 minutes
c)
10 minute windows sliding every 1 minute
d)
1 minute windows sliding every 10 minutes
e)
5 minute session windows with 10 minute gaps
56.
In the same figure, what is said about when result tables are produced?
a)
Result tables are shown after 5 minute triggers
b)
Result tables appear only after 10 minute triggers
c)
Result tables are produced only when streaming stops
d)
Result tables appear only when a failure occurs
e)
Result tables appear only for non-overlapping windows
57.
In the late-data illustration, late data is generated at an earlier event time but arrives much later. What is the key issue being managed?
a)
Out-of-order arrival relative to event time
b)
Lack of schema for the input stream
c)
Inability to run any aggregations
d)
Need to convert RDDs into DataFrames
e)
Requirement to use HDFS as the only sink
58.
In the slide's example, which window is explicitly shown as receiving the count increment from the late data?
a)
Window 12:00-12:10
b)
Window 12:05-12:15
c)
Window 12:10-12:20
d)
No window; late data is always dropped
e)
All overlapping windows equally
59.
Which definition of event time matches the watermarking slide?
a)
The timestamp associated with when an event occurred in the real world
b)
The time when Spark finishes processing a batch
c)
The time when output is written to the sink
d)
The time interval used to checkpoint state
e)
The time when the driver schedules a task
60.
According to the slide, what does a watermark specify in structured streaming?
a)
A threshold for how late data is allowed to be
b)
A guarantee that late data will always be reprocessed
c)
A fixed batch duration for discretized processing
d)
A method for persisting RDDs to disk
e)
A requirement to write complete output every trigger
61.
In group aggregation with update mode, what is the typical content written to the sink after each trigger?
a)
Only the rows that were updated since the previous trigger
b)
Only brand-new rows that have never been seen before
c)
The full result table every trigger regardless of changes
d)
Only the watermark value and no result rows
e)
Only rows that arrived after the watermark
62.
In the update-mode diagram, what happens to intermediate state for a window once the watermark passes the window end?
a)
It is dropped, so further updates for that window stop
b)
It is kept forever to accept arbitrarily late data
c)
It is immediately written as complete output every trigger
d)
It is converted into an RDD lineage graph for recomputation
e)
It is broadcast to all executors for faster joins
63.
In group aggregation with append mode, when are results for a window emitted to the sink?
a)
When the watermark indicates the window is complete (safe to finalize)
b)
On every trigger regardless of watermark
c)
Only when a failure occurs
d)
Immediately when the first record for the window arrives
e)
Only after a manual flush is invoked on the driver
64.
In the append-mode figure, what is described as happening to late data that arrives after the watermark?
a)
It is dropped and not counted
b)
It is always appended as a correction row
c)
It triggers recomputation of all past windows
d)
It is stored as a broadcast variable for later use
e)
It is written to the sink in complete mode
65.
Which description matches a tumbling window as illustrated on the slide?
a)
Fixed-size, non-overlapping windows that cover time contiguously
b)
Overlapping windows that slide by less than their length
c)
Windows that close based on inactivity gaps
d)
Windows defined only by processing time, not event time
e)
Windows that expand indefinitely until streaming stops
66.
In session windows (gap duration shown as 5 minutes on the slide), when is a session considered closed?
a)
After a period of inactivity equal to the gap duration since the last event
b)
Exactly at fixed 5-minute boundaries regardless of activity
c)
Whenever the trigger interval fires
d)
Only when the watermark passes the session start
e)
Only after the job writes complete output
67.
In the batch ETL DataFrame example, which call is used to read a JSON dataset from a path?
a)
load("source-path")
b)
stream("source-path")
c)
startStream("source-path")
d)
save("source-path")
e)
withWatermark("source-path")
68.
In the same batch ETL example, which call writes the result to Parquet at the destination path?
a)
save("dest-path")
b)
startStream("dest-path")
c)
stream("dest-path")
d)
load("dest-path")
e)
foreach("dest-path")
69.
In the streaming ETL DataFrame example, what replaces load() when reading input?
a)
stream()
b)
save()
c)
persist()
d)
countByValue()
e)
window()
70.
In the streaming ETL DataFrame example, what replaces save() when writing output and starting processing?
a)
startStream()
b)
load()
c)
flatMap()
d)
reduceByKey()
e)
withWatermark()
71.
According to the slide, which statement about read...stream() is correct?
a)
It creates a streaming DataFrame but does not start the computation
b)
It immediately starts processing and writes output to the sink
c)
It converts the stream into a cached RDD and forces persistence
d)
It finalizes all event-time windows by advancing the watermark
e)
It writes the full result table every trigger
72.
According to the slide, what does write...startStream() do in the streaming ETL example?
a)
It defines where and how to output the data and starts the processing
b)
It only defines the schema of the streaming DataFrame
c)
It only registers the stream as a temporary view
d)
It only sets the batch duration for discretized processing
e)
It only configures replication of input batches in memory
73.
In the streaming ETL diagram, what conceptual model is used for the evolving result of the query?
a)
A result table that is updated as new input arrives
b)
A single immutable RDD that never changes
c)
A fixed bounded table that is rewritten only once
d)
A broadcast value that is overwritten on each executor
e)
A sink-only dataset that exists only after output
74.
In the diagram, which output behavior aligns with append mode semantics?
a)
Write only new rows in the result since the previous trigger
b)
Rewrite the full result table every trigger
c)
Rewrite only the schema when new data arrives
d)
Update previously written sink rows in place without any new rows
e)
Emit only watermark timestamps and drop all data
75.
A streaming query needs to continuously compute the average signal separately for each device-type. Which expression best matches this intent?
a)
input.avg('signal')
b)
input.groupBy('device-type').avg('signal')
c)
input.select('device-type', 'signal')
d)
input.groupBy('signal').avg('device-type')
e)
input.groupBy('device').avg('signal')
76.
In the slide's examples, which pair correctly matches the aggregation scope to the intended continuous result?
a)
input.avg('signal') yields a per device-type average
b)
input.groupBy('device-type').avg('signal') yields a single global average
c)
input.avg('signal') yields a single global average
d)
Both expressions yield identical grouping and output cardinality
e)
Neither expression computes an average over 'signal'
77.
The query groups by device-type and window(event-time-col, '10 min') then averages 'signal'. What is the result keyed by at each trigger?
a)
Only device-type, with a lifetime-to-date average
b)
Only a 10-minute window, aggregated across all device types
c)
(device-type, 10-minute event-time window), with an average signal
d)
(device, 10-minute processing-time window), with an average signal
e)
(device-type, session), with a median signal
78.
Which claim is explicitly made about this event-time windowed stream processing capability?
a)
It is possible in DStreams but not in Structured Streaming
b)
It simplifies event-time stream processing and is not possible in DStreams
c)
It requires separate code paths for streaming jobs and batch jobs
d)
It only works for batch jobs and cannot run continuously
e)
It requires joining with static data via JDBC
79.
In the join example, which data source is used as the streaming input and which is used as the static lookup?
a)
Kafka is static; JDBC is streaming
b)
Kafka is streaming; JDBC is static
c)
Both Kafka and JDBC are streaming sources
d)
Both Kafka and JDBC are static sources
e)
Neither Kafka nor JDBC is used; both datasets come from files
80.
The code joins kafkaDataset with staticDataset using 'device-type'. What is the purpose of this operation according to the slide?
a)
To convert Kafka offsets into HDFS files
b)
To enrich streaming data with static device information
c)
To compute the global average signal across all devices
d)
To define the trigger interval for streaming execution
e)
To make the static dataset replayable
81.
The slide distinguishes output modes by query type. Which pairing is stated?
a)
Append mode with aggregation queries; Complete mode with non-aggregation queries
b)
Append mode with non-aggregation queries; Complete mode with aggregation queries
c)
Append mode is required for all queries
d)
Complete mode is required for all queries
e)
Output modes are unrelated to query structure
82.
Output modes 'define what is outputted every time there is a trigger'. For an aggregation query written in complete mode, what is the most consistent interpretation?
a)
Only newly arrived input rows are written
b)
Only rows whose aggregate changed are written
c)
The entire aggregation result is output each trigger
d)
No output is produced until the query is stopped
e)
Only the schema is written to the destination path
83.
In the slide, 'query' is described as a handle to the running streaming computation. Which action is explicitly listed as supported via this handle?
a)
Repartition the source topic
b)
Stop it
c)
Rewrite the logical plan manually
d)
Change Kafka offsets without restarting
e)
Disable the Catalyst optimizer
84.
Which pair of calls is shown as query-level introspection for source and sink status?
a)
query.sourceStatuses() and query.sinkStatus()
b)
query.sources() and query.sinks()
c)
query.inputStatus() and query.outputStatus()
d)
query.offsets() and query.commits()
e)
query.catalog() and query.optimizer()
85.
The slide separates query execution into 'logically' and 'physically'. Which statement matches the 'logically' description?
a)
Spark runs continuous incremental execution plans
b)
Dataset operations on a table are as easy to understand as batch
c)
Offsets are written to a WAL in HDFS
d)
Sinks must be idempotent by design
e)
A planner polls Kafka for new data
86.
In the diagram, which component connects DataFrame -> Logical Plan -> Continuous, incremental execution?
a)
Project Tungsten
b)
Catalyst optimizer
c)
SparkContext
d)
JDBC driver
e)
StreamingContext
87.
Structured Streaming is described as a high-level streaming API built on which abstraction?
a)
DStreams only
b)
RDDs only
c)
Datasets/DataFrames
d)
HDFS files
e)
Kafka offsets
88.
Which capability is explicitly highlighted as part of Structured Streaming's feature set?
a)
Manual offset checkpointing in user code
b)
Event time, windowing, sessions, sources and sinks
c)
Batch-only SQL execution
d)
Non-replayable sources to improve latency
e)
Exactly-once is not supported end-to-end
89.
The slide states Structured Streaming 'unifies streaming, interactive and batch queries'. Which example on the slide aligns with that claim?
a)
Aggregate data in a stream, then serve using JDBC
b)
Only write Parquet files in append mode
c)
Convert DStreams into RDDs via manual polling
d)
Disable stateful operations to avoid checkpoints
e)
Replace DataFrames with raw sockets for input
90.
Which runtime operation is explicitly mentioned as supported for Structured Streaming queries?
a)
Add, remove, or change queries at runtime
b)
Change HDFS block size at runtime
c)
Rewrite the Spark SQL planner at runtime
d)
Change the JVM bytecode generator at runtime
e)
Disable fault tolerance at runtime
91.
A slide titled 'Internal execution' most directly signals a focus on which topic?
a)
Managing output directories
b)
How streaming queries execute under the hood
c)
Writing OAuth credentials to Twitter
d)
Selecting output modes for Parquet sinks
e)
Designing a problem statement for sentiment analysis
92.
In the illustrated Spark SQL batch pipeline, which stage comes immediately after 'Analysis'?
a)
Code Generation
b)
Logical Optimization
c)
Cost Model
d)
RDDs
e)
SQL AST
93.
According to the diagram, where is a 'Cost Model' used?
a)
To transform SQL AST into DataFrame
b)
To pick a Selected Physical Plan from Physical Plans
c)
To convert a DataFrame into an Unresolved Logical Plan
d)
To create a Catalog entry for a Dataset
e)
To compute offsets for incremental execution
94.
The slide labels 'Project Tungsten - Phase 1 and 2' and separates optimizations into two categories. Which item is listed under Memory Optimizations?
a)
Bytecode generation
b)
JVM intrinsics, vectorization
c)
Operations on serialized data
d)
Offheap memory
e)
Logical planning
95.
In the top pipeline (DataFrame/Dataset -> Logical Plan -> Planner -> Execution Plan), what is produced by the Planner in this slide?
a)
SQL AST
b)
Unresolved Logical Plan
c)
Execution Plan
d)
Selected Physical Plan
e)
Catalog
96.
The slide states that the planner converts streaming logical plans into what?
a)
A single static execution plan that never changes
b)
A continuous series of incremental execution plans
c)
A set of JDBC queries executed once
d)
A write-ahead log stored in Kafka
e)
A set of batch jobs that ignore new data
97.
In the diagram with Incremental Execution Plan 1..4, each plan is described as processing what?
a)
The entire historical dataset
b)
Only static reference data
c)
The next chunk of streaming data
d)
Only the schema of the DataFrame
e)
Only the latest window boundary
98.
The slide shows Incremental Execution 1 consuming offsets [19-105] and producing count 87. What is the most consistent role of the offset range in this depiction?
a)
A JDBC connection string
b)
A write-ahead log file name
c)
The specific span of source data processed in that execution
d)
A Spark executor memory limit
e)
A batch job identifier unrelated to sources
99.
Which statement matches the depicted control flow: planner, sources, incremental executions, and sink?
a)
The sink polls the planner and pushes offsets to Kafka
b)
The planner polls for new data from sources and writes results to the sink
c)
The sources write results directly to the sink without any execution
d)
Incremental executions decide their own offsets without the planner
e)
Counts are written back to the source as new offsets
100.
The slide states that a running aggregate is maintained as in-memory state backed by what for fault tolerance?
a)
A WAL in the file system
b)
A JDBC table
c)
A static dataset join
d)
A catalog-only metadata store
e)
A non-deterministic sink
101.
In the diagram, state from Incremental Execution 1 (state: 87) is used in Incremental Execution 2. What does this illustrate?
a)
State data is generated and used across incremental executions
b)
Each execution recomputes aggregates from scratch without state
c)
Offsets are ignored when computing counts
d)
Counts are written as new offsets to Kafka
e)
HDFS is used only for batch jobs
102.
The slide titled 'Fault-tolerance' asserts a requirement for the system. What must be recoverable or replayable?
a)
Only the sink output files
b)
Only the Spark UI metrics
c)
All data and metadata in the system
d)
Only the user's application code
e)
Only the catalog entries for tables
103.
Given the slide's statement, which failure scenario is most directly addressed by making data and metadata replayable?
a)
Avoiding JDBC driver upgrades
b)
Re-executing after failures using previously seen inputs
c)
Preventing any need for windowing
d)
Ensuring that joins always become broadcast joins
e)
Removing the need for a planner
104.
According to the slide, the planner tracks offsets by writing what to a WAL in HDFS?
a)
The offset range of each execution
b)
The full input data payload
c)
The final Parquet schema
d)
The list of executor hostnames
e)
The JDBC credentials
105.
In the diagram annotation, offsets are written to the fault-tolerant WAL at what point relative to execution?
a)
After execution completes and the sink commits
b)
Before execution
c)
Only when a failure happens
d)
Only for static datasets
e)
Only in complete output mode
106.
The slide shows a 'Failed Planner' causing a 'Failed Execution'. Which component is indicated as failing the current execution?
a)
Sink
b)
Source
c)
Planner
d)
State store
e)
Catalog
107.
Given the slide's offset tracking approach, what is the key asset available even if the planner fails during the current execution?
a)
A cost model for physical plans
b)
An offset range recorded in the WAL
c)
A static JDBC snapshot of the stream
d)
A list of Spark SQL AST nodes
e)
A new device-type grouping key
108.
The slide states that the planner reads the log to recover from failures and then does what?
a)
Skips the failed offsets to reduce latency
b)
Re-executes the exact range of offsets
c)
Replaces event time with processing time
d)
Switches from Kafka to JDBC for streaming
e)
Disables stateful processing
109.
In the recovery diagram, the restarted planner uses 'Offsets read back from WAL'. What is the intended outcome shown at the bottom?
a)
Same executions regenerated from offsets
b)
New offsets invented to avoid replay
c)
A new schema generated for each restart
d)
A non-deterministic sink to merge duplicates
e)
A switch to batch-only execution
110.
The slide lists examples of replayable structured streaming sources. Which option is included in that list?
a)
Kafka
b)
Email inbox
c)
Web browser cache
d)
GPU memory
e)
Catalog
111.
Fault-tolerant sources are said to 'generate the exactly same data' given what?
a)
A different random seed each run
b)
Offsets recovered by the planner
c)
A new cost model
d)
A changed device-type taxonomy
e)
A JDBC transaction id
112.
Intermediate 'state data' is described as maintained in what structure on Spark workers?
a)
Immutable SQL AST nodes
b)
Versioned, keyvalue maps
c)
JDBC tables only
d)
Kafka partitions only
e)
Parquet schemas only
113.
The slide states that the planner ensures the 'correct version' of state is used for what purpose?
a)
To serve results using JDBC
b)
To re-execute after failure
c)
To disable windowing
d)
To force append output mode
e)
To convert batch jobs into streaming jobs
114.
The slide states that sinks are by design idempotent (deterministic). What problem is this meant to handle?
a)
Reducing the number of Kafka partitions
b)
Avoiding double committing the output during re-executions
c)
Eliminating the need for offsets
d)
Preventing joins with static data
e)
Avoiding any use of HDFS
115.
Which sink characteristic is explicitly paired with 'idempotent' on the slide?
a)
Non-deterministic
b)
Deterministic
c)
Session-based
d)
Windowless
e)
Unresolved
116.
The slide summarizes end-to-end exactly-once guarantees as a sum of three components. Which set matches the slide?
a)
Offset tracking in WAL + state management + fault-tolerant sources and sinks
b)
SQL AST + code generation + catalog
c)
Windowing + sessions + JDBC serving
d)
Append mode + complete mode + parquet format
e)
SparkConf + SparkContext + StreamingContext
117.
According to the slide's equation, which missing component would directly undermine end-to-end exactly-once guarantees?
a)
Output directory screenshots
b)
Fault-tolerant sources and sinks
c)
The use case URL
d)
Twitter token authorization
e)
Importing packages
118.
The slide describes Structured Streaming as 'Fast, fault-tolerant, exactly-once stateful stream processing'. Which term is NOT part of that phrase?
a)
Fast
b)
Fault-tolerant
c)
Exactly-once
d)
Stateful
e)
Stateless
119.
The same slide claims this capability is delivered 'without having to reason about streaming'. Which interpretation best matches that wording?
a)
Users must manually write WAL and offset recovery logic
b)
Users can express computations as table-like operations rather than low-level streaming mechanics
c)
Users must implement custom replayable sources
d)
Users must write their own deterministic sink commits
e)
Users must avoid all stateful operations
120.
A slide lists a URL containing several keywords. Which combination of topics is explicitly present in that URL text?
a)
Real-time analysis, popular Uber locations, Spark Structured Streaming, machine learning, Kafka, MapR-DB
b)
Batch processing, airline schedules, Cassandra, REST APIs
c)
Graph analytics, PageRank, HBase, Flink
d)
Image classification, CNN training, GPUs, TensorRT
e)
Log parsing, regex extraction, CSV formatting, Excel
121.
The slide distinguishes 'Sentiment' from 'Sentiment Analysis'. Which definition matches 'Sentiment' on the slide?
a)
Categorising tweets related to a particular topic
b)
The emotion behind a social media mention online
c)
A write-ahead log stored in HDFS
d)
A 10-minute event-time window
e)
A deterministic sink commit protocol
122.
According to the slide, which pair of outcomes is tied to using trending topics and sentiment analytics?
a)
Create campaigns and attract a larger audience
b)
Rewrite Spark SQL physical plans
c)
Crisis management and service adjusting
d)
Replace JDBC with files
e)
Disable windowing and sessions
123.
The problem statement proposes designing a Twitter Sentiment Analysis System to populate real-time sentiments for which set of business functions?
a)
Code generation, logical optimization, physical planning
b)
Crisis management, service adjusting, target marketing
c)
Append mode, complete mode, parquet output
d)
Offsets, counts, incremental execution
e)
Device-type averages, windowed averages, joins
124.
The slide lists several uses of Sentiment Analysis. Which item is included?
a)
Decide whether to invest in a certain company
b)
Tune JVM intrinsics and vectorization
c)
Recover offsets from a WAL
d)
Generate the exactly same data given offsets
e)
Join Kafka with JDBC by device-type
125.
The importing slide includes a grouped import 'org.apache.spark.streaming.{Seconds, StreamingContext}'. What does this pairing most directly enable in the example program?
a)
Project Tungsten memory optimizations
b)
A streaming context configured with a time interval in seconds
c)
A JDBC join with static device info
d)
A cost model for physical plan selection
e)
A Parquet sink in complete mode
126.
Which imported type on the slide most directly suggests configuring persistence levels for Spark data?
a)
org.apache.spark.sql
b)
org.apache.spark.storage.StorageLevel
c)
scala.io.Source
d)
java.io.File
e)
org.apache.spark.SparkConf
127.
In the token authorization code, what is the minimum number of command-line arguments required before the program proceeds?
a)
0
b)
2
c)
4
d)
5
e)
8
128.
The usage string shown on the slide has four required tokens plus optional filters. Which set corresponds to the required tokens?
a)
consumer key, consumer secret, access token, access token secret
b)
device-type, event-time-col, signal, window
c)
offset start, offset end, count, sink
d)
SQL AST, logical plan, physical plan, RDDs
e)
sourceStatuses, sinkStatus, exception, stop
129.
The slide sets several system properties with keys like 'twitter4j.oauth.consumerKey'. What is the purpose of these properties in the shown code?
a)
Configure output mode and Parquet format
b)
Provide OAuth credentials for the Twitter4j library
c)
Store offsets in a WAL
d)
Define a 10-minute event-time window
e)
Enable Project Tungsten offheap memory
130.
In the shown transformation, tags are built using stream.flatMap over status.getHashtagEntities. What is extracted from each hashtag entity?
a)
The entire tweet text
b)
The hashtag text
c)
The sentiment label
d)
The Kafka offset
e)
The JDBC row id
131.
In the RDD-based snippet, tags.countByValue().foreachRDD(...) is followed by sortBy and map(x => (x, now)). What is the role of 'now' in the mapped output?
a)
It replaces the hashtag text
b)
It is appended as a timestamp alongside the counted tag
c)
It becomes the Kafka offset range
d)
It is used as a StorageLevel
e)
It is the JDBC table name
132.
The DStream-based snippet constructs tags from t.getText by split and then filter(_.startsWith('#')). What does this logic aim to retain?
a)
Only tokens that begin with '#', interpreted as hashtags
b)
Only tokens that begin with 'http', interpreted as URLs
c)
Only tokens that begin with '@', interpreted as usernames
d)
Only tokens equal to 'device-type'
e)
Only tokens equal to 'event-time-col'
133.
In the sentiment extraction mapping, what three elements are combined into each output record?
a)
(consumerKey, consumerSecret, accessTokenSecret)
b)
(offsetStart, offsetEnd, count)
c)
(tweet text, sentiment as string, hashtags as string)
d)
(logical plan, physical plan, selected plan)
e)
(device-type, window, average signal)
134.
The slide shows saving output using data.saveAsTextFiles('~/twitterss','20000'). Which detail is explicitly present?
a)
The files are written to a path starting with ~/twitterss
b)
The sink is parquet in complete mode
c)
Offsets are written to a WAL in HDFS
d)
The window length is 10 minutes
e)
The join key is device-type
135.
The results screenshot highlights three sentiment labels using color-coded tags. Which set matches the legend shown?
a)
Positive, Neutral, Negative
b)
Good, Average, Bad
c)
High, Medium, Low
d)
Pass, Fail, Unknown
e)
Buy, Hold, Sell
136.
The figure caption on the slide identifies where the sentiment analysis output is shown. What environment is named?
a)
Jupyter Notebook
b)
Eclipse IDE
c)
Command Prompt
d)
Kafka UI
e)
HDFS NameNode Web UI
137.
The output directory slide labels two distinct kinds of folders within the 'twitter' project folder. Which description matches one of those labels?
a)
Output folder containing Twitter profiles for analysis
b)
Output folder containing Spark SQL logical plans
c)
Output folder containing WAL offset ranges
d)
Output folder containing JDBC connection pools
e)
Output folder containing DataFrame schemas only
138.
On the same slide, a highlighted output folder is annotated as containing what?
a)
Offsets and execution plans
b)
Tags and their sentiments
c)
Device-type averages
d)
OAuth keys and secrets
e)
Catalog entries and cost models
139.
The output usernames slide shows lines like ((OneBestsFans,2),2017-02-09T10:22:27.347+05:30). What does the outer structure most closely represent?
a)
A (username,count) pair associated with a timestamp
b)
A (tweet text,sentiment) pair associated with a timestamp
c)
A (offset range,count) pair associated with a sink path
d)
A (device-type,average) pair associated with a window
e)
A (consumerKey,secret) pair associated with a filter
140.
The slide caption describes the file contents. Which combination is explicitly stated?
a)
Twitter username and timestamp
b)
Hashtag entities and offsets
c)
Window boundaries and averages
d)
Planner state versions and WAL paths
e)
JVM intrinsics and bytecode
141.
The output tweets and sentiments slide shows an output file where each line ends with a sentiment label. What is the file described as containing?
a)
Tweet and its sentiment
b)
Username and its timestamp
c)
Offset range and its count
d)
Device-type and its average signal
e)
SQL AST and its optimized plan
142.
In the screenshot, sentiment labels (e.g., NEUTRAL, POSITIVE, NEGATIVE) appear inline with each tweet. What does the adjacent legend reinforce?
a)
There are exactly three sentiment categories
b)
There are exactly five sentiment categories
c)
Sentiment is computed as a numeric score only
d)
Sentiment is stored only in a JDBC table
e)
Sentiment is available only after query termination
143.
The slide highlights a 'Trump' keyword in the filtering logic. What is the implied purpose of that keyword selection?
a)
Restrict processing to tweets that match the chosen topic keyword
b)
Switch from event-time to processing-time windows
c)
Force complete output mode for aggregations
d)
Write offsets to a WAL in HDFS
e)
Join Kafka with a JDBC static dataset
144.
The figure caption states 'Performing Sentiment Analysis on Tweets with 'Trump' Keyword'. Which outcome is illustrated in the lower console output?
a)
Tweets labeled as POSITIVE, NEUTRAL, or NEGATIVE
b)
Kafka offsets converted into RDDs
c)
A cost model selecting a physical plan
d)
A device-type windowed average signal
e)
A list of imported packages
145.
The slide states that sentiment analytics can be used in several business contexts. Which set matches the slide text?
a)
Crisis management, service adjusting, target marketing
b)
SQL AST analysis, logical optimization, code generation
c)
Offset tracking, state recovery, sink idempotence
d)
Windowing, sessions, device-type grouping
e)
Bytecode generation, vectorization, offheap memory
146.
According to the slide, companies using Spark Streaming for sentiment analysis apply the approach to achieve which outcome?
a)
Enhancing the customer experience
b)
Eliminating the need for any sources and sinks
c)
Replacing SparkConf with JDBC
d)
Avoiding stateful stream processing
e)
Preventing any re-executions after failure
100 %
