Font size
WorksheetsHadoop-the-Definitive-Guide-3rd-Edition-Chapter-1 - Meet Hadoop
Total questions: 75
Worksheet time: 38mins
According to Grace Hopper, what should we focus on instead of trying to build bigger computers?
Building more systems of computers
Building faster networks
Increasing storage capacity
Improving user interfaces
What is the estimated size of the "digital universe" in 2006 according to IDC?
0.18 zettabytes
1.8 zettabytes
10 terabytes
2.5 petabytes
Which of the following best describes a zettabyte?
One thousand exabytes, one million petabytes, or one billion terabytes
One thousand gigabytes, one million megabytes, or one billion kilobytes
One thousand terabytes, one million gigabytes, or one billion megabytes
One thousand petabytes, one million terabytes, or one billion gigabytes
Which source generates about one terabyte of new trade data per day?
The New York Stock Exchange
Ancestry.com
The Internet Archive
Which of the following statements is true about the Internet Archive?
It stores around 2 petabytes of data and is growing at a rate of 20 terabytes per month.
It generates about one terabyte of new trade data per day.
It stores around 2.5 petabytes of data.
It produces about 15 petabytes of data per year.
If the digital universe was 0.18 zettabytes in 2006 and is forecasted to grow tenfold by 2011, what would its estimated size be in 2011?
1.8 zettabytes
0.36 zettabytes
18 zettabytes
2.5 zettabytes
Analyze the implications of the rapid growth in data volume for organizations and suggest one strategy they could use to manage this growth effectively.
Organizations must adopt scalable storage and processing solutions, such as distributed computing systems like Hadoop, to handle the increasing data volume.
Organizations should reduce the amount of data they collect to avoid storage issues.
Organizations can ignore data growth as technology will automatically adapt.
Organizations should only store data on local machines.
What was the purpose of Microsoft Research’s MyLifeBits project?
To develop new camera technology
To archive personal information electronically for later access
To create a new social media platform
To analyze retail transactions
Which of the following is NOT mentioned as a contributor to the growing mountain of data?
Machine logs
Vehicle GPS traces
Printed books
Sensor networks
What is the significance of initiatives like Public Data Sets on Amazon Web Services and Infochimps.org?
They restrict access to data for security reasons.
They foster the “information commons” by making data available for anyone to download and analyze.
They only provide data to large corporations.
They focus solely on financial data.
How does the Astrometry.net project utilize data in a way that was not anticipated by the creator?
By creating new photographic images
By analyzing tagged photographic images to identify celestial bodies
By selling images to the public
By deleting old images
What is one method described to speed up reading data from storage?
Using only one disk at a time
Reading from multiple disks in parallel
Compressing all data before reading
Deleting unnecessary files
What is a common way to avoid data loss when using many disks?
Using faster disks
Replication of data
Deleting old data
Encrypting all data
Which distributed file system is mentioned as taking a different approach to hardware failure?
NTFS
FAT32
Hadoop Distributed Filesystem (HDFS)
EXT4
What is the main challenge when combining data from multiple disks for analysis?
Data is always lost
Data must be combined correctly from multiple sources
Disks are too slow
Data cannot be replicated
How does MapReduce help with data analysis in distributed systems?
It increases disk speed
It abstracts the problem from disk reads and writes
It compresses all data
It deletes unnecessary files
What are the two core components of Hadoop's kernel?
HDFS and MapReduce
SQL and NoSQL
Spark and Hive
Pig and HBase
What is the main advantage of using MapReduce for data analysis?
It processes only a small portion of the dataset for each query
It allows for batch processing and ad hoc queries on large datasets
It is only suitable for real-time data processing
It requires no storage system
What is the impact of using MapReduce at Rackspace?
It reduced the need for data analysis
It allowed engineers to gain new insights and improve customer service
It made data analysis slower and more complex
It replaced the need for all other data storage systems
Why is MapReduce considered transformative in the way data is analyzed?
It only works with small datasets
It unlocks previously archived data and enables new insights
It eliminates the need for any computation
It is only used for email processing
Why might traditional RDBMS with lots of disks not be sufficient for large-scale batch analysis?
RDBMS cannot store any data
RDBMS are not designed for large-scale batch processing like MapReduce
RDBMS are faster than MapReduce for all tasks
RDBMS do not support any queries
What is the main difference between seek time and transfer rate in disk drives?
Seek time is the process of moving the disk’s head to a particular place, while transfer rate is the speed of data transfer.
Seek time is the speed of data transfer, while transfer rate is the process of moving the disk’s head.
Seek time and transfer rate are the same thing.
Seek time is only relevant for SSDs, while transfer rate is for HDDs.
Which data structure is commonly used in traditional relational databases for updating a small proportion of records?
B-Tree
Hash Table
Linked List
Trie
Which of the following is a key advantage of MapReduce over traditional RDBMS for data analysis?
It is better for batch processing of large datasets.
It provides higher integrity.
It is optimized for point queries.
It uses a static schema.
Refer to the comparison table between RDBMS and MapReduce. Which system is characterized by linear scaling?
MapReduce
Traditional RDBMS
Both
Neither
Which type of data does MapReduce handle particularly well?
Unstructured or semi-structured data
Only structured data
Only encrypted data
Only image data
A relational database (RDBMS) might be preferred over MapReduce for certain applications because (a) .
What is the main characteristic of structured data?
It is organized into entities with a defined format.
It has no particular internal structure.
It is always in the form of plain text.
It cannot be stored in XML documents.
Which of the following best describes the integrity of MapReduce compared to traditional RDBMS?
MapReduce has lower integrity than RDBMS.
MapReduce has higher integrity than RDBMS.
Both have the same level of integrity.
Integrity is not relevant to either system.
How does MapReduce interpret the input keys and values for processing data?
They are chosen by the person analyzing the data.
They are intrinsic properties of the data.
They are always defined by a static schema.
They are automatically generated by the system.
Which of the following best describes why normalization poses problems for MapReduce?
It makes reading a record a non-local operation.
It increases the speed of streaming reads and writes.
It simplifies the structure of web server logs.
It allows for easier analysis of logfiles.
What is a key characteristic of a web server log that makes it well-suited for analysis with MapReduce?
It is not normalized, so the same client may appear many times.
It is always stored in a relational database.
It contains only unique client hostnames.
It is normalized for high-speed access.
Which statement about the scalability of MapReduce is correct?
Doubling the size of the cluster and the input data will result in the job running as fast as the original one.
Doubling the input data will always make the job run faster.
MapReduce cannot handle large datasets efficiently.
SQL queries scale linearly like MapReduce jobs.
How do the map and reduce functions in MapReduce operate with respect to the size of the data?
They are oblivious to the size of the data and can be used unchanged for any dataset size.
They must be rewritten for each new dataset size.
They only work for small datasets.
They require manual adjustment for large clusters.
Which of the following is a challenge faced by High Performance Computing (HPC) when processing large datasets?
Network bandwidth becomes the bottleneck as data size increases.
Compute nodes are always fully utilized.
HPC cannot access shared filesystems.
HPC is not suitable for compute-intensive jobs.
Over time, what trend is observed between relational databases and MapReduce systems?
The differences between them are likely to blur as they adopt each other's features.
Relational databases are becoming less popular.
MapReduce is being replaced by SQL queries.
They are becoming more distinct and incompatible.
What is the main reason for the good performance of MapReduce in a data center environment?
Data locality
High CPU usage
Complex network topology
Explicit data flow management
Which of the following best describes the architecture of MapReduce?
Shared-nothing architecture
Centralized architecture
Client-server architecture
Peer-to-peer architecture
In MapReduce, what is the primary way it handles process failure in a large-scale distributed computation?
By detecting failed map or reduce tasks and rescheduling replacements on healthy machines
By stopping the entire computation
By requiring manual intervention
By using a backup server for every process
Which of the following is a key difference between MapReduce and MPI from the programmer’s perspective?
MapReduce hides the mechanics of data flow, while MPI requires explicit handling
MapReduce requires explicit socket programming, while MPI does not
MPI is only used for database applications, while MapReduce is not
MapReduce cannot handle large-scale computations, while MPI can
Why was MapReduce originally invented by engineers at Google?
To build production search indexes by solving repetitive problems efficiently
To create a new programming language
To replace all database systems
To improve graphical user interfaces
How does MapReduce differ from MPI in terms of failure handling?
MapReduce automatically detects and recovers from failures, while MPI requires explicit check-pointing and recovery by the programmer
Both require manual intervention for failure recovery
MPI automatically handles failures, while MapReduce does not
Neither can handle failures in distributed systems
What is the primary purpose of the SETI@home project?
To analyze radio telescope data for signs of intelligent life outside earth
To search for large prime numbers
To understand protein folding and its relation to disease
To build machine learning libraries
Which of the following best describes a "work unit" in volunteer computing projects like SETI@home?
A chunk of data sent to computers for analysis
A type of computer used for volunteer computing
A software program that manages volunteers
A network protocol for data transfer
How does SETI@home differ from MapReduce in terms of computation and network requirements?
SETI@home uses untrusted machines with variable connections, while MapReduce uses trusted, dedicated hardware with high bandwidth
SETI@home requires high bandwidth, while MapReduce does not
MapReduce runs perpetual computations, while SETI@home runs short jobs
SETI@home is only used for machine learning, while MapReduce is not
Why is SETI@home suitable for running on hundreds of thousands of computers across the world?
Because the problem is very CPU-intensive and does not require high bandwidth
Because it requires a lot of memory
Because it needs high-speed internet connections
Because it only runs on dedicated servers
What is a precaution taken by SETI@home to combat cheating in volunteer computing?
Each work unit is sent to three different machines and at least two results must agree to be accepted
Volunteers are required to register with government ID
Data is encrypted before being sent
Only trusted computers are allowed to participate
Who created Hadoop?
Doug Cutting
Mike Cafarella
Howard Gobioff
Sanjay Ghemawat
What is the origin of the name "Hadoop"?
It is a made-up name given by Doug Cutting's child to a stuffed yellow elephant.
It is an acronym for "Highly Available Distributed Object Oriented Processing."
It is derived from the name of a famous mathematician.
It is a combination of the words "data" and "loop."
Which open source web search engine project did Hadoop originate from?
Apache Nutch
Lucene
Pig
MapReduce
What principle is followed in naming subprojects and modules in Hadoop?
Names are often unrelated to their function and may use animal themes.
Names must always be acronyms.
Names must be based on famous scientists.
Names must be related to data storage.
What was the main challenge faced by the creators of Nutch when scaling to billions of web pages?
Their architecture wouldn’t scale to the required size.
They lacked a programming language.
They had no access to the internet.
They could not find enough developers.
The Google File System (GFS) was important to the development of Hadoop because (a) .
The function of the "jobtracker" in Hadoop is to (a) .
What was the main motivation for developing the Nutch Distributed Filesystem (NDFS)?
To solve storage needs for very large files generated by web crawling and indexing.
To create a new programming language.
To replace the internet.
To develop a new search algorithm.
In what year did Google publish the paper that introduced MapReduce to the world?
2002
2004
2006
2008
Which of the following companies is NOT mentioned as a user of Hadoop by early 2008?
Last.fm
The New York Times
Describe how the New York Times used Amazon’s EC2 and Hadoop in a notable project.
To build a new search engine
To crunch through four terabytes of scanned archives and convert them to PDFs for the Web
To analyze social media data
To develop a new programming language
Match the following world records with the corresponding achievements related to Hadoop as of April 2008.
Fastest web search
Record for the quickest web search performance
Largest database
Record for the biggest database size
Fastest system to sort a terabyte of data
Record for sorting a terabyte of data in the shortest time
Most users on a cluster
Record for the highest number of users on a single cluster
How long did it take Hadoop to sort one terabyte of data on a 910-node cluster in April 2008?
62 seconds
209 seconds
297 seconds
601 seconds
Evaluate the impact of Hadoop’s easy-to-use parallel programming model on large-scale data processing, using the New York Times example as evidence.
It made data processing slower and more expensive
It allowed organizations to process large datasets quickly and cost-effectively, as shown by the NYT’s ability to process four terabytes in less than 24 hours
It was only useful for small datasets
It required specialized hardware
What was the main reason for redesigning the Dreadnaught infrastructure for the WebMap at Yahoo! in early 2005?
To improve search accuracy
To scale up to more nodes
To reduce memory usage
To enhance user interface
Match each programming language to the framework or system it was used to prototype or develop, based on the context of distributed computing frameworks.
Java
Used by Eric Baldeschwieler’s team to prototype the new framework modeled after GFS and MapReduce
Python
Popular language for prototyping and scripting in data science and machine learning frameworks
C++
Commonly used for high-performance distributed systems like Apache Hadoop’s native components
Ruby
Known for its use in web application frameworks such as Ruby on Rails
Why was it easier for Yahoo! to adopt Hadoop compared to their prototype?
Hadoop was already open source
Hadoop required fewer servers
Hadoop was faster to install
Hadoop had better documentation
According to the timeline, what significant event happened in February 2006 regarding Hadoop?
Doug Cutting joined Yahoo!
Nutch was ported to the new framework
Apache Hadoop project officially started to support the stand-alone development of MapReduce and HDFS
Initial versions of Hadoop Distributed Filesystem and MapReduce were implemented
How did the adoption of Hadoop benefit Yahoo! researchers compared to the previous prototype?
It allowed researchers to use a new framework sooner and leverage open source advantages
It reduced the number of nodes required for research
It improved the graphical interface for researchers
It eliminated the need for legal permissions
Analyze why the WebMap applications did not require extensive refactoring to fit into MapReduce.
Most of the WebMap phases were already based on sorting key-value pairs, which corresponded to MapReduce
WebMap was already using Hadoop
WebMap was designed for small-scale data
WebMap did not use distributed computing
Which of the following best describes the function of the 'Common' project in the Hadoop ecosystem?
A set of components and interfaces for distributed filesystems and general I/O
A distributed data processing model and execution environment
A serialization system for efficient, cross-language RPC
A database management system for Hadoop
What is the primary purpose of the Avro project in the Hadoop ecosystem?
To provide a distributed filesystem
To offer a serialization system for efficient, cross-language RPC, and persistent data storage
To manage cluster resources
To execute SQL queries on Hadoop
What is the main function of MapReduce in the Hadoop ecosystem?
To provide a user interface for Hadoop
To manage network security
To serve as a distributed data processing model and execution environment for large clusters
To store large files in a database
Which component of the Hadoop ecosystem is responsible for providing a distributed filesystem that runs on large clusters of commodity machines?
HDFS
Pig
Hive
ZooKeeper
What is the primary function of Pig in the Hadoop ecosystem?
It is a distributed filesystem.
It is a data flow language and execution environment for exploring large datasets.
It is a distributed, column-oriented database.
It is a coordination service for distributed applications.
Which Hadoop component provides a query language based on SQL for managing data stored in HDFS?
HBase
Hive
Sqoop
Pig
What is the main purpose of ZooKeeper in the Hadoop ecosystem?
To move data between relational databases and HDFS
To provide a distributed, highly available coordination service
To manage data stored in HDFS using SQL
To execute MapReduce jobs
Suppose you need to efficiently move data between a relational database and HDFS. Which Hadoop tool would you use?
Hive
Sqoop
HBase
Pig
