Data Fundamentals Post-Test

Which of the following is NOT a characteristic of unstructured data?

• It often requires advanced AI/ML techniques for processing.

• It is typically organized into tables with rows and columns.

• Examples include images, audio files, and plain text documents.

• It accounts for the majority of data generated in the world.

You are designing a data pipeline where the schema of your incoming data (e.g., user profiles) might change frequently, with new optional fields being added over time, but you need to ensure backward compatibility for existing consumers. Which data storage format would be most suitable for this scenario?

• CSV

• JSON

• Parquet

• Avro

A data engineer is building a system to generate daily sales reports by summing up individual transaction amounts from the previous day. Which data ingestion and processing approach is most appropriate for this task?

• Real-time streaming and data enrichment.

• Batch processing and data aggregation.

• Continuous ingestion and data cleaning.

• Manual data entry and normalization.

Why is viewing data as a valuable 'organizational asset' important for a junior data engineer?

• It helps them choose the cheapest data storage solutions.

• It enables them to avoid working with complex data types.

• It provides context for their technical work, highlighting how data quality and availability impact business decisions and competitive advantage.

• It ensures they only work with structured data.

Describe a real-world scenario where a company would likely combine all three data types (structured, semi-structured, and unstructured data). Explain briefly how each type might be used in that scenario.

• Scenario Example (E-commerce Company)

Compare and contrast Parquet and Avro data storage formats. Discuss at least two key differences in their design philosophies or primary use cases.

You are tasked with building a data pipeline for a company. Data arrives from an external API as JSON files, needs to be combined with customer data from a relational database, cleaned, and then loaded into a data warehouse for analytical reporting. Identify at least three distinct transformation steps you would consider applying in this pipeline.