Azure Data Factory for Beginners - Build Data Ingestion - Fixing the Duplicating Batch Ingestions

The video tutorial addresses issues in an orchestration pipeline, specifically duplicate entries in the pipeline log. It guides viewers through querying data in Azure Data Studio to confirm the problem, modifying a stored procedure to include a dataset filter, and adjusting parameters in the Azure Data Factory ingestion pipeline. The tutorial concludes with testing the fixes and emphasizes the importance of problem-solving in framework development.

What issue was identified in the pipeline log?

• Incorrect data format

• Unauthorized access

• Duplicate entries

• Missing entries

What was the main problem with the stored procedure?

• It was missing a dataset parameter for filtering

• It was not saving data correctly

• It was running too slowly

• It was not connecting to the database

What type of parameter was added to the stored procedure to fix the issue?

• Boolean

• Varchar

• Date

• Integer

What was the first step in fixing the ingestion pipeline?

• Adding a new dataset

• Deleting the entire pipeline

• Cleaning out the pipeline log

• Creating a new stored procedure

Which activity in Azure Data Factory needed updating to fix the pipeline?

• Copy activity

• Lookup activity

• Delete activity

• Filter activity

What was the final step after updating the pipeline parameters?

• Creating a new dataset

• Publishing the changes

• Deleting old logs

• Running a full system backup

What is emphasized as important when building a framework from scratch?

• Relying on default settings

• Avoiding any changes

• Ignoring errors

• Troubleshooting and fixing issues