IST2334 - Lecture 3

What is web crawling?

• Extracting data from a website

• Indexing web pages for search engines

• Creating a new website

• Sending emails to website owners

Which of the following ethical considerations should be kept in mind while web scraping?

• Scraping any data without permission is always ethical.

• It's not necessary to give credit to the source website.

• Scraping personal and sensitive data is acceptable.

• Respect website terms of service and robots.txt.

What is the purpose of robots.txt file on a website?

• To collect analytics data

• To tell search engine crawlers which URLs the crawler can access on your site.

• To display a custom 404 error page

• To improve website security

What is the primary difference between web crawling and web scraping?

• Web crawling is used to extract structured data, while web scraping is used to follow links.

• Web crawling is focused on indexing, while web scraping is focused on extracting data.

• Web crawling requires human interaction, while web scraping is automated.

• Web crawling and web scraping are synonymous terms.

Which of the following is NOT a type of web crawler?

• Focused Web Crawler

• Parallel Web Crawler

• Incremental Web Crawler

• Convolutional Web Crawler

Select the three principles followed by all web scraping bots when scraping web data.

• Making an HTTP request to a server

• Add URL links to frontier

• Extracting and parsing the website’s code

• Saving the relevant data locally

What is the term for automatically following links from one web page to another during web crawling?

• Link forwarding

• Link traversal

• Link indexing

• Link extraction

Which of the following programming language is a commonly used for web scraping? (Select 2)

• HTML

• Python

• CSS

• R