Structuring Machine Learning Projects

This example is adapted from a real production application, but with details disguised to protect confidentiality.

You are a famous researcher in the City of Peacetopia. The people of Peacetopia have a common characteristic: they are afraid of birds. To save them, you have to build an algorithm that will detect any bird flying over Peacetopia and alert the population.

The City Council gives you a dataset of 10,000,000 images of the sky above Peacetopia, taken from the city’s security cameras. They are labeled:

• y = 0: There is no bird on the image

• y = 1: There is a bird on the image

• Your goal is to build an algorithm able to classify new images taken by security cameras from Peacetopia.

  There are a lot of decisions to make:

  • What is the evaluation metric?

  • How do you structure your data into train/dev/test sets?

  Metric of success

  The City Council tells you the following that they want an algorithm that

  1. Has high accuracy.

  2. Runs quickly and takes only a short time to classify a new image.

  3. Can fit in a small amount of memory, so that it can run in a small processor that the city will attach to many different security cameras.

  You meet with them and ask for just one evaluation metric. True/False?

• True

• False

The city revises its criteria to:

• "We need an algorithm that can let us know a bird is flying over Peacetopia as accurately as possible."

• "We want the trained model to take no more than 10 sec to classify a new image.”

• “We want the model to fit in 10MB of memory.”

Given models with different accuracies, runtimes, and memory sizes, how would you choose one?

• Create one metric by combining the three metrics and choose the best performing model

• Find the subset of models that meet the runtime and memory criteria. Then, choose the highest accuracy

• Take the model with the smallest runtime because that will provide the most overhead to increase accuracy

• Accuracy is an optimizing metric, therefore the most accurate model is the best choice

Based on the city’s requests, which of the following would you say is true?

• Accuracy is an optimizing metric; running time and memory size are satisfying metrics

• Accuracy, running time and memory size are all optimizing metrics because you want to do well on all three

• Accuracy, running time and memory size are all satisfying metrics because you have to do sufficiently well on all three for your system to be acceptable

• Accuracy is a satisfying metric; running time and memory size are an optimizing metric

Structuring your data

Before implementing your algorithm, you need to split your data into train/dev/test sets. Which of these do you think is the best choice?

• Train: 3,333,334

  Dev: 3,333,334

  Test 3,333,334

• Train: 6,000,000

  Dev: 1,000,000

  Test: 3,000,000

• Train: 6,000,000

  Dev: 3,000,000

  Test: 1,000,000

• Train: 9,500,000

  Dev: 250,000

  Test: 250,000

After setting up your train/dev/test sets, the City Council comes across another 1,000,000 images, called the “citizens’ data”. Apparently the citizens of Peacetopia are so scared of birds that they volunteered to take pictures of the sky and label them, thus contributing these additional 1,000,000 images. These images are different from the distribution of images the City Council had originally given you, but you think it could help your algorithm.

Notice that adding this additional data to the training set will make the distribution of the training set different from the distributions of the dev and test sets.

Is the following statement true or false?

"You should not add the citizens' data to the training set, because if the training distribution is different from the dev and test sets, then this will not allow the model to perform well on the test set."

• True

• False

Answer explanation

Sometimes we'll need to train the model on the data that is available, and its distribution may not be the same as the data that will occur in production. Also, adding training data that differs from the dev set may still help the model improve performance on the dev set. What matters is that the dev and test set have the same distribution.

One member of the City Council knows a little about machine learning, and thinks you should add the 1,000,000 citizens’ data images to the test set. You object because:

• This would cause the dev and test set distributions to become different. This is a bad idea because you're not aiming where you want to hit

• The test set no longer reflects the distribution of data (security cameras) you most care about

• The 1,000,000 citizens' data images do not have a consistent x-->y mapping as the rest of the data

• A bigger test set will slow down the speed of iterating because of the computational expense of evaluating models on the test set

If your goal is to have “human-level performance” be a proxy (or estimate) for Bayes error, how would you define “human-level performance”?

• The performance of the head of the City Council

• The best performance of a specialist (ornithologist) or possibly a group of specialists

• The performance of the average citizen of Peacetopia

• The performance of their volunteer amateur ornithologists

Which of the following statements do you agree with?

• A learning algorithm's performance can be better than human-level performance but it can never be better than Bayes error.

• A learning algorithm's performance can be better than human-level performance and better than Bayes error

• A learning algorithm's performance can never be better than human-level performance but it can be better than Bayes error

• A learning algorithm's performance can never be better than human-level performance nor better than Bayes error

You find that a team of ornithologists debating and discussing an image gets an even better 0.1% performance, so you define that as “human-level performance.” After working further on your algorithm, you end up with the following:

Based on the evidence you have, which two of the following four options seem the most promising to try? (Check two options.)

• Train a bigger model to try to do better on the training set

• Try increasing regularization

• Get a bigger training set to reduce variance

• Try decreasing regularization

You’ve now also run your model on the test set and find that it is a 7.0% error compared to a 2.1% error for the dev set. What should you do? (Choose all that apply)

• Try increasing regularization to reduce overfitting to the dev set

• Increase the size of the dev set

• Try decreasing regularization for better generalization with the dev set

• Get a bigger test set to increase its accuracy