ANNS-09

The momentum method ...

• smooths the gradient over time

• converges to global optima

• overcomes smaller local optima

• can be controlled by a value of [-1,1]

Adaptive optimization methods

• require less learning rate tuning

• converge to global optima

• are superior to sgd for all tasks

• separate learning rates for each and every parameter

Autoencoders are trained in a [] manner.

• unsupervised

• supervised

• semi-supervised

• contrastive

Batch normalization standardize the [] to have 0 mean and 1 standard deviation.

• input

• output

• activation of hidden layers

• first layer output

Highway network uses [] to aggregate branches.

• gating

• addition

• linear projection

• recurrent neurons

Residual network uses [] to aggregate branches.

• gating

• addition

• linear projection

• recurrent neurons