Model Ensemble

1. Train ten models and ensemble them.

2. Keep good snapshots and ensemble them

3. Fancier version of snapshot ensemble: crazy high learning rate scheduler.

https://arxiv.org/pdf/1608.03983.pdf

https://arxiv.org/pdf/1704.00109.pdf

4. Moving average of the parameter vector.

http://www.meyn.ece.ufl.edu/archive/spm_files/Courses/ECE555-2011/555media/poljud92.pdf