> For the complete documentation index, see [llms.txt](https://sisyphus.gitbook.io/project/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://sisyphus.gitbook.io/project/deep-learning-basics/basics/regularizations.md).

# Regularizations

## **Common Regularization:**

Overall, we want a simpler model to get rid of overfitting.

1. **L2 regularization:** SUMk SUMl ( Wk l^ 2 )  
   * It works in this following way: by using L2 regularization, W \[0.25, 0.25, 0.25, 0.25] is more preferred than W \[1, 0, 0, 0], so that the **decision made would be counting on all 4 input features**, and the final guess will have looked into more features rather than only one.
   * L2 regularization also corresponds MAP inference using a Gaussian prior on W
2. **L1 regularization:** SUMk SUMl ( | Wkl | ). L1 will force the model to be more sparse
   * In the other way, L1 regularization kind of have the opposite L2 interpretation. And we would prefer W \[1, 0, 0, 0] more than W \[1, 1, 1, 1].
3. **Elastic net** ( L1 + L2 ) : SUMk SUMl ( beta \* Wkl^2 + | Wkl |)
4. **Dropout:** set random activations zero (for FC layer), or random channels to zero for Convolution layers.

![](https://443921002-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LGHUhl6VYqrZm4Re77O%2F-LUimnxqyEHs3-GWCeO9%2F-LUimsQ-WeYTmZed7PYL%2FScreen%20Shot%202018-12-27%20at%2012.32.15%20AM.png?alt=media\&token=c5fc87fe-eb7c-4c74-a06f-7b5839f20e69)

![](https://443921002-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LGHUhl6VYqrZm4Re77O%2F-LGHXVMfoEBVK5KRs-fG%2F-LGHYjueOMD69mx9rZR2%2FScreen%20Shot%202018-06-26%20at%204.04.10%20PM.png?alt=media\&token=d22cdd18-e3ce-44ac-8d56-a7dbbeb5f214)