SVM

Maximize Margin

Hinge loss (SVM loss)

1. Definition: The correct score should be larger than a margin (1, and the 1 here actually can be arbitrary) than wrong scores

2. Min 0, Max infinity

3. At the initial time, every class scores should be similar and the expected loss is C-1 ( each wrong class has loss 1, and there are C - 1 wrong classes)

4. Squared hinge loss is different from hinge loss, squared loss focusing more on the bad cases, a small difference may have huge loss value at the end.

SVM Loss vs Cross Entropy Loss

Differences

Kernels

Gaussian Kernels

Need to do feature normalization before using multivariate SVM

Other Kernels