📉 1. Multi-Class SVM Loss

• The Multi-Class SVM Loss measures how badly the model got things wrong:

  Loss = max(0, score_j - score_correct + 1)
  

• If the prediction is already correct enough (inside the margin), the loss stays at zero.

• The minimum possible loss is 0 — perfect classification.

• If all class scores are roughly equal, the loss will be (number of classes - 1) — a useful debug check.

• Using the mean instead of sum doesn’t change the behavior; it just scales it.

🗂️ 2. Choosing a Loss Function

• How you choose a loss function defines what types of errors you care about.
• For example, you can add a squared term to SVM Loss to make large mistakes hurt more or less — the curve shape changes how “wrong” gets penalized.

🗑️ 3. Zero Loss & Weights

• If the loss is zero, increasing weights W doesn’t change anything — the model already classifies that example correctly.

⚖️ 4. Regularization

• Regularization keeps the model simple to prevent overfitting.
• Common techniques:
  • L1 Norm: Encourages sparsity.
  • L2 Norm: Penalizes large weights smoothly.
  • Max Norm: Caps the maximum norm of the weights.
• Add a lambda λ hyperparameter as a penalty term to balance fit and simplicity.