1. Origin of Logistic Regression

• Logistic Regression actually originated from Naive Bayes, by converting its terms into probabilities of 1 and -1.

• Then, we take the logarithm on both sides, which transforms the formulation into a linear form.

• This leads to the condition:

  (w^t)x+b>0

which represents a linear equation.

2. Naive Bayes Intuition vs Logistic Regression

• Naive Bayes works by finding the best hyperplane to separate probability distributions, not directly positive vs negative points.
• In contrast, Logistic Regression explicitly models the probability of a class (0 or 1) using the sigmoid function.

3. MAP vs MLE in Logistic Regression

• Maximum Likelihood Estimation (MLE):

  • Estimates parameters by maximizing the likelihood that the observed data came from the model.
  • In logistic regression, this means finding weights \( w \) that maximize the probability of correct classification.

• Maximum A Posteriori (MAP):

  • Extends MLE by including a prior distribution over parameters.
  • This allows regularization and prevents overfitting.
  • MAP estimation finds:

  P(w∣D)= (P(D∣w)P(w))/P(D)

  where \( D \) is the dataset.

4. Difference Between Gaussian Naive Bayes and Logistic Regression

• Gaussian Naive Bayes (GNB):
  • Assumes each feature follows a Gaussian distribution.
  • Uses Bayes' theorem to compute class probabilities.
  • Decision boundaries are influenced by distributional assumptions.
• Logistic Regression (LR):
  • Does not assume feature distributions.
  • Directly models the probability of class membership using the sigmoid function.
  • Decision boundary is linear in the feature space.

Demo Sketch Idea (to visualize in notes or on paper)

• Gaussian Naive Bayes: Show two bell-curve distributions for positive and negative classes, with a decision threshold in between.