Custom Logistic Regression with Implementation

Logistic regression is a statistical method for predicting a categorical outcome with two possible classes.

Logistic regression estimates the probability of an event occurring such as voting or not voting based on a given dataset of independent variables since the outcome is a probability. The independent variable is bounded between 0 and 1.

Line equation

• Ax + βx + C = 0

For more columns

• Ax + βx + C + d = 0

Perceptron Trick:

• This method chooses a random point and sets a line by transformation by looping.

How to find the positive and negative region.

• ax + by + c > 0 → Positive
• ax + by + c < 0 → Negative

Custom Logistic regression in python

import numpy as np

class CustomLogisticRegression:
    def __init__(self, learning_rate=0.01, num_iterations=1000):
        self.learning_rate = learning_rate
        self.num_iterations = num_iterations
        self.weights = None
        self.bias = None

    def sigmoid(self, z):
        return 1 / (1 + np.exp(-z))

    def initialize_weights(self, num_features):
        self.weights = np.zeros((num_features, 1))
        self.bias = 0

    def fit(self, X, y):
        m, n = X.shape
        self.initialize_weights(n)

        for _ in range(self.num_iterations):
            # Forward pass
            z = np.dot(X, self.weights) + self.bias
            predictions = self.sigmoid(z)

            # Compute gradients
            dz = predictions - y
            dw = np.dot(X.T, dz) / m
            db = np.sum(dz) / m

            # Update weights and bias
            self.weights -= self.learning_rate * dw
            self.bias -= self.learning_rate * db

    def predict(self, X):
        z = np.dot(X, self.weights) + self.bias
        predictions = self.sigmoid(z)
        return np.round(predictions)

# Example usage:
if __name__ == "__main__":
    # Generate some sample data
    np.random.seed(42)
    X = np.random.rand(100, 2)
    y = (X[:, 0] + X[:, 1] > 1).astype(int)

    # Reshape y to be a column vector
    y = y.reshape(-1, 1)

    # Initialize and train the model
    model = CustomLogisticRegression(learning_rate=0.01, num_iterations=1000)
    model.fit(X, y)

    # Make predictions
    predictions = model.predict(X)

    # Print accuracy
    accuracy = np.mean(predictions == y)
    print(f"Accuracy: {accuracy * 100:.2f}%")

• Learning Rate: It is the step size for moving the line in a specific direction. It moving towards a loss function’s minimum.
• num_iteration: How many times move the line in loop
• weights: weights is a parameter that the algorithm learns during the training. here we have initialized by zero after that it changes in iterations.
• bias: also called the intercept is a constant term that is independent of the input features. It is added to the weights sum of input features to shift the decision boundary.
• The sigmoid function is to convert numbers between 0 to 1.
• The fit function initializes the weights and uses the dot product to find the z and return the updated weights and bias.
• The predict function returns the predictions by using the last updated weights.