Making the neural network from scratch

Building a neural network from scratch requires a good understanding of the underlying principles and mathematics involved in neural networks. Here's a step-by-step guide to building a simple neural network from scratch using Python:

1. Import necessary libraries: We will need NumPy to perform matrix operations and generate random weights.

import numpy as np

1. Define the activation function: We will use the sigmoid function as our activation function, which maps any input value to a value between 0 and 1.

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

1. Define the derivative of the activation function: We will also need to compute the derivative of the sigmoid function during backpropagation.

def sigmoid_derivative(x):
    return x * (1 - x)

1. Define the neural network architecture: We will create a simple neural network with one hidden layer containing three neurons, and an output layer with one neuron.

class NeuralNetwork:
    def __init__(self, input_dim, hidden_dim, output_dim):
        self.input_dim = input_dim
        self.hidden_dim = hidden_dim
        self.output_dim = output_dim

        # Initialize weights randomly
        self.weights1 = np.random.randn(self.input_dim, self.hidden_dim)
        self.weights2 = np.random.randn(self.hidden_dim, self.output_dim)

1. Define the forward propagation function: The forward propagation function takes in the input data and computes the output of the neural network.

def forward(self, X):
    # Calculate output of the first layer
    self.hidden_layer = sigmoid(np.dot(X, self.weights1))

    # Calculate output of the second layer (output layer)
    output = sigmoid(np.dot(self.hidden_layer, self.weights2))

    return output

1. Define the backpropagation function: The backpropagation function computes the gradients of the weights with respect to the loss function, which is used to update the weights.

def backward(self, X, y, output):
    # Compute the error of the output layer
    output_error = y - output

    # Compute the derivative of the output layer
    output_delta = output_error * sigmoid_derivative(output)

    # Compute the error of the hidden layer
    hidden_error = np.dot(output_delta, self.weights2.T)

    # Compute the derivative of the hidden layer
    hidden_delta = hidden_error * sigmoid_derivative(self.hidden_layer)

    # Update the weights
    self.weights2 += np.dot(self.hidden_layer.T, output_delta)
    self.weights1 += np.dot(X.T, hidden_delta)

1. Define the training function: The training function iteratively computes the output of the neural network, computes the gradients using backpropagation, and updates the weights.

def train(self, X, y, epochs):
    for epoch in range(epochs):
        # Forward propagation
        output = self.forward(X)

        # Backpropagation
        self.backward(X, y, output)

1. Test the neural network: To test the neural network, we can generate some test data and input it into the neural network.

# Initialize the neural network
nn = NeuralNetwork(input_dim=2, hidden_dim=3, output_dim=1)

# Generate some test data
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([[0], [1], [1], [0]])

# Train the neural network
nn.train(X, y, epochs=10000)

# Make predictions on the test data
output = nn.forward(X)
print(output)