How MCP Neurons and Perceptrons Mirror Real Neurons

How do neurons work?

1. Dendrites: these are like thousands of tiny antennas, each receive signals from other neurons

2. Signals: these are electrochemical signals and one thing to note is each signal has it’s importance meaning one signal can be less important than other. Signals can be excitatory (encouraging the neuron to fire) or inhibitory (discouraging it)

3. Cell body/Soma: All the incoming signals gather up at the body and it decides whether to fire the signal or not. If the combines importance of the excitatory signal minus the inhibitory signal exceeds this threshold -> Boom the neuron fires.

4. Axon & Terminals: This is like a dedicated output cable, carrying the decision. Axon terminals are like broadcast stations, sending the signal to dendrites of other neurons

Neuron working analogy

Think of it like a bar with the world’s most mathematical bouncer. Let’s break down how this party (I mean, neuron ;) works:

1. Every person (signal) has a specific importance value (weight)
2. Normal people count as 1
3. VIPs count as 2 (they’re twice as important!)
4. Police/party-poopers count as -2 (they actually work against the party starting)
5. The party only starts when the total “party value” hits a certain threshold

Threshold for biological neurons

**You must be wondering who set’s biological neuron’s threshold? **

In biological neurons, the “threshold” is called Action Potential and it’s determined by the neuron’s properties and the environment.

The threshold can change based on chemicals in the brain, fatigue, or recent activity (for example, after firing a lot, a neuron might temporarily raise its threshold to avoid overloading).

So, in the brain, the threshold is a dynamic property, influenced by genetics, experience, and the neuron’s current state.

Activation function

An activation function is like a decision-maker for a neuron, like the Soma (cell body).

It takes total signal a neuron receives (after adding all of the weighted inputs) and determines what the output should be, which is usually transformed into something useful like 0 or 1, or a number between 0 and 1.

The MCP neuron uses Step Function i.e it returns binary output based on the input meeting the given a threshold $θ(theta)$ criteria.

$y$ being the output of the activation function and $u$ being input (total signal/weighted sum) or in other terms $u$ is the output of the weighted sum function.

This is also used in perceptron as we’ll discuss further. But the question is what can we do with it?

Implementing MCP Neuron

Let’s implement it in code. We’ll use Python to create a simple implementation of the MCP neuron.

class MCPNeuron:
    def __init__(self, weights, theta):
        self.weights = weights
        self.theta = theta
 
    def weighted_sum(self, inputs):
        return sum(i * w for i, w in zip(inputs, self.weights))
 
    def step_function(self, u):
        return int(u >= self.theta)  # returns 1 if threshold met, else 0
 
    def activate(self, inputs):
        u = self.weighted_sum(inputs)
        return self.step_function(u)

Implementing Logic Gates using MCP

Why are we suddenly talking about logic gates? Well, it turns out MCP neurons can do something pretty neat: they can simulate basic logical operations. This was one of the first practical applications that got researchers excited.

Just by choosing different weights and thresholds, we can make it perform logical operations. Let’s see how:

weights = [1,1]
threshold = 2
andGate = MCPNeuron(weights,threshold)
 
inputs = [1,1] # try out other inputs
print(andGate.activate(inputs))

weights = [1,1]
threshold = 1
orGate = MCPNeuron(weights,threshold)
 
inputs = [1,1] # try out other inputs
print(orGate.activate(inputs))

weights = [-1]
threshold = 0
notGate = MCPNeuron(weights,threshold)
 
inputs = [1] # try out other inputs
print(notGate.activate(inputs))

Linear Separability

Let’s visualize the whole process on a graph. Suppose we have two inputs: temperature and humidity and we need decide if we go outside (output 1) or stay in (output 0).

To recap, MCP neuron takes these inputs, gives them weights (how important they are), adds them up, and checks against a threshold.

This is the equation of a line! say $t$ as $x$ axis and $h$ as $y$ axis, this is what the graph looks like:

but wait we have an inequality to handle:

here is the graph of the inequality:

Now the weights (importance) of the inputs influences whether to go out or not.

say weight of $temperature$ is $0$ and weight of $humidity$ is $1$. This means we don’t care about temperature and only care about humidity, so the line will be parallel to $t$ axis and will pass through $h = 2$

Here is a visual of how weights influence the line:

Alright maybe I played too much there 💀 but you get the point. Go ahead play around with the weights and see how it changes the line: MCP Neuron plot

Limitations

Even though the MCP neuron was a breakthrough, it comes with some important limitations:

• No learning or adaptation: The weights and threshold (unlike biological neurons) are fixed and must be set manually, so the neuron can’t adjust or improve itself based on experience or new data.
• Works only with simple, linearly separable problems: It can only handle binary signals and can solve logic gates like AND, OR, and NOT, but fails for more complex cases like XOR that can’t be separated by a straight line.

This made Frank Rosenblatt think: “How about making the MCP neuron learn?”

How do biological neurons learn?

Neurons that fire together, wire together.

Neurons talk to each other through connections called synapses. When two neurons send signals (or “fire”) at the same time a lot, their connection gets stronger. When they don’t fire together often, their connection gets weaker.

While the main idea is about making connections stronger or weaker, a lot more complex stuff happens in the brain:

• Add more “receivers” (receptors) at a synapse to catch signals better.
• Change how many chemical messengers (neurotransmitters) are sent between neurons.
• Even grow new synapses or get rid of old ones

Deriving Perceptron from MCP Neuron (mathematically)

It’ll get a bit mathy here, but I promise it’ll be worth it. So we want our neuron to fulfill following conditions:

1. Adjust threshold based on learning (unlike MCP neuron, where threshold and weights are fixed)
2. Adjust weights based on learning

Let’s break this down step by step.

Iterations, Epochs, and Convergence

Training a perceptron is a bit like practicing a new skill: you rarely get everything perfect on your first try. When training, the perceptron goes through the entire dataset, making predictions and adjusting its weights whenever it makes a mistake.

But why do we need multiple epoch / passes through the data?

Remember learning rate? Well usually a small learning rate is used for a stable learning. This means that the perceptron will make small changes to its weights each time it makes a mistake at each example.

By going through the data multiple times, the perceptron can slowly and steadily improve, fixing mistakes it couldn’t fix in just one round.

So we keep doing homework sheets (epochs) until the student(perceptron) masters the material (convergence)!

Linear Separability

Just like the MCP neuron, the perceptron can only solve problems that are linearly separable.

Implementing Perceptron

Now, let’s implement a perceptron in Python.

import numpy as np
 
class Perceptron:
    def __init__(self, learning_rate=0.1, epochs=100):
      self.weights = None
      self.bias = None
      self.learning_rate = learning_rate
      self.epochs = epochs
 
    def activate(self, x):
      # Step function: output 1 if sum >= 0, else 0
      return 1 if x >= 0 else 0
    
    def weighted_sum(self, x):
      return np.dot(self.weights, x) + self.bias
 
    def predict(self, x):
      # Weighted sum of inputs + bias
      return self.activate(self.weighted_sum(x))    

So far it’s same as the MCP neuron with addition of bias and in activation function we are comparing the weighted sum with 0 instead of threshold.

Now we implement the training algorithm, from our understanding so far we need:

• Initialize weights and bias with some random values (preferably small)
• We need to iterate over the whole training data epochs times
• Now in each epoch we need to go through each example and check if the prediction is correct
• If prediction is incorrect, we update the weights and bias using the perceptron learning rule
• We also need to keep track of the number of errors in each epoch and stop if it’s 0

import numpy as np
 
class Perceptron:
    ...
    def train(self, x, y):
      training_data_length = x.shape[1]
      self.weights = np.random.randn(training_data_length) * 0.01
      self.bias = np.random.randn() * 0.01
 
      for epoch in range(self.epochs):
        total_error = 0
        for i in range(len(x)):
          prediction = self.predict(x[i])
          error = y[i] - prediction
          if error != 0:
              self.weights += self.learning_rate * error * x[i]
              self.bias += self.learning_rate * error #notice
          total_error += abs(error)
 
        # Stop early if no errors in this epoch
        if total_error == 0:
          print(f"Converged after {epoch+1} epochs")
          break

Learning AND gate

Let’s make the perceptron learn the AND gate. The AND gate takes two binary inputs and outputs 1 only if both inputs are 1. Here’s how we can set up the training data and test the perceptron.

import numpy as np
 
# Define training data for AND gate
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])  # Input pairs
y = np.array([0, 0, 0, 1])                      # Expected outputs
 
# Initialize and train the perceptron
perceptron = Perceptron(learning_rate=0.1, epochs=100)
perceptron.train(X, y)
 
# Test the perceptron
print("Testing AND gate:")
for i in range(len(X)):
    prediction = perceptron.predict(X[i])
    print(f"Input: {X[i]}, Predicted: {prediction}, Expected: {y[i]}")
 

I hope this doesn’t require much explanation, pretty straightforward.

Converged after 8 epochs
Testing AND gate:
Input: [0 0], Predicted: 0, Expected: 0
Input: [0 1], Predicted: 0, Expected: 0
Input: [1 0], Predicted: 0, Expected: 0
Input: [1 1], Predicted: 1, Expected: 1

Activity

Hints:

• For the XOR gate, you may notice the perceptron struggles. This is because XOR is not linearly separable, meaning a single perceptron cannot solve it.
• For the addition task, the perceptron may not learn the sum perfectly since it outputs binary values (0 or 1) and struggles with multi-valued outputs.

Limitations

The perceptron is a powerful yet simple model, but it has key limitations:

1. Linear Separability: The perceptron can only solve problems where the data can be separated by a single straight line (in 2D) or a hyperplane (in higher dimensions). For example:
   • It can learn AND and OR gates, which are linearly separable.
   • It cannot learn the XOR gate, as the classes (0 and 1) cannot be separated by a single line.
2. Binary Output: The perceptron outputs only 0 or 1 due to the step activation function, making it unsuitable for tasks requiring continuous or multi-valued outputs (e.g., predicting exact sums in the addition task).

Example (XOR Issue):

# Trying XOR gate
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([0, 1, 1, 0])  # XOR outputs
 
perceptron = Perceptron(learning_rate=0.1, epochs=100)
perceptron.train(X, y)
 
# Test XOR
print("Testing XOR gate:")
for i in range(len(X)):
    prediction = perceptron.predict(X[i])
    print(f"Input: {X[i]}, Predicted: {prediction}, Expected: {y[i]}")

Running this will likely show incorrect predictions because XOR requires a non-linear decision boundary, which a single perceptron cannot achieve.

What’s Next? To address these limitations, we can explore:

• Multi-layer Perceptrons (MLPs): Combining multiple perceptrons in layers allows learning non-linear patterns like XOR.
• Different Activation Functions: Using functions like sigmoid or ReLU enables continuous outputs.
• Advanced Models: Neural networks or other algorithms can handle more complex tasks.

Try the exercises above and experiment with the perceptron to see these limitations in action!