Luke Richardson - Electrical Engineer

1. Neural Networks Overview

Neural networks are computational models inspired by the human brain's structure and function. They consist of interconnected nodes (neurons) organized in layers that process information through weighted connections and activation functions.

Core Components

Neurons: Basic processing units that receive inputs, apply weights, and produce outputs
Layers: Groups of neurons - input, hidden, and output layers
Weights: Learnable parameters that determine connection strength between neurons
Biases: Additional parameters that allow neurons to adjust their activation threshold
Activation Functions: Non-linear functions that introduce complexity to the model

Learning Process

Neural networks learn through backpropagation, a process where:

Input data flows forward through the network
The network produces predictions
Error is calculated by comparing predictions to actual targets
Gradients are computed and propagated backward
Weights and biases are updated to minimize error

2. Mathematical Foundation

Forward Propagation

For a neuron in layer $l$ , the activation is computed as:

a^{(l)} = f(W^{(l)} \cdot a^{(l-1)} + b^{(l)})

Where:

$a^{(l)}$ is the activation of layer $l$
$W^{(l)}$ is the weight matrix connecting layers $l-1$ and $l$
$b^{(l)}$ is the bias vector for layer $l$
$f$ is the activation function

Activation Functions

This playground supports three activation functions:

Sigmoid

\sigma(x) = \frac{1}{1 + e^{-x}}

Output range: (0, 1)

Tanh

\tanh(x) = \frac{e^{x} - e^{-x}}{e^{x} + e^{-x}}

Output range: (-1, 1)

ReLU

ReLU(x) = \max(0, x)

Output range: [0, ∞)

Backpropagation

The gradient of the loss function with respect to weights is computed using the chain rule:

\frac{\partial L}{\partial W^{(l)}} = \frac{\partial L}{\partial a^{(l)}} \cdot \frac{\partial a^{(l)}}{\partial z^{(l)}} \cdot \frac{\partial z^{(l)}}{\partial W^{(l)}}

Where $z^{(l)} = W^{(l)} \cdot a^{(l-1)} + b^{(l)}$ is the pre-activation output.

3. Implementation Details

Neural Network Class Structure

The core neural network is implemented as a TypeScript class with methods for forward propagation, backpropagation, and training.

NeuralNetwork.ts

export class NeuralNetwork {
  layers: number[];
  weights: number[][][];
  biases: number[][];

  constructor(layers: number[]) {
    this.layers = layers;
    this.weights = [];
    this.biases = [];
    this.initializeNetwork();
  }

  initializeNetwork() {
    for (let i = 0; i < this.layers.length - 1; i++) {
      const layerWeights: number[][] = [];
      const layerBiases: number[] = [];

      for (let j = 0; j < this.layers[i + 1]; j++) {
        const neuronWeights: number[] = [];
        for (let k = 0; k < this.layers[i]; k++) {
          // Random initialization between -1 and 1
          neuronWeights.push(Math.random() * 2 - 1);
        }
        layerWeights.push(neuronWeights);
        layerBiases.push(Math.random() * 2 - 1);
      }

      this.weights.push(layerWeights);
      this.biases.push(layerBiases);
    }
  }
}

Forward Propagation Implementation

The forward pass computes activations for each layer sequentially:

NeuralNetwork.ts - Forward Pass

forward(input: number[], activationFunction: string = 'tanh') {
  let activation = input;
  const activations = [activation];
  const zs: number[][] = [];

  for (let i = 0; i < this.weights.length; i++) {
    const layerZ: number[] = [];
    const layerOutput: number[] = [];

    for (let j = 0; j < this.weights[i].length; j++) {
      // Compute weighted sum + bias
      let z = this.biases[i][j];
      for (let k = 0; k < this.weights[i][j].length; k++) {
        z += this.weights[i][j][k] * activation[k];
      }
      layerZ.push(z);
      // Apply activation function
      layerOutput.push(this.activate(z, activationFunction));
    }

    zs.push(layerZ);
    activation = layerOutput;
    activations.push(activation);
  }

  return { activations, zs };
}

Backpropagation Algorithm

The backpropagation method implements gradient descent to update weights and biases:

NeuralNetwork.ts - Backpropagation

backpropagate(input: number[], target: number[], learningRate: number, activationFunction: string): number {
  // Forward pass
  const { activations, zs } = this.forward(input, activationFunction);

  // Initialize gradient arrays
  const nabla_b = this.biases.map(layer => new Array(layer.length).fill(0));
  const nabla_w = this.weights.map(layer =>
    layer.map(neuron => new Array(neuron.length).fill(0))
  );

  // Compute output layer error
  const outputActivations = activations[activations.length - 1];
  const outputZs = zs[zs.length - 1];
  let delta = outputActivations.map((output, i) =>
    (output - target[i]) * this.activateDerivative(outputZs[i], activationFunction)
  );

  nabla_b[nabla_b.length - 1] = delta;

  // Backpropagate error through hidden layers
  for (let l = this.weights.length - 2; l >= 0; l--) {
    const z = zs[l];
    const sp = z.map(z => this.activateDerivative(z, activationFunction));
    const delta_next = delta;

    delta = new Array(this.weights[l].length).fill(0);
    for (let i = 0; i < delta.length; i++) {
      let sum = 0;
      for (let j = 0; j < delta_next.length; j++) {
        sum += this.weights[l + 1][j][i] * delta_next[j];
      }
      delta[i] = sum * sp[i];
    }

    nabla_b[l] = delta;
    
    // Compute weight gradients
    for (let j = 0; j < this.weights[l].length; j++) {
      for (let k = 0; k < this.weights[l][j].length; k++) {
        nabla_w[l][j][k] = delta[j] * activations[l][k];
      }
    }
  }

  // Update weights and biases
  for (let i = 0; i < this.weights.length; i++) {
    for (let j = 0; j < this.weights[i].length; j++) {
      for (let k = 0; k < this.weights[i][j].length; k++) {
        this.weights[i][j][k] -= learningRate * nabla_w[i][j][k];
      }
      this.biases[i][j] -= learningRate * nabla_b[i][j];
    }
  }

  return this.calculateError(outputActivations, target);
}

4. Project Architecture

The project is structured using React components with clear separation of concerns:

Component Structure

page.tsx - Main component with state management
NeuralNetwork.ts - Core neural network implementation
NetworkCanvas.tsx - Network structure visualization
ResultsCanvas.tsx - Classification results display
SettingsPanel.tsx - Configuration controls
Legend.tsx - Visual legends for canvases

State Management

The main component manages all application state using React hooks:

page.tsx - State Management

export default function NeuralNetworkPlayground() {
  const [network, setNetwork] = useState<NeuralNetwork | null>(null);
  const [isTraining, setIsTraining] = useState(false);
  const [isPaused, setIsPaused] = useState(false);
  const [epoch, setEpoch] = useState(0);
  const [error, setError] = useState(0);
  const [accuracy, setAccuracy] = useState(0);
  const [problemType, setProblemType] = useState<ProblemType>("spiral");
  const [hiddenLayers, setHiddenLayers] = useState(3);
  const [neuronsPerLayer, setNeuronsPerLayer] = useState(8);
  const [learningRate, setLearningRate] = useState(0.03);
  const [activationFunction, setActivationFunction] = useState("tanh");
  const [isClient, setIsClient] = useState(false);
  
  const animationFrameRef = useRef<number | undefined>(undefined);
  
  // Training loop using requestAnimationFrame for smooth animation
  useEffect(() => {
    if (!isTraining || isPaused || !network || !isClient) return;

    const animate = () => {
      trainStep();
      animationFrameRef.current = window.requestAnimationFrame(animate);
    };

    animationFrameRef.current = window.requestAnimationFrame(animate);

    return () => {
      if (animationFrameRef.current) {
        window.cancelAnimationFrame(animationFrameRef.current);
      }
    };
  }, [isTraining, isPaused, network, learningRate, activationFunction, problemType, isClient]);
}

5. Visualization Components

Network Structure Canvas

The NetworkCanvas component visualizes the neural network structure with neurons and weighted connections:

NetworkCanvas.tsx - Rendering Logic

const render = () => {
  const rect = canvas.getBoundingClientRect();
  const dpr = window.devicePixelRatio || 1;
  
  canvas.width = rect.width * dpr;
  canvas.height = rect.height * dpr;
  ctx.scale(dpr, dpr);
  
  // Clear canvas
  ctx.fillStyle = '#0a0a0a';
  ctx.fillRect(0, 0, rect.width, rect.height);
  
  const margin = 40;
  const layerSpacing = (rect.width - margin * 2) / Math.max(1, network.layers.length - 1);
  
  // Draw connections with weight-based styling
  for (let i = 0; i < network.layers.length - 1; i++) {
    const layer1X = margin + layerSpacing * i;
    const layer2X = margin + layerSpacing * (i + 1);
    
    for (let j = 0; j < network.layers[i]; j++) {
      for (let k = 0; k < network.layers[i + 1]; k++) {
        const weight = network.weights[i][k][j];
        
        ctx.beginPath();
        const opacity = Math.min(Math.abs(weight), 1);
        if (weight > 0) {
          ctx.strokeStyle = `rgba(255, 255, 255, ${opacity * 0.5})`;
        } else {
          ctx.strokeStyle = `rgba(115, 115, 115, ${opacity * 0.5})`;
        }
        ctx.lineWidth = Math.min(Math.abs(weight) * 3, 4);
        ctx.moveTo(layer1X, neuron1Y);
        ctx.lineTo(layer2X, neuron2Y);
        ctx.stroke();
      }
    }
  }
  
  // Draw neurons with glow effects
  for (let i = 0; i < network.layers.length; i++) {
    // ... neuron rendering code
  }
};

Classification Results Canvas

The ResultsCanvas shows the decision boundary and training data points:

ResultsCanvas.tsx - Decision Boundary

// Draw the decision boundary
const resolution = 50;
const step = size / resolution;

for (let i = 0; i < resolution; i++) {
  for (let j = 0; j < resolution; j++) {
    const x = (i / resolution) * 2 - 1;
    const y = (j / resolution) * 2 - 1;
    
    // Get network prediction for this point
    const output = network.forward([x, y], 'tanh').activations.slice(-1)[0][0];
    
    // Color based on prediction
    if (output > 0.5) {
      ctx.fillStyle = 'rgba(255, 255, 255, 0.2)';
    } else {
      ctx.fillStyle = 'rgba(115, 115, 115, 0.2)';
    }
    
    ctx.fillRect(
      offsetX + i * step,
      offsetY + j * step,
      step + 1,
      step + 1
    );
  }
}

6. Training Process

Problem Types

The playground includes two classic classification problems:

Circle Classification

Points inside a circle belong to Class 1, points outside belong to Class 0.

Problems.ts - Circle

circle: {
  generateData(): TrainingData[] {
    const data: TrainingData[] = [];
    for (let i = 0; i < 100; i++) {
      const x = Math.random() * 2 - 1;
      const y = Math.random() * 2 - 1;
      const target = x * x + y * y < 0.5 ? [1] : [0];
      data.push({ input: [x, y], target });
    }
    return data;
  }
}

Spiral Classification

Two interleaving spirals represent different classes.

Problems.ts - Spiral

spiral: {
  generateData(): TrainingData[] {
    const data: TrainingData[] = [];
    const n = 100;
    
    for (let i = 0; i < n; i++) {
      for (let j = 0; j < 2; j++) {
        const r = (i / n) * maxRadius;
        const theta = (i / n) * 4 * Math.PI + j * Math.PI;
        
        const x = r * Math.cos(theta);
        const y = r * Math.sin(theta);
        
        data.push({
          input: [x, y],
          target: [j]
        });
      }
    }
    return data;
  }
}

Real-time Training Loop

Training occurs in real-time using requestAnimationFrame for smooth visualization:

page.tsx - Training Step

const trainStep = () => {
  if (!network) return;
  
  const trainingData = Problems[problemType].generateData();
  let totalError = 0;
  let correctPredictions = 0;

  trainingData.forEach(({ input, target }) => {
    const error = network.backpropagate(input, target, learningRate, activationFunction);
    totalError += error;

    const output = network.forward(input, activationFunction).activations.slice(-1)[0][0];
    if (Math.round(output) === target[0]) {
      correctPredictions++;
    }
  });

  const newAccuracy = (correctPredictions / trainingData.length * 100);
  const avgError = totalError / trainingData.length;

  setEpoch(prev => prev + 1);
  setError(avgError);
  setAccuracy(newAccuracy);
};

Performance Considerations

Client-side rendering: All computations run in the browser for instant feedback
Efficient canvas updates: Double buffering and device pixel ratio handling
Optimized training loop: Batch processing of training data each frame
Memory management: Proper cleanup of animation frames and event listeners

Documentation

Table of Contents