Neural Networks

Internal

• Machine Learning

Individual Unit

Individual neural network units are computational units that read input features, represented as an unidimensional vector x₁ ... x_n in the diagram below, and calculate the hypothesis function as output of the unit. Note that x₀ is not part of the feature vector, but it represents a bias value for the unit. The output value of the hypothesis function is also called the "activation" of the unit.

A common option is to use a logistic function as hypothesis, thus the unit is referred to as a logistic unit with a sigmoid (logistic) activation function.

The θ vector represents the model's parameters (model's weights). For a multi-layer neural network, the model parameters are collected in matrices named Θ, which will be describe below.

The x0 input node is called the bias unit, and it is optional. When provided, it is equal with 1.

Multi-Layer Neural Network

Notations and Conventions

a_i^(j) represents the "activation" of unit i in layer j.

Θ^(j) represents the matrix of parameters (weights) that controls function mapping from layer j to layer j + 1.

The Input Layer

The input layer, conventionally named "layer 1", consists of input nodes. The input layer provides the training values. A training set contains a number of samples (m), and each sample has a number of features (n). The features of the training set are conventionally represented as a matrix X.

The Hidden Layers

Paramenter Matrix Θ Notation Convention

If the layer j has p units, not counting the bias unit, and layer j + 1 has q units, not counting the bias unit, then the parameter matrix Θ^(j) controlling function mapping from layer j to layer j + 1 has q x (p + 1) elements. The "+1" comes from the addition of the bias node in layer j.

Layer j + 1 Unit Activation Values

The Output Layer