A linear layer: weighted sums

Each output adds up the inputs, each with its own weight

Start with three input numbers, x₁, x₂ and x₃. They might be the three coordinates of a point, or three measurements of anything. One output of the layer is computed like this:

y₁ = w₁₁x₁ + w₁₂x₂ + w₁₃x₃ + b₁

Each weight w says how much one input counts toward this output. A large weight makes that input matter a lot. A negative weight makes the input pull the output down. A weight of zero ignores the input. The bias b is a constant added at the end. It sets the output's value when every input is zero.

A layer usually has several outputs. Each output has its own weights and its own bias, so each one adds up the same inputs in a different way.

Try it. In the figure, the inputs x₁ = 1 and x₃ = 2 stay fixed. Predict from the weights which value of x₂ makes y₁ exactly 0. Then move the slider to check.

A matrix holds all the weights, one row per output

Write the weights as a weight matrix W: row 1 holds output 1's weights, row 2 holds output 2's. Put the biases in a vector b. Then the whole layer is a matrix times a vector, plus a vector:

y = Wx + b,   W = [[2, −1, 0.5], [0, 1, 1]],   b = (1, −2)

A layer with n inputs and m outputs has an m × n weight matrix: m rows, n columns. The input size and output size are free choices. A layer can make the list longer, shorter, or keep its length.

Training picks the weights, and there are many of them

Nobody types in the weights of a real network. They start as small random numbers. Training then adjusts them, step by step, so that the network's outputs get closer to the right answers. The unit on gradient descent shows how. Numbers that training adjusts are called the model's parameters.

A layer with n inputs and m outputs has n × m weights and m biases. The small layer above has 3 × 2 + 2 = 8 parameters. A layer from 512 numbers to 2,048 numbers has 512 × 2,048 + 2,048 = 1,050,624.

In code, a layer handles a whole batch of inputs at once

Programs usually pass many inputs through a layer together: one input per row of an array. For a batch X with one input per row, the layer computes X @ W.T + b. Each row of the result is the output for that row's input. This is the same arithmetic, done for all rows at once.

linear_layer_numpy.py

import numpy as np

W = np.array([[2.0, -1.0, 0.5],
              [0.0,  1.0, 1.0]])   # 2 outputs, 3 inputs
b = np.array([1.0, -2.0])
x = np.array([1.0, 3.0, 2.0])

y = W @ x + b
print(y)                # one input vector

X = np.array([[1.0, 3.0, 2.0],
              [0.0, 0.0, 0.0]])  # a batch: one input per row
print(X @ W.T + b)      # one output per row

[1. 3.]
[[ 1.  3.]
 [ 1. -2.]]

PyTorch's nn.Linear holds W and b for you and starts them at random values. It stores W with one row per output, as above.

linear_layer_torch.py

import torch

layer = torch.nn.Linear(3, 2)   # 3 inputs, 2 outputs
print(layer.weight.shape, layer.bias.shape)
print(sum([p.numel() for p in layer.parameters()]))  # count the numbers in W and b

with torch.no_grad():            # set the weights by hand
    layer.weight.copy_(torch.tensor([[2.0, -1.0, 0.5],
                                     [0.0,  1.0, 1.0]]))
    layer.bias.copy_(torch.tensor([1.0, -2.0]))
print(layer(torch.tensor([1.0, 3.0, 2.0])).detach())

torch.Size([2, 3]) torch.Size([2])
8
tensor([1., 3.])

torch.tensor turns a list into a tensor, PyTorch's version of an array (see tensors on the GPU). PyTorch records every step that uses the layer's weights, so that it can compute gradients later. Inside torch.no_grad() it stops recording (see automatic gradients), so the code may overwrite the weights with copy_. .detach() gives the output without that record, so only its numbers print.