Fourier features: positions as waves

A raw coordinate makes nearby positions hard to tell apart

Suppose a shape has an edge at x = 0.31: positions just left of it are inside, positions just right of it are outside. A network must give opposite answers for x = 0.30 and x = 0.32. Yet the two inputs differ by only 0.02. The first layer computes weighted sums of x, and those sums differ by only 0.02 times a weight. To make the two sums very different, the layer needs very large weights.

In practice, networks trained on raw coordinates learn the smooth, slowly changing parts of a function first, and sharp detail very slowly. This tendency is called spectral bias. Fourier features work around it by handing the network inputs that already change fast.

Each frequency turns a position into a sine and a cosine

Pick a frequency f: the number of full waves per unit of position. For a position x, compute the pair sin(2πfx) and cos(2πfx), as in the sine waves unit. The factor 2π makes f count full turns: as x runs from 0 to 1, the pair goes around a circle f times. With k frequencies, the Fourier features of x are 2k numbers. We write them as φ(x), with the Greek letter phi:

φ(x) = [ sin(2πf₁x), cos(2πf₁x), …, sin(2πf_kx), cos(2πf_kx) ]

The name honors Joseph Fourier, who showed that sums of sines and cosines can build almost any repeating curve. A layer that reads φ(x) can mix these waves with its weights, so fast-changing functions of x become easy to make.

One high frequency alone repeats itself

Predict first. Frequency 16 separates the two positions better than frequency 4. Will frequency 50 separate them better still?

So a single frequency is ambiguous: any two positions a whole number of waves apart look the same to it. This is why Fourier features use several frequencies, from low to high. The low frequencies tell distant positions apart without repeating. The high frequencies tell close positions apart. Together they pin down the position, like the hands of a clock.

Models choose the frequencies in one of three common ways

The lowest frequency should not repeat inside the range of the data. With f = 1 as the lowest, positions 0 and 1 get the same features, so the data should lie in [0, 1). The highest frequency sets the finest detail the network can easily make.

A 3D point p = (x, y, z) works the same way. Either encode each coordinate separately, or choose frequency vectors b and use sin(2π b · p) and cos(2π b · p), where · is the dot product. A model that must know the time t of a process, such as a step in a generative model, encodes t the same way (see time conditioning).

fourier_features.py

import numpy as np

def fourier_features(x, freqs):
    angles = 2 * np.pi * np.outer(x, freqs)    # row i: 2π x[i] times each frequency
    # The sine columns, then the cosine columns, side by side in one array.
    return np.concatenate([np.sin(angles), np.cos(angles)], axis=1)

freqs = np.array([1.0, 4.0, 16.0])
F = fourier_features(np.array([0.30, 0.32]), freqs)
print(np.round(F, 3))                          # sines first, then cosines
print(round(np.linalg.norm(F[0] - F[1]), 3))   # gap between the feature vectors

[[ 0.951  0.951 -0.951 -0.309  0.309  0.309]
 [ 0.905  0.982  0.685 -0.426 -0.187  0.729]]
1.765