Adam stores one value per parameter.
A full second-moment accumulator has the same shape as the parameter matrix, so optimizer memory grows with rows times columns.
Second-moment state
State memory0 MB
Memory reduction0x
Update RMS0
Clipped fraction0%
Approximate scale, preserve the direction of useful updates.
Adafactor stores row and column marginals.
The outer product of two compact statistics approximates the full matrix of squared-gradient scales.
Factoring favors large matrices.
State size falls from O(rows × columns) to O(rows + columns).
Update clipping limits surprises.
RMS clipping rescales unusually large normalized updates rather than clipping each coordinate independently.
Relative steps follow parameter scale.
A parameter-relative schedule can adapt the effective step to the RMS magnitude of the weights.