Adam

Understand the term. Connect the concept.

A practical dictionary of 512 AI, machine learning, mathematics, and programming terms from HerbDev’s AI Vocabulary app. Start with a definition, explore an example, then follow the concepts behind it.

Linked concepts open in a new tab. Audio reads the definition and example.

Optimization · Level 2 of 5

An optimizer using moving estimates of gradient means and squared gradients.

It rescales updates coordinatewise and corrects initial moment-estimate bias.

Example

A trainer maintains two running gradient statistics for each parameter.

Listen to the definition and example

Audio transcript

Adam. An optimizer using moving estimates of gradient means and squared gradients. It rescales updates coordinatewise and corrects initial moment-estimate bias. For example: A trainer maintains two running gradient statistics for each parameter.

Explore this concept

A useful analogy

A walker that smooths recent directions and adjusts the scale separately for each coordinate.

Why it matters

This helps you understand what parameter updates are trying to improve and when they can fail.

Technical detail

Adam tracks m_t = β₁m_(t−1)+(1−β₁)g_t and v_t = β₂v_(t−1)+(1−β₂)g_t², then uses bias-corrected moments.

Common misconception

Adam’s adaptive steps do not remove the need to choose a learning rate.

Start with

Related concepts

Quick recall question

Try answering before looking back at the definition.