Problème
Adam keeps two moving averages per weight, of the gradient (m) and of the squared gradient (v), and uses them to give each weight its own step size.
Start with m = 0 and v = 0 for every weight. For each step t = 1, 2, 3, … with gradient g:
- m ← beta1 × m + (1 - beta1) × g
- v ← beta2 × v + (1 - beta2) × g²
- m̂ = m / (1 - beta1ᵗ) and v̂ = v / (1 - beta2ᵗ) (bias correction)
- w ← w - learning_rate × m̂ / (√v̂ + epsilon)
Implement adam_steps(gradients, weights, learning_rate, beta1, beta2, epsilon), applying one update per row of gradients, in order.
Input
gradients: One row per step; each row has one gradient per weight.
weights: The starting weights.
learning_rate, beta1, beta2, epsilon: Adam's hyperparameters (commonly 0.001, 0.9, 0.999 and 0.00000001).
Output
Return the weights after all steps.
Contraintes
0 ≤ steps ≤ 1,000, 1 ≤ weights ≤ 100
0 < learning_rate ≤ 1, 0 ≤ beta1, beta2 < 1, 0 < epsilon ≤ 0.001
-100 ≤ each value ≤ 100
Exemples
Entrée :
beta1 = 0.9
beta2 = 0.999
epsilon = 1.0e-08
weights = [1.0]
gradients = [
[2.0]
]
learning_rate = 0.1
Sortie attendue :
expected = [0.9]Entrée :
double beta1 = 0.9;
double beta2 = 0.999;
double epsilon = 1.0e-08;
std::vector<double> weights = {1.0};
std::vector<std::vector<double>> gradients = {
{2.0}
};
double learning_rate = 0.1;
Sortie attendue :
std::vector<double> expected = {0.9};Entrée :
double beta1 = 0.9;
double beta2 = 0.999;
double epsilon = 1.0e-08;
double[] weights = new double[] {1.0};
double[][] gradients = new double[][] {
new double[] {2.0}
};
double learning_rate = 0.1;
Sortie attendue :
double[] expected = new double[] {0.9};Explication
m = 0.1 × 2 = 0.2 and v = 0.001 × 4 = 0.004. Bias correction gives m̂ = 0.2 / 0.1 = 2 and v̂ = 0.004 / 0.001 = 4, so the step is 0.1 × 2 / √4 = 0.1: w = 0.9. On its first step Adam moves each weight by about the learning rate, whatever the gradient's size.
Entrée :
beta1 = 0.9
beta2 = 0.999
epsilon = 1.0e-08
weights = [0.0, 0.0]
gradients = [
[100.0, 0.01]
]
learning_rate = 0.01
Sortie attendue :
expected = [-0.01, -0.01]Entrée :
double beta1 = 0.9;
double beta2 = 0.999;
double epsilon = 1.0e-08;
std::vector<double> weights = {0.0, 0.0};
std::vector<std::vector<double>> gradients = {
{100.0, 0.01}
};
double learning_rate = 0.01;
Sortie attendue :
std::vector<double> expected = {-0.01, -0.01};Entrée :
double beta1 = 0.9;
double beta2 = 0.999;
double epsilon = 1.0e-08;
double[] weights = new double[] {0.0, 0.0};
double[][] gradients = new double[][] {
new double[] {100.0, 0.01}
};
double learning_rate = 0.01;
Sortie attendue :
double[] expected = new double[] {-0.01, -0.01};Explication
The gradients differ by a factor of 10,000, yet both weights move by about 0.01, to [-0.01, -0.01]: Adam scales each step by the gradient's own size.
Références
Ce défi a été adapté pour CodeUtility. Suivez l'énoncé et les contraintes ci-dessus.