Self-Tuning Networks: Amortizing the Hypergradient Computation for Hyperparameter Optimization
Optimization of many deep learning hyperparameters can be formulated as a bilevel optimization problem. While most black-box and gradient-based approaches require many independent training runs, we aim to adapt hyperparameters online as the network trains. The main challenge is to approximate the response Jacobian, which captures how the minimum of the inner objective changes as the hyperparameters are perturbed. To do this, we introduce the self-tuning network (STN), which fits a hypernetwork to approximate the best response function in the vicinity of the current hyperparameters. Differentiating through the hypernetwork lets us efficiently approximate the gradient of the validation loss with respect to the hyperparameters. We train the hypernetwork and hyperparameters jointly. Empirically, we can find hyperparameter settings competitive with Bayesian Optimization in a single run of training, and in some cases find hyperparameter schedules that outperform any fixed hyperparameter value.
Roger Grosse i
10 views
25
7
4 weeks ago 00:59:48 2
[1hr Talk] Intro to Large Language Models
3 months ago 00:30:15 1
S&W Model 66 Combat Magnum
3 months ago 01:00:51 1
Is the Universe Alive? - Kabbalah Explained Simply
3 months ago 00:10:41 1
“David Icke’s Revelation: The Shocking Prophecy for 2025 That Threatens Humanity’s Existence!“
4 months ago 00:08:14 16
OpenAI Eve Humanoid Robot: The Most Versatile and Autonomous Humanoid Robot Ever Created