Parameter Prediction for Unseen Deep Architectures (w/ First Author Boris Knyazev)
#deeplearning #neuralarchitecturesearch #metalearning
Deep Neural Networks are usually trained from a given parameter initialization using SGD until convergence at a local optimum. This paper goes a different route: Given a novel network architecture for a known dataset, can we predict the final network parameters without ever training them? The authors build a Graph-Hypernetwork and train on a novel dataset of various DNN-architectures to predict high-performing weights. The results show that not only can the GHN predict weights with non-trivial performance, but it can also generalize beyond the distribution of training architectures to predict weights for networks that are much larger, deeper, or wider than ever seen in training.
OUTLINE:
0:00 - Intro & Overview
6:20 - DeepNets-1M Dataset
13:25 - How to train the Hypernetwork
17:30 - Recap on Graph Neural Networks
23:40 - Message Passing mirrors forward and backward propagation
25:20 - How to deal with different output shapes
28:45 - Differentiable Normal
6 views
5
1
1 day ago 00:14:18 1
How Milankovitch Cycles Are Causing Earth’s Climate To Change
1 week ago 00:00:13 1
Allen Bradley MEASUREMENT MODULE 1440 SPD02 01RB
1 month ago 00:02:48 1
AES256 Encryption - DMR Radio
2 months ago 00:04:20 1
82. Central Limit Theorem.
2 months ago 00:17:35 1
What Do Neural Networks Really Learn? Exploring the Brain of an AI Model
4 months ago 00:05:06 9
Magnitude 4.1 Earthquake Hits Los Angeles California - May 1st, 2024
4 months ago 00:09:37 1
Is Lottery Defeater a Scam? (SCAM?⚠️) LOTTERY DEFEATER – Lottery Defeated – Lottery Defeater Reviews
5 months ago 01:39:49 1
Digitakt 2 - Beginner’s MEGA TUTORIAL
5 months ago 00:09:19 2
How to Understand What Black Holes Look Like
5 months ago 00:18:28 1
Как похудеть на интуитивном питании? Что можно есть и что нельзя есть на интуитивном питании?
6 months ago 00:20:05 2
Самое важное для костей l Остеопороз - Лечение l Минералы и Витамины l Osteoporosis - Treatment
7 months ago 00:03:11 1
Confidence Interval
7 months ago 00:07:05 1
Deep Video Portraits - SIGGRAPH 2018
8 months ago 01:53:11 1
The Science & Health Benefits of Deliberate Heat Exposure | Huberman Lab Podcast #69
8 months ago 00:07:40 1
Mw6.4 / 210km deep Earthquake in Afghanistan caught by GlobalQuake - Jan 11, 2024