Accelerating Transformers via Kernel Density Estimation Insu Han
A Google TechTalk, presented by Insu Han, 2023/05/30
A Google Algorithms Seminar. ABSTRACT: Dot-product attention mechanism plays a crucial role in modern deep architectures (e.g., Transformer) for sequence modeling, however, naïve exact computation of this model incurs quadratic time and memory complexities in sequence length, hindering the training of long-sequence models. Critical bottlenecks are due to the computation of partition functions in the denominator of softmax function as well as the multiplication of the softmax matrix with the matrix of values. Our key observation is that the former can be reduced to a variant of the kernel density estimation (KDE) problem, and an efficient KDE solver can be further utilized to accelerate the latter via subsampling-based fast matrix products. Our proposed KDEformer can approximate the attention in sub-quadratic time with provable spectral norm bounds, while all prior results merely provide entry-wise error bounds. Empirically, we verify that KDEformer outperforms other attention approximations in terms of accuracy, memory, and runtime on various pre-trained models.
Bio: Insu Han is a postdoctoral research fellow at Yale University, hosted by Amin Karbasi. He completed his Ph.D. degree in the School of Electrical Engineering at the Korea Advanced Institute of Science and Technology (KAIST) in 2021, under the supervision of Jinwoo Shin. Before that, he obtained his Bachelor’s degree in Electrical Engineering and minored in Mathematics at KAIST. He has worked on developing and analyzing approximate algorithms for large-scale machine learning problems and their applications. His most recent work focuses on accelerating the attention mechanism in large language models via fast kernel density estimation methods. In 2019, he was the recipient of the Microsoft Research Asia Fellowship.
1 view
0
0
5 hours ago 00:02:41 1
cat soup - cursed [video]
8 hours ago 00:12:17 1
PRÉPAREZ-VOUS POUR LA NOUVELLE AUBE : Les Pléiadiens
1 day ago 00:45:45 1
All 9 Solfeggio Frequencies in 45 Minute | Complete Restoration of Body & Mind | Black Screen
5 days ago 00:05:58 1
California High-Speed Rail Construction Package 4 | HNTB
5 days ago 00:19:01 1
🚨 URGENT: Les BRICS ont tué le Dollar ! Quel impact pour nous?
6 days ago 00:49:00 1
IT’S COMING! 4 SEPTEMBER 2024! The New Moon 5D portal Opening tonight!
7 days ago 00:18:51 2
CHOQUANT Découvrez pourquoi le temps s’accélère
1 week ago 00:29:07 1
7 Spiritual Signs You’re Quantum Leaping (and How to Embrace It)
1 week ago 00:03:03 1
WOOSUNG – Phase Me | Official MV
2 weeks ago 00:45:25 1
[Romance,Family] Best Choice Ever EP1 | Starring: Yang Zi, Xu Kai | ENG SUB
2 weeks ago 00:00:10 1
“Daily Chuckles: Your Source for Hilarious Laughter!“
2 weeks ago 00:50:07 1
Dairy Alternatives – The Intersection of Success and Sorrow- ’update’ Session 3
2 weeks ago 11:55:00 1
Money Will Flow To You Non-Stop, Let The Universe Send You Money, Huge Amounts Of Money Very Quickly