HPC Approaches to Training Neural Networks in Deep Learning

Patrick Legresley, Baidu Parallel computing is critical to achieving cost-effective, fast-turnaround for training models in deep learning. In this talk I will give a brief overview of algorithms for deep learning using neural networks, and describe parallelization of model training for speech recognition. Our work uses a High Performance Computing (HPC) approach: a cluster of multi-GPU servers, linked via an InfiniBand interconnect, and using CUDA aware Message Passing Interface (MPI) for communication. Th