Optimization Track. Denis Timonin: Fast training with AMP/TF32 using TensorCores on NVIDIA GPU

We will talk about one of the most powerful methodologies to speed-up Training and Inference at the current time. In my presentation, we will dive into details of the research paper “Mixed Precision Training” by NVIDIA and Baidu Research and into detail of TensorFloat32 precision format. We will discuss algorithms that are used in Mixed Precision training and also we will talk about hardware that can provide high speed for that data formats in Neural Networks. I will try to simplify all of this information.