bfloat16 - how it improves AI chip designs
Floating point calculations are slow for computers (specifically CPUs); possibly representing the same struggle for many humans. :) I remember a time when a FPU (floating point unit) was an upgrade and one had to pay extra to get one. Very useful when you needed that extra precision in computing - and in my head, it always seemed like the Turbo button. :) For most #ML workloads and computations, precision isn’t the most important criteria; with every increasing data and parameters (looking at you GPT-3 with 45 TB of data and 175 billion parameters!...