reCAPTCHA WAF Session Token

Video Highlights: Ultimate Guide To Scaling ML Models – Megatron-LM | ZeRO | DeepSpeed | Mixed Precision

In this video presentation, Aleksa Gordić explains what it takes to scale ML models up to trillions of parameters! He covers the fundamental ideas behind all of the recent big ML models like Meta’s OPT-175B, BigScience BLOOM 176B, EleutherAI’s GPT-NeoX-20B, GPT-J, OpenAI’s GPT-3, Google’s PaLM, DeepMind’s Chinchilla/Gopher models, etc. He covers the ideas of data parallelism, model/pipeline parallelism (e.g. GPipe, PipeDream, etc.), model/tensor parallelism (Megatron-LM), activation checkpointing, mixed precision training, ZeRO (zero redundancy optimizer) from Microsoft’s DeepSpeed library and many more. Along the way, many top research papers are highlighted. The video presentation is sponsored by AssemblyAI.


✅ Megatron-LM paper:

✅ ZeRO (DeepSpeed) paper:

✅ Mixed precision training paper:

✅ Gpipe (pipeline parallelism) paper:


✅ Collective ops:…

✅ IEEE float16 format:…

✅ Google Brain’s bfloat16 format:…

Sign up for the free insideBIGDATA newsletter.

Join us on Twitter:

Join us on LinkedIn:

Join us on Facebook:

Source link

Leave a Reply

Your email address will not be published. Required fields are marked *

WP Twitter Auto Publish Powered By :