2020 was no ordinary year. We have gone through a lot, but there were a lot of exciting things happened in ML.
I want to close the year through a Hall of Fame like section where I will share some of the “Best X” in ml. As there is a long weekend coming, these would keep you busy.
Best Paper
GPT-3 Paper outlines the data, model and training methodology for GPT-3 model. It is a great read both from modeling perspective as well as the model size and operations.
Best Book
Deep Dive into Deep Learning is available online and it has a number of excellent code samples both in PyTorch and MxNet. It also covers very recent research and very applicable for industry use cases.
Best Conference Session
Challenges in Deploying&Monitoring Machine Learning Systems was the best session that I have seen throughout the year. It included a variety of topics that target machine learning in production.
Best Tutorial
EMLNP Tutorial: High Performance NLP is “the tutorial”. The video is also worth watching. It has a number of tricks/methods for model optimization as well as how to leverage some of the very large models under a latency/memory constraint. The tutorial is long, but you can watch the area that you are interested in most through the slides.
Best Class
NYU’s Deep Learning Class was excellent this year. The videos are definitely worth watching.
Best Library
Hands down, HuggingFace’s Transformers library.
Best Book Update
Seminal book Speech and Language Processing got a great update on 30th December. 9th Chapter includes a very good Transformers section.
Others for Year in Review
Microsoft Research published a lengthy post for their year review(a significant portion is AI unsurprisingly).
Papers With Code did an excellent review for ml research.