This week, we have an excellent new library that targets for GPU programming from OpenAI, called Triton.
NVIDIA wrote a good post on various different recommender libraries and advancements around session based features. If you are working in this domain, be sure to check that out as well.
Without further ado, let’s dive in the material right away!
Bugra
Articles
OpenAI open-sourced a new library called Triton to make it easy to program in GPU. They benchmark for kernels for FP16 and they claim upto 2x performance improvements comparing to PyTorch implementation. The documentation is here and code is here.
NVIDIA wrote about how to build a deep learning based recommender systems in this post. The post covers both the original implementation of Wide and Deep Recommender systems from Google and DLRM(Deep Learning Recommender Model). They further cover on session based recommender systems where incorporation of the user features over time becomes “session features”.
Twitter published a post announcing their first bounty for algorithmic bias. If you remember, the image cropping algorithm that they pushed to production was rolled back after a number of use cases pointed out various biases due to dataset/lack of validation in the datasets and I wrote about this decision in this newsletter.
Testing and validating models is very early stages to cover all of the fairness use cases as this field is very nascent. This news is a right step in the right direction.
Gradient wrote about Machine Translation and how its usage “shifts power”. It provides a historical context on how machine translation started and how large scale language models are being used to do machine translation and how these could be biased to do machine translation.
I think the post is well intentioned, but it is not clearly articulating the issues with the large language models, but it is a good read on possible abuses of these systems in a number of different settings. The better way to articulate these problems from language perspective where “low resource” languages(where there are not a lot of articles/books not available) suffer against “high resource” languages(spanish, english) as through ML models, larger the dataset, the better the translation would be.
Google open-sourced two datasets that can be used to build conversational NLP.
Papers
How to avoid machine learning pitfalls: a guide for academic researchers gives a concise and actionable guidance on what to do and what not to do in ml research. It is very approachable and great read for 17 pages paper.
Measuring Robustness to Natural Distribution Shifts in Image Classification looks at the model robustness in the face of distribution in the dataset. It uses imagenet as a testbed and library that they open-source supports PyTorch image models for imagenet.
Libraries
torchdyn is a neural differential equation and implicit neural models written in PyTorch.
Reverse Engineering GMs is a library that implements the following paper. It tries to extract GM(Generative Model) parameters from the generated images.
Non Parametric Transformers is a library that implements the following paper. It is written in PyTorch.
Notebooks
Graph Attention Networks is a nice notebook that shows how to implement GATs in PyTorch.
Keras team published a notebook for Vector-Quantized Variational Encoders.
Katherine Crowson published a notebook that combines OpenAI’s CLIP and Guided Diffusion models to connect prompts with images.