Libraries
Recommender Systems Tutorial is a tutorial that shows you how to build recommender systems in Keras and Tensorflow.
Vector Quantization - Pytorch is a vector quantization library originally transcribed from Tensorflow implementation.
Transformers-Recipe is a repository of a lot of introductory information for Transformers.
DocArray is a library for nested, unstructured data such as text, image, audio, video, 3D mesh. It allows deep learning engineers to efficiently process, embed, search, recommend, store, transfer the data with Pythonic API.
Articles
Graph ML in 2022: Where are we now? gives a good overview of Graph ML trends and advancements in 2021 and an outlook for 2022.
It is an excellent overview of what has happened in 2021 which covers:
Graph Transformers
Positional Features
Equivariant Graph Neural Networks
Knowledge Graphs
It also covers a number of libraries that are related to Graph ML as well:
TensorFlow GNN — GNNs as first-class citizens in the Tensorflow world.
TorchDrug — PyTorch-based GNN library for molecular and KG tasks
PyG 2.0 — now supporting heterogeneous graphs, GraphGym, and a flurry of improvements and new models
DGL 0.7 — graph sampling on a GPU, faster kernels, more models
PyKEEN 1.6 — the go-to library for training KG embeddings: more models, datasets, metrics, and NodePiece support!
Jraph — GNNs for JAX aficionados, check this fresh intro by Lisa Wang (DeepMind) and Nikola Jovanović (ETH Zurich) on building and evaluating GNNs
Michael Bronstein is very passionate and champions this area, you might also follow him in Medium and Twitter.
Google wrote on how they do weather predictions based on deep learning models.
HF wrote a blog post on Perceiver model and how it can be used within HF. I have already shared a blog post regarding DeepMind blog post, but just for a refresher around Perceiver:
The compute and memory requirements of the self-attention mechanism don't depend on the size of the inputs and outputs.
Despite its task-agnostic architecture, the model can get great results on modalities such as language, vision and multimodal data.
Google wrote about their new architecture GLAM.
GLaM is a mixture of experts (MoE) model, a type of model that can be thought of as having different submodels (or experts) that are each specialized for different inputs. The experts in each layer are controlled by a gating network that activates experts based on the input data. For each token (generally a word or part of a word), the gating network selects the two most appropriate experts to process the data.
Google wrote a paper about learning tokens in this blog post. It tries to answer to the following question:
With VIT(Vision Transformer) and Transformer computation, computational complexity increases quadratically with the number of tokens. This makes Transformer training to be harder and sometimes intractable even with a large amount of computational power. Instead of compromising on the architecture, can we somehow reduce the tokens? How do we process less number of tokens so that we do not have to compute all of the tokens in each layer?
They came up with an TokenLearner module that integrates both ViT and Transformer architectures overall.
TokenLearner is a learnable module that takes an image-like tensor (i.e., input) and generates a small set of tokens. This module could be placed at various different locations within the model of interest, significantly reducing the number of tokens to be handled in all subsequent layers.
Papers
LabML has a good number of pages that they provide papers with implementations. The home page also has a number of famous modeling architectures in PyTorch.
PapersWithCode gave a an excellent review on papers that are published in 2021.
Videos
Austin Huang from Fidelity gave an overview on how they move various models research to production in this video.
Building Recommender Systems with PyTorch is a good video series that explains how to build recommender systems in PyTorch.
Privacy and Security in ML Group has a number of videos that are in Privacy and Security space.
BAIRS common Symposium made the whole event available in here.