Slack's Recommendation System, Amazon's new GNN training algorithm NVIDIA introduces HugeCTR RecSys
PCA detailed and explained, Visual Prompting for Image Inpainting
What do you think that are most important intellectual breakthroughs? Please send me an email with your response!
Articles
Peter Bloom wrote a number of good articles on PCA(Principal Component Analysis):
Part 1: introduces the basic concepts and terminology that is being used later in the articles
Part 2: introduces eigenvectors, eigenvalues and other optimization algorithms to reduce the loss function when we need to optimize the loss function when we are finding the eigenvectors and eigenvalues of the covariance matrix. It also talks about Spectral theorem and other types of concepts that would be useful to understand PCA.
Part 3: Further explains spectral theorem, determinants, characteristic polynomial,
Part 4: Talks about singular value vectors, singular value decomposition(SVD) and how SVD can be used to do PCA.
Slack explains their recommend api in this blog post by dissecting different dimensions. Their focus of MLOps and productionization of the system, articulating this in the blog post is well done.
Amazon introduces a new algorithm for Graph Neural Network to speed up the training process: DistDGLv2, which has three main components:
a distributed key-value database (KVStore) to store node/edge features and learnable embeddings;
a distributed graph store to keep the partitioned graphs for minibatch sampling; and
a set of trainers to run forward and backward computation on minibatches to estimate the gradients of the model parameters.
Nvidia wrote about their recommender systems in this blog post where they talk about common challenges for large scale recommender systems both from modeling and infrastructure perspective. HugeCTR and Hierarchical Parameter Serving Infrastructure are the things that they have developed to respond some of these challenges in response
.UC Berkeley and Tel Aviv university researchers create a new way to do image inpainting through visual prompting and they have a nice research project page.
During training, an input image is patchified, masked and fed into a masked auto-encoder. For each masked token, the decoder outputs a distribution over a pretrained VQGAN codebook. The model is trained using cross entropy loss on random crops from our datasets.
Everyone’s favorite visualization library seaborn has a new interface which is similar to ggplot and adopts a much more functional approach, has a number of examples in its page.
HuggingFace wrote their Tensorflow philosophy and their support model in the following blog post.
Libraries
Netron is a nice model visualizer and it is open-source in GitHub.
Universal Transformer is an extension to the Transformer models which combines the parallelizability and global receptive field of the Transformer model with the recurrent inductive bias of RNNs, which seems to be better suited to a range of algorithmic and natural language understanding sequence-to-sequence problems. Besides, as the name implies, in contrast to the standard Transformer, under certain assumptions the Universal Transformer can be shown to be computationally universal.
Tutorials
Causal Fairness is presented in ICML this year gives a good overview on the
Outline of Causal Fairness Analysis
Introduction to Structural Causal Models and Causal Graphs.
Fundamental Problem of Causal Fairness Analysis (FPCFA)
Theory of Decomposing Variations
Bias Detection: Fairness Cookbook
Fair Prediction: Fair Prediction Theorem
Failure of Optimal Transport Methods
Causal Optimal Transport (Causal Individual Fairness)
Causal Inference Bootcamp talks about causal inference concepts
Andrey Karpathy released another tutorial in basics of deep learning in the following video: