Articles
Google published a nice blog post on open-domain question answering and how they recently tackled this domain through an architecture using multi-head projection and self attention.
Google provided a number of interactive visualizations for paper Diversity and Inclusion Metrics in Subset Selection. The paper is tackling various fairness issues in a variety different settings. They first try to measure diversity through a metric and similarly for inclusion. The paper gives these concepts and how they are applied to a dataset. In the blog post, they extended the examples from the paper to recommendation systems. There is an amazing notebook that explains these concepts in very detailed way. There is also another good interactive visualization for measuring fairness. That page follows this paper. If you are interested in this area, there is also a guidebook from Google which explains these concepts and has a number of pointers to interactive pages and papers.
Sabrina Mielke wrote a long and detailed blog post on how Jax and PyTorch is compared with some of the design decisions on how PyTorch behaves and how Jax behaves.
MIT wrote about how major ML datasets have a number of labeling errors, especially some of the findings that they have for ImageNet is pretty interesting. There is a cleanlab Python package that finds these mislabeled examples. If you want to find out more about the package, there is another blog post that explains the motivation and how it can be used. There is a demo page that shows these problematic labels in here.
Papers
MasakhaNER: Named Entity Recognition for African Languages build a NER(Named Entity Recognition) for African languages. The code is available in here. The languages that are available: Amharic, Hausa, Igbo, Kinyarwanda, Luganda, Luo, Naija Pidgin, Swahili, Wolof and Yoruba.
Stephen Boyd published a document for Crimes Against Matrices. It is hilarious and worth a reading for common mistakes in linear algebra.
Real-time Data Infrastructure at Uber talks how Uber builds data infrastructure to support various applications. It has a strong focus on the low-latency as most of the applications need real-time data to be able to serve customers like dynamic pricing.
Do Transformer Modifications Transfer Across Implementations and Applications? argues that a number of incremental updates to the transformers do not really improve performance on the vanilla transformer. In order to prove this, they do have a lot of empirical results for modifications that do not improve the results substantially.
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval proposes a transformer based system to enable image/video retrieval for a given text query. The proposed approach beats SOTA with Recall@1, @5, @10 and MedianRank measures.
Libraries
WeightWatcher is a library that analyzes neural network weight layers and summarizes a number of statistics per layers for different types of neural networks. It supports both PyTorch and Tensorflow.
Elegy is a trainer framework based on Jax which is agnostic to model libraries of Jax. You can use Flax, Haiku, but also you can also use Elegy to build the model.
Notebooks
Siamese Network for image similarity is a very good notebook in Keras showing how you can use siamese network and triplet loss estimation to compare how similar two images are.
Videos
Ana Klimovic presents her work on tf.data. She first motivates the reader why data input layer is such a crucial part of efficiency in deep learning training jobs and then talks about how they speed up the data jobs through tf.data. Paper is also very interesting to read.
Christina Delimitrou presents her work on Seer. Seer is an end to end deep learning system that allows inference for performance predictability in a micro service environment. It ingests distributed tracing logs and predicts where there might be issues. The idea is that instead of paging a number of engineers after the incident occurs and Quality of Service(QoS) is impacted, proactive measures can be taken with the system. The paper is in here.