Jigsaw, Locked Image Tuning, Ithaca
Scaling Mixture Of Experts from Google, Goodhart's Law from OpenAI
Articles
Microsoft wrote about Jigsaw based on the following paper: Jigsaw: Large Language Models meet Program Synthesis, they introduce a new tool that can improve the performance of these large language models. Jigsaw deploys post-processing techniques that understand the programs’ syntax and semantics and then leverages user feedback to improve future performance. Jigsaw is designed to synthesize code for Python Pandas API using multi-modal inputs. After Codex from OpenAI becomes very successful, I think we will see more and more various domain use cases of these helper tools for SWEs and MLEs.
Use of conditional computation: rather than activating the whole network for every single input, different parts of the model are activated for different inputs. This paradigm has been featured in the Pathways vision and recent works on large language models, while it has not been well explored in the context of computer vision.
Google published a blog post on scaling vision in mixture of experts. They present V-MoE, a new vision architecture based on a sparse mixture of experts. They have also open-sourced the code to train sparse models and provided several pre-trained models.
Google wrote a blog post on adding language understanding to image models called Locked Image Tuning(LIT). There is an another website that allows you to play with the model as well. They have also a good notebook in here.
LinkedIn wrote about how they use Tensorflow.js to personalize performance of their system so that they can make various adjustments in their delivery system based on a number factors from the user in this blog post.
Deepmind’s blog post gives an overview of Ithaca, the deep neural network that can restore the missing text of damaged inscriptions, identify their original location, and help establish the date they were created. Ithaca is named after the Greek island in Homer’s Odyssey and builds upon and extends Pythia, their previous system that focused on textual restoration. The tool is also available in here for further experimentation.
OpenAI wrote about Goodhart’s law in the blog post. In here, they are talking about how important metrics are, but when that metric becomes a goal, it ceases to become useful to optimize and then they choose a proxy metric to resemble. However, especially in Reinforcement learning domain, this also turns out to be somehow hard to do, then they come up with the best-of-n sampling, also known as rejection sampling or reranking. They simply sample n times and take the one that scores the highest according to the proxy objective.
Libraries
FastTreeSHAP package is built based on the paper Fast TreeSHAP: Accelerating SHAP Value Computation for Trees published in NeurIPS 2021 XAI4Debugging Workshop. It is a fast implementation of the TreeSHAP algorithm in the SHAP package.
Novelty MiniGrid (NovGrid) is an extension of MiniGrid environment that allows for the world properties and dynamics to change according to a generalized novelty generator. The MiniGrid environment is a grid-world that facilitates reinforcement learning algorithm development with low environment integration overhead, which allows for rapid iteration and testing. In addition to necessary grid world objects of agents, floor, walls, and goals, MiniGrid implements actionable objects including doors, keys, balls, and boxes.
Ancient History relies on disciplines such as Epigraphy, the study of inscribed texts known as "inscriptions", for evidence of the thought, language, society and history of past civilizations. However, over the centuries many inscriptions have been damaged to the point of illegibility, transported far from their original location, and their date of writing is steeped in uncertainty. We present Ithaca, the first Deep Neural Network for the textual restoration, geographical and chronological attribution of ancient Greek inscriptions. Ithaca is designed to assist and expand the historian’s workflow: its architecture focuses on collaboration, decision support, and interpretability.
Neo is a visual analytics system that enables practitioners to flexibly author and interact with hierarchical and multi-output confusion matrices, visualize derived metrics, renormalize confusions, and share matrix specifications.