Parti: Pathways AutoRegressive Text-to-Image

Techniques for training large neural networks

Articles

OpenAI wrote about different types of parallelisms and some of the best practices of training large deep neural networks in this blog post.

It talks about various parallelisms such as:

• Data parallelism—run different subsets of the batch on different GPUs;

• Pipeline parallelism—run different layers of the model on different GPUs;

• Tensor parallelism—break up the math for a single operation such as a matrix multiplication to be split across GPUs;

• Mixture-of-Experts—process each example by only a fraction of each layer.

Google released another high-image model generator based on input similar to Imagegen, called Parti(Pathways Autoregressive Text-to-Image model).

This model exploits sequence to sequence model in an autoregressive way rather than diffusion which is what Imagegen model was leveraging.

It is interesting that both methods leverage ViT-VQGAN to learn image tokens in a vocabulary and transformer encoder-decoder duo takes care of the input sequence which is input text to an output sequence which is the image.

Google also does further analysis based on the parameter numbers within the model, results are not as surprising. The larger the model parameters, the better model is.

Detailed comparisons of four scales of Parti models – 350M, 750M, 3B and 20B – and observe:

• Consistent and substantial improvements in model capabilities and output image quality.

• When comparing the 3B and 20B models, human evaluators preferred the latter most of the time, specifically:

  • 63.2% for image realism/quality

  • 75.9% for image-text match

• The 20B model especially excels at prompts that are abstract, require world knowledge, specific perspectives, or writing and symbol rendering.

Some of the examples(albeit cherry-picked) are very impressive.

HuggingFace wrote a very good article on the paper of Denoising Diffusion Probabilistic Models(DDPM). They have a number of good code snippets that go along with that post as well.

Netflix wrote about all of the work that they do for causal inference in this blog post. If you want to get a recent work that has been happening for causal inference, it is an excellent survey.