Sparser, better, faster, stronger: automatic differentiation with a lot of zeros

This talk is an introduction to the efficient computation of Jacobian and Hessian matrices. Common wisdom tells us that such matrices are impractical because they require quadratic storage (unlike gradient vectors). However, for many applications in scientific machine learning, Jacobians and Hessians exhibit sparsity. This feature can both speed up their computation and lighten their storage requirements, unlocking second-order optimization in high dimension.

First I will recall the basics of automatic differentiation, and explain how Jacobians and Hessians are computed (one column at a time). Then I will show how sparsity unlocks the joint retrieval of multiple columns, drawing connections with graph coloring problems. Finally, I will describe the first pipeline for automatic sparse differentiation in a high-level open-source language (Julia), as well as a prototype in JAX.