A Flexible Distributed Optimization Scheme with Asynchronous, Scarse, and Sparse Communications

We present an asynchronous algorithm for distributed convex optimization when the objective writes a sum of smooth functions, local to each worker, and a non-smooth function. This kind of problem typically appears when learning over distributed data. Unlike many existing methods, our algorithm is adjustable to various levels of communication cost, delays, machines computational power. Moreover, for L1-regularized problems, this algorithm identifies near-optimal sparsity patterns and leverages on it to improve the efficiency of communications. (joint work with Dmitry Grishchenko, Konstantin Mishchenko, Jerome Malick, Massih Amini)