Title: Federated Low-rank Learning

Speaker: Diego Klabjan, Northwestern University

Abstract: Federated Learning (FL) faces significant challenges related to communication efficiency and performance reduction when scaling to many clients. To address these issues, we explore the potential of using low-rank updates and provide the first theoretical study of rank properties in FL. Our theoretical analysis shows that a client's loss exhibits a higher-rank structure (i.e., gradients span higher-rank subspaces of the Hessian) compared to the server's loss, and that low-rank approximations of the clients' gradients have greater similarity. Consequently, we propose FedLoRU, a general low-rank update framework for FL. Our framework enforces low-rank client-side updates and accumulates these updates to form a higher rank model. Experimental results demonstrate that FedLoRU performs comparably to full-rank algorithms and exhibits robustness.

Joint work with Haemin Park, PhD candidate in IEMS

Zoom: https://northwestern.zoom.us/j/98732771280

-----

To subscribe to the Applied Mathematics Colloquia List send a message to LISTSERV@LISTSERV.IT.NORTHWESTERN.EDU with the command:

SUBSCRIBE esam-seminar FirstName LastName