NeurIPS-20 Workshop on SpicyFL
Schedule: Saturday, December 12, 2020
Recorded Video: https://nips.cc/virtual/2020/protected/workshop_16123.html
All events are in Pacific Standard Time (PST)，UTC -8
|08:20AM – 08:30AM||Opening Remarks||General/Program Chairs|
|08:30AM – 09:00AM||Keynote Talk 1: Dawn Song Towards a Principled and Practical Approach for Data Valuation in Federated Learning [Bio] [Abstract] [Video]
Dawn Song is a Professor in the Department of Electrical Engineering and Computer Science at UC Berkeley. Her research interest lies in AI and deep learning, security and privacy. She is the recipient of various awards including the MacArthur Fellowship, the Guggenheim Fellowship, the NSF CAREER Award, the Alfred P. Sloan Research Fellowship, the MIT Technology Review TR-35 Award, and Best Paper Awards from top conferences in Computer Security and Deep Learning. She is an ACM Fellow and an IEEE Fellow. She is ranked the most cited scholar in computer security (AMiner Award). She obtained her Ph.D. degree from UC Berkeley. Prior to joining UC Berkeley as a faculty, she was a faculty at Carnegie Mellon University from 2002 to 2007. She is also a serial entrepreneur and has been named on the Female Founder 100 List by Inc. and Wired25 List of Innovators.
Data is valuable and a key driver of modern economy. However, how should data be valued? In this talk, we will present some recent work on data valuation. We will start by introducing a principled notion for data valuation and then present a suite of algorithms that we developed to efficiently compute the data value. We will then discuss how to adapt this notion to value data in federated learning settings. Finally, we discuss the implications of the proposed data value notion to enhance system robustness, security, and efficiency.
|09:00AM – 10:00AM||
|10:00AM – 10:30AM||Keynote Talk 2: H. Brendan McMahan Advances and Challenges in Private Cross-Device Federated Learning [Bio] [Abstract] [Video]
Brendan McMahan is a research scientist at Google, where he leads efforts on decentralized and privacy-preserving machine learning. His team pioneered the concept offederated learning, and continues to push the boundaries of what is possible when working with decentralized data using privacy-preserving techniques. Previously, he has worked in the fields of online learning, large-scale convex optimization, and reinforcement learning. Brendan received his Ph.D. in computer science from Carnegie Mellon University.
Privacy for users is a central goal of cross-device federated learning.This talk will begin with a broad view of privacy, highlighting key principles and threat models. We will then deep-dive into some recent advances in providing stronger privacy guarantees for cross-device federated learning, as well as highlight some perhaps under-appreciated challenges that arise when applying advanced techniques from differential privacy in the cross-device setting.
|10:30AM – 10:50AM||
|10:50AM – 11:20AM||Keynote Talk 3: Ruslan Salakhutdinov Efficient Transformers in Reinforcement Learning using Actor-Learner Distillation [Bio] [Abstract] [Video]
Ruslan Salakhutdinov received his PhD in machine learning (computer science) from the University of Toronto in 2009. After spending two post-doctoral years at the Massachusetts Institute of Technology Artificial Intelligence Lab, he joined the University of Toronto as an Assistant Professor in the Department of Computer Science and Department of Statistics. In February of 2016, he joined the Machine Learning Department at Carnegie Mellon University.Ruslan's primary interests lie in deep learning, machine learning, and large-scale optimization. His main research goal is to understand the computational and statistical principles required for discovering structure in large amounts of data. He is an action editor of the Journal of Machine Learning Research and served on the senior programme committee of several learning conferences including NIPS and ICML. He is an Alfred P. Sloan Research Fellow, Microsoft Research Faculty Fellow, Canada Research Chair in Statistical Machine Learning, a recipient of the Early Researcher Award, Connaught New Researcher Award, Google Faculty Award, Nvidia's Pioneers of AI award, and is a Senior Fellow of the Canadian Institute for Advanced Research.
Many real-world applications such as robotics provide hard constraints on power and compute that limit the viable model complexity of Reinforcement Learning (RL) agents. Similarly, in many distributed RL settings, acting is done on unaccelerated hardware such as CPUs, which likewise restricts model size to prevent intractable experiment run times. These “actor-latency” constrained settings present a major obstruction to the scaling up of model complexity that has recently been extremely successful in supervised learning. To be able to utilize large model capacity while still operating within the limits imposed by the system during acting, we develop an “Actor-Learner Distillation” (ALD) procedure that leverages a continual form of distillation that transfers learning progress from a large capacity learner model to a small capacity actor model. As a case study, we develop this procedure in the context of partially-observable environments, where transformer models have had large improvements over LSTMs recently, at the cost of significantly higher computational complexity. With transformer models as the learner and LSTMs as the actor, we demonstrate in several challenging memory environments that using Actor-Learner Distillation recovers the clear sample-efficiency gains of the transformer learner model while maintaining the fast inference and reduced total training time of the LSTM actor model.
(joint work with Emilio Parisotto)
|11:20AM – 11:50AM||
|11:50AM – 12:20PM||Keynote Talk 4: Virginia Smith On Heterogeneity in Federated Settings [Bio] [Abstract] [Video]
Virginia Smith is an assistant professor in the Machine Learning Department at Carnegie Mellon University, and a courtesy faculty member in the Electrical and Computer Engineering Department. Her research interests lie at the intersection of machine learning, optimization, and computer systems. A unifying theme of her research is to develop machine learning methods and theory that effectively leverage prior knowledge and account for practical constraints (e.g., hardware capabilities, network capacity, statistical structure). Specific topics include: large-scale machine learning, distributed optimization, resource-constrained learning, multi-task learning, transfer learning, and data augmentation.
A defining characteristic of federated learning is the presence of heterogeneity, i.e., that data and compute may differ significantly across the network. In this talk I show that the challenge of heterogeneity pervades the machine learning process in federated settings, affecting issues such as optimization, modeling, and fairness. In terms of optimization, I discuss FedProx, a distributed optimization method that offers robustness to systems and statistical heterogeneity. I then explore the role that heterogeneity plays in delivering models that are accurate and fair to all users/devices in the network. Our work here extends classical ideas in multi-task learning and alpha-fairness to large-scale heterogeneous networks, enabling flexible, accurate, and fair federated learning.
|12:20PM – 12:36PM||
|12:36PM – 01:30PM||Poster Session 1 (Papers presented in the morning)||Yanlin Zhou|
|01:30PM – 02:00PM||Keynote Talk 5: John C. Duchi Some thoughts on Optimality in Optimization [Bio] [Abstract] [Video]
John Duchi is an assistant professor of Statistics and Electrical Engineering and (by courtesy) Computer Science at Stanford University. His work spans statistical learning, optimization, information theory, and computation, with a few driving goals. (1) To discover statistical learning procedures that optimally trade between real-world resources---computation, communication, privacy provided to study participants---while maintaining statistical efficiency. (2) To build efficient large-scale optimization methods that address the spectrum of optimization, machine learning, and data analysis problems we face, allowing us to move beyond bespoke solutions to methods that robustly work. (3) To develop tools to assess and guarantee the validity of---and confidence we should have in---machine-learned systems.
I will survey and give perspective on a few recent (and some not so recent) results on optimality in optimization problems. These have some bearing on optimization problems arising in federated learning, where I will try to make a few polemical connections and suggestions for future work.
|02:00PM – 03:00PM||
|03:00PM – 03:30PM||Break|
|03:30PM – 04:00PM||Keynote Talk 6: Tao Yang: Privacy-aware Ranking for Document Search on the Cloud [Bio] [Abstract] [Video]
Tao Yang is a full Professor of Computer Science at University of California at Santa Barbara. His most recent research is in the areas of information retrieval and privacy. His past work has been in the areas of web search and parallel/distributed computing. He was Chief Scientist for Ask.com from 2001 to earlier 2010, and its Senior Vice President and Vice President of Engineering as the head of its search division. He visited Microsoft Bing for search technology R&D from 2010 to 2011.
As sensitive information is increasingly centralized into the cloud, for the protection of data privacy, such data is often encrypted, which makes effective data indexing and search a very challenging task. Hiding feature information through encryption prevents the server from performing effective scoring and result comparison. On the other hand, unencrypted feature values can lend themselves to privacy attacks. This talk will present our recent efforts in privacy-aware searching of large datasets with linear, tree-based, and neural ranking methods.
|04:00PM – 04:20 PM||
|04:20PM – 04:50PM||Keynote Talk 7: Tong Zhang [Bio] [Abstract] [Video]
Tong Zhang is a professor of Computer Science and Mathematics at The Hong Kong University of Science and Technology. Previously, he was a professor at Rutgers university, and worked at IBM, Yahoo, Baidu, and Tencent.
|04:50PM – 05:00PM||
|05:00PM – 06:00PM||Panel Discussion
Liefeng Bo [Bio]
Dr. Bo is chief scientist and head of AI Lab at JD Digits. He and his team are developing advanced AI technologies to improve the products for better serving hundreds of millions of JD Digits’ customers. He was principal scientist at Amazon, where he led a research team at Amazon Go for building a new grab-and-go shopping experience using computer vision, deep learning and sensor fusion technologies.
Xiaolin Li [Bio]
Heiko Ludwig [Bio]
Heiko Ludwig is a Principle Research Staff Member and Senior Manager of the AI Platforms department in IBM’s Almaden Research Center in San Jose, CA. Heiko is leading research work on computational platforms for AI, focusing on security, privacy, performance and reliability of machine learning and inference. He leads IBM Research’s work on federated learning. Heiko has worked on problems of distributed systems and artificial intelligence in his career, publishing more than 100 refereed articles and conference papers. He is an ACM Distinguished Engineer and a managing editor of the International Journal of Cooperative Information Systems. The results of his work contributes to various IBM lines of business and open source projects. Prior to the Almaden Research Center, Heiko held different positions at IBM Research in Switzerland, the US, Argentina and Brazil. He holds a Master’s (Diplom) degree and a PhD in information systems from Otto-Friedrich University Bamberg, Germany.
Jason Martin [Bio]
Jason Martin is a Principal Engineer in the Security Solutions Lab and manager of the Secure Intelligence Team at Intel Labs. He leads a team of diverse researchers to investigate machine learning security and privacy in a way that incorporates the latest research findings and Intel products. Jason’s interests include machine learning, authentication and identity, trusted execution technology, wearable computing, mobile security, and privacy. Prior to Intel Labs he spent several years as a security researcher performing security evaluations and penetration tests on Intel’s products.
|06:00PM – 07:00PM||Poster Session 2 (Papers presented in the afternoon)||Yanlin Zhou|
|07:00PM – 07:10PM||Closing Remarks||General/Program Chairs|
• Paper List of Lightning Talks
|10:30AM – 10:50AM||Lightning Talk Session 1|
|12:20PM – 12:36PM||Ligtning Talk Session 2|
|04:00PM – 04:20 PM||Ligtning Talk Session 3|
|04:50PM – 05:00PM||Ligtning Talk Session 4|
• Poster Sessions
- Poster Session 1 includes all papers presented before 12:36 PM
- Poster Session 2 includes all papers presented before 18:00 PM