We introduce a framework based on bilevel programming that unifies gradient-based hyperparameter optimization and meta-learning. We show that an approximate version of the bilevel problem can be solved by taking into explicit account the optimization dynamics for the inner objective. Depending on the specific setting, the outer variables take either the meaning of hyperparameters in a supervised learning problem or parameters of a meta-learner. We provide sufficient conditions under which solutions of the approximate problem converge to those of the exact problem. We instantiate our approach for meta-learning in the case of deep learning where representation layers are treated as hyperparameters shared across a set of training episodes. In experiments, we confirm our theoretical findings, present encouraging results for few-shot learning and contrast the bilevel approach against classical approaches for learning-to-learn.

Bilevel Programming for Hyperparameter Optimization and Meta-Learning / Franceschi, Luca; Frasconi, Paolo; Salzo, Saverio; Grazzi, Riccardo; Pontil, Massimiliano. - ELETTRONICO. - 80:(2018), pp. 2537-2548. (Intervento presentato al convegno International Conference on Machine Learning tenutosi a Stockholm nel 10-15 July 2018).

Bilevel Programming for Hyperparameter Optimization and Meta-Learning

Frasconi, Paolo;
2018

Abstract

We introduce a framework based on bilevel programming that unifies gradient-based hyperparameter optimization and meta-learning. We show that an approximate version of the bilevel problem can be solved by taking into explicit account the optimization dynamics for the inner objective. Depending on the specific setting, the outer variables take either the meaning of hyperparameters in a supervised learning problem or parameters of a meta-learner. We provide sufficient conditions under which solutions of the approximate problem converge to those of the exact problem. We instantiate our approach for meta-learning in the case of deep learning where representation layers are treated as hyperparameters shared across a set of training episodes. In experiments, we confirm our theoretical findings, present encouraging results for few-shot learning and contrast the bilevel approach against classical approaches for learning-to-learn.
2018
Proceedings of the 35th International Conference on Machine Learning
International Conference on Machine Learning
Stockholm
10-15 July 2018
Franceschi, Luca; Frasconi, Paolo; Salzo, Saverio; Grazzi, Riccardo; Pontil, Massimiliano
File in questo prodotto:
File Dimensione Formato  
franceschi18a.pdf

accesso aperto

Descrizione: Articolo principale
Tipologia: Pdf editoriale (Version of record)
Licenza: Open Access
Dimensione 1.78 MB
Formato Adobe PDF
1.78 MB Adobe PDF

I documenti in FLORE sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificatore per citare o creare un link a questa risorsa: https://hdl.handle.net/2158/1150954
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 154
  • ???jsp.display-item.citation.isi??? ND
social impact