@[toc]

Manual Hyperparameter Tuning

• Start with a good baseline, e.g. default settings in high-quality toolkits, values reported in papers

• Tune a value, retrain the model to see the changes

• Repeat multiple times to gain insights about

  • Which hyperparameters are important

  • How sensitive the model to hyperparameters

  • What are the good ranges

• Needs careful experiment management

• Save your training logs and hyperparameters to compare, share and
  reproduce later

  • The simplest way is saving logs in text and put key metrics in Excel

  • Better options exist, e.g. tenesorboard and weights & bias

• Reproducing is hard, it relates to

  • Environment (hardware & library)

  • Code

  • Randomness (seed)

Automated Machine Learning (AutoML)

• Automate every step in applying ML to solve real-world problems: data cleaning, feature extraction, model selection…
• Hyperparameter optimization (HPO):find a good set of hyperparameters
  through search algorithms
• Neural architecture search (NAS):construct a good neural network model

Summary

• Hyperparameter tuning aims to find a set of good values
• It’s time consuming as data preprocessing
• There is a trend to use algorithm for tuning