Machine Learning with Systems Science


Fall 2019: 12.S592  9Cr(2-1-6) U/G

Lec: Fr. 1000-1200, 54-1623
Rec: Fr. 1300-1400, 54-1827

 

Sai Ravela (ravela@mit.edu)

Follow the current course  on Stellar by clicking here

This course looks closely at the interaction of theory and data in learning, interprets learning from a systems dynamics and optimization perspective, and applies learning to systems dynamics and optimization problems. In this sense, it is a distinct course that complements other courses at the institute and elsewhere. This course including its predeessor has been taught since 2012. It is an "infinite course," continuing each term round-robining among topics. The entire course is organized in four parts; Preparatory, Foundations, Methods and Advanced Topics. Preparatory topics and Foundations  are covered every term. Methods are covered in two ways: detailed discussion of one topical area in class and broad overview of the remaining (with implementation examples) rotated term-by-term from the list in the Recitation part. Advanced Topics are covered by specific participant interest. So, please attend the first two lectures, the topics set quickly!

Participation

Participants will be required to complete bi-weekly assignments or a project and they may choose to do both.  In this course, my job is to help bring you to the fundamentals and your job is to take that understanding to your project, application or field of study. Prior experience with Probability, Linear Algebra, Statistics and a Data Science course would be perfect or seek the permission of the instructor.

 

Preparatory Material (covered every term)

  • Supervised, Semi-supervised and Unsupervised Learning
  • Regression, Classification, Clustering
  • Loss, Empirical Loss
  • Generalization and Extrapolation
  • Training-Validation-Testing
  • Model Selection, Feature Selection, Data Selection
  • Solving hard problems by Sampling

Foundations

  • Learning Dilemmas: Bias-Variance, Invariance-Selectivity
  • Variational and Bayesian Inference
  • Localization (incl. Kernels, Nearest Neighbors etc.)
  • Measures of Information
  • Regularization via Reproducing Kernel Hilbert Spaces (incl. Smoothness, Tiknonov)
  • Sparsity, Parsimony and Natural Statistics
  • Information Loss and Gain
  • Invariance and Symmetry
  • Relevance Feedback
  • Dynamic Programming
  • Systems Science Problem Solving (select topics by participant interest): We are motivated at present  by problems in applications across the breadth of Nature pertaining to

    • Estimation
    • Detection
    • Control
    • Prediction (Model Reduction)
    • Parameterization
    • Upscaling/Downscaling
    • Characterization
    • Discovery
    • Decision Making. Detection

Methods (round-robin) 

  • PCA/LDA/ICA
  • MCMC, Hamiltonian-MC
  • Sequential and Hierarchical Bayesian estimation
  • Two point boundary value problems 
  • Exponential families, Mixtures, EM
  • Graphical Models
  • Kernel Machines
  • Manifold Learning
  • Graph Spectra
  • Ensemble Learning
  • Active, Incremental, Online Learning 
  • Relevance Feedback
  • Deep (Neural) Learning
  • Reinforcement Learning
  • Information Theoretic Learning
  • Causal Learning
  • Additional Topics
    • Transfer Learning
    • Basis and Dictionary Learning

Advanced Topics (selected topics by participant interest):

  • Dynamics of Learning
  • Optimization in Learning
  • Sizing Learning Machines
  • Invariance and Symmetry in Learning
  • Information Transfers in Learning
  • Predictability and Learnability
  • Learning and Renormalization
  • Theory-driven Learning

Applications

Data Assimilation, Autonomous Environmental Mapping, Model Reduction, Uncertainty Quantification, Sensor Planning, Prediction and Predictability, Planning for Risk Mitigation, Convective Super-parameterization, Radiative-Convective Equilibrium, Nonlocal Operators, Teleconnections, Particle Detection and Sizing, Species Characterization, Paleoclimate, Event Detection and Tracking in X (Voclanoes, Earthquakes, Hurricanes, Tsunamis, Storms and Transits). Super-resolution/Downscaling, Coherent Structures in Turbulence,  Seismic Imaging and Geomorphology, Porus Media, Reservoirs, Exoplanets. However, other Engineering, Science and Finance applications may be included depending on participant interest. Whilst our motivation and reach is broad, we will drill down each term to a few core applications that is set by participant interest.

Books

  1. C. M. Bishop, "Pattern Recognition and Machine Learning"
  2. T. Hastie et al. The Elements of Statistical  Learning
  3. I. Goodfellow et al., Deep Learning
  4. S. Raschka, Python Machine Learning

 

Sai Ravela (ravela@mit.edu)