Fall 2019: 12.S592 9Cr(2-1-6) U/G
Lec: Fr. 1000-1200, 54-1623
Rec: Fr. 1300-1400, 54-1827
Sai Ravela (ravela@mit.edu)
Follow the current course on Stellar by clicking here
This course looks closely at the interaction of theory and data in learning, interprets learning from a systems dynamics and optimization perspective, and applies learning to systems dynamics and optimization problems. In this sense, it is a distinct course that complements other courses at the institute and elsewhere. This course including its predeessor has been taught since 2012. It is an “infinite course,” continuing each term round-robining among topics. The entire course is organized in four parts; Preparatory, Foundations, Methods and Advanced Topics. Preparatory topics and Foundations are covered every term. Methods are covered in two ways: detailed discussion of one topical area in class and broad overview of the remaining (with implementation examples) rotated term-by-term from the list in the Recitation part. Advanced Topics are covered by specific participant interest. So, please attend the first two lectures, the topics set quickly!
Participation
Participants will be required to complete bi-weekly assignments or a project and they may choose to do both. In this course, my job is to help bring you to the fundamentals and your job is to take that understanding to your project, application or field of study. Prior experience with Probability, Linear Algebra, Statistics and a Data Science course would be perfect or seek the permission of the instructor.
Preparatory Material (covered every term)
- Supervised, Semi-supervised and Unsupervised Learning
- Regression, Classification, Clustering
- Loss, Empirical Loss
- Generalization and Extrapolation
- Training-Validation-Testing
- Model Selection, Feature Selection, Data Selection
- Solving hard problems by Sampling
Foundations
- Learning Dilemmas: Bias-Variance, Invariance-Selectivity
- Variational and Bayesian Inference
- Localization (incl. Kernels, Nearest Neighbors etc.)
- Measures of Information
- Regularization via Reproducing Kernel Hilbert Spaces (incl. Smoothness, Tiknonov)
- Sparsity, Parsimony and Natural Statistics
- Information Loss and Gain
- Invariance and Symmetry
- Relevance Feedback
- Dynamic Programming
Systems Science Problem Solving (select topics by participant interest): We are motivated at present by problems in applications across the breadth of Nature pertaining to
- Estimation
- Detection
- Control
- Prediction (Model Reduction)
- Parameterization
- Upscaling/Downscaling
- Characterization
- Discovery
- Decision Making. Detection
Methods (round-robin)
- PCA/LDA/ICA
- MCMC, Hamiltonian-MC
- Sequential and Hierarchical Bayesian estimation
- Two point boundary value problems
- Exponential families, Mixtures, EM
- Graphical Models
- Kernel Machines
- Manifold Learning
- Graph Spectra
- Ensemble Learning
- Active, Incremental, Online Learning
- Relevance Feedback
- Deep (Neural) Learning
- Deep Learning as a two-point boundary value problem
- Auto-Encoders
- Convolutional Networks
- Recurrent Networks
- Adversarial Learning
- Continuous-Time Networks
- Reinforcement Learning
- Information Theoretic Learning
- Causal Learning
- Additional Topics
- Transfer Learning
- Basis and Dictionary Learning
Advanced Topics (selected topics by participant interest):
- Dynamics of Learning
- Optimization in Learning
- Sizing Learning Machines
- Invariance and Symmetry in Learning
- Information Transfers in Learning
- Predictability and Learnability
- Learning and Renormalization
- Theory-driven Learning
Applications
Data Assimilation, Autonomous Environmental Mapping, Model Reduction, Uncertainty Quantification, Sensor Planning, Prediction and Predictability, Planning for Risk Mitigation, Convective Super-parameterization, Radiative-Convective Equilibrium, Nonlocal Operators, Teleconnections, Particle Detection and Sizing, Species Characterization, Paleoclimate, Event Detection and Tracking in X (Voclanoes, Earthquakes, Hurricanes, Tsunamis, Storms and Transits). Super-resolution/Downscaling, Coherent Structures in Turbulence, Seismic Imaging and Geomorphology, Porus Media, Reservoirs, Exoplanets. However, other Engineering, Science and Finance applications may be included depending on participant interest. Whilst our motivation and reach is broad, we will drill down each term to a few core applications that is set by participant interest.
Books
- C. M. Bishop, “Pattern Recognition and Machine Learning”
- T. Hastie et al. The Elements of Statistical Learning
- I. Goodfellow et al., Deep Learning
- S. Raschka, Python Machine Learning