DAY 1
Introduction to Deep Learning
AI versus ML versus Deep Learning • "Classical" Machine Learning • Deep Learning • Supervised Learning • Unsupervised Learning • Neural Networks • Cloud versus Edge Computing • Applications • Libraries/Frameworks for Training • Cloud Platforms for Training and Inference • Vendor Platforms for Deploying ML / DL
Jupyter Notebook
AWS Deep Learning AMI • Connect to Remote Machine using SSH • Using Jupyter Notebook • Basic Markdown • Output and Evaluating Expressions • Expanding, Collapsing, Hiding Output • Menus and Tool Bar
Python Basics
Functions, Variables, and Values • Control Statements • Operators • Imports • Instance Objects • Kwd arguments • Range • Format • Tuples and Lists • Functions returning Functions • Numpy Array • Multi-dimensional Numpy array • Reshape • Slices • Elementwise and Scalar Operations • One-Hot Encoding • Reduction Operations • matplotlib.pyplot • List Operations
Linear Regression
Regression Task • Defining a Model • Cost / Loss / Error Function • Cost as a Function of Trainable Parameters • Mathematical Optimization • Contour Plot of Cost Function • Gradient Descent • Stochastic Gradient Descent
Keras
TensorFlow and Keras • Versions of Keras • A Keras Sequential Model • Datasets • Gradient Descent in Keras • History Object • Plotting Progress • Model, Optimizer, and Weights
DAY 2
Logistic Regression
Classification Task • One-Hot Labels • The Hypothesis or Model • Calculating the Cost Function • Converting Scores to Probabilities • The Softmax Function • Compare using Cross-Entropy • Multinomial Logistic Regression • Plotting the Decision Boundary • Choosing the Loss Function
Neural Networks
Neural Networks • An Artificial Neuron • Common Activation Functions • A Deep Neural Network • Forward and Back-Propagation • Kinds of Neural Network
Non-linear Regression
Linear Regression • A Non-Linear Polynomial Model • The Rectified Linear Unit (ReLU) • Normalizing the Data • Exploding and Vanishing Gradients • Varying the Weight Distribution • Xavier Glorot Initialization • Non-Linear Keras Model • The Magic of Deep Neural Networks
Non-linear Classification
A Non-Linear Decision Boundary • Decision Boundary and Softmax • Non-Linear Neural Network for Classification • From ReLU to Decision Boundary • Softmax
Overfitting and Regularization
Training versus Test Datasets • Scikit-learn • Learning Curves • Matching the Network to the Problem • How to Reduce Overfitting? • More Data? • Regularization (L2 Regularization) • Choosing Lambda • L2 versus L1 Regularization
DAY 3
Stochastic Gradient Descent
Full-Batch vs Stochastic Gradient Descent • Mini-Batches • The Landscape of the Cost Function • Stationary Points • Learning Rate • Learning Rate Decay Schedule • Momentum • Nesterov Momentum • Adaptive Per-Parameter Learning Rates • Adam Algorithm
Splitting the Dataset
The MNIST Dataset • A Deep Neural Network for Classification • Hyperparameters • Training, Validation, and Test Datasets • K-Fold Cross-Validation • Validatation • Choose a Single Scalar Metric • Imbalanced Classes or Rare Events • ROC Curve • Trading off Precision and Recall
Convolutional Neural Networks
Convolution • Patch Size and Stride • Valid Padding versus Same Padding • Network Size • Multiple Features Maps • Pooling • Stride Versus Receptive Field • Hierarchical Feature Detection • Filter Visualization • Number of Parameters and Values • Plotting Convolution Filters
Visualization using TensorBoard
Visualizing a Graph in TensorBoard • Visualizing Scalars in TensorBoard • Visualizing Multiple Runs • Visualizing Weights and Activations • Highlighting a Histogram • Visualizing an Embedding in 3-D Space • Keras fit versus fit_generator • Using TensorBoard from Keras
PCA, t-SNE, and K-Means
Dimensionality Reduction • Principal Component Analysis • PCA for Visualization • t-SNE for Visualization • Clustering with K-Means • PCA for Dimensionality Reduction • Linear Regression Out-of-the-Box • Normalizing the Dataset
DAY 4
Data Preparation
The Deep Learning Process • CRISP-DM • Feature Engineering • Correlated Features, Missing or Bad Values • Choose Meaningful Features • Avoid Magic Values • Pandas Dataframe • Pandas Summary Statistics • Pandas Scatter Matrix • Cleaning Data with Pandas • TensorFlow Extended (TFX) • Error Analysis • Artificial Data Synthesis • Data Augmentation
Dropout and Batch Normalization
Dropout • When to use Dropout? • Batch Normalization • Scale-and-Shift • Benefits of Batch Normalization • Calculating the Scaling Factors
Inception and Residual Networks
General Principles of Network Architecture • Evolution of CNN Architectures • Principles of the Inception Architecture • Fully-Connected versus Sparse • Inception Module • Global Average Pooling • Fully Convolutional Network • Residual Networks • Matching Dimensions • Performance of Inception and ResNet
Transfer Learning
Why Transfer Learning? • Re-use trained weights • Simple Transfer Learning in Keras • A Pre-trained Inception Network • Fine-Tuning Previous Layers • Saving and Loading • Make Each Class Visually Distinct • Other Tips for Image Data
DAY 5
Pruning and Quantization
Inference Engines at the Edge • ML / DL Tool Flow for Edge Computing • Neural Network Exchange Formats • Network Pruning • Quantization • 8-Bit Quantization • TensorFlow Lite Post-Training Quantization • OpenCV • OpenVX • OpenCL
MobileNet
MobileNet • Depthwise-Separable Convolution • MobileNet V1 Architecture • MobileNet V2 Architecture • Hyperparameters • MobileNet Family • MobileNet Inference in TensorFlow • CNNs Compared
Encoding and Word Embedding
Coding Categorical Data • Binning • Text in a Neural Network • Word Embedding and Semantics • Hidden Layers and Latent Features • The Word2vec Algorithm • Negative Sampling Neural Network • Preparing the Training Dataset • Frequency Counts
Recurrent Neural Networks
Recurrent Neural Network (RNN) • RNN Applications • Long Short Term Memory – LSTM • LSTM Gates • LSTM Connections • Gated Recurrent Unit – GRU • Simple Character-Level RNN in Keras • LSTM trained on Linux Source Code • Bidirectional LSTM • Networks for Images, Sound, Text, Video
GANs
Putting Networks End-to-End • Generative Adversarial Network (GAN) • Deep Convolutional GAN Generator • Generated Images • GAN to Generate MNIST Digits
Object Detection
Datasets for Object Detection • Google Open Images Dataset • Networks for Object Detection • Object Detection – Faster R-CNN • Training a Faster R-CNN Network • Image Segmentation using Mask R-CNN • Pixel-Level Image Segmentation
