Monday 22 April 2019

Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow

Part I. The Fundamentals of Machine Learning 1. The Machine Learning Landscape. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 What Is Machine Learning? 4 Why Use Machine Learning? 4 Types of Machine Learning Systems 8 Supervised/Unsupervised Learning 8 Batch and Online Learning 15 Instance-Based Versus Model-Based Learning 18 Main Challenges of Machine Learning 24 Insufficient Quantity of Training Data 24 Nonrepresentative Training Data 26 Poor-Quality Data 27 Irrelevant Features 27 Overfitting the Training Data 28 Underfitting the Training Data 30 Stepping Back 30 Testing and Validating 31 Hyperparameter Tuning and Model Selection 32 Data Mismatch 33 Exercises 34 2. End-to-End Machine Learning Project. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Working with Real Data 38 Look at the Big Picture 39 iii Frame the Problem 39 Select a Performance Measure 42 Check the Assumptions 45 Get the Data 45 Create the Workspace 45 Download the Data 49 Take a Quick Look at the Data Structure 50 Create a Test Set 54 Discover and Visualize the Data to Gain Insights 58 Visualizing Geographical Data 59 Looking for Correlations 62 Experimenting with Attribute Combinations 65 Prepare the Data for Machine Learning Algorithms 66 Data Cleaning 67 Handling Text and Categorical Attributes 69 Custom Transformers 71 Feature Scaling 72 Transformation Pipelines 73 Select and Train a Model 75 Training and Evaluating on the Training Set 75 Better Evaluation Using Cross-Validation 76 Fine-Tune Your Model 79 Grid Search 79 Randomized Search 81 Ensemble Methods 82 Analyze the Best Models and Their Errors 82 Evaluate Your System on the Test Set 83 Launch, Monitor, and Maintain Your System 84 Try It Out! 85 Exercises 85 3. Classi€cation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 MNIST 87 Training a Binary Classifier 90 Performance Measures 90 Measuring Accuracy Using Cross-Validation 91 Confusion Matrix 92 Precision and Recall 94 Precision/Recall Tradeoff 95 The ROC Curve 99 Multiclass Classification 102 Error Analysis 104 iv | Table of Contents Multilabel Classification 108 Multioutput Classification 109 Exercises 110 4. Training Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 Linear Regression 114 The Normal Equation 116 Computational Complexity 119 Gradient Descent 119 Batch Gradient Descent 123 Stochastic Gradient Descent 126 Mini-batch Gradient Descent 129 Polynomial Regression 130 Learning Curves 132 Regularized Linear Models 136 Ridge Regression 137 Lasso Regression 139 Elastic Net 142 Early Stopping 142 Logistic Regression 144 Estimating Probabilities 144 Training and Cost Function 145 Decision Boundaries 146 Softmax Regression 149 Exercises 153 5. Support Vector Machines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 Linear SVM Classification 155 Soft Margin Classification 156 Nonlinear SVM Classification 159 Polynomial Kernel 160 Adding Similarity Features 161 Gaussian RBF Kernel 162 Computational Complexity 163 SVM Regression 164 Under the Hood 166 Decision Function and Predictions 166 Training Objective 167 Quadratic Programming 169 The Dual Problem 170 Kernelized SVM 171 Online SVMs 174 Table of Contents | v Exercises 175 6. Decision Trees. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 Training and Visualizing a Decision Tree 177 Making Predictions 179 Estimating Class Probabilities 181 The CART Training Algorithm 182 Computational Complexity 183 Gini Impurity or Entropy? 183 Regularization Hyperparameters 184 Regression 185 Instability 188 Exercises 189 7. Ensemble Learning and Random Forests. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 Voting Classifiers 192 Bagging and Pasting 195 Bagging and Pasting in Scikit-Learn 196 Out-of-Bag Evaluation 197 Random Patches and Random Subspaces 198 Random Forests 199 Extra-Trees 200 Feature Importance 200 Boosting 201 AdaBoost 202 Gradient Boosting 205 Stacking 210 Exercises 213 8. Dimensionality Reduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 The Curse of Dimensionality 216 Main Approaches for Dimensionality Reduction 218 Projection 218 Manifold Learning 220 PCA 222 Preserving the Variance 222 Principal Components 223 Projecting Down to d Dimensions 224 Using Scikit-Learn 224 Explained Variance Ratio 225 Choosing the Right Number of Dimensions 225 PCA for Compression 226 vi | Table of Contents Randomized PCA 227 Incremental PCA 227 Kernel PCA 228 Selecting a Kernel and Tuning Hyperparameters 229 LLE 232 Other Dimensionality Reduction Techniques 234 Exercises 235 9. Unsupervised Learning Techniques. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 Clustering 238 K-Means 240 Limits of K-Means 250 Using clustering for image segmentation 251 Using Clustering for Preprocessing 252 Using Clustering for Semi-Supervised Learning 254 DBSCAN 256 Other Clustering Algorithms 259 Gaussian Mixtures 260 Anomaly Detection using Gaussian Mixtures 266 Selecting the Number of Clusters 267 Bayesian Gaussian Mixture Models 270 Other Anomaly Detection and Novelty Detection Algorithms 274 Part II. Neural Networks and Deep Learning 10. Introduction to Arti€cial Neural Networks with Keras. . . . . . . . . . . . . . . . . . . . . . . . . . 277 From Biological to Artificial Neurons 278 Biological Neurons 279 Logical Computations with Neurons 281 The Perceptron 281 Multi-Layer Perceptron and Backpropagation 286 Regression MLPs 289 Classification MLPs 290 Implementing MLPs with Keras 292 Installing TensorFlow 2 293 Building an Image Classifier Using the Sequential API 294 Building a Regression MLP Using the Sequential API 303 Building Complex Models Using the Functional API 304 Building Dynamic Models Using the Subclassing API 309 Saving and Restoring a Model 311 Using Callbacks 311 Table of Contents | vii Visualization Using TensorBoard 313 Fine-Tuning Neural Network Hyperparameters 315 Number of Hidden Layers 319 Number of Neurons per Hidden Layer 320 Learning Rate, Batch Size and Other Hyperparameters 320 Exercises 322 11. Training Deep Neural Networks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325 Vanishing/Exploding Gradients Problems 326 Glorot and He Initialization 327 Nonsaturating Activation Functions 329 Batch Normalization 333 Gradient Clipping 338 Reusing Pretrained Layers 339 Transfer Learning With Keras 341 Unsupervised Pretraining 343 Pretraining on an Auxiliary Task 344 Faster Optimizers 344 Momentum Optimization 345 Nesterov Accelerated Gradient 346 AdaGrad 347 RMSProp 349 Adam and Nadam Optimization 349 Learning Rate Scheduling 352 Avoiding Overfitting Through Regularization 356 ℓ1 and ℓ2 Regularization 356 Dropout 357 Monte-Carlo (MC) Dropout 360 Max-Norm Regularization 362 Summary and Practical Guidelines 363 Exercises 364 12. Custom Models and Training with TensorFlow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367 A Quick Tour of TensorFlow 368 Using TensorFlow like NumPy 371 Tensors and Operations 371 Tensors and NumPy 373 Type Conversions 374 Variables 374 Other Data Structures 375 Customizing Models and Training Algorithms 376 Custom Loss Functions 376 viii | Table of Contents Saving and Loading Models That Contain Custom Components 377 Custom Activation Functions, Initializers, Regularizers, and Constraints 379 Custom Metrics 380 Custom Layers 383 Custom Models 386 Losses and Metrics Based on Model Internals 388 Computing Gradients Using Autodiff 389 Custom Training Loops 393 TensorFlow Functions and Graphs 396 Autograph and Tracing 398 TF Function Rules 400 13. Loading and Preprocessing Data with TensorFlow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403 The Data API 404 Chaining Transformations 405 Shuffling the Data 406 Preprocessing the Data 409 Putting Everything Together 410 Prefetching 411 Using the Dataset With tf.keras 413 The TFRecord Format 414 Compressed TFRecord Files 415 A Brief Introduction to Protocol Buffers 415 TensorFlow Protobufs 416 Loading and Parsing Examples 418 Handling Lists of Lists Using the SequenceExample Protobuf 419 The Features API 420 Categorical Features 421 Crossed Categorical Features 421 Encoding Categorical Features Using One-Hot Vectors 422 Encoding Categorical Features Using Embeddings 423 Using Feature Columns for Parsing 426 Using Feature Columns in Your Models 426 TF Transform 428 The TensorFlow Datasets (TFDS) Project 429 14. Deep Computer Vision Using Convolutional Neural Networks. . . . . . . . . . . . . . . . . . . 431 The Architecture of the Visual Cortex 432 Convolutional Layer 434 Filters 436 Stacking Multiple Feature Maps 437 TensorFlow Implementation 439 Table of Contents | ix Memory Requirements 441 Pooling Layer 442 TensorFlow Implementation 444 CNN Architectures 446 LeNet-5 449 AlexNet 450 GoogLeNet 452 VGGNet 456 ResNet 457 Xception 459 SENet 461 Implementing a ResNet-34 CNN Using Keras 464 Using Pretrained Models From Keras 465 Pretrained Models for Transfer Learning 467 Classification and Localization 469 Object Detection 471 Fully Convolutional Networks (FCNs) 473 You Only Look Once (YOLO) 475 Semantic Segmentation 478 Exercises 482

https://www.knowledgeisle.com/wp-content/uploads/2019/12/2-Aur%C3%A9lien-G%C3%A9ron-Hands-On-Machine-Learning-with-Scikit-Learn-Keras-and-Tensorflow_-Concepts-Tools-and-Techniques-to-Build-Intelligent-Systems-O%E2%80%99Reilly-Media-2019.pdf

Speech and Language Processing An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition

Speech and Language Processing An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition Third Editi...