#Stochastics For Derivatives Modelling Online Help | Explore Tumblr posts and blogs

joeyrob1 · 5 years ago

Text

Top 10 Most Cited Machine Learning Articles

(No Ratings Yet) Loading...

Here, we provide you with our list of Top 10 Most Cited Machine Learning Articles based on info in CiteSeer database as of 19 March 2015.

1. Statistical Learning Theory

Author: V Vapnik in 1998

Citations: 9898

Synopsis: Statistical learning theory was introduced in the late 1960’s. Until the 1990’s it was a purely theoretical analysis of the problem of function estimation from a given collection of data. In the middle of the 1990’s new types of learning algorithms (called support vector machines) based on the theory were proposed. This made statistical learning theory not only a tool for the theoretical analysis but also a tool for creating practical algorithms for estimating multidimensional functions. This article presents a very general overview of statistical learning theory including both theoretical and algorithmic aspects of the theory. The goal of this overview is to demonstrate how the abstract learning theory established conditions for generalization which are more general than those discussed in classical statistical paradigms and how the understanding of these conditions inspired new algorithmic approaches to function estimation problems.

2. A tutorial on hidden Markov models and selected applications in speech recognition

Author: L R Rabiner in 1989

Citations: 4585

Synopsis: Although initially introduced and studied in the late 1960s and early 1970s, statistical methods of Markov source or hidden Markov modeling have become increasingly popular in the last several years. There are two strong reasons why this has occurred. First the models are very rich in mathematical structure and hence can form the theoretical basis for use in a wide range of use cases. Second, the models, when applied properly, work very well in practice for several important use cases. In this paper we attempt to carefully and methodically review the theoretical aspects of this type of statistical modeling and show how they have been applied to selected problems in machine recognition of speech.

3. Reinforcement Learning, an introduction

Author: R Sutton, A Barto in 1998

Citations: 4148

Synopsis: In this article, we try to give a basic intuitive sense of what reinforcement learning is and how it differs and relates to other fields, e.g., supervised learning and neural networks, genetic algorithms and artificial life, control theory. Intuitively, RL is trial and error (variation and selection, search) plus learning (association, memory). We argue that RL is the only field that seriously addresses the special features of the problem of learning from interaction to achieve long-term goals.

4. Libsvm: a library for support vector machines

Author: C-C Chang, C-J Lin.

Citations: 3829

Synopsis: LIBSVM is a library for Support Vector Machines (SVMs). We have been actively building this package since the year 2000. The goal is to help users to easily apply SVM to their user cases. LIBSVM has gained wide popularity in machine learning and many other areas. In this article, we present all implementation details of LIBSVM. Issues such as solving SVM optimization problems, theoretical convergence, multi-class classification, probability estimates, and parameter selection are discussed in detail

5. Induction of decision trees

Author: J R Quinlan in 1986

Citations: 3634

Synopsis: The technology for building knowledge-based systems by inductive inference from examples has been demonstrated successfully in several practical use cases. This paper summarizes an approach to synthesizing decision trees that has been used in a variety of systems, and it describes one such system, ID3, in detail. Results from recent studies show ways in which the methodology can be modified to deal with information that is noisy and/or incomplete. A reported shortcoming of the basic algorithm is discussed and two means of overcoming it are compared. The paper concludes with illustrations of current research directions.

6. Bagging predictors

Author: L Breiman in 1996

Citations: 2751

Synopsis: Bagging predictors is a method for generating multiple versions of a predictor and using these to get an aggregated predictor. The aggregation averages over the versions when predicting a numerical outcome and does a plurality vote when predicting a class. The multiple versions are formed by making bootstrap replicates of the learning set and using these as new learning sets. Tests on real and simulated data sets using classification and regression trees and subset selection in linear regression show that bagging can give substantial gains in accuracy. The vital element is the instability of the prediction method. If perturbing the learning set can cause significant changes in the predictor constructed, then bagging can improve accuracy.

7. A tutorial on support vector machines for pattern recognition

Author: C J C Burges in 1998

Citations: 2486

Synopsis: The tutorial starts with an overview of the concepts of VC dimension and structural risk minimization. We then describe linear Support Vector Machines (SVMs) for separable and non-separable data, working through a non-trivial example in detail. We describe a mechanical analogy, and discuss when SVM solutions are unique and when they are global. We describe how support vector training can be practically implemented, and discuss in detail the kernel mapping technique which is used to construct SVM solutions which are nonlinear in the data. We show how Support Vector machines can have very large (even infinite) VC dimension by computing the VC dimension for homogeneous polynomial and Gaussian radial basis function kernels. While very high VC dimension would normally bode ill for generalization performance, and while at present there exists no theory which shows that good generalization performance is guaranteed for SVMs, there are several arguments which support the observed high accuracy of SVMs, which we review. Results of some experiments which were inspired by these arguments are also presented. We give numerous examples and proofs of most of the key theorems. There is new material, and I hope that the reader will find that even old material is cast in a fresh light.

8. Support vector networks

Author: C Cortes, V Vapnik in 1995

Citations: 2375

Synopsis: The support-vector network is a new learning machine for two-group classification problems. The machine conceptually implements the following idea: input vectors are non-linearly mapped to a very high-dimension feature space. In this feature space a linear decision surface is constructed. Special properties of the decision surface ensures high generalization ability of the learning machine. The idea behind the supportvector network was previously implemented for the restricted case where the training data can be separated without errors. We here extend this result to non-separable training data.

9. Learning with kernels

Author: B Schölkopf, A J Smola in 2002

Citations: 2234

Synopsis: Kernel based algorithms such as support vector machines have achieved considerable success in various problems in the batch setting where all of the training data is available in advance. Support vector machines combine the so-called kernel trick with the large margin idea. There has been little use of these methods in an online setting suitable for real-time use cases. In this paper we consider online learning in a Reproducing Kernel Hilbert Space. By considering classical stochastic gradient descent within a feature space, and the use of some straightforward tricks, we build simple and computationally efficient algorithms for a wide range of problems such as classification, regression, and novelty detection. In addition to allowing the exploitation of the kernel trick in an online setting, we examine the value of large margins for classification in the online setting with a drifting target. We derive worst case loss bounds and moreover we show the convergence of the hypothesis to the minimiser of the regularised risk functional. We present some experimental results that support the theory as well as illustrating the power of the new algorithms for online novelty detection.

10. Text Categorization with Support Vector Machines: Learning with Many Relevant Features

Author: T Joachims in 1998

Citations: 1872

Synopsis: This paper explores the use of Support Vector Machines (SVMs) for learning text classifiers from examples. It analyzes the particular properties of learning with text data and identifies, why SVMs are appropriate for this task. Empirical results support the theoretical findings. SVMs achieve substantial improvements over the currently best performing methods and they behave robustly over a variety of different learning tasks. Furthermore, they are fully automatic, eliminating the need for manual parameter tuning.

Conclusion

Here, we provide you with our list of Top 10 Most Cited Machine Learning Articles based on info in CiteSeer database as of 19 March 2015.

Top 10 Most Cited Machine Learning Articles was originally published on RobustTechHouse - Mobile App Development Singapore

#Machine Learning #mobile #app #development #singapore

0 notes

theresawelchy · 6 years ago

Text

Neural Networks: Tricks of the Trade Review

Deep learning neural networks are challenging to configure and train.

There are decades of tips and tricks spread across hundreds of research papers, source code, and in the heads of academics and practitioners.

The book “Neural Networks: Tricks of the Trade” originally published in 1998 and updated in 2012 at the cusp of the deep learning renaissance ties together the disparate tips and tricks into a single volume. It includes advice that is required reading for all deep learning neural network practitioners.

In this post, you will discover the book “Neural Networks: Tricks of the Trade” that provides advice by neural network academics and practitioners on how to get the most out of your models.

After reading this post, you will know:

The motivation for why the book was written.

A breakdown of the chapters and topics in the first and second editions.

A list and summary of the must-read chapters for every neural network practitioner.

Let’s get started.

Neural Networks – Tricks of the Trade

Overview

Neural Networks: Tricks of the Trade is a collection of papers on techniques to get better performance from neural network models.

The first edition was published in 1998 comprised of five parts and 17 chapters. The second edition was published right on the cusp of the new deep learning renaissance in 2012 and includes three more parts and 13 new chapters.

If you are a deep learning practitioner, then it is a must read book.

I own and reference both editions.

Motivation

The motivation for the book was to collate the empirical and theoretically grounded tips, tricks, and best practices used to get the best performance from neural network models in practice.

The author’s concern is that many of the useful tips and tricks are tacit knowledge in the field, trapped in peoples heads, code bases, or at the end of conference papers and that beginners to the field should be aware of them.

It is our belief that researchers and practitioners acquire, through experience and word-of-mouth, techniques and heuristics that help them successfully apply neural networks to difficult real-world problems. […] they are usually hidden in people’s heads or in the back pages of space-constrained conference papers.

The book is an effort to try to group the tricks together, after the success of a workshop at the 1996 NIPS conference with the same name.

This book is an outgrowth of a 1996 NIPS workshop called Tricks of the Trade whose goal was to begin the process of gathering and documenting these tricks. The interest that the workshop generated motivated us to expand our collection and compile it into this book.

— Page 1, Neural Networks: Tricks of the Trade, Second Edition, 2012.

Breakdown of First Edition

The first edition of the book was put together (edited) by Genevieve Orr and Klaus-Robert Muller comprised of five parts and 17 chapters and was published 20 years ago in 1998.

Each part includes a useful preface that summarizes what to expect in the upcoming chapters, and each chapter written by one or more academics in the field.

The breakdown of this first edition was as follows:

Part 1: Speeding Learning

Chapter 1: Efficient BackProp

Part 2: Regularization Techniques to Improve Generalization

Chapter 2: Early Stopping – But When?

Chapter 3: A Simple Trick for Estimating the Weight Decay Parameter

Chapter 4: Controlling the Hyperparameter Search on MacKay’s Bayesian Neural Network Framework

Chapter 5: Adaptive Regularization in Neural Network Modeling

Chapter 6: Large Ensemble Averaging

Part 3: Improving Network Models and Algorithmic Tricks

Chapter 7: Square Unit Augmented, Radically Extended, Multilayer Perceptrons

Chapter 8: A Dozen Tricks with Multitask Learning

Chapter 9: Solving the Ill-Conditioning on Neural Network Learning

Chapter 10: Centering Neural Network Gradient Factors

Chapter 11: Avoiding Roundoff Error in Backpropagating Derivatives

Part 4: Representation and Incorporating PRior Knowledge in Neural Network Training

Chapter 12: Transformation Invariance in Pattern Recognition – Tangent Distance and Tangent Propagation

Chapter 13: Combining Neural Networks and Context-Driven Search for On-Line Printed Handwriting Recognition in the Newton

Chapter 14: Neural Network Classification and Prior Class Probabilities

Chapter 15: Applying Divide and Conquer to Large Scale Pattern Recognition Tasks

Part 5: Tricks for Time Series

Chapter 16: Forecasting the Economy with Neural Nets: A Survey of Challenges and Solutions

Chapter 17: How to Train Neural Networks

It is an expensive book, and if you can pick-up a cheap second-hand copy of this first edition, then I highly recommend it.

Want Better Results with Deep Learning?

Take my free 7-day email crash course now (with sample code).

Click to sign-up and also get a free PDF Ebook version of the course.

Download Your FREE Mini-Course

Additions in the Second Edition

The second edition of the book was released in 2012, seemingly right at the beginning of the large push that became “deep learning.” As such, the book captures the new techniques at the time such as layer-wise pretraining and restricted Boltzmann machines.

It was too early to focus on the ReLU, ImageNet with CNNs, and use of large LSTMs.

Nevertheless, the second edition included three new parts and 13 new chapters.

The breakdown of the additions in the second edition are as follows:

Part 6: Big Learning in Deep Neural Networks

Chapter 18: Stochastic Gradient Descent Tricks

Chapter 19: Practical Recommendations for Gradient-Based Training of Deep Architectures

Chapter 20: Training Deep and Recurrent Networks with Hessian-Free Optimization

Chapter 21: Implementing Neural Networks Efficiently

Part 7: Better Representations: Invariant, Disentangled and Reusable

Chapter 22: Learning Feature Representations with K-Means

Chapter 23: Deep Big Multilayer Perceptrons for Digit Recognition

Chapter 24: A Practical Guide to Training Restricted Boltzmann Machines

Chapter 25: Deep Boltzmann Machines and the Centering Trick

Chapter 26: Deep Learning via Semi-supervised Embedding

Part 8: Identifying Dynamical Systems for Forecasting and Control

Chapter 27: A Practical Guide to Applying Echo State Networks

Chapter 28: Forecasting with Recurrent Neural Networks: 12 Tricks

Chapter 29: Solving Partially Observable Reinforcement Learning Problems with Recurrent Neural Networks

Chapter 30: 10 Steps and Some Tricks to Set up Neural Reinforcement Controllers

Must-Read Chapters

The whole book is a good read, although I don’t recommend reading all of it if you are looking for quick and useful tips that you can use immediately.

This is because many of the chapters focus on the writers’ pet projects, or on highly specialized methods. Instead, I recommend reading four specific chapters, two from the first edition and two from the second.

The second edition of the book is worth purchasing for these four chapters alone, and I highly recommend picking up a copy for yourself, your team, or your office.

Fortunately, there are pre-print PDFs of these chapters available for free online.

The recommended chapters are:

Chapter 1: Efficient BackProp, by Yann LeCun, et al.

Chapter 2: Early Stopping – But When?, by Lutz Prechelt.

Chapter 18: Stochastic Gradient Descent Tricks, by Leon Bottou.

Chapter 19: Practical Recommendations for Gradient-Based Training of Deep Architectures, by Yoshua Bengio.

Let’s take a closer look at each of these chapters in turn.

Efficient BackProp

This chapter focuses on providing very specific tips to get the most out of the stochastic gradient descent optimization algorithm and the backpropagation weight update algorithm.

Many undesirable behaviors of backprop can be avoided with tricks that are rarely exposed in serious technical publications. This paper gives some of those tricks, and offers explanations of why they work.

— Page 9, Neural Networks: Tricks of the Trade, First Edition, 1998.

The chapter proceeds to provide a dense and theoretically supported list of tips for configuring the algorithm, preparing input data, and more.

The chapter is so dense that it is hard to summarize, although a good list of recommendations is provided in the “Discussion and Conclusion” section at the end, quoted from the book below:

– shuffle the examples – center the input variables by subtracting the mean – normalize the input variable to a standard deviation of 1 – if possible, decorrelate the input variables. – pick a network with the sigmoid function shown in figure 1.4 – set the target values within the range of the sigmoid, typically +1 and -1. – initialize the weights to random values as prescribed by 1.16.

The preferred method for training the network should be picked as follows: – if the training set is large (more than a few hundred samples) and redundant, and if the task is classification, use stochastic gradient with careful tuning, or use the stochastic diagonal Levenberg Marquardt method. – if the training set is not too large, or if the task is regression, use conjugate gradient.

— Pages 47-48, Neural Networks: Tricks of the Trade, First Edition, 1998.

The field of applied neural networks has come a long way in the twenty years since this was published (e.g. the comments on sigmoid activation functions are no longer relevant), yet the basics have not changed.

This chapter is required reading for all deep learning practitioners.

Early Stopping – But When?

This chapter describes the simple yet powerful regularization method called early stopping that will halt the training of a neural network when the performance of the model begins to degrade on a hold-out validation dataset.

Validation can be used to detect when overfitting starts during supervised training of a neural network; training is then stopped before convergence to avoid the overfitting (“early stopping”)

— Page 55, Neural Networks: Tricks of the Trade, First Edition, 1998.

The challenge of early stopping is the choice and configuration of the trigger used to stop the training process, and the systematic configuration of early stopping is the focus of the chapter.

The general early stopping criteria are described as:

GL: stop as soon as the generalization loss exceeds a specified threshold.

PQ: stop as soon as the quotient of generalization loss and progress exceeds a threshold.

UP: stop when the generalization error increases in strips.

Three recommendations are provided, e.g. “the trick“:

1. Use fast stopping criteria unless small improvements of network performance (e.g. 4%) are worth large increases of training time (e.g. factor 4). 2. To maximize the probability of finding a “good” solution (as opposed to maximizing the average quality of solutions), use a GL criterion. 3. To maximize the average quality of solutions, use a PQ criterion if the net- work overfits only very little or an UP criterion otherwise.

— Page 60, Neural Networks: Tricks of the Trade, First Edition, 1998.

The rules are analyzed empirically over a large number of training runs and test problems. The crux of the finding is that being more patient with the early stopping criteria results in better hold-out performance at the cost of additional computational complexity.

I conclude slower stopping criteria allow for small improvements in generalization (here: about 4% on average), but cost much more training time (here: about factor 4 longer on average).

— Page 55, Neural Networks: Tricks of the Trade, First Edition, 1998.

Stochastic Gradient Descent Tricks

This chapter focuses on a detailed review of the stochastic gradient descent optimization algorithm and tips to help get the most out of it.

This chapter provides background material, explains why SGD is a good learning algorithm when the training set is large, and provides useful recommendations.

— Page 421, Neural Networks: Tricks of the Trade, Second Edition, 2012.

There is a lot of overlap with Chapter 1: Efficient BackProp, and although the chapter calls out tips along the way with boxes, a useful list of tips is not summarized at the end of the chapter.

Nevertheless, it is a compulsory read for all neural network practitioners.

Below is my own summary of the tips called out in boxes throughout the chapter, mostly quoting directly from the second edition:

Use stochastic gradient descent (batch=1) when training time is the bottleneck.

Randomly shuffle the training examples.

Use preconditioning techniques.

Monitor both the training cost and the validation error.

Check the gradients using finite differences.

Experiment with the learning rates [with] a small sample of the training set.

Leverage the sparsity of the training examples.

Use a decaying learning rate.

Try averaged stochastic gradient (i.e. a specific variant of the algorithm).

Some of these tips are pithy without context; I recommend reading the chapter.

Practical Recommendations for Gradient-Based Training of Deep Architectures

This chapter focuses on the effective training of neural networks and early deep learning models.

It ties together the classical advice from Chapters 1 and 29 but adds comments on (at the time) recent deep learning developments like greedy layer-wise pretraining, modern hardware like GPUs, modern efficient code libraries like BLAS, and advice from real projects tuning the training of models, like the order to train hyperparameters.

This chapter is meant as a practical guide with recommendations for some of the most commonly used hyper-parameters, in particular in the context of learning algorithms based on backpropagated gradient and gradient-based optimization.

— Page 437, Neural Networks: Tricks of the Trade, Second Edition, 2012.

It’s also long, divided into six main sections:

Deep Learning Innovations. Including greedy layer-wise pretraining, denoising autoencoders, and online learning.

Gradients. Including mini-batch gradient descent and automatic differentiation.

Hyperparameters. Including learning rate, mini-batch size, epochs, momentum, nodes, weight regularization, activity regularization, hyperparameter search, and recommendations.

Debugging and Analysis. Including monitoring loss for overfitting, visualization, and statistics.

Other Recommendations. Including GPU hardware and use of efficient linear algebra libraries such as BLAS.

Open Questions. Including the difficulty of training deep models and adaptive learning rates.

There’s far too much for me to summarize; the chapter is dense with useful advice for configuring and tuning neural network models.

Without a doubt, this is required reading and provided the seeds for the recommendations later described in the 2016 book Deep Learning, of which Yoshua Bengio was one of three authors.

The chapter finishes on a strong, optimistic note.

The practice summarized here, coupled with the increase in available computing power, now allows researchers to train neural networks on a scale that is far beyond what was possible at the time of the first edition of this book, helping to move us closer to artificial intelligence.

— Page 473, Neural Networks: Tricks of the Trade, Second Edition, 2012.