Експериментальне обгрунтування якості градієнтних методів навчання нейронних мереж

Abstract

Мета дослідження – розробити програмний засіб, що дозволяє дослідити і порівняти експериментально ефективність різних варіантів програмної реалізації алгоритмів, що використовують градієнт в якості напрямку спуску. Варіанти, пов’язані з різноманітними методами побудови напрямків спуску з використанням напрямку, що визначає градієнт.

The greatest interest in the gradient method in recent years is due to the fact that gradient descents and their stochastic or randomized variants underlie almost all modern learning algorithms developed in data analysis. Most optimization algorithms come from propositions that have access to an exact gradient or hessian. In practice, there is usually only a noisy or even biased estimate of these values. Almost all deep learning algorithms are described on sample estimates, by extreme networks, in terms of the use of mini-packages of learning examples to calculate the gradient. It also happens that the objective function that minimizes is one that has no computational solution. In this case, there is usually no computational solution to the gradient calculation problem, and then only the approximate gradient remains. Such problems most often arise in complex models, for example, the algorithm of the compared distribution (contractual divergence) offers a method of approximation of gradient functions of logarithmic plausible mechanical engineering. Various neural network optimization algorithms have been developed to compensate for inaccurate gradient estimation. The problem can also be found by choosing a surrogate loss function that is even more approximate than true. In any case, gradient descents and their stochastic variants underlie almost all modern learning algorithms. The purpose of the study is to develop a software tool that allows you to investigate and compare experimentally the effectiveness of different options for software implementation of algorithms that use the gradient as the direction of descent. The options involve a variety of methods for constructing descent directions using a gradient direction. The object of research is the features of software implementation of algorithms that use the gradient as the direction of descent. The subject of the research is the issues related to the comparative analysis of the most common and wellfounded technologies of selection and calculation of directions and steps that use algorithms that correspond to the above scheme. The main research method that determines the research technology is the method of computational experiments applied to nonlinear unconditional optimization problems. Important tools for the implementation of research methods are also .NET programming technologies, software models for building visual graphical representations and data aggregation. Research methods are implemented with the help of programming patterns based on .NET technologies. Methods of unconditional optimization is one of the important directions in engineering practice. The basic numerical method of unconditional optimization are methods that use the gradient of the function as the direction of descent. The software product described in this paper allows to obtain and substantiate recommendations for the choice of a set of parameters of nonlinear optimization methods. The recommendations are based on experiments with the most complex functions for descent methods. Programs based on similar principles and having the above characteristics are not known.