Learning Algorithm

This area is usually called unconstrained optimization. It is a very useful technique in many natural sciences, social sciences, and engineering disciplines. In this framework, the goal is to minimize a real scalar objective function of $n$-dimensional vector $\vec x$ on the parameter space, where $n$ is the total number of free parameters.

A typical unconstrained optimization method can be better understood by imaging walking on an $n$-dimensional terrain described by the objective function and to look for the deepest valley from where one starts the exploration. One chooses every step in order to descend to a lower level. In general, one can eventually find a point at which the gradient is zero. Note that there is no guarantee to find the global minimum [86,41].

More precisely, given a real function $f(\vec x)$ of $n$ variables (i.e. $\vec x \in \mathbb{R}^n)$, we want to find a particular $\vec x_{min} \in \mathbb{R}^n$ such that

$$f(\vec x_{min})<f(\vec y)$$

for all $\vec y$ in the neighborhood of $\vec x_{min}$. $f(\vec x_{min})$ is called a local minimum of function $f$. Specifically, if $f$ is a continuous differentiable function, at $\vec x_{min}$ we must have,

$$\left. {\nabla f(\vec x)} \right\vert _{\vec x=\vec x_{min}}=0.$$