Implementation of Derivative-Free Optimization Methods Chapter One OBJECTIVE OF THE STUDY The overall aim of the study is to implement derivative-free algorithms in unconstrained problems. The specific objectives of this study are: to implement the finite difference approach for derivative in the Quasi-Newton algorithm

CHAPTER TWO

 LITERATURE REVIEW

 INTRODCTION

This section introduces the different algorithms that will be compared in this project. Consider an optimization problemMin   f(x) (2.1)

Several strategies can be considered when we are faced with the problems which do not allow utilization of direct derivatives of gradients. The first is to apply existing direct search optimization methods, like the well-known and widely used simplex reflection algorithm or its modern variants, of the parallel direct search algorithm. This first approach has the advantage of requiring little effort from the user; however, it may require substantial computing resources like a great number of function evaluations (Nocedal and Wright 2014). This is because it does not take into account the advantage of the objective function’s smoothness well enough.

The second approach is using automatic differentiation tools. Automatic differentiation is utilized to define computer programs which calculate the derivatives of a function by some procedures. These computer programs calculate the Jacobean of vector-valued functions which are from n-dimensional to m-dimensional Euclidean space, i.e., from m. On the other hand, if the function is scalar-valued, i.e., from n to R, then the computer program should calculate the gradient (and Hessian) of the function. However, such tools are not preferred in solving problems which we consider. This is mainly because in the automatic differentiation tools approach, the function to be differentiated is required to be the result of a callable program which cannot be treated as a black box (Xiaogang, Dong, Xingdong and Lu, 2006). A third possibility is to make use of finite difference approximation of the derivatives (gradients and possibly Hessian matrices). In general, given the cost of evaluating the objective function, evaluating its Hessian by finite differences is expensive. One can utilize quasi-Newton Hessian approximation techniques instead. Incorporating finite differences for computing gradients in conjunction with the quasi-Newton Hessian approximation techniques has proved to be helpful and sometimes surprisingly efficient. Indeed, the additional function evaluations required in the calculation of the derivatives is very costly and, most importantly, finite differencing can be unreliable in the presence of noise if the differentiation step is not adapted according to the noise level. The objective functions in the problems we consider are obtained from some simulation procedures; therefore, automatic differentiation tools are not applicable as mentioned above. This forces one to consider algorithms without preceding the approximation of the derivatives of the objective function at a given value. We will look at discrete gradients from non-smooth optimization, where the approach can be interpreted as an approximation or mimicking of derivatives. There are two important components of derivative free methods. Sampling better points in the iteration procedure is the first one of these components. The other one is searching appropriate subspaces where the chance of finding a minimum is relatively high. In order to be able to use the extensive convergence theory for derivative based methods, these derivative free methods need to satisfy some properties. For instance, to guarantee the convergence of a derivative free method, we need to ensure that the error in the gradient converges to zero when the trust-region or line-search steps are reduced. Hence, a descent step will be found if the gradient of the true function is not zero at the current iterate. To show this, one needs to prove that the linear or quadratic approximation models satisfy Taylor-like error bounds on the function value and the gradient. Finally, for our approach to derivative free optimization given by non-smooth optimization, we shall use so-called discrete gradients.

NESTEROV, (2011) work on Derivative Free optimization, and sighted Dixon, Himmelblau and Polyak for extensive discussions and references. These methods come essentially in four different classes, a classification strongly influenced by (Conn and Toint 1996). The first class contains algorithms which use finite-difference approximation of the objective function’s derivatives in the context of a gradient based method, such as nonlinear conjugate gradients or quasi-Newton methods (see, for instance (Nash and Sofer 1996)). He says that Derivative free optimization is a class of nonlinear optimization methods which usually comes to mind when one needs to apply optimization to complex systems. The complexity of those systems manifests itself in the lack of derivative information (exact or approximate) of the functions under consideration. What usually causes the lack of der