\[\newcommand{\u}[1]{\boldsymbol{\mathsf{#1}}} \renewcommand{\b}[1]{\boldsymbol{#1}} \newcommand{\t}[1]{\textsf{#1}} \newcommand{\m}[1]{\mathbb{#1}} \def\RR{\bf R} \def\bold#1{\bf #1} \def\mbf#1{\mathbf #1} \def\uv#1{\hat{\usf {#1}}} \def\dl#1{\underline{\underline{#1}}} \newcommand{\usf}[1]{\boldsymbol{\mathsf{#1}}} \def\bs#1{\usf #1}\]

Engineering Optimization: Basic definitions

(Global) minimizer, minimizer, (global) minimum point, minimum point: A point \((x^{i,*})\in \mathcal{D}\) is a minimizer of \(f\) on \(\mathcal{D}\) if

\(\begin{equation} f[(x^{i})] \ge f[(x^{i,*})] \end{equation}\) for all \((x^{i})\in \mathcal{D}\).

A function can have no, one, or many minimizers on its domain \(\mathcal{D}\).

• Many minimizers, \(f: [0,1]\to \mathbb{R}\), \(f(x)=\sin(100 x)\).

• No minimizers, \(f: (0,1]\to \mathbb{R}\), \(f(x)=x\).

• No minimizers, \(f: (-\pi/2,\pi/2)\to \mathbb{R}\), \(f(x)=\tan{x}\).

• Unique minimizers, \(f: [-1,1]\to \mathbb{R}\), \(f(x)=x^2\).

When the function has has no or many minimizers, then the problem of finding the minimizers is termed an ill-posed problem.

A function’s minimizers can lie either in the interior of the function’s domain or lie on its boundary.

• Minimizer lying on the boundary of the domain: \(f: [-1,1]\to \mathbb{R}\), \(f(x)=x^3\).

• Minimizer lying in the interior of the domain: \(f: [-1,1]\to \mathbb{R}\), \(f(x)=x^2\).

Strict (global) minimizer, strict minimizer, strict (global) minimum point, strict minimum point: A point \((x^{i,*})\in \mathcal{D}\) is a strict minimizer of \(f\) on \(\mathcal{D}\) if

A strict local extremizer is a local extremizer. However, a local extremizer need not be a strict local extreminizer.

\(\begin{equation} f[(x^{i})]> f[(x^{i,*})] \end{equation}\) for all \((x^{i})\neq (x^{i;*}) \in \mathcal{D}\).

local minimizer, local minimum point A point \((x^{i,*})\) is a local minimizer of the \(f\) on \(\mathcal{D}\) if there exists a \(\delta>0\) such that

\(\begin{equation} f[(x^{i})] \ge f[(x^{i,*})] \end{equation}\) for all \((x^{i})\in \mathcal{D}\cap B[(x^{i,*}),\delta]\), where

\(\begin{equation} B[(x^{i,*}),\delta] = \{ (x^{i})\in \mathbb{R}^n~|~\lVert (x^{i,*})-(x^{i}) \rVert <\delta \} \end{equation}\) is called the open ball centered around \((x^{i,*})\) of radius \(\delta\).

Minimizers Local and global minimizers

Maximizers Local and global maximizers

Extremizer minimizers and maximizers.

A global minimizer is also a local minimizer, but the converse (a local minimizer is a global minimizer) is not necessarily true.

Saddle point A stationary point of \(f\) that is neither a local minimizer or a maximizer is called a saddle point.

Let \((x^{i,*})\) be an accumulation point of \(\mathcal{D}\). Then, it can be both a local minimizer and a local maximizer. For example, consider a function in \(f :\mathbb{R}\to \mathbb{R}\), where f(x)=0. In this case \(\mathcal{D}=\mathbb{R}\). For this function every point in \(\mathcal{D}\) is a local minimizer and a local maximizer. However, an accumulation point cannot simultaneously be a strict local minimizer and strict-local maximizer.

Hessian matrix

\[\begin{equation} (H_{ij})[(x^{i})]=(D_{,i,j}f[(x^{i})]) \end{equation}\]

For the case of two variables, i.e., \(n=2\), the Hessian matrix reads

\[\begin{equation} (H_{ij})[(x^{1},x^{2})]= \left( \begin{array}{cc} f_{,11}[(x^{1},x^{2})] && f_{,12}[(x^{1},x^{2})] \\ f_{,21}[(x^{1},x^{2})] && f_{,22}[(x^{1},x^{2})] \end{array} \right) \end{equation}\]

Since \(\partial f_{,ij}[(x^k)]=\partial f_{,ji}[(x^k)]\), we can write

\[\begin{align} (H_{ij})[(x^{1;*},x^{2;*})] &= \left( \begin{array}{cc} A && B \\ B && C \end{array} \right) \end{align}\]

where \(A=f_{,11}[(x^{1;*},x^{2;*})]\), \(B=f_{,12}[(x^{1;*},x^{2;*})]\), and \(C=f_{,11}[(x^{1;*},x^{2;*})]\)

[edit]