Optimal Control A Review of Theory and Practice

AIAA Journa
VOLUME 3
NOVEMBER 1965
NUMBER 11
Optimal Control:
A Review of Theory and Practice
BERNARD PAIEWONSKY Institute for Defense Analyses, Arlington, Va.

I.
Downloaded by Beihang University (CNPIEC - XI'AN BRANCH) on May 12, 2013 | http://arc.aiaa.org | DOI: 10.2514/3.3307
Introduction
Scope of Review
HE objectives of this review are to summarize and to evaluate the present state of knowledge concerning optimal control synthesis. The primary areas of interest are flight mechanics and flight control. The design of optimal stabilization and control systems, the determination of optimal flight paths, and the calculation of optimal orbital transfers have a common mathematical foundation in the calculus of variations; this review is limited to problems requiring a variational treatment. The selection of optimum vehicle shapes and the determination of optimum staging and propellant loadings for rockets are not included in the scope of this work although they are variational problems. A general discussion of optimization techniques draws upon the accumulated experience and research results of many workers in different branches of engineering and applied mathematics. In recent years, there has been a confluence of control research in aeronautical, electrical, mechanical, and chemical engineering. The research efforts in specialized technical fields have produced results of general interest, but the splintering of the literature according to old traditions has sometimes retarded the application of the results. A principal task of this review is to discuss the synthesis of closed-loop optimal controllers. This review will not make a sharp distinction between optimal guidance and optimal control because the mathematical problems are nearly identical even though the time scales of the system being controlled may differ by several orders of magnitude. A
synthesis procedure for an optimal system is a set of design rules whose application results in a closed feedback loop comprised by the system being controlled and a computer to process the output measurements and to determine the optimal control law. This is illustrated in Fig. 1, which is a block diagram encompassing a very wide range of diverse problems extending from the optimal correction of interplanetary trajectories to optimal flight controllers for airplanes. These two apparently dissimilar technical problems are nevertheless nearly identical from a system synthesis viewpoint. The same basic design steps are required in each case. These include the selection of a performance criterion, the estimation of the statistical properties of noise and random inputs, the determination of the system state from noisy data, and the computation of the optimal control law. Each of these items will be taken up again later on. Scope of the Literature Survey The survey is oriented towards aeronautical or astronautical applications, although papers from other branches of the general engineering and scientific literature are included. Several texts have been published which give comprehensive coverage to certain aspects of this subject. Much essential material is concentrated there, and the problem of searching for individual papers in a large and widely scattered literature is reduced. The reader seeking more detailed descriptions of the material contained in this review should consult the books by Bellman,1 Chang,2 Lawden,3 Leitmann et al.,4 Miele,5 Pontriagin et al.,6 as well as conference proceedings such as those given in Refs. 7 and 8.
Bernard Paiewonsky received a B.Sc. in mathematics from the Massachusetts Institute of Technology, an M.A. in mathematics from Indiana University, and an M.S.E. and Ph.D. in Aeronautical Engineering from Princeton University. Dr. Paiewonsky came to the Institute for Defense Analyses in 1964 as a member of the Senior Technical Staff. Prior to this he was a consultant at Aeronautical Research Associates of Princeton, Inc., which he joined in 1958. He was a Legendre Fellow at Princeton in 1957-1958 and was a Lieutenant in the U. S. Air Force at the Flight Control Laboratory at Wright Field from 1955 to 1957. Dr. Paiewonsky was a senior member of the American Rocket Society and is a member of the AIAA, B.I.S., and the AIAA Astrodynamics Committee. He was an associate editor of the ARS Journal and AIAA Journal from 1959 to 1964.
Received June 11, 1965; revision received September 14, 1965. A portion of the work on which this paper is based was carried out while the author was at Aeronautical Research Associates of Princeton (ARAP). The work at ARAP was supported in part by the U. S. Air Force Flight Dynamics Laboratory under contract AF 33(657)7781 and in part by the U. S. Navy (BuShips) Contract Nonr 3703(00) administered by the David Taylor Model Basin (DTMB). I am happy to acknowledge the valuable assistance of Peter Woodrow of ARAP and John Mclntyre, formerly at Princeton University and now at North American Aviation, in collecting, examining, and discussing the material used in parts of this report. This review draws on the results of a 1963 survey presented in AF TDR-63-239 and on the results of research on The Bounded State Space Problem (ARAP Report #60, prepared for DTMB). Thanks are due to Patricia Maines of the Institute for Defense Analysis (IDA) who typed the manuscript.
1985
1986
Disturbances Commands
B. PAIEWONSKY
AIAA JOURNAL
Estimated state variables
Fig. 1 Closed-loop optimal control.
Section II contains a brief historical account of optimization research in areas of aeronautical and astronautical interest. These areas include 1) orbital flight mechanics and rocket steering, 2) aircraft performance, 3) flight controllers and automatic stabilization systems in general, and 4) optimal controllers. In Sec. Ill, there is a discussion of certain mathematical aspects of optimization. The discussion emphasizes the problems of inequality constraints on the control and on the state variables. This is followed by Sec. IV, on stochastic optimization, e.g., problems with random inputs, noisy measurements, and imperfectly specified system parameters. Variational problems usually lead to differential equations with split boundary conditions. Section V on two-point boundary problems describes methods of successive approximations for the usual situation where exact solutions in closed form are unavailable. Methods for solving the two-point boundary problems frequently require the solution of an ordinary minimization problem with several independent variables. Section VI is devoted to this, and some examples are given. In Sec. VII, the design of an optimal control closed-loop system is discussed. The final section is a list of the cited articles.
II.
Historical Aspects and Areas of Interest
Lawden,18 however, has found a spiral extremal arc employing intermediate thrust levels. Kopp and Mover19 have proved that the spiral arcs are nonoptimal in the timeopen case. McCue and Bender20 have made numerical studies and observed that, in the cases considered, the characteristic velocity for the spiral arc exceeds that for an impulsive transfer with the same boundary conditions. When an impulsive mode of transfer is postulated, the optimal locations, directions, and magnitudes of the impulses must be determined. There exists a very large body of references on this problem. Some of these results can be found in the papers of Lawden,21~27 Edelbaum,28'29 Munick et al.,30 Altman and Pistiner,31 Hoelker and Silber,32 Fimple,33 Gobetz,34 Horner,35 and Templeman,36 among others.37"43 The determination of optimal steering and thrusting programs for separately powered rockets is also a problem of continuing interest. (Edelbaum44 has examined the use of high and low thrust in combination.) The optimal operation of nuclear rockets has been studied by Leitmann,45 Bussard,46 Irving and Blum,47 and Wang et al.48 Optimal trajectories for interplanetary operations with continuous thrust systems have been investigated by Melbourne,49 Melbourne and Sauer,50 Fox,51 Kelley,52 Hinz and Moyer,53 and MacKay.54 The important problems of optimal rocket boosting in strong gravitational fields has been studied by Hibbs,55 Fried,66 Bryson and Ross,57 Ross,58 Miele,59 Stancil and Kulakowski,60'61 Breakwell,62 Leitmann,63 and many others. The optimal spacing of corrective thrusts on interplanetary flights has been studied by Lawden,64'65 Breakwell and Striebel,66 and Denham and Speyer.67 The field of optimal rocket trajectories has been reviewed by Lawden,68 Leitmann,69 Miele,70 and Bergqvist.71 Fuel-optimal and time-optimal rendezvous maneuvers have been studied by Goldstein et al.,72 Kelley and Dunn,73 Hinz,74 Mclntyre arid Crocco.75 The thrust and fuel constraints were applied separately. Paiewonsky and Woodrow76 treat a linearized time-optimal rendezvous in three dimensions with bounded thrust and limited fuel by application of Neustadt's method.77 Fuel-optimal lunar landings are studied by Meditch78 and Hall et al.79 Neustadt has recently prepared a general treatment of minimum-fuel space trajectories which includes many of the foregoing special problems.80
Aircraft Performance
The history of optimization in flight mechanics and control theory is long, and the literature is extensive. This section outlines some of the problem areas already explored. It is not intended to be a complete catalog of all past work. It should be viewed as a selection of research papers chosen to show the relations between adjacent areas of engineering analysis.
Rocket Steering Problems
The question of the optimal mode of operation for rockets was raised over 45 years ago by Goddard,9 who was interested in maximizing the altitude of sounding rockets. This problem, and other versions of it, have been treated subsequently by Malina,10 Tsien and Evans,11 Ross,12 Miele,13 Leitmann,14 and Garfinkel.15 There are several cases of importance, and these differ from each other mainly in the ways in which the drag is assumed to vary with velocity and the atmospheric density varies with altitude. The general problem of optimal interorbital transfer of a rocket is still unsettled. In 1925 Hohmann16 discussed orbital transfers and arrived at the conclusion that for coplanar circular orbits the cotangential ellipse with impulsive rocket thrust is optimal with respect to fuel. Other investigators in subsequent studies conjectured that", in problems of this type (i.e., chemical rockets with high thrust capability), the thrust level was either a maximum or a minimum, with no extremal arcs of intermediate thrust levels. Barrar17 has shown that the Hohmann transfer is the minimum two-impulse transfer.
Airplane performance is the subject of many optimization studies. The objective of the optimization is the improvement in the operational capabilities of the aircraft with respect to range, takeoff and landing distances, time-toclimb, etc. The minimization of the time-to-climb was treated originally by a quasi-steady analysis based on power available and power required.81 Lippisch82 modified the quasi-steady analysis by including the effect of accelerations along the flight path. The time-to-climb problem was subsequently treated by Lush,83 Garfinkel,84 Miele,85 and Cicala and Miele.86 In 1949 Hestenes87 formulated a very general problem in the calculus of variations and applied it to twoand three-dimensional time-optimal climbs. This work included an analysis of inequality constraints, e.g., bounds on the angle of attack. Mengel,88 using Hestenes7 work as a starting point, recast the equations in a form better suited to computational techniques and carried out a series of studies on an analog computer. The endpoints of the computed paths were observed to be very sensitive to variations in the initial choices of Lagrange multipliers, and it was very difficult to satisfy the boundary conditions of the problem. Kelley studied the optimal climb problem using in one case a classical variational approach,89 and more recently a gradient method.90 The latter study includes bounds on the state variables as well as on the controls. Bell has reviewed atmospheric flight optimization for the 1945-1960 period.91 The maximization of aircraft range was also treated originally by quasi-steady techniques, e.g., the well-known Breguet
NOVEMBER 1965
OPTIMAL CONTROL: A REVIEW OF THEORY AND PRACTICE
1987
formulas.81 The maximization of range for hypervelocity vehicles has been studied by Arens92 and Bell93 and Miele.94 Energy management, range control, and aerodynamic heating associated with space vehicles returning to the earth have been investigated by Bryson et al.,95 Levinsky,96 and others.97 Bryson et al.98 have also described the application of an optimal terminal controller to the tasks of tracking nominal re-entry paths. Optimal ship routing and optimal aeronavigation are the subjects of many fundamental investigations. Zermelo's minimum-time steering problem,99 is well known and is often used today as a prototype or example for control synthesis ideas.100-101 Minimum-time steering including the action of winds or ocean currents also has been studied by Von Mises,102 LeviCevita,103 Faulkner,104 and Haltiner et al.105 An exceptionally fine account of this subject appears in DeJong's monograph.106 This 1956 monograph presents an interesting development of the variational problem from the viewpoint of isochronal wave fronts and the Hamilton-Jacobi theory. It is regrettable that this work did not receive wider circulation at the time of its publication, as it certainly would have stimulated research along fruitful lines in control theory and flight mechanics.
Flight Controllers
Flight controllers, autopilots, and stability augmentation systems for manned aircraft are ordinarily designed using linearized analyses. Feedback gains or adjustable parameters are manipulated to obtain satisfactory system transient response to control inputs and to gust inputs. Linear systems subject to deterministic inputs are frequently optimized with respect to transient response, and auxiliary criteria based on transient rise time, peak overshoot, settling time, bandwidth, etc., are often used.107"109 These characteristics of the system response depend upon the locations of the poles and zeros of the system transfer function. Kalman et al.110'111 point out that a transfer function is not always adequate to describe the behavior of a system. The design of optimal linear feedback controllers has been thoroughly explored, and comprehensive reviews of this aspect of control theory have already appeared.112-113 The design of an optimal controller requires the selection of a performance criterion. In many aeronautical applications, the selection of a performance criterion is based on real physical considerations, e.g., payload, final velocity, total heat absorbed, etc. There are studies of optional controllers that are not based on extremizing physically meaningful quantities; the result may nevertheless be a good control system from the standpoint of the pole and zero locations of the transfer function.114-115 Sometimes it is possible to give a physical meaning to these criteria, but these appear to be artificial. The minimization of these functionals may be viewed as a helpful device, which selects' a set of~ system parameters producing a stable system with a good transient response, even though the physical significance of the optimization criterion is not obvious at all. The ultimate decision on the quality of the flight-control system in the case of manned aircraft is, of course, determined by pilot opinion. The relationship between generalized performance criteria for optimal controllers, aircraft handling qualities, and pilot opinion is still an open question for all but the simplest systems and even then only for certain special piloting tasks. The greatest difficulty in designing an optimal controller for a manned vehicle lies in finding out what the pilot really wants. A realistic mathematical model for the pilot has yet to be developed, although research on the characteristics of human pilots has been carried on for many years.116 A recent review117 shows that progress has been achieved in making a servomechanistic model for the pilot in certain tracking and pursuit tasks. In these studies, the higher cognitive processes are not considered at all; the pilot
is treated as a quasi-linear dynamical element in a closedloop servomechanism. This approach has been applied successfully to the elimination of pilot-induced oscillations118 and to the improvement of pilot-aircraft systems, e.g., carrier landings.119 Nonlinear autopilots that are not optimal but whose designs are based on the ideas of optimal control have been described in the literature. (See, for example, the works of Passera,120 Schmidt, et al.,121 and Flugge-Lotz.122) There have also been a number of studies of nonoptimal predictive terminal controllers. The predictive controllers bear a strong resemblance to the optimal controllers under discussion because two-point boundary problems occur in each case. Studies have been made of the application of these controllers to re-entry and automatic landing problems.123~129 The design of optimal autopilots for aircraft and missiles subjected to random inputs has received considerable attention. The minimum mean square error criterion is often used, but this is not the only one that has been considered.113 If the system is linear and if the noise has stationary statistical properties and is Gaussian, then the mean square error is easily found in terms of the input power spectral density or correlation function. Minimization problems of a similar nature occur in communication research, and an extensive theory of optimal filtering and prediction is available based on the work of Weiner130 and subsequent investigators.131 An approach frequently used in autopilot design postulates linear feedback of the state variables and seeks values of the feedback gains which minimize the penalty criterion, e.g., expected value of rms loads. The omission of control deflection limitations or a cost-of-control term from the problem formulation leads to a trivial result. The optimization of nonlinear control systems with random inputs and the design of optimal linear systems subjected to non-Gaussian random inputs requires an analysis of the probability distributions of the output variables. If the output variables form a Markov process, then the FokkerPlanck partial differential equation can be used to generate the time-varying probability densities.132 This technique has been used by Ruina and Van-Valkenberg133 in a study of radar tracking systems. Kalman and Bucy134 have studied the combined problems of optimal control and optimal filtering. The results of these studies are being applied to the guidance of space vehicles.135"137 This technique has also been investigated by Battin.138 The stochastic optimization of a terminal controller with a saturating element has been examined by Booton.139 The design of optimal autopilots subjected to random inputs was studied by Schwartz140 and others. Schwartz applied Booton's technique to the terrain clearance problem for lowaltitude aircraft flight with a g limit. The loads on aircraft in flight through turbulent air can be ameliorated by the use of control manipulations that change the aircraft's response characteristics. These corrective control motions may be due to the pilot or to an automatic control system. Measurements of atmospheric turbulence have been made, and the general shapes of the power spectra are known, although the level of intensity depends upon the local meteorological situation. The problem of high transient loadings due to turbulence is particularly severe for highspeed, elastic aircraft flying near the ground. Some optimization studies have been done in this area,141-142 but the problem is not at all solved; in fact, the surface has hardly been scratched. A closely related problem is the control of large flexible booster rockets flying through strong wind shear.143"145 The difficulty here is the limited control moment available for load relief and artificial stabilization. This is often aggravated by the coupling of the elastic body bending modes and the rigid body modes through the control system and the sensors.
1988
B. PAIEWONSKY
AIAA JOURNAL
The optimal attitude control of a tumbling satellite is usually treated in two steps. The body is first brought to rest by application of an optimal control that stops the rotation; the correct orientation is then sought by control actions (not necessarily optimal) that turn the body from one attitude to another. The optimization of the combined tasks (the rotational rates brought to zero at the proper orientation with minimum fuel or in minimum time from any initial condition) is very difficult, and the synthesis problem has not yet been solved. Athans146 and Lee147 have produced useful results on the problem of stopping the tumbling. Automatic stabilization systems (nonoptimal) for earth satellites and space vehicles have been designed based on approximations derived from optimal control theory. The controls try to keep the attitude errors within specified bounds and reduce the fuel or power expenditure.148"150 These control systems may be operating with a limit cycle close to the error null position, and component imperfections, e.g., relay and valve hysteresis, gear backlash, propellant ignition delays, friction, deadzones, etc., become important and must be taken into account. Satellite attitude control techniques are reviewed in detail by Ergin et al.150 and Roberson.151 A shorter review by Ergin has recently been published.152 Single axis attitude regulation is reviewed and discussed in Ref. 153.
Optimal Controllers
Many studies have been carried out to determine the behavior of closed-loop systems containing relays.180-181 The emphasis has been on the control of time-invariant linear processes with relays employing linear switching laws; e.g.,
Xj +
(1)
The discussion to this point has drawn mainly on aeronautical sources. This section outlines the development of optimal control research in other areas. The historical development of control theory in engineering emphasized stability. Although stability is always a consideration, it is not the main concern here. Optimization problems in the engineering literature are frequently treated separately, aside from stability considerations, and are ordinarily formulated in terms of calculus of variations. The optimal control papers in the electrical and mechanical engineering literature can be placed into two broad categories. The first category consists of abstract studies not directed towards any specific system or device. In the second category are studies of more concrete applications, e.g., highspeed servos, relay controllers,154"156 and space vehicle attitude controllers.157'158 The latter group may contain experimental tests, whereas papers in the first category are usually devoid of hardware considerations. The scope of the studies in both categories includes time-optimal control,159"161 optimal regulation,162"164 optimal sampled data systems,165'166 and optimal stochastic systems.167"169 Application of optimal controllers to chemical processes has also been studied extensively.170"172 The problem of finding optimal locations of static operating points or "set-points" for chemical processes led to research on "optimalize^."173-174 These devices (analog or digital computers) are mechanized hill-climbers that. search for maxima of functions of several variables in a systematic way. The algorithms used to speed up the hill climbing processes are of interest, but static optimization will not be treated here at all. We note in passing that early original work on optimalizers was done by Draper and Li175 and applied to the control of reciprocating engines for aircraft This work is also described in Tsien's book.176 Early work on engineering problems with bounded controls assumed a relay-type operation, and a large engineering literature exists for bang-bang or relay controllers. The relay, a deceptively simple device, can be employed easily as a logical element in an automatic controller. Some engineering studies177"179 of relay controllers begin with the stipulation that a two-position or three-position control be used. This requirement is imposed prior to any optimality considerations. The relay switching function can be a linear combination of state variables, or it can be a nonlinear (not necessarily optimizing) function of the state variables.
The over-all behavior of the closed-loop system depends on the slope of the switching hyperplane (determined by the c/) and the elements of the matrix A and vector b. The function "sgn" is discontinuous, and consequently the system does not possess the qualities needed to insure existence of smooth solutions from a mathematical standpoint. The solutions of this system of equations can exhibit peculiar behavior known as "chattering" and "endpoint." The behavior of systems with relay-like discontinuities has been studied extensively by Flugge-Lotz,182 Andre and Seibert,183-184 Ayzerman and Gantmacher,185 Filippov,186-187 and Tsypkin.188 The fact that actual relays have hysteresis, dead zones, and time delays prevents the endpoint phenomenon from being observed, but in its place, there is a chattering phenomenon and a limit cycle about the error null position. These results are fairly well known, and the fundamental ideas are summarized in Tsien's book176 and in more recent texts. The development of high-speed switching amplifiers and high-speed relays has reduced the effects of component imperfections, but the use of bang-off-bang jet reaction controls with valve dead zones, time delays, and hysteresis has reintroduced these problems.189"191 Many engineering studies of relay controllers are directed at second-order systems because the phase-plane permits graphical synthesis techniques to be developed easily. The graphical phase-space approach has been extended to some third- and fourth-order systems by studying the projections of the phase motions onto coordinate planes.192 The resulting techniques are relatively complicated, and only limited applications have been attempted. Bass193 pointed out that synthesis of stable relay controllers for systems of order higher than two could be accomplished via Liapunov's second method. Kalman and Bertram discuss this in detail and give examples.194 Some effort has been directed at physically realizing nonlinear switching functions by means of diodes and other devices.195-196 Attempts to improve the time response of servomechanisms by adding nonlinear elements to the feedback circuit have been made by McDonald,197 Schieber,198 Ku,199 and many others. The 1961 article by Flugge-Lotz200 provides an excellent review of relay controllers; her monograph201 provides a comprehensive and detailed account of a class of discontinuous controllers. An older review by Kazda202 provides a critical survey of this area up to 1957. The work on second-order systems beginning with McDonald's 1950 paper203 is thoroughly discussed. The optimization of linear systems with bounded controls and limited control effort is important to the engineer as well as to the mathematician because the linearized versions of many physical problems can be easily forced into this general formulation. Time-optimal control is one of the more interesting of these problems because it is actually possible to get solutions in some cases, and these solutions do provide insight into more general aspects of optimization. The literature on time-optimal control was reviewed in 1963 by Kreindler.204 The problem is usually stated as follows. Given a dynamical system of nth order described by a linear time-varying vector differential equation,
Xi = AH (t)xj + Bik (f)uK
(2)
find a piecewise continuous control vector w(t), which
NOVEMBER 1965
1989
satisfies the following requirements: 1) u lies in U, where U is a compact, convex set containing the origin as an interior point, e.g., Uk\ < 1, and 2) the system is transferred from an initial state Xi (0) to a desired terminal state # (T) in the least possible time, by the application of u(T). The control may also be required to satisfy an integral constraint of the form
these, and it has been applied to rocket steering,225 rendezvous,76 midcourse guidance,226 and lunar landings.78 Neustadt220 has extended this method to include minimum effort control where the effort is defined by an integral of a functional of the control, i.e.,
<p(u)dt < I
<p(u)k-dt < I
in addition to, or in place of, requirement 1. Bushaw205 obtained solutions for the second-order timeoptimal regulator problem [x(T) = 0], u\ < 1 by using geometrical constructions in the phase plane. These results provided a rational synthesis procedure, for the first time, for systems with oscillatory characteristics. Nonoscillatory, stable, second-order systems can be brought to rest at the origin from any initial phase point by a bang-bang scalar control having at most one change of sign. Oscillatory systems may require a large number of switches; this depends on the size of the initial errors in velocity and position. It may not always be possible to bring the state to the origin if the system is unstable. Bellman, Glicksberg, and Gross published a topological approach to the time-optimal regulation problem in n dimensions.206-207 The main results of their study can be summarized as follows: If all solutions of x = Ax go to zero as t -*- co ? then there is a time-optimal control vector u(t), and it is bang-bang, i.e., Uk = 1. If all of the characteristic roots of A are real, distinct, and negative, then there exists a minimizing control vector u, uk = 1, and uk changes sign at most (n 1) times where n is the order of the system. These results are very important, yet they do not say anything about systems with complex eigenvalues or real nonnegative eigenvalues. Bushaw did not impose these restrictions, but he was looking only at second-order systems. LaSalle208 extended these results to nth-order time-varying systems and showed that the control is bang-bang and unique (a.e.) if the system (2) is normal, i.e., no component of the expression n-X" 1 ^) B(2) vanishes over an interval of finite length for any vector n.* If the system is normal then the optimal control as a function of time is u = sgn [tiX"1^) B()]. Similar results were obtained by Gamkrelidze209 for the autonomous time-optimal control problem. This problem was treated via the maximum principle by Pontriagin et al.6 and Rozonoer.211 Krasovskii,212 Kulikowski,213 and others214"216 applied functional analysis to the timeoptimal problem and obtained formal solutions. Kulikowski observed that some optimal control problems can be solved by application of the theory of approximations217; e.g., solutions for the switching times were obtained as zeros of Tschebychev polynomials. The time-optimal problem was formulated by Desoer218 using the calculus of variations. The results of the different formulations are nearly identical. In each case, the timeoptimal control of Eq. (1) is bang-bang, and the switching times for the components of the control vector are the zero crossing times of a function formed from adjoint equations, i.e., u = sgnnX~1(OB0). This fact was observed earlier by Bass.219 The initial conditions for the adjoint, in effect, determine the switch times; the boundary conditions are satisfied by applying the appropriate initial conditions to the adjoint. These initial conditions are generally unknown and must be found for each initial state point. The adjoint variables (also called costate variables) correspond to the Lagrange multipliers of the variational formulation. Neustadt,77 Gamkrelidze,221 Krasovskii,222 Eaton, 223 Knudson,162 Ho,224 and many others have proposed methods for finding the initial conditions of the adjoint for the bang-bang problem. Neustadt's method appears to be the best known of
* This rules out singular subarcs.
Time-optimal problems with fuel constraints as well as minimum fuel problems can be solved in this way. The synthesis procedures listed so far are based on continuous models of the process being controlled. There are many situations in which the nature of the problem makes it desirable or necessary to use a discrete model. The model may be based on sampling a continuous system, or it may arise from a process that is properly described by difference equations. This is an important subject area because it is quite likely that major applications of closed-loop optimal control in flight mechanics will appear in a sampled data or discrete form. The use of digital and hybrid analog-digital computers leads in a natural way to a discrete formulation. Closed-loop optimal controllers that repetitively solve an open-loop continuous problem between sampling instants obviously fall into the class of discrete optimal systems. The optimal spacing of interplanetary corrective thrust, mentioned earlier, is a problem in this category. Discrete time-optimal problems have been studied by Desoer and Wing,227 Neustadt,228 Polak,229 and others. There is a large literature on sampled data systems which will not be revie wed here.230-231 A complete and consistent theory of discrete optimal stochastic feedback systems has not yet been developed, although considerable progress has been made in solving prerequisite deterministic problems. Some aspects of this are discussed in the section on stochastic optimal control.
III.
Mathematical Treatment of Optimization
Introduction
Mathematical formulations of optimal control problems have been made by Hestenes,87 Warga,232 Marcus and Lee,210 Breakwell,62 Krasovskii,222 Bellman,1 LaSalle,208 Neustadt,220 Pontriagin,260 and many others. Many of the papers are directed at the time-optimal problem for linear systems with bounded controls. Hestenes233 has recently reviewed the variational formulations of optimization theory. Optimization problems containing a state space inequality constraint are by no means new. Weierstrass' lectures, reported by Bolza,234 develop the usual necessary conditions plus the "corner" conditions for the problem of Lagrange in two dimensions. A fourth necessary condition (analogous to the Jacobi conjugate point condition) was developed by subsequent investigators and used in a sufficiency proof by Bliss235 for the problem of Lagrange in two dimensions and by Bliss and Underhill236 for the three-dimensional Lagrange problem. Sufficiency proofs were developed under the assumption that the problem is "regular"; this condition requires that the extremal be smooth. Later, Mancill237 was able to remove the "regular" condition, thereby allowing solutions that are not tangent at the boundary surface and boundary surfaces that have corners. The problem was also discussed by De Jong.106 In more recent treatments of the problem, sufficiency considerations have been virtually ignored. The emphasis has been placed on computational aspects and also on methods for including control constraints that restrict the admissible control to a bounded control space. One of the earliest works on a bounded control problem was by Valentine238 who treated a general problem in calculus of variations. More recently, Isaacs239 considered bounds on the state variables in his papers on differential games. The constraints in state
1990
B. PAIEWONSKY
AIAA JOURNAL
variables are taken care of by imposing a special restriction on the class of allowable controls. That is, whenever the system state is about to strike a boundary, the range of possible control strength is suitably reduced. This is to be done in such a way that the system becomes incapable of violating the restrictions. The problem, in principle, may then be treated by modifications of known methods. Berkovitz240 rewrote the boundaries as an additional system of first-order differential equations and applied Valentine's techniques. Chang241 and Kipiniak242 each apply a penalty to the performance criterion if the boundaries are violated. The penalty function is made large and "sharp-edged," and then a limiting process is invoked. In addition, Bryson95 and Levinsky96 have treated variational problems of re-entry flight. The main contribution in the Russian literature on this problem, to date, is a paper by Gamkrelidze.243 Additional Lagrange multipliers are introduced, which may be discontinuous, and additional Euler equations are required. Gamkrelidze, Chang, and Berkovitz deal solely with the theoretical aspects of the problem, developing necessary conditions that the solution must satisfy under the additional restriction that the control u be in a specified set U. Both theoretical and computational aspects are considered in papers by Dreyfus,244 Denham,245 Bryson et al.,246 and Bryson and Denham,247 with the computational procedure being essentially an extension of the gradient or steepest-ascent method frequently used in unconstrained problems. Kahne,248 Ho and Brentani,249 and Kelley et al.250 deal exclusively with the computational aspects of the problem while developing methods of solution which are rather indirect. Garfinkel and McAllister251 point out some unusual cases where an extremal striking the boundary and satisfying the corner conditions may have no possible continuation.
Necessary Conditions: Maximum Principle
and where the pi, called adjoint variables, costate variables or multipliers, satisfy the differential equations
(7)
In order for the function S in Eq. (4) to be a minimum, the control action u must be selected so that
The bounded state space problem can be stated mathematically as follows: We are given the system
Xi = fi(x, t, U)
(8) at each point along the trajectory. The symbol u denotes the optimal control, and u denotes any other control, both of which are contained in the set U. It is important to remember that the maximum principle and the necessary conditions obtained from classical calculus of variations produce, not optimal trajectories, but extremals. It is still required to show that the extremal is optimal. A singular arc or subarc occurs whenever the inequality (8) becomes an equality over a finite interval of time, and H becomes independent of u. If, for example, H is linear in Uk, then there may be a singular subarc whenever the coefficient of u^ ftHfbuk vanishes over a finite interval. The maximum principle fails to select the optimum control in this case. The singular control is found by using the requirement that the system remain on a path such that bH/buk 0. Singular arcs occur in the Goddard problem,15 linear problems with quadratic criteria,252'255 orbital transfers,18'19 and in terrestrial flight mechanics.5 The difficult part of the analysis lies in determining whether a singular arc is optimal. The singular arc problem has been studied by Kelley253 and Hermes254 and discussed by Johnson and Gibson.252 Miele5 has used Green's theorem successfully on problems with two state variables. Kopp and Moryer have obtained a test for determining the optimality of a singular arc. A formulation along the classical variational lines is possible if it is assumed that the constraint, that the control u lie in the set U, can be expressed analytically by an inequality of the form
0 (9)
#(p, x, u) > # (p, x, u)
1, . . . n
(3)
where x is an ^-dimensional state vector, and u is an rdimensional control vector. We must find a piecewise continuous control u in the set U such that the function of the final state
'8 = S(xf,tf)
(4)
where the boundary curve, 4>(u) = 0, is piecewise differentiable. For example, Valentine's method applied to constraints on the control of the form a < u < ft leads to the function (u a) (ft u) 772, where rj is real. The functions F and / are then formed as in the Mayer problem,256'257 where
takes on its minimum value subject to the condition that x stays within a specified region of the state space given by the inequality
0
(10)
(5)
,)+ f t f F d t
J to
(11)
t = l,r (12)
The set U is assumed to be compact, convex, and to contain the origin as an interior point. The functions /, S, and G are assumed to be of class C" with respect to all tneir arguments. This assumption on the smoothness of G(x) can be relaxed to allow for the occurrence of corners on the boundary surface.6 It should be noted that G(x) is not explicitly dependent on the control action. The optimal trajectory is composed of two types of segments: interior segments and boundary segments. It may happen that there are no boundary segments or no interior segments, but in the general case both types will be present. An interior segment satisfies the same necessary conditions as an optimal trajectory in the unconstrained problem. The unconstrained problem [i.e., without inequality (5)] may be set up in several different ways. The maximum principle requires the formation of a function H, called the Hamiltonian:
Setting 5J = 0, we obtain the Euler equations
The control action u and the multiplier A are determined from Eq. (12) and the constraint
X^Cw) = 0 (13)
The Weierstrass necessary condition must also be satisfied. This yields an equation identical to Eq. (8) in the Maximum Principle formulation. Application of the Legendre-Clebsch condition using the device of Valentine provides the additional inequality
X >0
(14)
H = #(p,x,u) =
(6)
An interior segment of a bounded state space problem must satisfy the foregoing necessary conditions. Along a boundary segment, the inequality (5) becomes an equality, and the optimization takes place subject to the
NOVEMBER 1965

Time optimal switching curve
1991
constraint
G(x) = 0
(15)
over the entire segment. Many classical problems in the calculus of variations contain constraints of this form (e.g., geodesies on surfaces), and the optimization procedures are well known. In fact there are two equivalent procedures for treating this problem.257 The first and more direct approach consists in adjoining the constraint G to the unconstrained version of the problem with an additional multiplier /*i. The second method consists in adjoining the total derivative of the constraint dG/dt to the unconstrained problem with a multiplier u%. If the function G(x) vanishes at the initial point of a boundary segment, then adjoining the total derivative and making it vanish along the segment is equivalent to adjoining the constraint itself. It has been shown258 that /t2 = Ai, and ^ < 0. It has been pointed out that an optimal trajectory in a bounded state space problem consists of two types of segments, boundary segments and interior segments, and the optimization procedure for each type segment is well known. It still remains to determine the manner in which the individual segments join together to form the composite optimal trajectory. A point at which an interior segment joins a boundary segment is called a corner point. The conditions that must be satisfied so that the resulting path is optimal are called the corner conditions. It has been shown6'258 that at a corner point the multipliers satisfy the relation
Switching curve
x2
G>0
+ v
(16)
/ /
where z>o is a constant that depends on the end conditions. The Hamiltonian is continuous, i.e.,
Optimal trajectory
\
Neighboring \ trajectories
G <0
, x,
(17)
The derivation of this condition assumes that the variations in the position and time of the corner (dxt and dt) are arbitrary. There are two cases of interest for which this assumption is not valid, and the corner conditions appear in a different form. The first case occurs when the nth derivative of the constraint (dnG/dtn) is the first derivative that contains the control u explicitly. In order to satisfy the constraint G < 0, it is required to make the first (n 1) derivatives vanish at the junction point. The conditions now are
b)
Fig. 2 Minimum time bang-bang problem with a velocity limit.
(18)
with the control constraint \u < 1, and the state constraint
G = x2 - k < 0
n-l
5
(21)
**
(19)
where VQ and vi are constants whose values are to be determined from the boundary conditions, and u denotes the optimal control. If the first derivative dG/dt does not depend explicitly upon the control, then a trajectory striking the boundary may go past it and violate the constraint. An example of this is given by Dreyfus.244 For example, in trajectory optimization problems with a constraint on the maximum altitude, the rate of change of altitude does not depend explicitly on the control (which in this case is the thrust vector). The vehicle will pass the altitude limit unless the rate of climb is zero at that altitude. The second case occurs when the optimal trajectory does not lie in a sufficiently rich neighborhood of extremals. For example, suppose the system is
= X2
X2 = U
The problem is to take the system from the initial point and bring it to the origin in a minimum time. The problem without the state variable constraint has been treated in the literature with the optimal trajectory shown in Fig. 2. The constrained solution takes the form shown in Fig. 2b with the corner points denoted by 1 and 2. Note that all neighboring trajectories lie to the right of the optimal as indicated in Fig. 2b. There is no trajectory in the small neighborhood of the optimal which falls to the left of the corner point 1. Furthermore, all the neighboring trajectories strike the boundary at a time later than the original optimal. Thus, the differentials dxi and dt, at = h, satisfy the conditions
i <0
dt > 0
(22)
(20)
The corner conditions for this case given by Eqs. (16) and (17) do not necessarily hold. There is no differential condition or local test for determining when or where to leave the boundary and to return to the interior of the domain. It is necessary to make successive approximations to find the junction points and satisfy the
1992
B. PAIEWONSKY
AIAA JOURNAL
boundary conditions. Optimal control in a bounded state space is discussed in more detail in Ref. 258.
IV.
Stochastic Problems
Statement of the Problem
Major theoretical problems arise as a result of feedback considerations in stochastic control systems. Some of these problems are discussed in a recent review by Kushner.263 The literature dealing with optimization of stochastic systems, in contrast to the deterministic case, is rather sketchy; certain points are still obscure, and much work remains to be done. The published papers, for the most part, present either carefully selected simple problems having closed-form solutions or theorems of moderate generality which are too complicated for routine engineering applications. There are, however, some noteworthy exceptions, particularly the problem of optimizing linear systems with quadratic performance criteria. The difficulty of problem formulation is caused in part by the fact that the transition from deterministic to stochastic performance criteria is not direct. A meaningful problem may result from transforming the minimization of quantity Q in the deterministic case to the minimization of the expected value, or mean value, of quantity Q in the stochastic case, but in other cases this transition provides a poor index for measuring system performance. The design of an "optimal controller" is frequently obtained by treating the problem from an open-loop point of view. The initial state of the system and its governing equations are specified, and a control law is calculated which transfers the system from its initial state to a terminal state in an optimum manner with respect to the performance criterion. The process is assumed to be completely deterministic, and the effects of uncertainties in process dynamics and random disturbances are neglected. The control law obtained under these assumptions unfolds only as a function of the initial state without recognition of the subsequent states of the process. In actual practice, however, the system will operate, not on an open-loop basis, but on a closed-loop basis: the system state being monitored either continuously or discretely with the control, depending on the estimates of the state variables. The control law in this case unfolds in time as the random process itself unfolds, and the control tends to counteract and smooth the effects of random disturbances. The question now arises as to how an optimum control law should be designed to include the effects of feedback in stochastic systems. One approach is to recompute continually the open-loop control law based on a deterministic performance criterion but using the observed or estimated values of the state variables as new initial conditions. The control law can be computed as a function of the initial conditions, with the initial conditions themselves updated and the time origin adjusted to match the existing state of the system. Such an approach is proposed by Kelley351 and Breakwell, Speyer, and Bryson,352 along with a second-order theory to facilitate adjustments in the control law.
Problem Formulation and Literature Survey
where s is the observation vector, and 97 is a second random vector denoting input and/or measurement errors. The dimensions of s and x need not be equal. Many papers on optimal stochastic control do not treat observation errors; that is, the system is assumed to be either perfectly observable with rj == 0 and s == x, or perfectly unobservable where no observations are made and no observation equation exists. An important contrast is the case of partial observability when 77 ^ 0, and Eq. (24) holds. It is convenient to treat these two cases separately. Uncertainties in system parameters can also be included in this formulation.
Perfectly observable or perfectly unobservable systems
The state of the process is given by

x = F(x, u, , 0
(25)
along with specified initial conditions at time t = t0 and a probability distribution for the random variable f . It is desired to minimize the expected value of some function of the terminal state <j>[x(T)}. The control u is to be chosen such that the integral
= f P[x(T),
J*
(26)
is minimized. In this equation, exp is the expectation operator corresponding to the random variable f, and P[x(T), T] denotes the probability of the vector x having a value between x(T) \dx and x(T) + \dx at time T, the final time. The meaning of the term P[x(T), T] must be examined in the light of the particular problem. For example, the probability density could mean the probability density conditioned on the state having the specified initial condition x(to) at time to. For this case, the probability density function is
P[x(T), T] = P[x(T),
\ tQ]
(27)
The probability density may be conditioned not only by the initial state x(to) but by observations made on the state at times between to and T] e.g.,
P[x(T), T] = P[x(T), T\x(t,\ t,' x(tj, ti; . . . ] (28)

where ^, fe, . . . are the observation times. Knowledge of the state at times subsequent to the initial time is used in computing the optimal control law. In contrast, Eq. (27) represents the feed-forward or open-loop type of control. The classical approach, which results in the control law u = u[x(to), t], applies to the feed-forward case, where no observations are made on the system. Kipiniak242 utilizes a variational approach for a class of problems in which the variational and expectational operators commute. Formulations utilizing the functional equation technique of dynamic programing treat the stochastic system as a closed-loop or perfectly observable system with a P function in the form of Eq. (28). The dynamic programing formulation of the stochastic control problem has been used many times.261"266 The most complete exposition of the method is given by Bbllman.267 For the most part, the problems treated by dynamic programing have linear dynamics with quadratic cost. The stochastic analogue of the bang-bang regulator is treated by Aoki.268 Krasovskii269 included the effects of time lag in the system, and Zadeh and Eaton270 consider the stochastic problem in the absence of a state transition equation. Some overlapping of the areas of optimal and adaptive control occurs in the cases where the controller is used to optimize and to probe the system for the purpose of improving knowledge of system parameters.271-272 Lass259 and Drenick and Shaw260 have treated optimizations with random parameters. The disturbances are assumed to possess rather simple statistical descriptions in most of the cases studied. The
Figure 1 illustrates a typical model of a stochastic system under control. The process is assumed to be governed by the vector differential equation
x = F(x, u, , t)
(23)
where x is the state vector, u the control, and a random vector representing dynamic disturbances and/or uncertainties in process parameters. The relation between the observed and actual state vector is given by the equation
= G(x, 77)
(24)
NOVEMBER 1965
1993
values of the random process are usually considered to be independent in nonoverlapping intervals of time. This simplifying assumption allows the system to be treated as a Markoff process where the state at time t + dt depends on its past history only through its dependence on the state at time t. An elegant exposition of dynamic programing solutions of Markoffian processes is given by Howard.273 Dreyfus274 has recently discussed the problem of closed-loop optimal control. This paper provides a clear account of the advantages and disadvantages of dynamic programing, visa-vis other approaches. The formulations based on the classical variational methods usually approach the problem from an open-loop or "perfectly unobservable" point of view. If the system state is perfectly unobservable, then it is not possible to use feedback, and the control is necessarily open-loop. Several open-loop optimal synthesis procedures are based on the idea of using the control to modify the probability density function P[x(T), Tx(to), t0] of Eq. (27). In order to do this, it is necessary to determine the relation between P[x(T), T\x(to), to] and u(t), t0 <t <T. We assume that the system without control is disturbed by additive noise. The equations describing the process are (29)
where the are the random disturbances as before. The probability that x(t) belongs to a given region of the state space 0 at time t is
rived by dynamic programing by Kushner281 and by Wonham.282 Krasovskii and Lidski283 and Kramer284 employ a generalized Liapunov function for optimizing stochastic systems. These papers demonstrate the difficulty encountered in formulating the optimal feed-forward problem. The problem is sometimes simplified and made easier to solve by eliminating the time dependence of P and assuming the existence of a steady state. The details of the transient solution are lost in this simplification, and, in many problems with finite time of action (e.g., missile interceptions), the transient solution contains the interesting information.The question of the value of asymptotic control applied to problems of finite duration has been studied285 but has not been completely settled, i.e., what happens when a "steadystate" control is applied to a transient situation? It can easily be appreciated, from what has already beensaid, that the open-loop stochastic problem is equivalent toa multidimensional optimization problem with a partial differential equation as a side condition. Constraints of this kind, particularly partial differential equations of parabolic type (e.g., the heat conduction equation), appear in the optimization of industrial and chemical processes with distributed parameters, and a maximum principle has already been derived.286 A solution or an advance in computational techniques in either one of these areas (stochastic problems or distributed parameter problems) should find rapid application in the other.
Partially observable systems
P(x, t) dx
If the initial conditions are known precisely, then
A partially observable system is one governed by the equations
x = F(x, u, ,t)
where the 5?s indicate Dirac 5-functions. In the case where the initial conditions are random, the appropriate statistical information must be supplied. If the process is a Markoff process, then
s = G(x, 77)
(31)
P(x, t + dt) dx
is determined uniquely by the state of the system given at an earlier time r. In that case, it can be shown275 that P[x} t\x(0)] satisfies the following second-order parabolic partial differential equation, known as the Fokker-Planck equation
(30)
where the coefficients and ba can be derived from Eq. (29) and the statistical properties of ;. This means that Eq. (30), together with a description of the initial state, contains all of the information about the future of the process that we can get from the system (29). The evolution in time of the probability density of the state variables depends on the action of the control, and this relation must be incorporated into the formulation of the optimal control problem. Several investigators276"278 have taken the approach of using the Fokker-Planck equation or, alternatively, the "backward" equation or Kolmogorov equation. Katz279 and Florentin280 derive the Fokker-Planck equation for the probability density diffusion in space and time (assuming the process to be Markoffian) and employ the Weierstrass condition to determine the optimal control law. The difference between the feed-forward and feed-back approach is discussed in,detail by Katz, and some examples are given. A stochastic version of the Hamilton-Jacobi equation and a corresponding maximum principle have been formally de-
where s is the observation vector contaminated by noise 77. Research is being done on the individual problems of linear optimal filtering and optimal control as well as on the problem of optimizing the combined filter and controller. The problem of synthesizing a combined optimal nonlinear filter and control is very difficult, and fewer results are available than in the linear case. A consistent general theory of optimizing partially observable nonlinear systems has not yet been developed. A paper by Florentin,287 on the optimization of partially observable systems, considers a system with linear dynamics, quadratic cost function, independent Gaussian observation noise 77, and disturbances . The observation vector s is a linear function of the state x and of dimension less than or equal to that of x. The method centers on the use of Bayes' Theorem to update the probability distribution of the future behavior of the system, based on prior knowledge of probability distributions and present observations. In this way, the control law unfolds as a function of all the past observations. Another approach to the partially observable system involves filtering the observation signal to subtract out as much as possible of the unwanted noise. The problem of designing an optimal filter is well known and one for which a consistent theory exists, with the restriction that the filter be linear and that the input signal and noise have stationary statistical properties.288"291 The design of nonlinear filters is studied by Bose,292 and the state space approach to linear filtering has been developed by Kalman.293'134 Kalman's work is quite well known and has been applied in many different circumstances. There is a very large body of literature reporting work on the optimal estimation of system state variables and parameters. The optimal estimation problem has been reviewed elsewhere294-295 and will not be treated here. Kalman and Bucy134 have investigated optimal controllers for linear systems and obtained solutions to the combined
1994
B. PAIEWONSKY
AIAA JOURNAL
optimal control and filtering problem. They study the system dx/dt = Fz + Gu
formance criterion is an integral of a quadratic form
(32)
1) A flooding technique is used, whereby a great number of optimal trajectories are obtained emanating from some point of interest, either forwards or backwards in the inde-
pendent variable. Some examples are available in the work
of Kipiniak. It appears that this procedure is being used in chemical or industrial process control. The generation of switching surfaces by backward integration is in effect a flooding technique. The flooding techniques have received only 27 = \im{[x(T),Sx(T)] + fT [\\y\\*Q + \\u\\Wr} (33) r-*-oo / to limited attention in the aeronautical literature because the 2 wide range of initial and terminal conditions usually enwhere \\y\\ Q is the quadratic form y'Qy. countered makes it necessary to provide for the storage of The optimal regulatory control is shown to be linear in many paths. In some special cases, e.g., rocket boost and the state, i.e., injection into orbit, it is possible to limit the range of end u(t) = (R-Kr'P):*; = K(t)x (34) conditions and use flooding technique to generate the optimal control as a function of the state variables and time-to-go. The matrix P in Eq. (34) is the solution of the following maThis has been studied by Miner303 and others304""306 under trix equation: the name of path adaptive guidance. It is necessary to store ~dP/dt = F'P + PF - PGR-1 G'P + H'QH (35) one or more functions of several variables in a manner that allows rapid readout of the control. It is possible to comSuppose the system is perturbed by noise press the information by using suitable approximations, and some specific suggestions are available in the literature.307 x = F# + Gu + NT; (36) 2) A trial and error search technique, whereby an initial guess at the unknown parameters is made and successive reThe deterministic procedure is still optimal if 77 is white finements taken until the errors in the boundary conditions Gaussian noise in the sense that it will minimize the expected
The measurable output is y = H#. The deterministic per-
value of the deterministic performance index. This formulation does not take into account saturation of the control,
although the control manipulations can be reduced by put-
are reduced to tolerable values.308"312
pectation of the deterministic performance criterion.296 This
ting a large penalty on the control via the matrix R. The term "certainty equivalence" has been applied to the class of control laws which optimize both the deterministic problem and the stochastic version formed by taking the ex-
notion of certainty equivalence can be extended to include

more general questions, e.g., the relations between optimal controls for the same dynamic system and different probabil-
ity density distributions for additive noise. Some work has been done in this area, and also in the related area of the inverse problem of optimal control297 (e.g., for what class of problems is a given control optimal?), but these questions have not yet been completely and thoroughly explored.
V. Two-Point Boundary-Value Problems
Linear Problems
The optimization problem formulated earlier contains a number of unspecified parameters, which must be adjusted so that the boundary conditions are met. In a few cases this is not difficult to do, but, in the great majority of cases, it is an extremely difficult task. In the rare event that both the system equations and the adjoint equations are linear, and if the control is also linear in the state and adjoint variables, and if the range of independent variable is prescribed, then a method based on Green7s functions can be applied. This is described in detail in Courant-Hilbert.298 Kalman has studied the problem of optimal control of linear systems with quadratic cost criteria and has developed a synthesis procedure described in Sec. IV. It has recently been applied to aircraft autopilots by Rynaski et al.114 and Tyler.299 The existence of an optimal control in this case
depends upon complete controllability; uniqueness requires
complete observability. Optimal control of linear systems with quadratic criteria and bounded controls has also been studied extensively. 255,300-302 This problem is interesting because there is a bang-bang mode, a linear feed-back mode, and also a singular solution.
Direct Integration
The general idea of the direct integration method is fairly simple. The initial (or terminal) conditions of the adjoint are to be found by successive approximations such that the boundary conditions are satisfied when the coupled system equations and adjoint equations are integrated. In some cases, it is more convenient to integrate the adjoint equations backwards, but suppose that both equations are being integrated forward. The end conditions are observed after each run, a correction is made to the initial values of the adjoint, and the equations are integrated again. In order for this to work, it is necessary that small changes in initial conditions produce changes in the terminal conditions. Examples taken from elementary problems show that this does not always occur.312 What is more, the end conditions are usually extremely sensitive functions of the initial conditions, making the search very difficult. The direct integration method has found application in flight mechanics, particularly in trajectory optimizations. Breakwell,62 in an excellent paper in 1959, presented examples of trajectory optimization, including the maximum range boost of a ballistic missile, boost to maximum velocity, and minimum time-to-climb. Both forward and backward integrations are discussed, and some numerical tests are described. The equations are integrated numerically by a first-order predictor-corrector routine, and adjustments of the guessed values are made using linear interpolation. The procedure converged, but it was pointed out that some educated guessing was required before this occurred. A seconddegree interpolation scheme was suggested to reduce the "educated" guessing, but no numerical results of this were presented. An interesting collection of trajectory optimization papers is contained in Ref. 313. A technique employed in several of these papers is direct integration with a gradient method used to drive the adjoint variables (or Lagrange multipliers) to the correct value, f It was reported that success depended, to a very large extent, on the initial guess used to start things off and on the procedure used to adjust the step size during a descent. For the most part, the range of initial and terminal conditions studied was limited so that, once one optimal trajectory was obtained, some neighboring trajectories, if any existed, might be found by perturbing the initial conditions. An extensive computation facility was required
A method commonly used for solving differential equations

with two-point boundary conditions is direct integration of
in all cases.
t The gradient methods mentioned here are not the gradient methods in function space described in the next part of this section.
the system equations and adjoint equations. The solutions are usually obtained in one of the following two ways.
NOVEMBER 1965
1995
is a function of the coefficients c. A minimization of this function with respect to the c yields a set of equations to be solved for the coefficients. If the functional being extremized and the <l>i(x) satisfy certain conditions (e.g, form a complete set), then the limit as n -> <, if it exists, will be an exact Gradient Methods in Function Space solution. The question of the convergence of the approxiThe method of gradients or, as it is also called, the method mations is very important and should not be taken for of steepest descent, has been applied in the field of optimal granted. For example, the fact that a minimizing sequence control problems. The basic idea in the method of gradients of allowable functions converges does not mean that the is to search for the optimizing function in a class of functions limit function lies in the class of allowable functions. which satisfy the boundary conditions. In some cases, the The Rayleigh-Ritz method has been applied to the calcularequirement that the boundary conditions be satisfied can tion of optimal interplanetary trajectories by Saltzer and be relaxed and an additional term included in the performFetherorT.320 They consider a power limited rocket operatance criterion to account for errors in the end conditions. ing at the maximum power level and seek to maximize the The searching proceeds by successive approximations taken payload. They assume that no coasting arcs will occur. in the direction of steepest descent in the function space. A gradient method is used to solve the nonlinear equations The idea can be grasped by looking at an elementary case. for the coefficients of the approximating series. For example, the simplest problem of the calculus of variIn Galerkin's method, the equations are first integrated by ations seeks the extremum of the functional parts, and then the homogeneous boundary conditions are applied. Galerkin7s method has been applied to optimal (37) } = fL(x,y,y'}dx control problems by Boyanovitch.321'322 The gradient of the functional F[y(x) ] is Ho and Brentani249 have investigated a successive approximation procedure based on discretization of the problem. r (38) ((d/dx)F, - F,]|,w Another direct method applied to much simpler problems without inequality constraints is described in Courantand the Euler equations are obtained by requiring the funcHilbert298 and Elsgolc.350 tional to be stationary, i.e., the gradient vanishes when r In all of these direct methods, the problem becomes a (d/dx)(Fy ) Fy = 0. Instead of solving the Euler equaminimization of a function of many variables. tions, suppose that we guess a solution y(x) and evaluate There are many other attempts at finding satisfactory gradient [F], i.e., approximate methods for optimal control problems and two(d/dx)Fy' - Fy\yo(x) (39) point boundary problems. Some of these are listed in the references, but none will be discussed in any detail here. A Suppose a maximum of F is desired. An improvement plan that is often suggested solves a simplified version of the 8F in the value of F is sought by taking a small step dy in the problem and uses the resulting initial values for the Lagrange direction of the gradient so that multipliers to start the integration of the Euler equations for the original problem. The success ;of this scheme depends (40) = f[vF]-dy(x)dx
Studies of optimal rocket boost trajectories using direct integration have been made by Stancil and Kulikowski.60'61 Other studies of optimal trajectories are listed in the bibliography. Atmospheric flight problems are particularly difficult because the presence of dissipative force causes the Euler equations to be unstable. A study of optimal climb trajectories was carried out by Mengel88 on an analog computer. However, as mentioned earlier, the extreme sensitivity of the end conditions of the trajectory to the initial conditions of the adjoint made it difficult to satisfy the boundary conditions. The literature of the chemical and process industries contains accounts of attempts to optimize operations170'171 by direct integration of the equations. Brunner314 presents an iterative method designed for analog or hybrid analogdigital computation. Smith et al.315 apply the Newton-Raphson method and method of steepest descent to optimization problems. They point out several of the difficulties involved. Partial derivatives are evaluated numerically after integrating the coupled equations, and because of this the integration errors must be taken into account. If the changes in the initial values of the adjoint are too small, then integration errors will make the resulting partials meaningless. If the increments are too large, then the partial derivatives are not correct. A compromise must be sought between the numerical integration errors and the numerical differentiation errors. The conclusion drawn from the results of these studies, and from studying simple examples, is that direct integration of the system and adjoint equations and searching for the correct adjoint values can be done, but success requires 1) good starting values for the adjoint, and 2) a reliable means of finding an extreme value of a function of several variables. The difficulty in doing this obviously depends on the particular problem and also on the particular initial and terminal conditions. It has been found that some places are harder to reach than others. Descriptions of unsuccessful attempts rarely appear in print.316 This observation can ajso be applied to the flooding technique and the "switching surface" technique.
It is necessary to include an explicit statement that the step size be small, e.g., fdy*(x)dx < 1. The computational techniques, although not conceptually difficult, require careful development, and the reader is referred to the literature in Refs. 95,245-247,250,317,318 for details. The method was developed and applied to aeronautical problems by Kelley250 and Bryson and Denham.95 A tutorial account by Kelley with several examples is contained in Ref. 317. The method works well, although in some cases the convergence of the iterations near the optimum is reported to be slow as the minimum is approached. A generalized Newton-Raphson method, which is reported to give relatively rapid convergence, has been successfully applied to aeronautical problems by McGill and Kenneth.318
Direct Methods
The objective of the direct methods is to change the variational problem into a problem of ordinary maxima and minima. The solution of the variational problem and the two-point boundary-value problem is obtained by approximating the unknown function by a truncated series and performing an ordinary maximization over the coefficients of the series. The techniques most commonly used are the Rayleigh-Ritz and Galerkin methods.298'319 The Rayleigh-Ritz method is based on the idea of approximating the minimizing function by a linear combination of functions <t>i(x) over [a, 6]; <;(a) = 0(6) = 0, i = 1 . . . n: (41)
where <po(x) satisfies the boundary conditions. The functional to be minimized, e.g., b F(x, yn, yn')dx i
1996
B. PAIEWONSKY
AIAA JOURNAL
on how well the gradients of the payoff surfaces of the original problem correspond to the simplified versions. Some work along these lines has been done by Gragg,355 who applied his results to systems with quadratic performance criteria and orbital transfers. It will be interesting to see how well the initial adjoint vectors for optimal impulsive tranfers serve as starting values for low thrust orbital transfer iterations.
Optimal Evolution
The concept of optimal evolution of a dynamic system and the notion of attainable sets provide a convenient framework for examination of closed-loop optimal systems. The discussion in this section is limited to the nonstochastic case. At present, optimal synthesis based on these notions can be achieved only for a certain class of problems for linear systems; nevertheless, the basic idea is extremely powerful. Studies of optimal processes in terms of the principle of optimal evolution have been made by deJong,106 Halkin,323 Roxin,324 Isaacs,239 Contensou,325 and Leitmann and Blaquiere.326 Consider the optimization problem stated previously. The system equations are
Xi = fi(x,
i = 0, . . . n
(42)
The control u is contained in convex compact set U and is piecewise continuous. Find u(t) such that 1) there exists T > 0 and x(T) = x1} etc., and 2)
f
Jo
L(xi, uh)dt
Is minimized. The space t X xn+1 is of dimension n + 2. Define a reachable point as a point of t X xn+l (t, XQ, xiy . . . xn) attained by application of an allowable control and time t > 0, subject to initial conditions z(0) = [0, #0(0), . . . , &n(0)]. We call this set of reachable points R[t, x3-(Q)], j = 1... n. The reachable set is the set of all points that can be reached by the system using all permissible controls in a given time. The reachable sets are defined with respect to the initial point. The principle of optimal evolution states simply that each event of an optimal trajectory belongs to the boundary of the reachable set. The referenced articles show that the adjoint vector p employed in the definition of the maximum principle corresponds to the normal to a support plane at the boundary of the reachable sets. The synthesis problem can be changed into the problem of generating the reachable sets by solving a partial differential equation. The evolution or growth of the reachable set can be thought of in terms of a generalized Huygens principle. That is, elementary wavelets originating from points on the boundary of the reachable set at time t possess an envelope, and, at time t + dt, this envelope in turn becomes the new boundary of the reachable set, and the process of growth or evolution continues. We state without proof that the "motion" of the boundary of the reachable set in n + 1 dimension (x, T) space can be described by the following partial differential equation (Hamilton-Jacobi) if the derivatives exist. Let s be the optimal value of the payoff:
bs
Limiting processes whereby dynamic programing equations formulated in discrete terms are carried over into partial differential equations should be handled with great care. Pines330 points out that, finding all the constants of motion of a general dynamical system, is equivalent to solving the Hamilton-Jacobi equation. He has applied this observation to the solution of an orbital transfer problem for which he has found the constants of motion. Although the description given here is in terms of time running forwards, it is possible to develop this idea with time running backwards and the sets nested about the target point. This approach is also directly related to the ideas of dynamic programing and differential game theory. An early study with time reversed can be found in the report by Anderson.328 Studies of the geometry of the optimal payoff surfaces and their relation to the control law for the time-optimal problem have been made by Paiewonsky,312 Fliigge-Lotz and Halkin,331 and Leitmann and Blaquiere.326 The convexity of the set U and the fact that the origin is an interior point of U are important because the convexity of the reachable set, the piecewise continuity of the control, and the attainability of the extremum depend on these properties. A number of interesting examples illustrating the effect of nonconvexity of U can be found in Isaac's Rand studies.239 The usual practice is to replace the set U by the smallest convex set encompassing it and to solve the problem as if this were the set of admissible functions; this provides a bound on the payoff function. More recently, Neustadt332 has treated the optimal control problem in the absence of convexity conditions.
VI.
Minimizing Functions of Several Variables
max [Vs-x(x,u, t ) ] . ueU
(43)
The relationships between the Hamilton-Jacobi equation, optimization problems, the maximum principle, and Liapunov stability have been discussed by Leitmann and Blaquiere,326 Pontriagin et al.,6 APbrekht,327 Geiss et al.,322 Aoki,329 and Kalman and Bertram.194 Discrete forms of this equation resemble the main equations of the method of dynamic programing. It is important to remember that the payoff surfaces, i.e., surfaces of s = const, are not always smooth and that sharp corners and edges may occur.312
The methods for solving two-point boundary-value problems depend to a large extent on being able to find extreme values of functions of several variables. The minimization of a function of several variables is hardly ever an easy task, and in many cases it presents a formidable challenge. The functions are often troublesome to evaluate, and the derivatives may be unavailable except by numerical approximation. It is important to keep in mind the distinction between the problems of finding the minimum value accurately and finding the location of the minimum. In Neustadt's method, for example, the location of the maximum of a certain function determines the optimal initial conditions for the adjoint equations. The function being maximized is rather flat, and it is difficult to get an accurate estimate of the location of the maximum point, although the value of the maximum is easy to obtain. The problem we discuss is a minimization, but, clearly, finding maxima can be treated by making obvious modifications to the methods. Furthermore, the methods described here find local minima only. A minimum value of a function f(xi, x2, . . . Xrj) is to be found, subject perhaps to constraints of the form &(#) = 0, j = 1, . . . a, hk(xi) < 0, k = a + 1, . . . 77 1. The inequality constraints are sometimes appended to the problem in the form of a verbal description of the domain of definition of the function /. It is assumed that, wherever derivatives of f(x) exist, they are available to a computer, either by evaluating an explicit expression for d//d#, or by numerical differentiation. The function / is assumed to be continuous so the minimum is actually obtained. Minima may occur on the boundaries of the domain of definition, at interior points where bF/dxi = (b/cteO (/ + A^) = 0, and the constraints </;fe) = 0 are satisfied simultaneously, and det d^F/dxibX] is > 0, or at points where the derivatives do not exist; e.g., cusps and sharp edges. The necessary conditions provide no guarantee that there will be an absolute minimum.
NOVEMBER 1965
1997
The solution of the simultaneous equations for the points where the gradient vanishes is generally difficult to obtain, and the method seems practical for only the simplest problems. Even in the case where the equations are linear, it is still a nontrivial task to solve for the minimizing x when the number of equations is large. The problem of solving simultaneous linear equations is sometimes transformed into a minimization problem just to avoid the difficulties associated with matrix inversions. These remarks indicate that it is often impractical to find the minimum of a function analytically, and, because of this, methods have been developed to do this computationally. The problem is to find an algorithm for a computer such that a good approximation to the minimum is obtained after a reasonable number of steps. We concentrate attention on minimizing a continuous and differentiate function F(x) by iterative procedures based on selecting an initial choice x, and then seeking x1 such that F(x1) < F(x) and so on. In these iterative methods, an initial point x = (XIQ, x2, . . ., Xr,Q) is chosen, and the algorithm determines a vector gk (the direction of the step) and a real number hk (the size of the step) . The (k + 1) 'st point in the iteration is xk+1 = xk + hk gk. Sometimes a good guess for the starting value, or the first point tested, can be made based on intuition or some general properties of the problem. For example, in applying Neustadt's method to time-optimal control of linear systems, the convexity property of the optimal isochrones indicates that a satisfactory rule for starting the iterations is to select the direction of initial adjoint vector opposite to the system state vector. The simplest of the iterative methods is the univariate method, where the components of xk are varied one at a time. Suppose dF/dXi(xk) > &F/dxj(x*) for j = 1, . . . , i - 1, t + 1, . . . , n, i.e., the change in F in the direction of the ith coordinate is greater than along any other coordinate axis. Then we let gk = (0, 0, . . . , 0, -1, 0, . . . , 0), where -1 is in the ith place. To determine hk, we write
i OXj
(xk)
(44)
and minimize f(xk+1) with respect to hk:
d/dhkF(xk+l)
This gives
= 0
(45)
and Courant.343 We shall briefly sketch the general ideas underlying some of the very many convergence acceleration schemes for gradient methods. More detailed accounts are contained in the references. The variations of the steep descent idea differ according to the rules used to generate the step hk and the direction of the step gk. A simple rule for choosing the step size is to make it a constant. This rule, although simple to apply, may lead to difficulties in obtaining convergent iterations. We know that F(xk+l) will be less than F(xk) at the kth step [xk+l = xk hgkty*)], if h is sufficiently small. On the other hand, the process will move very slowly if h is too small, and a great many steps will be required to bring x close to the minimum. A technique in common use in overcoming problems of this kind is to choose a step size at each stage according to values of the function and its gradient evaluated at several trial points. One of the simplest step-changing routines makes an initial choice of h and, subsequently, doubles or halves the step size depending on the improvement obtained in F(x). As steps are taken in a given direction, say in the direction of VF, the values of F(x) (in that direction) will decrease initially but may begin to increase again for a large enough step away from the initial point. The step hk* corresponding to the point where the rate of change of F(x) along the line of march vanishes is called the optimum step. That is, the optimum step corresponds to finding the first local minimum along a line whose direction is specified by the rule of the gradient method being used. An obvious way to compute the optimum step is to choose some small quantity dh and successively compute F in the direction given by gk at the points xk + dh gk, xk + 2 dh gk, . . . . Sooner or later F will increase; if this happens at the point xk + m-dh gk, go one step back and halve dh, and try the point xk + (m %)-8h-gk), etc. The scheme is time consuming. Booth335 has developed a method that reduces the amount of computation but at the cost of decreased accuracy. The idea is to pass a quadratic polynomial through F(xk) and two other points along the line and to find the minimum of the polynomial analytically. One of the best available methods to find 'hk* for steepest descent and other gradient methods makes use of the fact that the derivative of the function along the line of march,
d/dh{F[xk + h vF(xk)}} = vF[xk +

(46)
hVF(xk)]-vF(xk)
vanishes at hk ~ hk*. We have to solve the equation
(47)
The scheme requires b^/ctoi2 to be nonvanishing. The convergence properties of the method are believed to be weaker than methods based on the gradient, and it is not recommended.333 The terms "gradient method" and "steep descent'' method as used here apply to minimization techniques that take successive steps in directions obtained by linear transformations of the local gradient vector.* The method of steepest descent is the special case wherein steps are always taken in the direction opposite to the local gradient vector. It is well known that the convergence of steep descent iterations can be slow. Special techniques have been developed to speed up the rate of convergence and eliminate time consuming oscillations about ridge crests. These accelerated gradient methods determine the direction of successive steps from observations of the gradient vectors at several points instead of using only the local gradient (for specific examples, see Refs. 334-337). There are many modifications of the idea of steep descent, and accounts of these are given in reviews by Edelbaum,338 Spang,339 Woodrow,333 Wilde,340 Shah et al.,341 Tompkins,342
J These methods are similar to, but not the same as, the gradient methods in function space developed by Bryson,95 Kelley,90 and others.
VF[xk + hk*'VF(xk)]-vF(xk)
= 0
(48)
for hk*, the optimal step size. March out from the starting point xk in the direction AF as shown in Fig. 3. As we move along the line, the direction of the local gradient is found at successive points. The
VF[x k +h*VF(x k )]
VF(x k + hVF(x k ))
Starting point Contour line
Fig. 3 The gradient direction method for finding the optimum step.
1998
B. PAIEWONSKY
AIAA JOURNAL
The discussion of closed-loop systems in this section approaches the problem from the viewpoint of recomputing the optimal control at successive instants of time, and perhaps taking into account the possibility of more disturbances occurring later on. It is possible to treat stochastic problems from the extremal field viewpoint, and discussions have appeared using dynamic programing to formulate functional equations, but very few tangible results have been published. The notion of a field of extremals is essential to understanding closed-loop optimal controllers. The rigorous formulation of the properties of fields of extremals can be found in Bliss,256 Bolza,234 or Elsgolc.350 Suppose we have a domain D in an n-dimensional space. If there exists one and only one curve of a family through 1 l m in x = ^Ffbxi ^Xj]~ -vF(x) = [H~ ]yF (49) each point, then we call this family a proper field in D. A curve is a member of a family if its defining equations conThis formula can be used to generate an iterative method for tain one or more parameters. If all the curves pass through finding the minimum point of a nonquadratic function by a common point (i.e., form a pencil, or bundle of rays), then using an optimum step routine to find the minimum in the they do not form a proper field in D. They form what is direction of H~1(xk) - VF(xk), recomputing the Hessian, called a central field, provided that there are no other points and repeating the procedure until the magnitude of ^F(x) of intersection within the domain. At all other points of is below a preselected value. the domain except the center of the pencil, the bundle of The Hessian of a general function is not a constant, but rays form a proper field. The size of the domain is related is a function of position, and the successive approximation to the question of sufficiency of the conditions for extrema. routine must provide for the computation of H~1 at each A field (proper or central) composed of a family of exstep. Furthermore, a closed form analytical expression tremals of a variational problem is called a field of extremals. for the Hessian may not be available, and numerical differIf one extremal of this field x(t) is known, we would like to entiation may be needed. find other neighboring members of this family of extremals The computation may be simplified by substituting a in order to develop an optimal guidance scheme. positive definite symmetric matrix in place of the unknown Suppose that an optimal path has been obtained to a Hessian. For example, the choice of the identity or unit deterministic optimization problem. Disturbances may be matrix as a replacement for the Hessian in Eq. (49) gives expected to occur which will force the system off its nominal the method of steepest descent, i.e., path in state space; we seek neighboring optimal paths (50) Xk+i _ Xk = -vp that will enable the requirements of the problem to be met. The successful application of these iterative methods deIt is clear that we may not want to return to the original path but might prefer to find a new optimal path starting pends on the location of the initial point being fairly close to from the current state. the minimum. If the initial guess is very bad, the method may not converge. Many of the gradient and accelerated Recent papers by Kelley,351 Breakwell, Speyer, and Bryson,352 and Merriam353 present a second-order variagradient methods are sensitive to scale changes, and the tional theory allowing the development of neighboring path possibility of improving the convergence rates by coordinate transformations should not be overlooked. Some converguidance schemes. Dreyfus100-274 and Rang354 have also gence acceleration methods do, in fact, generate coordinate studied the problem of closed-loop optimal control. The
arrows show the angle between ^F [xk + hkVF(x)] and V^ (xk) at a typical point. The point xk where f(xk) is perpendicular to the line of march is the desired stopping point. The solution of Eq. (48) can be obtained by trial and error, Newton's method, or any other method for finding a zero of a function of a single variable. The dot product in Eq. (47) is required to be differentiable with respect to hh, and there are simple examples in which the derivative along the line of march does not exist. The presence of grossly unequal eigenvalues of the quadratic form obtained from the second-order terms of the Taylor series expansion of F(x) (a ridge or ravine) may cause very unpleasant oscillations in the iterations. Gel'fand336 has developed a method for handling this, but reports of computational experience are scarce. The oscillations in the value of F(x) at successive steps may occur even when optimum step routines are employed if the other aspects of numerical analysis are not treated with care. Step-size adjustment schemes based on comparisons of F(x) at successive trials may fail to work simply because the magnitude of the difference between the observed values of the function is on the same order as the noise. In this case, the sign and value of the scalar product V[F(XI + hvF(xi)]-VF(xi) are useful guides for selecting the step size. A glance at the literature will show that a good deal of effort has been spent on accelerating the convergence of steep-descent methods and reducing the amount of computation necessary to get close to the minimum point. In particular, very powerful methods have been developed for minimizing quadratic polynomial functions, and convergence proofs and estimates of the number of steps required to reach the minimum have been obtained for the ideal case of a quadratic form in n variables. The hope is that similar behavior will be obtained when the method is applied to other functions. In many cases of interest, the function being minimized can be approximated satisfactorily in the vicinity of the minimum point by a quadratic form in n variables. A considerable simplification in the analysis is now possible, as the gradient of the approximating polynomial is a linear function of the coordinates, and the Hessian matrix H = b^F/dxifrXj is constant. The minimum point xmin for a quadratic function can be found in one step if the elements of the Hessian are known. Specifically,
transformation as part of their successive approximation calculations.344 The inverse of the Hessian plays such a role in Eq. (49) if we view H~l as a coordinate transformation, which maps the quadratic form into an n sphere. The minimum is easily found by taking one step of length R = [H~l]-\^F\ in the direction opposite the gradient. In a method originally due to Davidon345 and later modified by Fletcher and Powell,346 the inverse of the Hessian is obtained by successive approximations. The process is stable, and the minimum of a quadratic form in n variables is obtained in n iterations. Powell347 has described an effective method, which is also guaranteed to converge to the minimum in the ideal case of a quadratic, positive definite polynomial in n variables. Powell's method is closely related to the parallel tangent or 'Tar-Tan," methods that are reviewed in the previously mentioned article by Shah et al. Some of the more important technical details involved in the programing of Powell's method for a computer are described in reports by Woodrow333 and Paiewonsky et al.348'76 A comparison of convergence rates for Po well's method, Davidon-Fletcher-Powell method, and steepest ascent applied to Neustadt's synthesis method appears in a report by Paiewonsky and Woodrow.76 A hybrid analogdigital mechanization of Neustadt's method using a modified steepest ascent is described in Ref. 349.
VII.
Closed-Loop Optimal Control
NOVEMBER 1965
1999
connection between the payoff surfaces and the optimal control law has already been mentioned. If we regard optimal processes in terms of the maximum principle, we will need to find new initial conditions for the adjoint corresponding to each new position in state space. Let the system be displaced from the nominal path by an amount fe(0- Tne new initial conditions for the adjoint and hence the new optimal control law are obtained from the gradient of the optimal payoff curve through the point x + dx. A particular problem may possess a very narrow field of extremals, and a perturbation may put the initial point outside the domain of admissible trajectories. Suppose that we attempt to represent the payoff surfaces s in the vicinity of the original optimal path by an expansion in terms of the perturbations, e.g., a Taylor series or perhaps an expansion in orthogonal polynominals. The new initial conditions for the adjoint then determine the optimal control through the requirement that the Hamiltonian
H =
i=Q
be maximized (or minimized) . Let the payoff surfaces be approximated in the vicinity of the nominal path by (52) s(x The partials bs/dx; are known already as functions of time along the nominal optimal path. A "second-order" theory then requires the knowledge of the partials ^s/bxidx^ or bP;/dx? along the nominal optimal trajectory. If time does not appear explicitly in a problem, then the optimal control can be computed as a function only of the location of the system in the state space. This can be an important consideration because it means that it may not be necessary to compare the observed or estimated state with the nominal state at each instant. This depends on how well the instantaneous state can be estimated. In an autonomous ideal, perfectly observable case, errors in time measurement and errors due to disturbances causing translations along the nominal path in state space would not require corrective control action. The solution to the Hamilton-Jacobi equation can provide the solution to the guidance problem wherever the payoff surfaces are smooth. It actually may provide more than we may need because the Hamilton-Jacobi equation can give all the smooth payoff surfaces over the entire attainable region of the state space, and we may want them only in the vicinity of a special curve. The papers by Kelley,351 Breakwell et al.,352 and Merriam353 show how the partial derivatives can be generated along a nominal optimal path. The computation and storage of neighboring optimal paths has also been studied by Vance307 and others for guidance of large rocket boosters. Ellert and Merriam356 and Bailey357 have applied these ideas to optimal aircraft landing, and guidance and control of a rocket booster. It is important to keep in mind that deterministic problems formulated in the manner just described may blow up when random disturbances are added, because it may become impossible to meet the terminal point conditions. This is a familiar problem in terminal control design, and there are methods that attempt to avoid it. The terminal point can be replaced by a terminal set, the feedback gains can be allowed to saturate or be reduced as the range to target decreases, an alternative mode for the controller may be activated at a specified minimum range, and so forth. In those instances where there is an effort constraint, it may be very undesirable to make a closed-loop optimal control from a succession of deterministic open-loop solutions along the lines of a predictive terminal controller. Certain orbital transfers, for example, may have a rocket thrust program with some very short periods of burning. The
times when these pulses occur are determined by estimates of the system state; the introduction of noise into the system may produce spurious pulses with the result that the effort constraint boundary is reached, or the propellant exhausted, before the terminal conditions can be satisfied. Behavior of this kind may be avoided by filtering, separating the steering program from the thrust program, and perhaps introducing an auxilliary controller to steer about the nominal optimal path. Kelley101 has investigated terminal accuracies achievable using several different guidance laws. He presents some examples for steering about solutions to Zermelo's problem which show that feedback based on errors transverse to the nominal optimal path is very attractive from the standpoint of ease of generation of the control as well as reduction of terminal errors. It may be that a dual-mode controller will offer a practical solution to those situations where a closedloop optimal control is desired. Nominal optimal paths could be generated from time to time as needed, and a secondary mode of continuous control could be employed for steering about these paths. Another approach is to formulate the optimal control problem from the beginning in stochastic terms. The difficulties in solving stochastic optimization problems have already been pointed out, and ad hoc solutions may have to do temporarily until the studies of stochastic problems reach the stage where they allow reasonable computational solutions. Very little work has been published in the open literature to date on the computational and operational aspects of closed-loop optimal controllers. Some studies have been published which do treat the computational problems and a few comparisons of the effectiveness of algorithms are available.344'349'358-360 The emphasis in the publications seems only now to be shifting from the problems of finding individual optimal solutions to the new task of applying these ideas to optimizing closed-loop controllers with noise, random disturbances, and imperfectly known parameters. This area of engineering research will become increasingly important because the advance in computer technology (from the standpoint of size, weight, power, reliability, and cost) offered by micro-circuitry361'362 brings the level of achievable computer performance close to that required for aeronautical and astronautical application of closed-loop optimal control.
References
Bellman, R., Adaptive Control Processes, A Guided Tour (Princeton University Press, Princeton, N. J., 1961); also Bellman, R. and Stuart, E. D., Applied Dynamic Programming (Princeton University Press, Princeton, N. J., 1962). 2 Chang, S. S. L., Synthesis of Optimal Control Systems (McGraw Hill Book Co., Inc., New York, 1961). 3 Lawden, D. F., Optimal Trajectories for Space Navigation (Butterworth's Mathematical Texts, London, 1963). 4 Leitmann, G., Optimization Techniques with Applications to Aerospace Systems (Academic Press Inc., New York, 1962). 5 Miele, A., Flight Mechanics: Theory of Flight Paths (Addison-Wesley Publishing Co., Inc., Reading, Mass., 1962), Vol. I. 6 Pontriagin, L. S., Boltryanskii, V. G., Gamkrelidze, R. V., and Mishchenko, E. F., The Mathematical Theory of Optimal Processes (John Wiley and Sons, Inc., New York, 1962). 7 Proceedings of the 5ih Joint Automatic Control Conference (Stanford University Press, Stanford, Calif., 1964). 8 Proceedings of the 1st IF AC Congress (Moscow, 1960); also Proceedings of the 2nd IF AC Congress (Butterworths Scientific Publications, London, 1963). 9 Goddard, R. H., "A method of reaching extreme altitudes/7 Smithsonian Inst. Misc. Collections 71, no. 2(1919). 10 Malina, F. J. and Smith, A. M. O., "Flight analysis of the sounding rocket," J. Aeronaut. Sci. 5, 199-202 (1938). 11 Tsien, H. S. and Evans, R. C., "Optimum thrust programming for a sounding rocket," ARS J. 21, 99-107 (1951).
1
2000
B. PAIEWONSKY
AIAA JOURNAL
12 Ross, S., "Minimality for problems in vertical and horizontal rocket flight," Jet Propulsion 28, 55-56 (1958). 13 Miele, A., "Generalized variational approach to the optimum thrust programming for the vertical flight of a rocket," Purdue Univ., School of Aerospace Engineering A-57-1 (March 1957); also Air Force Office of Scientific Research-TN-57-173 (1957). 14 Leitmann, G., "A calculus of variations solution of Goddard's problem," Astronaut. Acta II, 55-62 (1956). 15 Garfinkel, B., "A. solution of the Goddard problem," SIAM J. Series A: Control I, 349 (1963); also "Errata: A solution of the Goddard problem," SIAM J. Series A: Control 2, 135(1964). 16 Hohmann, W., Die Erreichbarkeit der Himmelskorper (Oldenbourg, Munich, 1925); also NASA translation TT-F-44, (1960). 17 Barrar, R. B., "An analytic proof that the Hohmann-type transfer is the true minimum two-impulse transfer," Astronaut. Acta IX, 1(1963). 18 Lawden, D. F., "Optimal intermediate-thrust arcs in a gravitational field," Astronaut. Acta VIII, no. 2, p. 106 (1962). 19 Kopp, R. E. and Moyer, H. G., "Necessary conditions for singular extremals," AIAA Paper 65-63 (January 1965). 20 Bender, D. F. and McCue, G. A., "Numerical demonstration of the non-optimality of the Lawden spiral," llth Technical Meeting, Space Flight and Guidance Theory, NASA Marshall Space Flight Center (December 1962); also AIAA J. (submitted for publication). 21 Lawden, D. F., "Minimal trajectories," J. Brit. Interplanet. Soc. 9,179-186 (July 1950). 22 Lawden, D. F., "The determination of minimal orbits," J. Brit. Interplanet. Soc., II, 216-224 (September 1952). 23 Lawden, D. F., "Minimal rocket trajectories," ARS J. 23,360-365(1953). 24 Lawden, D. F., "Fundamentals of space navigation," J. Brit. Interplanet, Soc. 13, 87-101 (March 1954). 25 Lawden, D. F., "Stationary rocket trajectories," Quart. J. Mech. Appl. Mech. 7, 488-504 (December 1954). 26 Lawden, D. F., "Optimal programming of rocket thrust direction," Astronaut. Acta I, 41-56 (1955). 27 Lawden, D. F., "Optimum launching of a rocket into an orbit around the earth," Astronaut. Acta I, 185-190 (1955). 28 Edelbaum, T. N., "Some extensions of the Hohmann transfer maneuver," ARS J. 29, 864 (1959). 29 Carstens, J. P. and Edelbaum, T. N., "Optimum maneuvers for launching satellites into circular orbits of arbitrary radius and inclination," ARS J. 31, 943-949 (1961). 30 Munick, H., McGill, R., and Taylor, G. E., "Minimization of characteristic velocity for two-impulse orbital transfer," ARS J. 30, 638-639 (1960). 31 Altman, S. P. and Pistiner, J. S., "Minimum velocity increment solution for two-impulse coplanar orbital transfer," AIAA J. 1,435-442(1963). 32 Hoelker, R. F. and Silber, R., "The bi-elliptical transfer between circular coplanar orbits," Dept. of the Army Tech. Memo 2-59, Army Ballistic Missile Agency (1959). 33 Fimple, W. R., "Optimum midcourse plane changes for ballistic interplanetary trajectories," AIAA J. 1, 430-434 (1963). 34 Gobetz, F. W., "Optimum transfers between hyperbolic asymptotes," AIAA J. 1, 2034-2041 (1963). 35 Horner, J. M., "Optimum impulsive orbital transfers between coplanar orbits," ARS J. 32, 1082-1089 (1962). 36 Templeman, W. H., "Comparison of minimum energy paths and apogee designation paths," ARS J. 29, 865-868 (1959). 37 Progress Reports on Studies in the Fields of Space Flight and Guidance Theory (George C. Marshall Space Flight Center, Huntsville, Ala.) 38 Fosdick, G. E. arid Anthony, M. L., "3-d pulse optimization for vehicles disorbiting from circular orbits," Astronaut. Acta VIII, 343 (1962). 39 Eckel, K., "Optimum transfer between non-coplanar elliptical orbits," Astronaut. Acta. VIII, 177 (1962). 40 Godal, T., "Method for determining the initial velocity corresponding to a given time of free flight transfer between given points for a simple gravitational field," Astronaut. (Svenska)2,183(1961). ' 41 Long, R. S., "Transfer between non-planar elliptic orbits, small eccentricities," Astronaut. Acta. VI, 167 (1960). 42 Wang, W., "Minimum time transfer between co-planar,
circular orbits by two impulses and the propulsion requirements," Astronaut. Acta IX, 12 (1963). 43 Ting, L., "Optimum orbital transfer by several impulses," Astronaut. Acta VI, 256 (1960). 44 Edelbaum, T. N., "The use of high- & low-thrust propulsion in combination for space missions," J. Astronaut. Sci. IX, no. 2 (1962). 45 Leitmann, G., "Some remarks on the optimum operation of a nuclear rocket," 10th International Astronautical Congress, London (Springer Verlag, Berlin, 1959). 46 Bussard, R. W., "Some topics in nuclear rocket optimization," Optimization Techniques (Academic Press Inc., New York), Chap. 14. 47 Irving, J. H. and Blum, E. K., "Comparative performance of ballistic and low thrust vehicles for flight to Mars," Vistas in Astron. (Pergamon Press, New York, 1959). 48 Wang, C. J., Anthony, G. W., and Lawrence, H. R., "Thrust optimization of a nuclear rocket of variable specific impulse," ARS J. 29, 341 (1959). 49 Melbourne, W. G., "Three-dimensional optimum thrust trajectories for power limited propulsion systems," ARS J. 31,1723(1961). 50 Melbourne, W. G. and Sauer, C. G., Jr., "Optimum thrust programs for power-limited propulsion systems," Astronaut. Acta VIII (Institute of Aerospace Sciences, New York, 1962). 51 Fox, R. H., "Powered trajectory studies for low thrust space vehicles," ARS J. 31, 28 (1961). 52 Kelley, H. J., "Successive approximation techniques for trajectory optimization," Proceedings of the I AS Symposium on Vehicle Systems Optimization (Institute of Aerospace Sciences, New York, 1961), p. 10. 53 Hinz, H. K. and Moyer, H. G., "Three-dimensional low thrust interplanetary trajectory optimization," Proceedings, 2nd IF AC Congress (Basle, 1963). 54 MacKay, John S., "A variational method for the optimization of interplanetary round-trip trajectories," NASA TN D1660. 55 Hibbs, A. R., "Optimum burning program for horizontal flight," ARS J. 22, 204-212 (1952). 56 Fried, B. D., "On the powered flight trajectory of an earth satellite," Jet Propulsion 27, 641-643 (1957). 57 Bryson, A. E. and Ross, S. E., "Optimum rocket trajectories with aerodynamic drag," Jet Propulsion 28, 465-469 (1958). 58 Ross, S., "Composite trajectories yielding maximum coasting apogee velocity," ARS J. 29, 843-848 (1959). 59 Miele, A., "Mathematical theory of the optimum trajectories of a rocketfinal report," Purdue Univ., School Aeronautical Engineering Rept. A-58-10, p. 35 (November 1958); also Rept. AD-206 361; also Air Force Office of Scientific Research TR 58-154. 60 Kulakowski, L. J. and Stancil, R. L. "Rocket boost trajectories for maximum burnout velocity," ARS J. 30, 612-618 (1960). 61 Stancil, R. T. and Kulakowski, L. J., "Rocket boost vehicle mission optimizations," ARS J. 31, 935 (1961). 62 Breakwell, J. V., "The optimization of trajectories," J. Soc. Ind. Appl. Math. 7, 215-247 (June 1959). 63 Leitmann, G., "Optimal thrust direction for maximum range," J. Brit. Interplanet. Soc. 16, 503-507 (1958). 64 Lawden, D. F. and Long, R. S., "The theory of correctional maneuvers in interplanetary space," Astronaut. Acta VI, 48 (1960). 65 Lawden, D. F., "Optimal program for correctional maneuvers," Astronaut. Acta VI, 195 (1960). 66 Striebel, C. T. and Breakwell, J. V., "Minimum effort control in interplanetary guidance," I AS Paper 63-80 (January 1963). 67 Denham, W. F. and Speyer, J. L., "Optimal measurement and velocity correction programs for midcourse guidance," AIAA J. 2, 896 (1964). 68 Lawden, D. F., "Interplanetary rocket trajectories" Advances in Space Sciences, edited by F. I. Ordway III (Academic Press, New York, 1959), p. 1053, Vol. 1. 69 Leitmann, G., "The optimization of rocket trajectories," Progress in the Astronautical Sciences (North Holland Publishing Co., Amsterdam, Holland, 1962), Vol. I. 70 Miele, A., "Some recent advances in the mechanics of terrestrial flight," Jet Propulsion 28, 581-587 (1958). 71 Bergqvist, B., "The optimization problem for rocket vehicles subjected to medium and high accelerations; a literature survey,"
NOVEMBER 1965
2001
Svenska Interplanetariska Sallskapet Astronautik 1, 101 (1959). 72 Goldstein, A. A., Greene, A. H., and Johnson, A. J., "Fuel optimization in orbital rendezvous/7 AIAA Progress in Astronautics and Aeronautics: Guidance and Control II, edited by R. C. Langford and C. J. Mundo (Academic Press, New York, 1964), Vol. 13, pp. 823-844. 73 Kelley, H. J. and Dunn, J. C., Proceedings, 2nd IF AC Congress on Automatic Control (Butterworths Scientific Publications, London, September 1963). 74 Hinz, H. K., "Optimal low-thrust near-circular orbital transfer," AIAA J. 1, 1367-1371 (1963). 75 Mclntyre, J. and Crocco, L., "Optimal transfer between close circular orbits," AIAA Paper 64-29 (January 1964). 76 Paiewonsky, B. H. and Woodrow, P. J., "Study of time optimal rendezvous in three dimensions," Wright Patterson Air Force Base, Ohio, TR-65-20, Vol. I (August 1964). 77 Neustadt, L. W., "Synthesizing time optimal control systems," J. Math. Anal. Appl. 1, 4 (December 1960). 78 Meditch, J. S., "On the problem of optimal thrust programming for a lunar soft landing," Inst. Elec. Electron. Engrs., Trans. Auto. Control AC-9, 477 (1964). 79 Hall, B. A., Dietrich, R. G., and Tiernan, K. E., "A minimum fuel vertical touchdown lunar landing guidance," AIAA Progress in Astronautics and Aeronautics; Guidance and Control II, edited by R. C. Langford and C. J. Mundo (Academic Press, New York, 1964), Vol. 13, pp. 965-994. 80 Neustadt, L. W., "A general theory of minimum-fuel space trajectories," University of Michigan TR06181-1-T (November 1964). 81 Perkins, C. D. and Hage, Airplane Performance, Stability and Controls (John Wiley and Sons, Inc., New York, 1949). 82 Lippisch, A., "Performance theory of airplanes with jet propulsion," Headquarters, Air Material Command, Transportation Rept. F-TS-685-RE (1946). 83 Lush, K. J., "Optimum climb theory and techniques of determining climb schedules from flight tests," NATO AGARD Flight Test Manual (1954), Vol. I, Pt. I, Chap. 7; also AFFTC-
92 Arens, M., "Some requirements for the efficient attainment of range by airborne vehicles," ARS J. 30, (August 1960). 93 Bell, D. L, "The determination of maximum range for an unpowered atmospheric vehicle," J. Roy. Aeron. Soc. 68, 111 (February 1964). 94 Miele, A., "On the flight path of a hypervelocity glider boosted by rockets," Proceedings of the 10th International Astronautical Congress (Springer Verlag, Berlin, 1959). 95 Bryson, A. E., Denham, W. F., Carroll, F. J. and Mikami, M., "Determination of lift or drag programs to minimize reentry heating," J. Aerospace Sci. 29 (April 1962). 96 Levinsky, E. S., "Application of inequality constraints to variational problems of lifting re-entry," I AS Paper 61-21 (January 1961); also Wright Air Development Div. Rept. TR 60-369 (July 1960). 97 "Flight control study of a manned re-entry vehicle," General Electric Co., Wright Air Development Div. TR 60-695 (July 1960), Vol. 1. 98 Bryson, A. E. and W. F. Denham, "Multivariable terminal control for minimum mean square deviation from a nominal path," Raytheon Co. .Rept. No. BR-1333 (September 1960); also Proceedings of the I AS Symposium on Vehicle Systems Optimization (New York, 1961).
TN-56-13(1956). 84 Garfinkel, B., "Minimal problems in airplane performance," Quart. Appl. Math. 9, no. 2 (1951). 85 Miele, A., "On the non-steady climb of turbo-jet aircraft," J. Aeronaut. Sci. 21, 781-783 (November 1954). 86 Cicala, P. and Miele, A., "Brachistochronic maneuvers of a variable mass aircraft in a vertical plane," J. Aeronaut. Sci. 22, 577-578 (August 1955). 87 Hestenes, M. R., "A general problem in the calculus of variations with applications to paths of least time," RAND RM 100, Rand Corp., Santa Monica, Calif. (March 1949). 88 Mengel, A. S., "Optimum trajectories," Proceedings of Project Cyclone Symposium I on REAC Techniques (1951). 89 Kelley, H. J., "An investigation of optimal zoom-climb techniques," J. Aerospace Sci. 26, 794-802, 824 (1959). 90 Kelley, H. J., "Gradient theory of optimal flight paths," ARSJ. 30, 947-954 (1960). 91 Bell, D. J., "A review of flight path optimization . . . in the period 1945-1960," J. Roy. Aeron. Soc. 67, 119 (February 1963).
99 Zermelo, E., "Uber das navigations Problem bei ruhender oder veranderlicher wind Verteilung," Z. Angew Math Mech 11,115(1931). 100 Dreyfus, S. E. and Elliott, J. R., "An optimal linear feedback guidance scheme," J. Math. Anal. Appl. 8, 364-386 (1964). 101 Kelley, H. J., "An optimal guidance approximation theory," Inst. Elec., Electron. Engrs. Trans. Auto. Control AC-9, 375 (October 1964). 102 yon Mises, R., "Zum navigations Problem der Luftfahrt," Z. Angew. Math. Mech. 11, 373 (1931). 103 Levi-Cevita, T., "Uber Zermelo's luftfahrt Problem," Z. Angew. Math. Mech. 11, 314A (August 1931). 104 Faulkner, F. D., "Optimum ship routing," Navigation (1963-1964). 105 Haltiner, G. J., Hamilton, H. D., and Arnason, G., "Minimal-time ship routing," J. Appl. Meteorol. 1, 1-7 (March 1962). 106 DeJong, H. M., "Theoretical aspects of aeronavigation and its application in aviation meteorology," Koninkl. Ned. Meteorol. Inst., no. 64(1956). 107 Rekasius, Z. V., "A general performance index for analytical design of control systems," Inst. Radio Engrs., Trans. Auto. Control AC-6 (May 1961). 108 Zaborsky, J., "The development of performance criteria for automatic control systems," Am. Inst. Elec. Engrs. Inst. Tech. Groups Auto. Control 1, no. 2 (1962). 109 Gibson, J. E., et. al., "A set of standard specifications for linear automatic control systems," Am. Inst. Elec. Engrs. Trans., pt. 11(1961). 110 Kalman, R. E., Ho, Y. C., and Narendra, K. S., "Controllability of linear dynamical systems," Contributions to Differential Equations (The Macmillan Co., New York, 1961), Vol. 1. 111 Kalman, R. E., "Mathematical description of linear dynamical systems," Research Institute for Advanced Studies, TR 62-18 (November 1962). 112 Walkovitch, J., Magdaleno, R. E., McRuer, D. T., Graham, F. D., and McDonnell, J. D., "Performance criteria for linear constant-coefficient systems with deterministic inputs," Wright Patterson Air Force Base, Ohio, Aeronautical Systems Div. TR 61-501 (1961). 113 Magdaleno, R. and Walkovitch, J., "Performance criteria for linear constant-coefficient systems with random inputs," Wright Patterson Air Force Base, Ohio, Aeronautical Systems Div. ASD-TDR-62-470 (1962). 114 Reynolds, P. A. and Rynaski, E. G., "Application of optimal linear control theory of the design of aerospace vehicle control systems," Proceedings of the ASD Optimal System Synthesis Conference (1962). 115 Rynaski, E. G., Reynolds, P. A., and Shed, W. H., "Design of linear flight control systems using optimal control theory," Wright Patterson Air Force Base, Ohio, Aeronautical Systems Div. ASD-TDR-63-376 (April 1964). 116 Sheridan, T. B., "The human operator in control instrumentation," Progress in Control Engineering (Heywood and Co., London, 1962), Vol. 1, pp. 143-197. 117 McRuer, D. T. and Graham, D., "Pilot-vehicle control system analysis," AIAA Progress in Astronautics and Aeronautics: Guidance and ControlII, edited by R. C. Langford and C. J. Mundo (Academic Press Inc., New York, 1964), Vol. 13, pp. 603-621. 118 Ashkenas, J. L., Jex, H. R., and McRuer, D. T., "Pilotinduced oscillations; their cause and cure," Systems Technology Inc., Rept. TR-239-2 (June 1964); also A'Harrah, R. C., J. Aircraft 1(1964). 119 Cromwell, C. H., Ill, and Ashkenas, J. L., Systems Technology Inc., TR-124-1 (June 1962); also Durand, T. S. and Jex, H. R., Wright Patterson Air Force Base, Ohio, Air Systems Div., ASD-TR-62-507 (May 1962). 120 Passera, A. L. and Willoch, R. G., Jr., "An optimum switching criterion for a third order contactor acceleration control system," NACA TN 3743 (August 1956). 121 Schmidt, S. F. and Harper, E. V., "The design of feedback control systems containing a saturation type nonlinearity," NASA TN D-324 (September 1960). 122 Flugge-Lotz, I., Taylor, C. F., and Lindberg, H. E., "Investigation of a nonlinear control system," NACA Rept. 1391 (1958). 123 Ziebolz, H. and Paynter, H. M., "Possibilities of a twotime scale computing system for control and simulation of dynamic systems," Proc. Nat. Electron. Conf. 9, (1954). 124 Coales, J. F. and Noton, A. R. M., "A non-off servomecha-
2002
B. PAIEWONSKY
AIAA JOURNAL
nism with predicted change-over," Proc. Inst. Elec. Engrs. (London) 103 (July 1955). 125 Chestnut, H., Sollecito, W. E., and Troutman, P. H., "Predictive control application," Am. Inst. Elec. Engrs. Paper 61-12 (July 1961). 126 Dow, P. C., Fields, D. P., and Scammell, F. H., "Automatic re-entry guidance at escape velocity," AIAA Progress in Astronautics and Rocketry: Guidance and Control, edited by R. E. Roberson and J. S. Farrior (Academic Press Inc., New York, 1962), Vol. 8, pp. 271-308. 127 Morris, R. V. and Steeg, C. W., "Multicondition terminal control systems for aircraft," J. Aerospace Sci. 712 (1960). 128 O'Hern, E. A. and Smyth, R. K., "Terminal control system applications," Inst. Radio Engrs., Trans. Auto. Control AC-6 (May 1961). 129 Wingrove, R. C. and Coate, R. E., "Piloted simulator tests of a guidance system which can continuously predict landing point of a low L/D vehicle during atmosphere re-entry," NASA TN D 787 (March 1961); also Wingrove, R. C., "Survey of atmosphere re-entry guidance and control methods," AIAA J. 1,2019-2029(1963). 130 Weiner, N., The Extrapolation, Interpolation and Smoothing of Stationary Time Series (John Wiley and Sons, Inc., New York 1949). 131 Zadeh, L. and Ragazzini, J., "An extension of Weiner's theory of prediction," J. Appl. Phys. 21, 645-655 (1950). 132 Bharucha-Reid, A. T., Elements of the Theory of Markov Processes and Their Applications (McGraw-Hill Book Co., Inc., New York, 1960). 133 Ruina, J. P. and Van Valkenburg, M. E.; "Stochastic analysis of automatic tracking systems," International Federation on Automatic Control Conference, Moscow, 1960 (Butterworths Scientific Publication, London, 1962). 134 Kalman, R. E. and Bucy, R. S., "New results in linear filtering and prediction theory," J. Basic Eng. 95 (March 1961). 135 Stewart, E. C., "An explicit linear filtering solution for the optimization of guidance systems with statistical inputs," NASA TN D-685 (February 1961). 136 McLean, J. D., Schmidt, S. F., and McGee, L. A., "Optimal filtering and linear prediction applied to a midcourse navigation system for the circumlunar mission," NASA TN D-1208 (March 1962). 137 Tung, F., "Linear control theory applied to interplanetary guidance," Inst. Elec. Electron. Engrs. Trans. Auto. Control AC-9, 82 (January 1964). 138 Battin, R., "Optimizing a statistical navigation procedure for space flight," ARS J. 32, 1681-1696 (1962). 139 Booton, R. C., "The analysis of non-linear control systems with random inputs," Proceedings of the Symposium on Nonlinear Circuit Analysis (Polytechnic Press, Brooklyn, N. Y., 1953), pp. 402-411. 140 Schwartz, L., "Optimum filter technique for terrain avoidance under G-limiting constraint," Wright Air Development Div. TR 60-709 (October 1960). 141 Swaim, R. L., Inst. Elec. Electron. Engrs., Trans. Auto. Control AC-9, 508 (October 1964); also Houbolt, J. C., Steiner, R., Pratt, K, G., NASA TR R-199, p. 115 (June 1964). 142 Quinlivan, R. P., Tye, G., Westerhold, H. H., "Analytical investigation of control requirements for high speed low altitude penetration, final report brief," Flight Dynamics Lab., Research and Technology Div., Wright Patterson Air Force Base, Ohio, FDL-TDR-64-104 (1964). 143 Moore, F. B. and Brooks, M., "Saturn ascending phase guidance and control techniques," ARS Preprint 2458-62 (July 1962). 144 Geissler, E., "Problems in attitude stabilization of large guided missiles," Aerospace Eng. 19 (October 1960). 145 Burke, H. H., "Aerodynamic load reduction techniques for large elastic launch vehicles," Inst. Elec. Electron. Engrs., Trans. Auto. Control AC-9, 565 (October 1964). 146 Athans, M., Falb, P. L., Lacoss, R. T., "Optimal control of self-adjoint systems," Inst. Elec. Electron. Engrs., Trans. Appl. Ind. 83 (May 1964); also see Ref. 160, 147 Lee, E. B., "Discussion of satellite attitude control," ARS J. 32, 981 (1962). 148 Suddath, J. H. and Carney, T. M., 'Technique for synthesis of constant linear dynamical systems with a bang-bang controller," NASA TR R-200, p. 33 (August 1964). 149 Pistiner, J. S., "On-off control system for attitude stabilization of a space vehicle," ARS J. 29, 283 (April 1959).
150 Ergin, E. L, Norum, V. D., and Windeknecht, T. G., "Techniques for analysis of nonlinear attitude control systems for space vehicles," Aeronautical Systems Div. ASD-TDR-62-208, Vols.I-IV(1962). 151 Roberson, R. E., "Methods for the control of satellites and space vehicles," Wright Air Development Div. TR-60-643 (July 1960). 152 Ergin, E. L, "Current status of progress in attitude control," AIAA Progress in Astronautics and Aeronautics: Guidance and Control-II, edited by R. C. Langford and C. J. Mundo (Academic Press, New York, 1964), Vol. 13, p. 7. 153 "Single axis attitude regulation of extra atmospheric vehicles," Aeronautical Systems Div., ASD TR 61-129 (February 1962). 154 Weiss, H. K., "Analysis of relay servomechanisms," J. Aeronaut. Sci. 13, 364 (July 1946). 155 Wang, P. K. C., "Analytical design of electrohydraulic servomechanisms with near time-optimal response," Inst. Elec. Electron, Engrs., Trans. Auto. Control AC-8 (January 1963). 156 Sawaragi, Y., Hajime A., and Tsuyoshi O., "On-off control systems operating on sampled data," Bull.-Japan Soc. Mech. Engrs. 4, 489-499 (August 1961). 157 Flugge-Lotz, I. and Marbach, H., "The optimal control of some attitude control systems for different performance criteria," Proceedings of the Joint Automatic Control Conference (1962). 158 Meditch, J. S., "On minimal-fuel satellite attitude controls," Inst. Elec. Electron. Engrs., Trans. Appl. Ind., no, 71. p. 120 (March 1964). 159 Friedland, B., "A minimum response-time controller for amplitude and energy constraints," Inst. Radio Engrs., Trans. Auto. Control AC-7 (January 1962). 160 Athans, M., Falb, P. L., and Lacoss, R. T., "Time-, fuel-, and energy-optimal control of nonlinear norm-invariant systems," Inst. Elec. Electron. Engrs., Trans. Auto. Control AC-8 (July 1963). 161 Chang, S. S. L., "Minimal time control with multiple saturation limits," Inst. Elec. Electron. Engrs., Trans. Auto. Control AC-8 (January 1963). 162 Knudson, H. K., "An iterative procedure for computing time optimal controls," Inst. Elec. Electron. Engrs., Trans. Auto. Control AC-9, 23 (January 1964). 163 Kalman, R. E., "The theory of optimal control and the calculus of variations," Research Institute for Advanced Studies, Baltimore, Md., TR 61-3 (1961). 164 Lee, E. B., "Geometric properties and optimal controllers for linear systems," Inst. Elec. Electron. Engrs., Trans. Auto. Control AC-8, 379 (October 1963). 168 Kalman, R. E. and Koepche, R. W., "Optimal synthesis of linear sampling control systems using generalized performance indices," Trans. Am. Soc. Mech. Engrs., p. 1120 (November 1958). 166 Katz, S., "A discrete version of Pontriagin's maximum principle," J. Electron. Control XIII, 179 (August 1962). 167 Hopkin, A. M. and Wang, P. K. C., "Further studies of relay-type feedback control systems designed for random inputs," International Federation on Automatic Control Conference, Moscow (Butterworths Scientific Publications, London, 1960); also Hopkin, A. M. and Wang, P. K. C., "A relay type feedback system control design for random inputs," Trans. Am. Inst. Elec. Engrs. 78, pt. II 228, (September 1959). 168 Ho, Y. C., "On the stochastic approximation method and optimal filtering theory," J. Math. Anal. Appl. 6, 152-154 (1962). 169 Aoki, M., "On minimum of maximum expected deviation from an unstable equilibrium position of a randomly perturbed control system," Inst. Radio Engrs., Trans. Auto. Control AC-7 (March 1962). 170 Aris, R., The Optimal Design of Chemical Reactors (Academic Press Inc., New York, 1961). 171 Lupfer, D. E. and Johnson, M. L., "Automatic control of distillation column to achieve optimum operation," Proceedings of the 1963 Joint Automatic Control Conference (1963), p. 145; also Kipiniak, W. and Gould, L. A., Am. Inst. Elec. Engrs. Trans., pt. I, 79 (January 1961). 172 Perry's Chemical Engineering Handbook (McGraw-Hill Book Co., Inc., New York), 4th ed., pp. 22-51 to 22-60. 173 Chestnut, H., Duresch, R. R., and Gaines, W. M., "Automatic optimizing of a poorly defined process, part I," Joint Automatic Control Conference, Paper 8-1 (1962); also "Auto-
NOVEMBER 1965
2003
matic optimizing of a poorly defined process, part II," Proceedings of the 1963 Joint Automatic Control Conference (1963), p. 54. 174 Box, G. E. P. and Wilson, K. B., "On the experimental attainment of optimum conditions," J. Royal. Statistical Soc. B. B. 1 (1951); also Box, G. E. P., "The exploration & exploitation of response surfaces: Some general considerations & examples," Biometrics 10,16 (1954). 175 Draper, C. and Li, Y., "Principles of optimizing control systems and an application to the internal combustion engine," Mech. Eng. 74, 145 (February 1952). 176 Tsien, H. S., Engineering Cybernetics (McGraw-Hill Book Co., Inc., New York, 1954). 177 Rose, N. J., "Optimum switching criteria for discontinuous automatic controls," Institute of Radio Engineers Convention Record, pt. 4, p. 61 (1956). 178 Fuller, A. T., "Relay control systems optimized for various performance criteria," Proceedings of the International Federation on Automatic Control Congress (Butterworths Scientific Publications, Ltd., London, 1960). 179 Flugge-Lotz, I. and Lindbert, H. E., "Studies of second and third order contactor control systems," pt. I, TR 116, Div. of Engineering Mechanics, Stanford Univ. (1959). iso King, R. E., "Passive optimization of a relay control system subjected to random inputs," J. Electron. Control XV, 617 (December 1963). 181 Izawa, K. and Weaver, L., "Relay type feedback control systems with dead time and sampling," Trans. Am. Inst. Elec. Electron. Engrs., p. 49 (May 1959). 182 Flugge-Lotz, L, Taylor, C. F., and Lindberg, H. E., "Investigation of a nonlinear control system," NACA Rept. 1391 (1958). 183 Andre, J. and Seibert, P., "The local theory of piecewise continuous differential equations," Contributions to the Theory of Nonlinear Oscillations (Princeton University Press, Princeton, N.J.,1960),Vol.5. 184 Andre, J. and Seibert, P., "After endpoint motions of general discontinuous control systems and their stability properties," Proceedings of the International Federation on Automatic Control Congress (Butterworths Scientific Publications, Ltd., London, 1960). iss Ayzerman, M. A. and Gantmacher, F. R., "Some aspects of the theory of a nonlinear automatic control system with discontinuous characteristics," Proceedings of the International Federation on Automatic Control (Butterworths Scientific Publications, Ltd., London, 1960). 186 Filippov, A. G., "Application of the theory of differential equations with discontinuous right-hand sides to non-linear problems in automatic control," Proceedings of the International Federation on Automatic Control Congress (Butterworths Scientific Publications, Ltd., London, 1960). 187 Filippov, A. G., "Differential equations with discontinuous right hand sides," Rec. Math. 51 (1960). 188 Tsypkin, la. Z., "VIIIANIE Sluchainykh Pomekh Na Periodicheskii Rezhim V Releinykh Avtomaticheskikh Sistemakh," USSR, An, Dokl. 139, 570-573 (July 21, 1961). 189 Molitor, J. H., "Ion propulsion system for stationarysatellite control," J. Spacecraft Rockets 1, 170-175 (1964). 190 Brown, S. C., "Predicted performance of on-off systems for precise satellite attitude control," NASA TN-D-1040 (July 1961). 191 Abzug, M. J., Martin, L. W., and Ohgi, F. T., "A time optimal attitude control system designed to overcome certain sensor imperfections," AIAA Progress in Astronautics and Aeronautics: Guidance and ControlII, edited by R. C. Langford and C. J. Mundo (Academic Press Inc., New York, 1964), pp. 231-259. 192 Kuba, R. E. and Kazda, L. F., "A phase-space method for the synthesis of nonlinear servomechanisms," Am. Inst. Elec. Engrs. Trans., Appl. Ind., p. 282 (November 1956). 193 Bass, R. W., "Discussion of a paper by A. M. Letov, proc. Heidelberg conf. on automatic control," Regelungstechnik: Moderne Theorien und ihre Verwendbarkeit (R. Oldenbourg, Munich, 1957), pp. 209-210. 194 Kalman, R. E. and Bertram, J. E., "Control system analysis and design via the 'second method' of Liapunov," J. Basic Eng., pt. I, p. 371 (June 1960). 195 Rauch, L. L. and Howe, R. M., "A servo with linear operation in a region about the optimum discontinuous switching curve," Proceedings of the Symposium on Nonlinear Circuit Analysis (Polytechnic Press, Brooklyn, N. Y., 1956). 196 Doll, H. G. and Stout, T. M., "Design and analog computer
analysis of an optimum third order nonlinear servomechanism," Trans. Am. Soc. Mech. Engrs. 79, p. 513 (1957). 197 McDonald, D. C., "Intentional non-linearization of servomechanisms," Proceedings of the Symposium on Nonlinear Circuit Analysis (Polytechnic Press, Brooklyn, N.Y., 1953), pp. 402-411. 198 Schieber, L., "Variably damped servomechanisms," Am. Inst. Elec. Engrs. Trans., Appl. Ind., p. 414 (January 1957). 199 Ku, Y. H., "Analysis of servomechanisms with nonlinear feedback control," Trans. Am. Inst. Elec. Engrs., Appl. Ind. (January 1957). 200 Flugge-Lotz, L, "Discontinuous automatic control," Appl. Mech. Rev. 14, p. 581 (1961). 201 Flugge-Lotz, I., Discontinuous Automatic Control (Princeton University Press, Princeton, N. J., 1953). 202 Kazda, L., "Control system optimization using computers as control system elements," Proceedings of Computer in Control Systems Conference (American Institute of Electrical Engineers, New York, 1958). 203 McDonald, D., "Nonlinear techniques for improving servo performance," Proc. Nat. Electron. Conf. 6, 400-421 (1950). 204 Kreindler, E., "Contributions to the theory of time-optimal control," J. Franklin Inst. 275 (April 1963). 205 Bushaw, D., "Optimal discontinuous forcing terms," Contributions to the Theory of Nonlinear Oscillations (Princeton University Press, Princeton, N. J., 1958), Vol. IV. 206 Bellman, R., Glicksberg, L, and Gross, O., "On the bangbang control problem," Quart. Appl. Math. 14, 11-18 (1956). 207 Bellman, R. F., Glicksberg, L, and Gross, O. A., "Some aspects of the mathematical theory of control processes," Rand Rept. R-313 (January 1958). 208 LaSalle, J. P., "The time optimal control problem," Contributions to the Theory of Nonlinear Oscillations (Princeton University Press, Princeton, N. J., 1960), Vol. V. 209 Gamkrelidze, R. V., "The theory of time optimal processes in linear systems," Bull. Acad. Sci. USSR English Transl. 2, 449-474(1958). 210 Markus, L. and Lee, E. B., "On the existence of optimal controls," J. Basic Eng. 84, 13-20 (March 1961); also Arch Rational Mech. Analysis 8, 36-58 (1961). 211 Rozonoer, L. L, "L. S. Pontriagin's maximum principle in the theory of optimum systems, I, II, III," Avtomat. i Telemeh. 20, 1320-1334, 1441-1458, 1561-1578 (October, November, December 1959); Automation Remote Control 20, 1288-1302, 1405-1421, 1517-1532 (June, July, August 1960); transl. 212 Krasovskii, N. N., "On the theory of optimum regulation," Automation Remote Control 18, 1005 (November 1957). 213 Kulikowski, R., "Optimizing processes and synthesis of optimizing automatic control systems with nonlinear invariable elements," Proceedings of the International Federation on Automatic Control Congress (Butterworths Scientific Publications, Ltd., London, 1960). 214 Kranc, G. M. and Sarachik, P. E., "An application of functional analysis to the optimal control problem," J. Basic Eng. 143 (June 1963). 215 Lee, E. B., Inst. Radio Engrs., Trans. Auto. Control AC-5, 283 (1960). 216 Balakrishnan, A. V., SIAM J. Control 12, 109-127 (1963). 217 Achieser, N. L, Theory of Approximation (Frederick Ungar Publishing Co., New York, 1956). 218 Desoer, C. A., "The bang-bang servo problem treated by variational technique," Inform. Control 2, 333-348 (December 1959). 219 Bass, R. W., "Equivalent linearizationnonlinear circuit synthesis, and the stabilization and optimization of control systems," Proc. Symp. Nonlinear Circuit Analysis, Polytechnic Inst. of Brooklyn 16,163-198 (1956). 220 Neustadt, L. W., "Minimum effort control systems," Soc. Ind. Appl. Math. J. Control 1, (1963). 221 Gamkrelidze, R. V., "Theory of time-optimal processes in linear systems," Izv. Akad. Nauk K. SSR, Ser. Mat. i Mekhan. 22, 449 (1958); Rept. 61-7, Univ. of California, Los Angeles (January 1961); transl. 222 Krasovskii, N. N., Prikl. Mat Met. 23, 625-639 (1959). 223 Eaton, J. H., "An iterative solution to time-optimal control," J. Math. Anal. Appl. 5, 329-344 (1962). 224 Ho, Y. C., "A successive approximation technique for optimal control systems subject to input saturation," J. Basic Eng. 84, 33-40 (March 1962). 225 Paiewonsky, B. H. and Woodrow,~P. J., "The synthesis
2004
B. PAIEWONSKY
AIAA JOURNAL
of optimal controls for a class of rocket steering problems/' AIAA Paper 63-224 (June 1963). 226 Meditch, J. S. and Neustadt, L. W,, "An application of optimal control to midcourse guidance," Proceedings of the 2nd International Federation on Automatic Control Congress (1963). 227 Desoer, C. A. and Wing, J., "A minimal time discrete system," Inst. Radio Engrs., Trans. Auto. Control AC-6 (May 1961). 228 Neustadt, L. W., "Discrete time optimal control systems," Proceedings of Office of Scientific Research-Research Institute for Advanced Studies Symposium (1961); also Nonlinear Differential Equations and Nonlinear Mechanics (Academic Press, Inc., New York, 1963). 229 Polak, J., "On the equivalence of discrete systems in time optimal control," J. Basic Engr. 205 (June 1963); also Polak, E., "An application of discrete optimal control theory," J. Franklin Inst. 276, 118 (August 1963). 230 Clement, P. R., "On the bang-bang regulator problem for linear sampled-data systems," Inst. Elec. Electron. Engrs., Trans. Auto. Control AC-8 (April 1963). 231 "Sampled-data control systems and Z transform applications," (Bibliography) Inst. Radio Engrs., Trans. Auto. Control 4, 28-30 (March 1958); also Jury, E. I., "Synthesis and critical study of sampled-data control systems," Am. Inst. Elec. Engrs., Trans. Appl. Ind. 141 (July 1956);, also Gunckel, T. L., Ill and Franklin, G. F., "A general solution for linear sampled data control," J. Basic Eng. 197 (June 1963). 232 Warga, J., "Relaxed variational problems," J. Math. Anal. Appl. 4,111-128 (1962). 233 Hestenes, M. R., "Variational theory and optimal control theory," Computing Methods in Optimization Problems, edited by A. V. Balakrishnan and L. W. Neustadt (Academic Press, Inc., New York, 1964). 234 Bolza, O., Lectures on the Calculus of Variations (Dover Publications, Inc., New York, 1961). 235 Bliss, G. A., "Sufficient conditions for a minimum with respect to one-sided variations," Trans. Am. Math. Soc. 5, 477-492(1904). 236 Bliss, B. A. and Underbill, A. L., "The minimum of a definite integral for unilateral variations in space," Trans. Am. Math. Soc. 15, 291-310 (1914). 237 Mancill, J. D., "The minimum of a definite integral with respect to unilateral variations," Contributions to the Calculus of Variations, 1933-1937 (University of Chicago Press, Chicago, 111., 1937). 238 Valentine, F. A., "The problem of Lagrange with differential inequalities as added side conditions," Contributions to the Calculus of Variations, 1933-1937 (University of Chicago Press, Chicago, 111., 1937). 239 Isaacs, R., "Differential games I, II, III, IV," Rand Corp. Research Memo. RM-1391, RM-1399, RM-1411, RM-1486 (1954). 240 Berkovitz, L. D., "On control problems with bounded state variables," J. Math. Anal. Appl. 5 (December 1962). 241 Chang, S. S. L., "An extension of Ascoli's theorem and its applications to the theory of optimal control," New York Univ., TR 400-51 (January 1962). 242 Kipiniak, W., Dynamic Optimization and Control: A Variational Approach, (Technical Press, Massachusetts Institute of Technology, Cambridge, Mass, and John Wiley and Sons, Inc., New York, 1961). 243 Gamkrelidze, R. V., "Time optimal processes with bounded phase coordinates," Dokl. Akad. Nauk. SSSR 125, 475-478 (1959); also Pontriagin, L. S., Boltyanskii, V. G., Gamkrelidze, R. V., and Mishchenko, E. F., "The mathematical theory of optimal processes," transl. by K. N. Trirogoff and L. W. Neustadt (Interscience Publishers, Inc., New York, 1962), Chap. 6. 244 Dreyfus, S., "Variational problems with state variable inequality constraints," Rand Corp. Rept. P-2605 (July 1962). 245 Denham, W. F., "Steepest-ascent solution of optimal programming problems," Raytheon. Co., Bedford, Mass., Rept. BR-2393 (April 1963); also Bryson, A. E. and Denham, W. F. J. Appl. Mech. 29 (June 1962). 246 Bryson, A. E., Denham, W. F., and Dreyfus, S. E., "Optimal programming problems with inequality constraints I: necessary conditions for extremal solutions," AIAA J. 1, 25442550(1963). 247 Denham, W. F. and Bryson, A. E., "Optimal programming problems with inequality constraints II: Solution by steepestascent," AIAA J. 2, 25-34 (1964).
248 Kahne, S. J., "On direct fixed-time optimization of invertible systems," Rept. R-162, Coordinated Science Lab., Univ. of Illinois, Urbana, 111. (June 1963). 249 Ho, Y. C. and Brentani, P. B., "On computing optimal control with inequality constraints," Rept. 1529 TR-5, Minneapolis-Honeywell, Boston, Mass. (March 1962); also, SIAM J. Control 1,319 (1963). 250 Kelley, H. J., Falco, M., and Ball, D. J., "Air vehicle flight path optimization," Rept. AFOSR/DRA-62-4, Vol. I, Air Force Office of Scientific Research, Holloman Air Force Base, N. M. (February 1962). 251 Garfinkel, B. and McAllister, G. T., "Singularities in a variational problem with an inequality," Rept. 1207, Ballistic Research Labs., Aberdeen Proving Ground, Md. (July 1963). 252 Johnson, C. D. and Gibson, J. E., "Singular solutions in problems of optimal control," Inst. Elec. Electron. Engrs., Trans. Auto Control AC-8, 4-74 (January 1963). 253 Kelley, H. I., "A transformation approach to singular subarcs in optimal trajectory and control problems," SIAM J. Control 2, 234(1964). 254 Hermes, H., "Controllability and the singular problem," SIAM J. Control 2, 241 (1964). 255 Wonham, W. M. and Johnson, C. D., Proceedings of the 1963 Joint Automatic Control Conference (1963). 256 Bliss, G. A., Lectures on the Calculus of Variations (University of Chicago Press, Chicago, 111., 1946), p. 296. 257 Bliss, G. A., "The problem of Lagrange in the calculus of variations," Am. J. Math. 52, 673-744 (1930). 258 Mclntyre, J. and Paiewonsky, B. H., "On optimal control with bounded state variables," Aeronautical Research Associates of Princeton Inc. Rept. 60 (July 1964). 259 Lass, H., Jet Propulsion Lab. Space Program Summary 37-25, Vol. IV, pt. F. (1963). 260 Drenick, R. F. and Shaw, L., "Optimal control of linear plants with random parameters," Inst. Elec. Electron. Engrs. Trans. Auto. Control AC-9, 236 (1964). 261 Orford, R. J., "Optimal stochastic control systems," J. Math. Anal. Appl. 6 (June 1963). 262 Aoki, M., "Dynamic programming and numerical experimentation as applied to adaptive control systems," Univ. of California, Los Angeles, Rept. 60-16 (February 1960). 263 Kushner, M. J., "Some problems and recent results in stochastic control," Inst. Elec. Electron. Engrs. International Convention Record, pt. 6 (1965). 264 Bellman, R., "On the foundations of a theory of stochastic variational processes," Rand Corp. Rept P-1903 (February 1960). 265 Aoki, M., "Dynamic programming and numerical experimentation as applied to adaptive control systems," Univ. of California, Los Angeles, Engineering Dept. Rept. 60-16 (February 1960). 266 Bellman, R., "Dynamic programming and stochastic control processes," Info. Control I (September 1958). 267 Bellman, R., "On the application of the theory of dynamic programming to the study of control processes," Proceedings of the Symposium on Nonlinear Circuit Analysis (Polytechnic Press, Brooklyn, N. Y., 1956), pp. 199-213. 268 Aoki, M., "Stochastic time optimal control systems," Am. Inst. Elec. Engrs. Appl. Ind. (May 1961). 269 Krasovskii, N. N., "On optimum control in the presence of random disturbances," Trans. PMM J. 24, No. 1 (1960). 270 Eaton, J. H. and Zadeh, L. A., "Optimal pursuit strategies in discrete state probabilistic systems," Paper 61-JAC-ll, Proceedings of the Joint Automatic Control Conference (June 1961). 271 Florentin, J. J., "Optimal probing; adaptive control of a simple bayesian system," J. Electron. Control XIII, 165 (August 1962). 272 Ho, Y. C. and Lee, R. C. K., "A Bayesian approach to problems in stochastic estimation and control," Inst. Elec. Electron. Engrs. Trans. Auto. Control AC-9, 333 (October 1964). 273 Howard, R. A., Dynamic Programming and Markovian Processes (Technical Press, Massachusetts Institute of Technology, Cambridge, Mass., 1960). 274 Dreyfus, S. E., "Some types of optimal control of stochastic systems," SIAM J. Series A: Control 2, 120 (1964). 275 Wax, N. (ed.), Selected Papers on Noise and Stochastic Processes, (Dover Publications New York, 1954); also Wang M. C. and Uhlenbeck, G. E., Rev. Mod. Phys. 17, 323-342 (April-July 1945).
NOVEMBER 1965
2005
276 Blackmail, N. M., "On the effect of noise in a non-linear control system," Paper IFAC Conference, Moscow (1961). 277 Barrett, J. F., "Application of Kolmogorov's equation to randomly disturbed automatic control systems," Proceedings of the International Federation on Automatic Control Congress (IFAC) (Butterworth Scientific Publications, Ltd., London, 1960). 278 Chuang, K. and Kazda, L. F., "A study of non-linear systems with random inputs," Trans. Am. Inst. Elec. Engrs., Appl. and Ind. 42,100-105 (1959). 279 Katz, S., "On the best endpoint control of noisy systems," J. Electron. Control XII (April 1962). 280 Jlorentin, J. J., "Optimal control of continuous time Markovian Stochastic systems," J. Electron. Control (June 1961). 281 Kushner, H. J., "On the dynamical equations of conditional probability density functions, with applications to optimal stochastic control theory," J. Math. Anal. Appl. 8 (April 1964). 282 Wonham, W. M., "Stochastic analysis of a class of nonlinear control systems with random inputs," J. Basic Eng., p. 223 (June 1963). 283 Krasovskii, N. N. and Lidskii, E. A., "Analytical control design in systems with random properties," Automation and Remote Control (September-October-November 1961). 284 Kramer, J. D. R., "On control of linear systems with time lags," Inform. Control (December 1960). 285 Bellman, R. and Bucy, R., "Asymptotic control theory," SIAM J. Series A: Control 2, 11 (1964). 286 Butkovskii, A. G., "The maximum principle for optimum systems with distributed parameters," Automation and Remote Control 22 (March 1962); also Wang, P. K C. and Tung, F., "Optimum control of distributed parameter systems," Joint Automatic Control Conference, Minneapolis, Minn. (June 1963). 28? Florentin, J. J., "Partial observability and optimal control," J. Electron. Control XIII, 263 (September 1962). 288 Peterson, E. L., Statistical Analysis and Optimization of Systems (John Wiley and Sons, Inc., New York, 1961). 289 Davenport, W. B. and Root, W. L., An Introduction to the Theory of Random Signals and Noise (McGraw-Hill Book Co., Inc., New York, 1958). 290 Laning, J. H. and Battin, R. H., Random Processes in Automatic Control (McGraw-Hill Book Co., Inc., New York, 1956). 291 Solodovnikov, V. V., Introduction to Statistical Dynamics of Automatic Control Systems (Dover Publications, Inc., New York, 1960). 292 Bose, A. G., "A theory of nonlinear systems," Massachusetts Institute of Technology Research Lab. of Electronics TR 309 (May 1956). 293 Kalmari, R. E., "A new approach to linear filtering and prediction problems," J. Basic Eng. (March 1960). 294 Rideout, V. and Rajarman, V., "A digest of adaptive systems," Proc. Am. Inst. Elec. Engrs.-Inst. Tech. Groups Auto. Control I, 10-27 (1962); also Aseltine, J., Mancini, A., and Sarture, C., "A survey of adaptive control systems," Inst. Radio Engrs. Trans. Auto. Control AC-6 102 (December 1958); also Stromer, P., Inst. Radio Engrs. Trans. Auto. Control AC-4 (May 1959). 295 Cox, H., "On the estimation of state variables and parameters for noisy dynamic systems," Inst. Elec. Electron. Engrs. Trans. Auto. Control AC-9, 5 (January 1964). 296 Adorno, D. S., "Optimum control of certain linear systems with quadratic loss," Inform. Control 5, 1-12 (March 1962); also Adorno, D. S., "Studies in the asymptotic theory of control systems, II," Jet Propulsion Lab. TR 32-99 (June 1, 1961); also Adorno, D. S., "Studies in the asymptotic theory of control systems: I. Stochastic and deterministic N-stage processes," Jet Propulsion Lab. TR 32-21 (May 1960). 297 Kalman, R. E., "When is a linear control system optimal?" Proceedings of the Joint Automatic Control Conference (1963). 298 Courant, R. and Hilbert, D., Methods of Mathematical Physics, Vol. I (Interscience Publishers, Inc., New York, 1953). 299 Tyler, J. S., Jr., "The characteristics of model-following systems as synthesized by optimal control," Proceedings of the Joint Automatic Control Conference (1964), p. 41. 300 Krasovskii, N. N. and Letov, A. M., "The theory of analytic design of controllers," Automation and Remote Control 23 (June 1962). 301 Rekasius, Z. V. and Hsia, T. C., "On an inverse problem in optimal control," Inst. Elec. Electron. Engrs., Trans. Auto. Control AC-9 (October 1964). 302 Johnson, C. D. and Gibson, I. E., "Optimal control with
quadratic performance index and fixed terminal time," Inst. Elec. Electron. Engrs., Trans. Auto. Control AC-9 (October 1964). 303 Miner, W. E. and Schmeider, D. H., "The path adaptive mode for guiding space vehicles," ARS Preprint 1944-61 (August 1961). 304 Schmeider, D. H. and Braud, N. J., "Implementation of the path-adaptive guidance mode in steering techniques for Saturn multi-stage vehicles," ARS Paper 195-61 (August 1961). 305 Wheeler, R. E., "A mathematical model for an adaptive guidance mode system," On studies in the fields of space flight and guidance theory, NASA Progress Rept. 4, MTP AERO-6365(1963). 306 Hoelker, R. F., "Theory of artificial stabilization of missiles and space vehicles with exposition of four control principles," NASA TND-555 (June 1961). 307 Vance, R. L, "Implementation of an optimal adaptive guidance mode," AIAA J. 3, 141-142 (1965). 308 Hestenes. M. R., "Numerical methods of obtaining solutions of fixed endpoint problems in the calculus of variations," Rand Rept. RM-102 (August 1949). 309 Knapp, C. H. and Frost, P. A., "Determination of optimal control and trajectories using the maximum principle in association with a gradient technique," Proceedings of the Joint Automatic Control Conference (1964), p. 222fL 310 Jurovics, S. A. and Mclntyre, J. E., "The adjoint method and its application to trajectory optimization," ARS J. 32, 1354(1962). 311 Greenley, R. R., "Comments on 'the adjoint method and its application to trajectory optimization,' " AIAA J. 1, 1463 (1963). 312 Paiewonsky, B. H., "A study of time optimal control," Aeronautical Research Associates of Princeton Rept. 33 (June 1961); also Proceedings of International Symposium on Nonlinear Differential Equations and Non-linear Mechanics, edited by J. P. LaSalle and S. Lefshetz (Academic Press Inc., New York, 1963), pp. 333-365. sis Proceedings of the Vehicle Systems Optimization Symposium, Garden City, New York (November 28-29, 1961). 314 Brunner, W., "An iteration procedure for parametric model building and boundary value problems," Joint IRE-AIEE-ACM Computer Conference, Los Angeles, Calif., (May 9-11, 1961). 315 Smith, F. B., Jr. and Lovingood, J. A., "An application of time-optimal control theory to launch vehicle regulation," Proceedings of the Optimum System Synthesis Conference (September 11-13, 1962); also Aeronautical Systems Div. ASD-TDR63-119 (February 1963), p. 99; also Scharmack, D. K., "The equivalent minimization problem and the Newton-Raphson optimization method," Proceedings of the Optimum System Synthesis Conference (September 11-13, 1962); also Aeronautical Systems Div. ASD-TDR-63-119 (February 1963), p. 119. 316 Rang, E. R., "An unsuccessful nonlinear control design procedure," Inst. Elec. Electron. Engrs., Trans. Auto. Control 8 (January 1963). 317 Kelley, H., "Optimization techniques," Method of Gradients, edited by G. Leitman (Academic Press, Inc., New York, 1962), Chap. 6. 318 McGill, R. and Kenneth, P., "Solution of variational problems by means of a generalized Newton-Raphson operator," AIAA J. 2,1761(1964). 319 Kantorovich, L. V. and Krylov, V. I., Approximate Methods of Higher Analysis (Interscience Publishers, Inc., New York and P. Noordhoff Ltd., Groningen, The Netherlands, 1958). 320 Saltzer, C. and Fetheroff, C. W., "A direct variational method for the calculation of optimum thrust programs for power limited interplanetary flight," Astronaut. Acta 7, 8 (1961). 321 Boyanovitch, D., "The use of Galerkin's method in the optimal stabilization of control systems," Grumman Aircraft Engineering Co. RN-161 (July 1963). 322 Geiss, G. R., "The analysis and design of nonlinear control systems via Liapunov's direct method," RTD-TDR-634076, Wright-Patterson Air Force Base, Ohio (August 1964); also D. Boyanovitch, Appendix VII, p. 285. 323 Halkin, H., "The principle of optimal evaluation," Contributions to the Theory of Differential Equations (Academic Press, Inc., New York, 1962), Vol. III. 324 Roxin, E., "A geometric interpretation of Pontriagin's maximum principle, Research Institute for Advanced Studies TR 61-15 (December 1961).
2006
B. PAIEWONSKY
AIAA JOURNAL
325 Contensou, P., "Theoretical study of optimal trajectories in a gravitational field. Application in the case of a single centre of attraction," Astronaut. Acta, Grumman Research Translation, TR-22 (August 1962). 326 Blaquiere, A. and Leitmann, G., "On the geometry of optimal processespart I," Rept. AM-64-10, Institute of Engineering Research, Univ. of California, Berkeley, Calif. (June 1964); also Leitmann, G., "Some geometrical aspects of optimal processes," SIAM J., Series A: Control 3 (1965); also Blaquiere, A., "Further investigation into the geometry of optimal processes," SIAM J., Series A: Control 3 (1965). 327 Al'brekht, E. G., "On the optimal stabilization of nonlinear systems," J. Appl. Math. Mech. 25 (1961). 328 Anderson, C. A., "Switching criteria for certain contactor servos with arbitrary inputs," Univ. of Michigan, Ph.D. Thesis (1958). 329 Aoki, M., "Synthesis of optimal controllers for a class of maximization problems," Automatica I, 69 (January-March 1963); also Aoki, M., "On a successive approximation technique in solving some control system optimization problems," J. Basic. Engr. 177 (June 1963). 330 pmes> g >? "Constants of the motion for optimum thrust trajectories in a central force field," AIAA J. 2, 2010 (November 1964). *33i Flugge-Lotz, I. and Halkin, EL, "Pontriagin's maximum principle and optimal control," Stanford Univ. TR 130; also Air Force Office of Scientific Research TN 1489 (September 1961). 332 Neustadt, L. W., "The existence of optimal controls in the absence of convexity conditions," J. Math. Anal. Appl. 7, 110 (August 1963). 333 Woodrow, P. J., "Methods for locating the extrema of multivariate functions," Princeton Univ., Dept. of Electrical Engineering TR 4 (November 1962). 334 Hestenes, M. R. and Stiefel, E., "Method of conjugate gradients for solving linear systems," J. Res. Natl. Bur. Std. 49,409-436(1952). 335 Curry, H. B., "The method of steepest descent for nonlinear minimization problems," Quart. Appl. Math. II (October 1944). 336 Gel'fand, I. M. and Tsetlin, M. L., "Nonlocal search principle in problems of automatic optimization," Dokl. Akad. Nauk. SSSR137,295-298 (1961); English transl. US JPRS 4687. 337 Levenberg, K., "A method for the solution of certain nonlinear problems in least squares," Quart. Appl. Math. II (July 1944). 338 Edelbaum, T. N., "Theory of maxima and minima," Optimization Techniques, edited by G. Leitmann (Academic Press Inc., New York, 1962), Chap. L 339 Spang, H. A., Ill, "A review of minimization techniques for nonlinear functions," SIAM Review 4, (October 1962). 340 Wilde, D. J., Optimum Seeking Methods (Prentice-Hall, Inc., Englewood Cliffs, N. J. 1964). 341 Shah, B. V., Buehler, R. J., and Kempthorne, p., "Some algorithms for minimizing a function of several variables," J. SIAM 12, 74 (March 1964). 342 Tompkins, C. B., "Methods of steep descent," Modern Mathematics for the Engineer, edited by E. F. Beckenbach (McGraw-Hill Book Co., Inc., New York, 1956), Chap. 18. 343 Courant, R., "Variational methods for the solution of
problems of equilibrium and vibrations," Bull. Am. Math. Soc. 49 (January 1943). 344 Fadden, E. I. and Gilbert, E. G., "Computational aspects of the time-optimal control program," Computing Methods in Optimization Problems, edited by A. V. Balakrishnan and L, W. Neustadt (Academic Press Inc., New York, 1964). 345 Davidon, W. C., "Variable metric method for minimization," Atomic Energy Commission Research and Development Rept. ANL-5990, Argonne (1959). 346 Fletcher, R. and Powell, M. J. D., "A rapidly convergent descent method for minimization," Computer J. (July 1963). 347 Powell, M. J. D., "An iterative method for finding stationary values of a function of several variables," Computer J. (July 1962). 348 Paiewonsky, B. H., Woodrow, P., Terkelsen, F., and McIntyre, J., "A study of synthesis techniques for optimal control," Aeronautical Systems Div. ASD-TDR-63-239, WrightPatterson Air Force Base, Ohio (June 1964). 349 Paiewonsky, B., Woodrow, P., Brunner, W., and Halbert, P., "Synthesis of optimal controllers using hybrid analog-digital computers," Computing Methods in Optimization Problems, edited by A. V. Balakrishnan and L. W. Neustadt (Academic Press Inc., New York, 1964). 350 Elsgolc, L. E., Calculus of Variation (Addison-Wesley Publishing Company, Inc., Reading, Mass., 1962). 351 Kelley, H. J., "Guidance theory and extremal fields," Inst. Radio Engrs., Trans. Auto. Control AC-7 (October 1962). 352 Breakwell, J. V., Speyer, J. L., and Bryson, A. E., "Optimization and control of. nonlinear systems using the second variation," SIAM J. Series A: Control 1, p. 193 (1963). 353 Merriam, C. W., Ill, "Use of a mathematical error criterion in the design of adaptive control systems," Am. Inst. Elec. Engrs. Trans., Appl. Ind. 78, 506-12 (1959) (January 1960 Section); also "An algorithm for the iterative solution of a class of two-point boundary value problems," SIAM J. Series A: Control 2,1(1964). 354 Rang, E. R., "On error analysis for closed-loop optimal guidance systems," Honeywell, MP 691, Rept. 1546-TR1 (June 15,1964). 355 Gragg, B. B., "Computation of approximately optimal control," Stanford Univ. SUDAER 179 (January 1964). 356 Ellert, F. J. and Merriam, C. W., Ill, "Synthesis of feedback controls using optimization theoryan example," Inst. Elec. Electron. Engrs. Trans., Auto. Control AC-8 (April 1963). 357 Bailey, F. B., "The application of the parametric expansion method of control optimization to the guidance and control problem of a rigid booster," Inst. Elec. Electron. Engrs., Trans. Auto. Control 9 (January 1964). 358 Schwartz, L., "Optimization techniquesa comparison," U. S. Air Force Flight Dynamics Lab. TDR-64^21, WrightPatterson Air Force Base, Ohio (March 1964). 359 Peterson, E. L. and Remond, F. X., "Investigations of dynamic programming methods for flight control problems," U. S. Air Force Flight Dynamics Lab. TDR-64-11, WrightPattersonAir Force Base, Ohio (March 1964). 360 Balakrishnan, A. V. and Neustadt, C. W., "Computing methods in optimization problems," Proceedings of the January 1964 UCLA Conference (Academic Press Inc., New York, 1964). 361 Miller, B., "Microcircuitry production growth outpaces application," Aviation Week, 76-87 (November 16, 1964). 362 "Integrated circuitry: I," Interavia XX, 409-413 (Marck 1965).
This article has been cited by: 1. Marc Rouff. 1997. Analytical computation of differential equations involved in dynamical nonlinear optimal problems. Computer Methods in Applied Mechanics and Engineering 142:1-2, 153-163. [CrossRef] 2. Gene H. Golub, Dianne P. OLeary. 1989. Some History of the Conjugate Gradient and Lanczos Algorithms: 1948 1976. SIAM Review 31:1, 50. [CrossRef] 3. M. L. Audu, D. T. Davy. 1988. A Comparison of optimal control algorithms for complex bioengineering studies. Optimal Control Applications and Methods 9:1, 101-106. [CrossRef] 4. Marc Rouff, Franoise Lamnabhi-Lagarrigue. 1986. A new approach to nonlinear optimal feedback law. Systems & Control Letters 7:5, 411-417. [CrossRef] 5. F. L. Chernous'ko, V. B. Kolmanovskii. 1979. Computational and approximate methods of optimal control. Journal of Soviet Mathematics 12:3, 310-353. [CrossRef] 6. A.G.J. Macfarlane. 1972. A survey of some recent results in linear multivariable feedback theory. Automatica 8:4, 455-492. [CrossRef] 7. R. O. ROGERS, D. D. SWORDER. 1970. Approximate minimum energy control. AIAA Journal 8:12, 2242-2244. [Citation] [PDF] [PDF Plus] 8. I. Flgge-Lotz. 1970. The importance of optimal control for the practical engineer. Automatica 6:6, 749-753. [CrossRef] 9. J. LOOMISOptimal longitudinal control for obstacle clearance on takeoff . [Citation] [PDF] [PDF Plus] 10. J. R. DOLL, F. W. GOBETZ. 1969. A survey of impulsive trajectories. AIAA Journal 7:5, 801-834. [Citation] [PDF] [PDF Plus] 11. A. C. ROBINSON. 1969. Survey of dynamic analysis methods for flight control design. Journal of Aircraft 6:2, 81-98. [Citation] [PDF] [PDF Plus] 12. References 59, 231-233. [CrossRef] 13. A. E. SABROFF. 1968. Advanced spacecraft stabilization and control techniques. Journal of Spacecraft and Rockets 5:12, 1377-1393. [Citation] [PDF] [PDF Plus] 14. Menahem Sidar. 1968. An iterative algorithm for optimum control problems. International Journal of Non-Linear Mechanics 3:1, 1-16. [CrossRef] 15. A. E. BRYSON, JR.. 1967. Applications of optimal control theory in aerospace engineering. Journal of Spacecraft and Rockets 4:5, 545-553. [Citation] [PDF] [PDF Plus] 16. Eustratios Kounias. 1966. Optimal bounded control with linear stochastic equations and quadratic cost. Journal of Mathematical Analysis and Applications 16:3, 510-537. [CrossRef]

Optimal Control A Review of Theory and Practice

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Optimal Control A Review of Theory and Practice

Uploaded by

Copyright:

Available Formats

AIAA Journa

A Review of Theory and Practice

BERNARD PAIEWONSKY Institute for Defense Analyses, Arlington, Va.

Estimated state variables

Fig. 1 Closed-loop optimal control.

Historical Aspects and Areas of Interest

OPTIMAL CONTROL: A REVIEW OF THEORY AND PRACTICE

Xi = AH (t)xj + Bik (f)uK

find a piecewise continuous control vector w(t), which

OPTIMAL CONTROL: A REVIEW OF THEORY AND PRACTICE

Mathematical Treatment of Optimization

#(p, x, u) > # (p, x, u)

Setting 5J = 0, we obtain the Euler equations

OPTIMAL CONTROL: A REVIEW OF THEORY AND PRACTICE

with the control constraint \u < 1, and the state constraint

Statement of the Problem

The state of the process is given by

P[x(T), T] = P[x(T), T\x(t,\ t,' x(tj, ti; . . . ] (28)

OPTIMAL CONTROL: A REVIEW OF THEORY AND PRACTICE

A partially observable system is one governed by the equations

pendent variable. Some examples are available in the work

The measurable output is y = H#. The deterministic per-

are reduced to tolerable values.308"312

pectation of the deterministic performance criterion.296 This

notion of certainty equivalence can be extended to include

A method commonly used for solving differential equations

OPTIMAL CONTROL: A REVIEW OF THEORY AND PRACTICE

Minimizing Functions of Several Variables

max [Vs-x(x,u, t ) ] . ueU

OPTIMAL CONTROL: A REVIEW OF THEORY AND PRACTICE

and minimize f(xk+1) with respect to hk:

d/dh{F[xk + h vF(xk)}} = vF[xk +

Starting point Contour line

Closed-Loop Optimal Control

OPTIMAL CONTROL: A REVIEW OF THEORY AND PRACTICE

OPTIMAL CONTROL: A REVIEW OF THEORY AND PRACTICE

OPTIMAL CONTROL: A REVIEW OF THEORY AND PRACTICE

OPTIMAL CONTROL: A REVIEW OF THEORY AND PRACTICE

You might also like