Constrained Markov Decision Processes

by Eitan Altman
Editor Eitan Altman

Crc Press Llc | March 30, 1999 | Hardcover

Not yet rated | write a review
This book provides a unified approach for the study of constrained Markov decision processes with a finite state space and unbounded costs. Unlike the single controller case considered in many other books, the author considers a single controller with several objectives, such as minimizing delays and loss, probabilities, and maximization of throughputs. It is desirable to design a controller that minimizes one cost objective, subject to inequality constraints on other cost objectives. This framework describes dynamic decision problems arising frequently in many engineering fields. A thorough overview of these applications is presented in the introduction.
The book is then divided into three sections that build upon each other.
The first part explains the theory for the finite state space. The author characterizes the set of achievable expected occupation measures as well as performance vectors, and identifies simple classes of policies among which optimal policies exist. This allows the reduction of the original dynamic into a linear program. A Lagranian approach is then used to derive the dual linear program using dynamic programming techniques.
In the second part, these results are extended to the infinite state space and action spaces. The author provides two frameworks: the case where costs are bounded below and the contracting framework.
The third part builds upon the results of the first two parts and examines asymptotical results of the convergence of both the value and the policies in the time horizon and in the discount factor. Finally, several state truncation algorithms that enable the approximation of the solution of the original control problem via finite linear programs are given.

Format: Hardcover

Dimensions: 256 pages, 9.02 × 5.98 × 0.8 in

Published: March 30, 1999

Publisher: Crc Press Llc

Language: English

The following ISBNs are associated with this title:

ISBN - 10: 0849303826

ISBN - 13: 9780849303821

save 5%

  • Out of stock online

$229.50  ea

$229.50 List Price

This item is eligible for FREE SHIPPING on orders over $25.
See details

Easy, FREE returns. See details

Item can only be shipped in Canada

Downloads instantly to your kobo or other ereading device. See details

Reviews

– More About This Product –

Constrained Markov Decision Processes

Constrained Markov Decision Processes

by Eitan Altman
Editor Eitan Altman

Format: Hardcover

Dimensions: 256 pages, 9.02 × 5.98 × 0.8 in

Published: March 30, 1999

Publisher: Crc Press Llc

Language: English

The following ISBNs are associated with this title:

ISBN - 10: 0849303826

ISBN - 13: 9780849303821

Table of Contents

INTRODUCTIONExamples of Constrained Dynamic Control ProblemsOn Solution Approaches for CMDPs with Expected CostsOther Types of CMDPsCost Criteria and AssumptionsThe Convex Analytical Approach and Occupation MeasuresLinear Programming and Lagrangian Approach for CMDPsAbout the MethodologyThe Structure of the BookPART ONE: FINITE MDPSMARKOV DECISION PROCESSESThe ModelCost Criteria and the Constrained ProblemSome NotationThe Dominance of Markov PoliciesTHE DISCOUNTED COSTOccupation Measure and the Primal LPDynamic Programming and Dual LP: the Unconstrained CaseConstrained Control: Lagrangian ApproachThe Dual LPNumber of RandomizationsTHE EXPECTED AVERAGE COSTOccupation Measure and the Primal LPEquivalent Linear ProgramThe Dual ProgramNumber of RandomizationsFLOW AND SERVICE CONTROL IN A SINGLE-SERVER QUEUEThe ModelThe LagrangianThe Original Constrained ProblemStructure of Randomization and Implementation IssuesOn Coordination Between ControllersOpen QuestionsPART TWO: INFINITE MDPSMDPS WITH INFINITE STATE AND ACTION SPACESThe ModelCost CriteriaMixed Policies, and Topologic StructuresThe Dominance of Markov PoliciesAggregation of StatesExtra Randomization in the PoliciesEquivalent Quasi-Markov Model and Quasi-Markov PoliciesTHE TOTAL COST: CLASSIFICATION OF MDPSTransient and Absorbing MDPsMDPs With Uniform Lyapunov FunctionsEquivalence of MDP With Unbounded and bounded costsProperties of MDPs With Uniform Lyapunov FunctionsProperties for Fixed Initial DistributionExamples of Unif
read more read less

From the Publisher

This book provides a unified approach for the study of constrained Markov decision processes with a finite state space and unbounded costs. Unlike the single controller case considered in many other books, the author considers a single controller with several objectives, such as minimizing delays and loss, probabilities, and maximization of throughputs. It is desirable to design a controller that minimizes one cost objective, subject to inequality constraints on other cost objectives. This framework describes dynamic decision problems arising frequently in many engineering fields. A thorough overview of these applications is presented in the introduction.
The book is then divided into three sections that build upon each other.
The first part explains the theory for the finite state space. The author characterizes the set of achievable expected occupation measures as well as performance vectors, and identifies simple classes of policies among which optimal policies exist. This allows the reduction of the original dynamic into a linear program. A Lagranian approach is then used to derive the dual linear program using dynamic programming techniques.
In the second part, these results are extended to the infinite state space and action spaces. The author provides two frameworks: the case where costs are bounded below and the contracting framework.
The third part builds upon the results of the first two parts and examines asymptotical results of the convergence of both the value and the policies in the time horizon and in the discount factor. Finally, several state truncation algorithms that enable the approximation of the solution of the original control problem via finite linear programs are given.