Handbook of Markov Decision Processes

Methods and Applications

edited by
Eugene A. Feinberg
SUNY at Stony Brook, USA
Adam Shwartz
Technion Israel Institute of Technology, Haifa, Israel

Kluwer 2002 (565 pages)

Contents and Contributors 
(links to introduction of each chapter)

1. Introduction; E.A. Feinberg, A. Shwartz.

Part I: Finite State and Action Models.

2. Finite State and Action MDPs; L. Kallenberg.
3. Bias Optimality; M.E. Lewis, M.L. Puterman.
4. Singular Perturbations of Markov Chains and Decision Processes;
K.E. Avrachenkov, J. Filar, M. Haviv

Part II: Infinite State Models.

5. Average Reward Optimization Theory for Denumerable State Spaces; L.I. Sennott.
6. Total Reward Criteria; E.A. Feinberg.
7. Mixed Criteria; E.A. Feinberg, A. Shwartz.
8. Blackwell Optimality; A. Hordijk, A.A. Yushkevich.
9. The Poisson Equation for Countable Markov Chains: Probabilistic Methods and Interpretations;
A.M. Makowski, A. Shwartz.
10. Stability, Performance Evaluation, and Optimization; S.P. Meyn.
11. Convex Analytic Methods in Markov Decision Processes;  V.S. Borkar.
12. The Linear Programming Approach; O. Hernández-Lerma,  J.B. Lasserre.
13. Invariant Gambling Problems and Markov Decision Processes;
L.E. Dubins, A.P. Maitra, W.D. Sudderth

Part III: Applications.

14. Neuro-Dynamic Programming: Overview and Recent Trends;  B. Van Roy.
15. Markov Decision Processes in Finance and Dynamic Options;  M. Schäl.
16. Applications of Markov Decision Processes in Communication Networks; E. Altman.
17. Water Reservoir Applications of Markov Decision Processes;
B.F. Lamond, A. Boukhtouta.
Index.

 

Special Offer

Write to Editors

Errata Page

Handbook of Markov Decision Processes

Handbook of Markov Decision Processes

No.	Introduction; Title/pages	Errata/links	Updated
1	Introduction; pages 1–17
2	Finite State and Action MDPs; pages 21–88	Page 73 Reference [66]: the authors are E.B. Dynkin and A.A. Yushkevich	12/09/01
3	Bias Optimality; pages 89–112
4	Singular Perturbations of Markov Chains and Decision Processes; pages 113–150	Example 2.1:  p.132	11/27/01
5	Average Reward Optimization Theory for Denumerable State Spaces; pages 153–172	Page 159 formula 5.17–omit the “*”	12/04/01
6	Total Reward Criteria; pages 173–208
7	Mixed Criteria; pages 209–230
8	Blackwell Optimality; pages 231–268
9	The Poisson Equation for Countable Markov Chains: Probabilistic Methods and Interpretations; pages 269–304
10	Stability, Performance Evaluation, and Optimization; pages 305–346
11	Convex Analytic Methods in Markov Decision Processes; pages 347–376
12	The Linear Programming Approach; pages 377–408
13	Invariant Gambling Problems and Markov Decision Processes; pages 409–430
14	Neuro-Dynamic Programming: Overview and Recent Trends; pages 431–460
15	Markov Decision Processes in Finance and Dynamic Options; pages 461–488	Eq.(15.8): p. 464, some typos	1/22/02
16	Applications of Markov Decision Processes in Communication Networks; pages 489–536
17	Water Reservoir Applications of Markov Decision Processes; pages 537–558

Comments on Open Problems