Handbook of Markov Decision Processes

Methods and Applications

Handbook of Markov Decision Processes
Handbook of Markov Decision Processes

edited by
Eugene A. Feinberg
SUNY at Stony Brook, USA
Adam Shwartz
Technion Israel Institute of Technology, Haifa, Israel

Kluwer 2002 (565 pages)

Contents and Contributors 
(links to introduction of each chapter)

1. Introduction; E.A. Feinberg, A. Shwartz.

Part I: Finite State and Action Models.

2. Finite State and Action MDPs; L. Kallenberg.
3. Bias Optimality; M.E. Lewis, M.L. Puterman.
4. Singular Perturbations of Markov Chains and Decision Processes;
K.E. Avrachenkov, J. Filar, M. Haviv

Part II: Infinite State Models.

5. Average Reward Optimization Theory for Denumerable State Spaces; L.I. Sennott.
6. Total Reward Criteria; E.A. Feinberg.
7. Mixed Criteria; E.A. Feinberg, A. Shwartz.
8. Blackwell Optimality; A. Hordijk, A.A. Yushkevich.
9. The Poisson Equation for Countable Markov Chains: Probabilistic Methods and Interpretations;
A.M. Makowski, A. Shwartz.
10. Stability, Performance Evaluation, and Optimization; S.P. Meyn.
11. Convex Analytic Methods in Markov Decision ProcessesV.S. Borkar.
12. The Linear Programming Approach; O. Hernández-Lerma,  J.B. Lasserre.
13. Invariant Gambling Problems and Markov Decision Processes;
L.E. Dubins, A.P. Maitra, W.D. Sudderth

Part III: Applications.

14. Neuro-Dynamic Programming: Overview and Recent TrendsB. Van Roy.
15. Markov Decision Processes in Finance and Dynamic OptionsM. Schäl.
16. Applications of Markov Decision Processes in Communication Networks; E. Altman.
17. Water Reservoir Applications of Markov Decision Processes;
B.F. Lamond, A. Boukhtouta.
Index.

 

Special Offer

Write to Editors

Errata Page

Handbook of Markov Decision Processes

Handbook of Markov Decision Processes

No.

Introduction; Title/pages

Errata/links

Updated

1

Introduction;
pages 1–17

2

Finite State and Action MDPs;
pages 21–88

Page 73 Reference [66]: the authors are E.B. Dynkin and A.A. Yushkevich

12/09/01

3

Bias Optimality;
pages 89–112

4

Singular Perturbations of Markov Chains and Decision Processes;
pages 113–150

Example 2.1:  p.132

11/27/01

5

Average Reward Optimization Theory for Denumerable State Spaces;
pages 153–172

Page 159 formula 5.17–omit the “*”

12/04/01

6

Total Reward Criteria;
pages 173–208

7

Mixed Criteria;
pages 209–230

8

Blackwell Optimality;
pages 231–268

9

The Poisson Equation for Countable Markov Chains: Probabilistic Methods and Interpretations;
pages 269–304

10

Stability, Performance Evaluation, and Optimization;
pages 305–346

11

Convex Analytic Methods in Markov Decision Processes;
pages 347–376

12

The Linear Programming Approach;
pages 377–408

13

Invariant Gambling Problems and Markov Decision Processes;
pages 409–430

14

Neuro-Dynamic Programming: Overview and Recent Trends;
pages 431–460

15

Markov Decision Processes in Finance and Dynamic Options;
pages 461–488

Eq.(15.8): p. 464, some typos

1/22/02

16

Applications of Markov Decision Processes in Communication Networks;
pages 489–536

17

Water Reservoir Applications of Markov Decision Processes; pages 537–558

Comments on Open Problems