In this lecture, we put together many of the topics covered throughout the term: martingales; Markov chains; countable state Markov processes; reversibility for Markov processes; random walks; and Wald's identity for two thresholds.

Lecture 25: Putting It All Together

Sequential hypothesis testing is viewed as a random walk example.  Threshold hypothesis tests are distinguished from random walk thresholds.  Random walk threshold probabilities are analyzed by Chernoff bounds.

Lecture 21: Hypothesis Testing and Random Walks

This lecture continues our conversation on Martingales and covers stopped martingales, Kolmogorov submartingale inequality, martingale convergence theorem, and more.

Lecture 24: Martingales: Stopping and Converging

After reviewing Wald's identity, we introduce martingales and show they include many processes already studied.  Next, submartingales, supermartingales, and stopped (simple, sub, super) martingales are introduced.

Lecture 23: Martingales (Plain, Sub, and Super)

This lecture covers topics including the Kingman bound for G/G/1, large deviations for hypothesis tests, sequential detection, and tilted random variables and proof of Wald's identity.

Lecture 22: Random Walks and Thresholds

After reviewing steady-state, this lecture discusses reversibility for Markov processes and for tandem M/M/1 queues.  Random walks and their applications are then introduced.

Lecture 20: Markov Processes and Random Walks

Markov processes with countable state-spaces are developed in terms of the embedded Markov chain.  The steady-state process probabilities and the steady-state transition probabilities are treated.

Lecture 19: Countable-state Markov Processes

In this lecture, the professor covers sample-time M/M/1 queue, Burke’s theorem, branching processes, and Markov processes with countable state spaces.

Lecture 18: Countable-state Markov Chains and Processes

This lecture reviews the previous 13 lectures in preparation for the upcoming quiz.

Lecture 14: Review

This lecture covers a variety of topics, including elementary renewal theorem, generalized stopping trials, the G/G/1 queue, Little's theorem, ensemble averages and more.

Lecture 13: Little, M/G/1, Ensemble Averages

Discrete Stochastic Processes

In this lecture, we continue our discussion of renewals and cover topics such as Markov chains and renewal processes, expected number of renewals, elementary renewal and Blackwell theorems, and delayed renewal processes.

Lecture 15: The Last Renewal

After reviewing the three major renewal theorems, we introduce Markov chains with countable state spaces.  The matrix approach for finite-state chains is replaced by renewals based on first-passage times.

Lecture 16: Renewals and Countable-state Markov

This lecture begins with a discussion of convergence WP1 related to a quiz problem.  Then positive and null recurrence, steady state, birth-death chains, and reversibility are covered.

Lecture 17: Countable-state Markov Chains

This lecture begins with the SLLN and the central limit theorem for renewal processes.  This is followed by the time-average behavior of reward functions such as residual life.

Lecture 11: Renewals: Strong Law and Rewards

This lecture covers rewards for Markov chains, expected first passage time, and aggregate rewards with a final reward. The professor then moves on to discuss dynamic programming and the dynamic programming algorithm.

Lecture 9: Markov Rewards and Dynamic Programming

The best podcast episodes of the most interesting stories and things. From skyping someone while they're in space, to tracking down and meeting a telephone scammer in person, to the story of a Galapagos. These episodes will keep you and all your friends entertained and captivated for hours. Learn something new in every episode! Add episodes you're interested into your queue below!

Best Episodes: Interesting Stories & Things

The best episodes for anyone looking to learn more about topics such as happiness, fulfillment, depression and psychology/mental health in general. With interviews with industry experts, personal stories of individuals, this topic has lessons applicable to everyone. Add episodes you're interested into your queue below!