In this lecture, we put together many of the topics covered throughout the term: martingales; Markov chains; countable state Markov processes; reversibility for Markov processes; random walks; and Wald's identity for two thresholds.

Lecture 25: Putting It All Together

Sequential hypothesis testing is viewed as a random walk example.  Threshold hypothesis tests are distinguished from random walk thresholds.  Random walk threshold probabilities are analyzed by Chernoff bounds.

Lecture 21: Hypothesis Testing and Random Walks

This lecture continues our conversation on Martingales and covers stopped martingales, Kolmogorov submartingale inequality, martingale convergence theorem, and more.

Lecture 24: Martingales: Stopping and Converging

After reviewing Wald's identity, we introduce martingales and show they include many processes already studied.  Next, submartingales, supermartingales, and stopped (simple, sub, super) martingales are introduced.

Lecture 23: Martingales (Plain, Sub, and Super)

This lecture covers topics including the Kingman bound for G/G/1, large deviations for hypothesis tests, sequential detection, and tilted random variables and proof of Wald's identity.

Lecture 22: Random Walks and Thresholds

After reviewing steady-state, this lecture discusses reversibility for Markov processes and for tandem M/M/1 queues.  Random walks and their applications are then introduced.

Lecture 20: Markov Processes and Random Walks

Markov processes with countable state-spaces are developed in terms of the embedded Markov chain.  The steady-state process probabilities and the steady-state transition probabilities are treated.

Lecture 19: Countable-state Markov Processes

In this lecture, the professor covers sample-time M/M/1 queue, Burke’s theorem, branching processes, and Markov processes with countable state spaces.

Lecture 18: Countable-state Markov Chains and Processes

This lecture reviews the previous 13 lectures in preparation for the upcoming quiz.

Lecture 14: Review

This lecture covers a variety of topics, including elementary renewal theorem, generalized stopping trials, the G/G/1 queue, Little's theorem, ensemble averages and more.

Lecture 13: Little, M/G/1, Ensemble Averages

After reviewing the three major renewal theorems, we introduce Markov chains with countable state spaces.  The matrix approach for finite-state chains is replaced by renewals based on first-passage times.

Lecture 16: Renewals and Countable-state Markov

In this lecture, we continue our discussion of renewals and cover topics such as Markov chains and renewal processes, expected number of renewals, elementary renewal and Blackwell theorems, and delayed renewal processes.

Lecture 15: The Last Renewal

This lecture begins with a discussion of convergence WP1 related to a quiz problem.  Then positive and null recurrence, steady state, birth-death chains, and reversibility are covered.

Lecture 17: Countable-state Markov Chains

This lecture begins with the SLLN and the central limit theorem for renewal processes.  This is followed by the time-average behavior of reward functions such as residual life.

Lecture 11: Renewals: Strong Law and Rewards

This lecture covers rewards for Markov chains, expected first passage time, and aggregate rewards with a final reward. The professor then moves on to discuss dynamic programming and the dynamic programming algorithm.

Lecture 9: Markov Rewards and Dynamic Programming

This lecture covers eigenvalues and eigenvectors of the transition matrix and the steady-state vector of Markov chains. It also includes an analysis of a 2-state Markov chain and a discussion of the Jordan form.

Lecture 8: Markov Eigenvalues and Eigenvectors

This lecture treats joint conditional densities for Poisson processes and then defines finite-state Markov chains. Recurrent and transient states, periodic states, and ergodic chains are discussed.(Courtesy of Mina Karzand.  Used with permission.)

This lecture treats joint conditional densities for Poisson processes and then defines finite-state Markov chains. Recurrent and transient states, periodic states, and ergodic chains are discussed.<br><br>(Courtesy of Mina Karzand.  Used with permission.)

Lecture 6: From Poisson to Markov

In this lecture, many problem solving techniques are developed using, first, combining and splitting of various Poisson processes, and, second, conditioning on the number of arrivals in an interval.

Lecture 5: Poisson Combining and Splitting

In this lecture, we learn about time-averages for renewal rewards, stopping trials for stochastic processes, and Wald's equality.

Lecture 12: Renewal Rewards, Stopping Trials, and Wald's Inequality

Renewal processes are introduced and their importance in analyzing other processes is explained.  Proofs about convergence with probability 1 (WP1) and the SLLN are given.

Lecture 10: Renewals and the Strong Law of Large Numbers

Discrete Stochastic Processes

What is this?
1 quote, 3 ideas & 1 question from a carefully curated podcast episode.
Every Monday & Wednesday.

Want a 3rd insight and episode every week? Email me at curation@owltail.com.

OwlTail

OwlTail 1-3-1

The best podcast episodes on physical health. Episodes talk about your diet, sleep, exercise and all the things that people do to improve their physical health and live a healthier life. This topic is for everyone, with applications in all of our everyday lives. Add episodes you're interested into your queue below!