Some probability processes have infinite expected value. Example: flip coin until heads. Expected flips = 2. But: pay if first heads is on flip . Expected payout = .
This is the St. Petersburg paradox. In DP, watch for recurrences like with probability 1 (infinite loop). The expected value is undefined. make sure your probability DP has absorbing states (termination) with positive probability from every state.