Topic outline

  • General

  • Definition and objetives of uncertain decision

    Decision-making under uncertainty deals with choice situations, where the decision-maker has to make decisions in an environment with the consequences of the decisions are not known exactly.

    The aim of the subject is to enable students to understand and use the methods and techniques  including,  Markov chain, Markov decision process (MDP), Bayesian Beleif Networks (BBN) and Hidden Markov model (HMM).

    This enable an intelligent agent to make the right decision on the basis of a number of observations, also known as perceptions. This makes it possible to automate decision-making and to have robots with reasoning similar to that of human.


    • Chapter 1: Markov chain

      Markov chain consists on the first technology to be studied in this subject, it describes a system whose state changes over time.  The  future of the system depends  only to  its present state, and not to the path by which the
      system got to this latter. A Markov chain is useful when we need to compute a probability for a sequence of observable events.

      The chapter is organized as  follows:

      1. Stochastic process

      2. Markov chains definition

      3. Transition Matrix and graph.

      4. Markov chain Distribution

      5. M-step transition

      6. Classification of states (recurrent, transient)

      7. Stationary distribution

    • Chapter 2

      In this chapter, we will studied one of machine learning technique which is Reinforcement Learning. This technique allows  an agent to make the correct decision in an environment guided by reward. In fact, the agent try to maximize the collect reward throw the action made in the environment. This allows the  agent to take  the  appropriate action for the  situation (state). For that. We use Markov Decision Process (MDP), that defines how an agent takes sequential actions from state in each environment, guided by reward, using uncertainty in how it transitions from state to state.

      This, the course is organized as follows:

       Part 1: Markov decision process
      • Markov decision process components
      • Policy
      • Bellmans equations
      • States and actions values
      Part 2: MDP solving
      • Value iteration
      • Policy iteration
      • Q-learning

       



    • Chapter 3: Bayesian beleif network(BBN))

      Bayesian networks make the key technology for dealing with probabilities in AI. They  are graphical models for reasoning under uncertainty, where the nodes represent variables (discrete or continuous) and arcs represent direct connections between them.These direct connections are often causal connections.

      In this chapter we provide a study of these networks as follows:

      • Introduction about probability theory
      • Bayesian network structor
      • joint probability
      • d-separation
      • Inference in Bayesian networks
               1. enumeration algorithm
               2. varaibles elimination algorithm


    • Hidden Markov Model(HMM)

      In contrast to Markov chain, which allows to to compute a probability for a sequence of observable events.

      A hidden Markov model (HMM) allows us to talk about both observed events and hidden events
      that we think of as causal factors in our probabilistic model.

      In this last chapter, we study HMM as follows:

      -Definition

      -Underlying problems

      -Decoding

      -Viterbi Algorithm