Maximum Entropy Inverse Reinforcement Learning: Algorithms and Computation
In the previous post we introduced inverse reinforcement learning. We defined the problem that is associated with this field, which is that of reconstructing a reward function given a set of demonstrations, and we saw what the ability to do this implies. In addition to this, we also saw came across some classification results as well as convergence guarantees from selected methods that were simply referred to in the post. There were some challenges with the classification results that we discussed, and although there were attempts to deal with these, there is still quite a lot that we did not talk about.
Maximum Entropy Inverse Reinforcement Learning
We shall now introduce a probabilistic approach based on what is known as the principle of maximum entropy, and this provides a well defined globally normalised distribution over decision sequences, while providing the same performance assurances as previously mentioned methods. This probabilistic approach allows moderate reasoning about uncertainty in the setting inverse reinforcement learning, and the assumptions further limits the space in which we search for solutions which we saw, last time, was quite massive.…
where the following is true:
is the set of states 
with 
.
is the set of actions 
with 
is the matrix of transition probabilities for taking action 
.
is the discount factor in the unit interval.
is defined as the reward function, and is taken as a function from 
.
, 
, evaluated at 
as ![V^\pi(s_1)=\mathbb{E}[\sum_{i=0}\gamma ^i R(s_i)|\pi]](http://s0.wp.com/latex.php?latex=V%5E%5Cpi%28s_1%29%3D%5Cmathbb%7BE%7D%5B%5Csum_%7Bi%3D0%7D%5Cgamma+%5Ei+R%28s_i%29%7C%5Cpi%5D&bg=ffffff&fg=000&s=0 "V^\pi(s_1)=\mathbb{E}[\sum_{i=0}\gamma ^i R(s_i)|\pi]")
, ![Q^\pi (s,a)=R(s)+\gamma \mathbb{E}_{s'\sim P_{sa}}[V^\pi (s')]](http://s0.wp.com/latex.php?latex=Q%5E%5Cpi+%28s%2Ca%29%3DR%28s%29%2B%5Cgamma+%5Cmathbb%7BE%7D_%7Bs%27%5Csim+P_%7Bsa%7D%7D%5BV%5E%5Cpi+%28s%27%29%5D&bg=ffffff&fg=000&s=0 "Q^\pi (s,a)=R(s)+\gamma \mathbb{E}_{s'\sim P_{sa}}[V^\pi (s')]")
. 
such that ![f:[0,T]\times \mathbb{R}^n\to \mathbb{R}^n](http://s0.wp.com/latex.php?latex=f%3A%5B0%2CT%5D%5Ctimes+%5Cmathbb%7BR%7D%5En%5Cto+%5Cmathbb%7BR%7D%5En&bg=ffffff&fg=000&s=0 "f:[0,T]\times \mathbb{R}^n\to \mathbb{R}^n")
is a continuous function. We can construct a rather simple differential equation given this in the following way. We let
such that this map satisfies the differential equation. 
. (Where we are taking the inner product of learned weights with a vector of features for each observation to predict the outcome).…
.