### Markov Process

#### Definition

• 元组 $(\mathcal{S,P})$
• $\mathcal{S}$是一个有限状态的集合
• $\mathcal{P}$是一个状态转移矩阵：$\mathcal{P_{ss’}}=\mathbb{P}[\mathcal{S_t+1}=s’|\mathcal{S_t}=s]$

### Markov Reward Process

• 在前者的基础上，增加了$(\mathcal{S,P,\color{red}{\mathcal{R,\gamma}}})$
• $\color{red}{\mathcal{R}是一个奖励函数，\mathcal{R_s}=\mathbb{E}[R_{t+1}|\mathcal{S}=s]}$
• $\color{red}\gamma是一个衰减因子，\gamma\in[0,1]$

### 价值函数-2

===>

$v_\pi(s)$是由当前状态$s$下，策略$\pi$可能的动作概率*该动作下得到的奖励值，累加而成

$q_\pi(s,a)$由两部分组成，及时回报和执行这个操作后可能到达所有状态$s’$的价值函数的累加

### 如何求解

_Fig. Summary of approaches in RL based on whether we want to model the value, policy, or the environment. (Image source: reproduced from David Silver’s RL course lecture 1.)_

### On-policy vs Off-policy

• Model-based: Rely on the model of the environment; either the model is known or the algorithm learns it explicitly.(_When we fully know the environment, we can find the optimal solution by Dynamic Programming (DP)._)
• Model-free: No dependency on the model during learning.
• Model-based尝试着 model 整个环境；先 model 了这个环境，基于该环境做出最优的策略；Model-free就是走一步看一步，在每一步中去尝试学习最优的策略。
• _The model-based learning uses environment, action and reward to get the most reward from the action. The model-free learning only uses its action and reward to infer the best action._

• On-policy: The agent learned and the agent interacting with the environment is the same.(自己和环境互动)

• Off-policy:The agent learned and the agent interacting with the environment is different.(自己看别人玩)

### Policy

Policy $\pi$,代表在状态$s$，会执行的动作$a$.(分为确定性和随机性)

• Deterministic: $\pi(s)=a.$
• Stochastic: $\pi(a|s)=\mathbb{P}_\pi[A=a|S=s]$
• $\pi_\theta(a|s)=\mathbb{P}_\pi[A=a|S=s,\theta]$

object function:

episodic environments:

$J_1(\theta)=V^{\pi_\theta}(s_1)=\mathbb{E}_{\pi_\theta}[V_1]$

continuing environments:

_连续型环境就没有初始状态一说了，那么就取平均，考虑 agent 在某时刻处于某个状态下的概率，即状态分布_

average value:

$J_{avV}(\theta)=\sum_sd^{\pi_\theta}(s)V^{\pi_\theta}(s)$

average reward per time-step:

$J_{avR}(\theta)=\sum_sd^{\pi_\theta}(s)\sum_a\pi_\theta(s,a)\mathcal{R}_s^a$

_注：用基于梯度的策略优化时，是基于序列结构片段，如果无穷地和环境交互，得到一个结果是没有意义的；就是和环境交互中的某一个序列，将其拿出来进行学习，然后优化策略_

one-step MDP(per time-step):

### Actor-Critic

AC 是在基于 Policy Gradient 的基础上进行了变化，融合了 Value based 的方法.

Actor:就是把$J(\theta)$最大化，但是梯度里面的$Q^{\pi_\theta}(s,a)$用 Value based 的方法来计算，结果如下：

### Reference

csdn-blog

David Silver 强化学习公开课

David Silver slides