↓↓↓點(diǎn)擊關(guān)注，回復(fù)資料，10個(gè)G的驚喜

來(lái)源：DeepHub IMBA
本文約5200字，建議閱讀10分鐘?本文為你舉例介紹MDP的基本概念。

馬爾可夫決策過(guò)程(Markov decision process, MDP)是人工智能中的一個(gè)重要概念，也是強(qiáng)化學(xué)習(xí)的理論基礎(chǔ)之一。在今天的文章中，我們使用來(lái)自Stuart Russell和Peter Norvig的《Artificial Intelligence: A Modern Approach》一書(shū)中的網(wǎng)格例子來(lái)介紹MDP的基本概念。

我們的吃豆人游戲

世界示例

MDP的定義

具有馬爾可夫轉(zhuǎn)移模型和附加獎(jiǎng)勵(lì)的完全可觀察的隨機(jī)環(huán)境的順序決策問(wèn)題稱(chēng)為馬爾可夫決策過(guò)程或 MDP，由一組狀態(tài)（具有初始狀態(tài) s?）組成；每個(gè)狀態(tài)下的一組動(dòng)作；一個(gè)轉(zhuǎn)換模型 P(s'| s, a);和獎(jiǎng)勵(lì)函數(shù) R(s)。

網(wǎng)格世界的代碼實(shí)現(xiàn)

0,0,0,10,3,0,20,0,0,0

import numpy as npimport matplotlib.pyplot as pltimport matplotlib.patches as patches

class GridWorld:  def __init__(self, filename, reward=None):      if reward is None:          reward = {0: -0.04, 1: 1.0, 2: -1.0, 3: np.NaN}      file = open(filename)      self.map = np.array(          [list(map(float, s.strip().split(","))) for s in file.readlines()]      )      self.num_rows = self.map.shape[0]      self.num_cols = self.map.shape[1]      self.num_states = self.num_rows * self.num_cols      self.num_actions = 4      self.reward = reward      self.reward_function = self.get_reward_table()      self.transition_model = self.get_transition_model()
  def get_state_from_pos(self, pos):      return pos[0] * self.num_cols + pos[1]
  def get_pos_from_state(self, state):      return state // self.num_cols, state % self.num_cols

獎(jiǎng)勵(lì)函數(shù)

class GridWorld:  def get_reward_function(self):      reward_table = np.zeros(self.num_states)      for r in range(self.num_rows):          for c in range(self.num_cols):              s = self.get_state_from_pos((r, c))              reward_table[s] = self.reward[self.map[r, c]]      return reward_table

轉(zhuǎn)換模型

class GridWorld:  def get_transition_model(self, random_rate=0.2):      transition_model = np.zeros((self.num_states, self.num_actions, self.num_states))      for r in range(self.num_rows):          for c in range(self.num_cols):              s = self.get_state_from_pos((r, c))              neighbor_s = np.zeros(self.num_actions)              for a in range(self.num_actions):                  new_r, new_c = r, c                  if a == 0:                      new_r = max(r - 1, 0)                  elif a == 1:                      new_c = min(c + 1, self.num_cols - 1)                  elif a == 2:                      new_r = min(r + 1, self.num_rows - 1)                  elif a == 3:                      new_c = max(c - 1, 0)                  if self.map[new_r, new_c] == 3:                      new_r, new_c = r, c                  s_prime = self.get_state_from_pos((new_r, new_c))                  neighbor_s[a] = s_prime              for a in range(self.num_actions):                  transition_model[s, a, int(neighbor_s[a])] += 1 - random_rate                  transition_model[s, a, int(neighbor_s[(a + 1) % self.num_actions])] += random_rate / 2.0                  transition_model[s, a, int(neighbor_s[(a - 1) % self.num_actions])] += random_rate / 2.0      return transition_model

策略

class GridWorld:def generate_random_policy(self):  return np.random.randint(self.num_actions, size=self.num_states)

策略的執(zhí)行

class GridWorld:def execute_policy(self, policy, start_pos=(2, 0)):  s = self.get_state_from_pos(start_pos)  total_reward = 0  state_history = [s]  while True:    temp = self.transition_model[s, policy[s]]    s_prime = np.random.choice(self.num_states, p=temp)    state_history.append(s_prime)    r = self.reward_function[s_prime]    total_reward += r    s = s_prime    if r == 1 or r == -1:      break  return total_reward

更多策略

如何找到最優(yōu)策略？

https://github.com/clumsyhandyman/mad-from-scratch

推薦閱讀
決策樹(shù)可視化，被驚艷到了！
開(kāi)發(fā)機(jī)器學(xué)習(xí)APP，太簡(jiǎn)單了
200 道經(jīng)典機(jī)器學(xué)習(xí)面試題總結(jié)
卷積神經(jīng)網(wǎng)絡(luò)（CNN）數(shù)學(xué)原理解析
收手吧，華強(qiáng)！我用機(jī)器學(xué)習(xí)幫你挑西瓜
為了這個(gè)GIF，我專(zhuān)門(mén)建了一個(gè)網(wǎng)站
【保姆級(jí)教程】白嫖老外的云服務(wù)器

三連在看，月入百萬(wàn)??