Abstract
這篇博客大概會(huì)記錄OpenAI gym的安裝以及使用的簡(jiǎn)要說(shuō)明。
在強(qiáng)化學(xué)習(xí)里面我們需要讓agent運(yùn)行在一個(gè)環(huán)境里面，然鵝手動(dòng)編環(huán)境是一件很耗時(shí)間的事情, 所以如果有能力使用別人已經(jīng)編好的環(huán)境, 可以節(jié)約我們很多時(shí)間。 OpenAI gym 就是這樣一個(gè)模塊, 他提供了我們很多優(yōu)秀的模擬環(huán)境. 我們的各種 RL 算法都能使用這些環(huán)境.。
不過 OpenAI gym 暫時(shí)只支持 MacOS 和 Linux 系統(tǒng)，如果是Win 10 用戶可以參考之前的[1]博客 安裝Win的Linux子系統(tǒng)。
安裝
首先需要安裝一些必要依賴，如果brew或者apt-get沒有安裝或者更新的話需要安裝更新一下：
# MacOS:
$ brew install cmake boost boost-python sdl2 swig wget
# Ubuntu 14.04:
$ apt-get install -y python-numpy python-dev cmake zlib1g-dev libjpeg-dev xvfb libav-tools
然后就可以使用pip安裝gym，如果要安裝gym的全部游戲需要把下面的gym替換成gym[all]
# python 2.7
$ pip install gym
# python 3.5
$ pip3 install gym
使用
我們先看一段簡(jiǎn)短的代碼：
demo1.py
import gym
env = gym.make('CartPole-v0')
for i_episode in range(20):
observation = env.reset()
for step in range(100):
env.render()
print(observation)
action = env.action_space.sample()
observation, reward, done, info = env.step(action)
if done:
print("Episode finished after {} timesteps".format(step+1))
break
- 首先是
gym.make('CartPole-v0')，gym會(huì)運(yùn)行CartPole-v0的游戲環(huán)境 - 在每個(gè)episode里面，
env.reset()會(huì)重置環(huán)境，即重新開始游戲，并返回觀測(cè)值 - 在每次的step里面，
env.render()會(huì)刷新畫面 -
env.action_space.sample()返回一個(gè)action的隨機(jī)sample，即隨機(jī)在動(dòng)作空間里面選擇一個(gè)動(dòng)作 -
env.step(action)返回值有四個(gè)：-
observation (object): an environment-specific object representing your observation of the environment. For example, pixel data from a camera, joint angles and joint velocities of a robot, or the board state in a board game.
特定于環(huán)境的對(duì)象表示人對(duì)環(huán)境的觀察。 例如，來(lái)自相機(jī)的像素?cái)?shù)據(jù)，機(jī)器人的關(guān)節(jié)角度和關(guān)節(jié)速度，或棋盤游戲中的棋盤狀態(tài) -
reward (float): amount of reward achieved by the previous action. The scale varies between environments, but the goal is always to increase your total reward.
上一個(gè)行動(dòng)獲得的獎(jiǎng)勵(lì)數(shù)額 -
done (boolean): whether it’s time to reset the environment again. Most (but not all) tasks are divided up into well-defined episodes, and done being True indicates the episode has terminated. (For example, perhaps the pole tipped too far, or you lost your last life.)
游戲是否已經(jīng)結(jié)束 -
info (dict): diagnostic information useful for debugging. It can sometimes be useful for learning (for example, it might contain the raw probabilities behind the environment’s last state change). However, official evaluations of your agent are not allowed to use this for learning.
調(diào)試用的診斷信息
-
observation (object): an environment-specific object representing your observation of the environment. For example, pixel data from a camera, joint angles and joint velocities of a robot, or the board state in a board game.
我們可以用下圖來(lái)表示agent和環(huán)境之間的關(guān)系：
image.png
運(yùn)行效果如http://s3-us-west-2.amazonaws.com/rl-gym-doc/cartpole-no-reset.mp4
Space
之后我們看一下上面代碼的action_space。每個(gè)游戲都有自己的action_space和observation_space，表示可以執(zhí)行的動(dòng)作空間與觀察空間。我們可以將其打印出來(lái)，看動(dòng)作空間和觀察空間的最大值或者最小值
import gym
env = gym.make('CartPole-v0')
print(env.action_space)
#> Discrete(2) 離散值0或1
print(env.observation_space)
#> Box(4,) 區(qū)間值，數(shù)組中包含四個(gè)數(shù)，取值如下
print(env.observation_space.high)
#> array([ 2.4 , inf, 0.20943951, inf])
print(env.observation_space.low)
#> array([-2.4 , -inf, -0.20943951, -inf])
