diff --git "a/git.diff" "b/git.diff" new file mode 100644--- /dev/null +++ "b/git.diff" @@ -0,0 +1,2744 @@ +diff --git a/notebooks/unit7/ml-agents b/notebooks/unit7/ml-agents +--- a/notebooks/unit7/ml-agents ++++ b/notebooks/unit7/ml-agents +@@ -1 +1 @@ +-Subproject commit 7db884323f8619b578fc1c8327d57fa087df27e7 ++Subproject commit 7db884323f8619b578fc1c8327d57fa087df27e7-dirty +diff --git a/notebooks/unit8/doom.ipynb b/notebooks/unit8/doom.ipynb +deleted file mode 100644 +index 86e29df..0000000 +--- a/notebooks/unit8/doom.ipynb ++++ /dev/null +@@ -1,690 +0,0 @@ +-{ +- "cells": [ +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "view-in-github", +- "colab_type": "text" +- }, +- "source": [ +- "\"Open" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "OVx1gdg9wt9t" +- }, +- "source": [ +- "# Unit 8 Part 2: Advanced Deep Reinforcement Learning. Using Sample Factory to play Doom from pixels\n", +- "\n", +- "\"Thumbnail\"/\n", +- "\n", +- "In this notebook, we will learn how to train a Deep Neural Network to collect objects in a 3D environment based on the game of Doom, a video of the resulting policy is shown below. We train this policy using [Sample Factory](https://www.samplefactory.dev/), an asynchronous implementation of the PPO algorithm.\n", +- "\n", +- "Please note the following points:\n", +- "\n", +- "* [Sample Factory](https://www.samplefactory.dev/) is an advanced RL framework and **only functions on Linux and Mac** (not Windows).\n", +- "\n", +- "* The framework performs best on a **GPU machine with many CPU cores**, where it can achieve speeds of 100k interactions per second. The resources available on a standard Colab notebook **limit the performance of this library**. So the speed in this setting **does not reflect the real-world performance**.\n", +- "* Benchmarks for Sample Factory are available in a number of settings, check out the [examples](https://github.com/alex-petrenko/sample-factory/tree/master/sf_examples) if you want to find out more.\n" +- ] +- }, +- { +- "cell_type": "code", +- "execution_count": null, +- "metadata": { +- "id": "I6_67HfI1CKg" +- }, +- "outputs": [], +- "source": [ +- "from IPython.display import HTML\n", +- "\n", +- "HTML(''''''\n", +- ")" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "DgHRAsYEXdyw" +- }, +- "source": [ +- "To validate this hands-on for the [certification process](https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process), you need to push one model:\n", +- "\n", +- "- `doom_health_gathering_supreme` get a result of >= 5.\n", +- "\n", +- "To find your result, go to the [leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) and find your model, **the result = mean_reward - std of reward**\n", +- "\n", +- "If you don't find your model, **go to the bottom of the page and click on the refresh button**\n", +- "\n", +- "For more information about the certification process, check this section 👉 https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "PU4FVzaoM6fC" +- }, +- "source": [ +- "## Set the GPU 💪\n", +- "- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`\n", +- "\n", +- "\"GPU" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "KV0NyFdQM9ZG" +- }, +- "source": [ +- "- `Hardware Accelerator > GPU`\n", +- "\n", +- "\"GPU" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "-fSy5HzUcMWB" +- }, +- "source": [ +- "Before starting to train our agent, let's **study the library and environments we're going to use**.\n", +- "\n", +- "## Sample Factory\n", +- "\n", +- "[Sample Factory](https://www.samplefactory.dev/) is one of the **fastest RL libraries focused on very efficient synchronous and asynchronous implementations of policy gradients (PPO)**.\n", +- "\n", +- "Sample Factory is thoroughly **tested, used by many researchers and practitioners**, and is actively maintained. Our implementation is known to **reach SOTA performance in a variety of domains while minimizing RL experiment training time and hardware requirements**.\n", +- "\n", +- "\"Sample\n", +- "\n", +- "\n", +- "\n", +- "### Key features\n", +- "\n", +- "- Highly optimized algorithm [architecture](https://www.samplefactory.dev/06-architecture/overview/) for maximum learning throughput\n", +- "- [Synchronous and asynchronous](https://www.samplefactory.dev/07-advanced-topics/sync-async/) training regimes\n", +- "- [Serial (single-process) mode](https://www.samplefactory.dev/07-advanced-topics/serial-mode/) for easy debugging\n", +- "- Optimal performance in both CPU-based and [GPU-accelerated environments](https://www.samplefactory.dev/09-environment-integrations/isaacgym/)\n", +- "- Single- & multi-agent training, self-play, supports [training multiple policies](https://www.samplefactory.dev/07-advanced-topics/multi-policy-training/) at once on one or many GPUs\n", +- "- Population-Based Training ([PBT](https://www.samplefactory.dev/07-advanced-topics/pbt/))\n", +- "- Discrete, continuous, hybrid action spaces\n", +- "- Vector-based, image-based, dictionary observation spaces\n", +- "- Automatically creates a model architecture by parsing action/observation space specification. Supports [custom model architectures](https://www.samplefactory.dev/03-customization/custom-models/)\n", +- "- Designed to be imported into other projects, [custom environments](https://www.samplefactory.dev/03-customization/custom-environments/) are first-class citizens\n", +- "- Detailed [WandB and Tensorboard summaries](https://www.samplefactory.dev/05-monitoring/metrics-reference/), [custom metrics](https://www.samplefactory.dev/05-monitoring/custom-metrics/)\n", +- "- [HuggingFace 🤗 integration](https://www.samplefactory.dev/10-huggingface/huggingface/) (upload trained models and metrics to the Hub)\n", +- "- [Multiple](https://www.samplefactory.dev/09-environment-integrations/mujoco/) [example](https://www.samplefactory.dev/09-environment-integrations/atari/) [environment](https://www.samplefactory.dev/09-environment-integrations/vizdoom/) [integrations](https://www.samplefactory.dev/09-environment-integrations/dmlab/) with tuned parameters and trained models\n", +- "\n", +- "All of the above policies are available on the 🤗 hub. Search for the tag [sample-factory](https://huggingface.co/models?library=sample-factory&sort=downloads)\n", +- "\n", +- "### How sample-factory works\n", +- "\n", +- "Sample-factory is one of the **most highly optimized RL implementations available to the community**.\n", +- "\n", +- "It works by **spawning multiple processes that run rollout workers, inference workers and a learner worker**.\n", +- "\n", +- "The *workers* **communicate through shared memory, which lowers the communication cost between processes**.\n", +- "\n", +- "The *rollout workers* interact with the environment and send observations to the *inference workers*.\n", +- "\n", +- "The *inferences workers* query a fixed version of the policy and **send actions back to the rollout worker**.\n", +- "\n", +- "After *k* steps the rollout works send a trajectory of experience to the learner worker, **which it uses to update the agent’s policy network**.\n", +- "\n", +- "\"Sample" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "nB68Eb9UgC94" +- }, +- "source": [ +- "### Actor Critic models in Sample-factory\n", +- "\n", +- "Actor Critic models in Sample Factory are composed of three components:\n", +- "\n", +- "- **Encoder** - Process input observations (images, vectors) and map them to a vector. This is the part of the model you will most likely want to customize.\n", +- "- **Core** - Intergrate vectors from one or more encoders, can optionally include a single- or multi-layer LSTM/GRU in a memory-based agent.\n", +- "- **Decoder** - Apply additional layers to the output of the model core before computing the policy and value outputs.\n", +- "\n", +- "The library has been designed to automatically support any observation and action spaces. Users can easily add their custom models. You can find out more in the [documentation](https://www.samplefactory.dev/03-customization/custom-models/#actor-critic-models-in-sample-factory)." +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "ez5UhUtYcWXF" +- }, +- "source": [ +- "## ViZDoom\n", +- "\n", +- "[ViZDoom](https://vizdoom.cs.put.edu.pl/) is an **open-source python interface for the Doom Engine**.\n", +- "\n", +- "The library was created in 2016 by Marek Wydmuch, Michal Kempka at the Institute of Computing Science, Poznan University of Technology, Poland.\n", +- "\n", +- "The library enables the **training of agents directly from the screen pixels in a number of scenarios**, including team deathmatch, shown in the video below. Because the ViZDoom environment is based on a game the was created in the 90s, it can be run on modern hardware at accelerated speeds, **allowing us to learn complex AI behaviors fairly quickly**.\n", +- "\n", +- "The library includes feature such as:\n", +- "\n", +- "- Multi-platform (Linux, macOS, Windows),\n", +- "- API for Python and C++,\n", +- "- [OpenAI Gym](https://www.gymlibrary.dev/) environment wrappers\n", +- "- Easy-to-create custom scenarios (visual editors, scripting language, and examples available),\n", +- "- Async and sync single-player and multiplayer modes,\n", +- "- Lightweight (few MBs) and fast (up to 7000 fps in sync mode, single-threaded),\n", +- "- Customizable resolution and rendering parameters,\n", +- "- Access to the depth buffer (3D vision),\n", +- "- Automatic labeling of game objects visible in the frame,\n", +- "- Access to the audio buffer\n", +- "- Access to the list of actors/objects and map geometry,\n", +- "- Off-screen rendering and episode recording,\n", +- "- Time scaling in async mode." +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "wAMwza0d5QVj" +- }, +- "source": [ +- "## We first need to install some dependencies that are required for the ViZDoom environment\n", +- "\n", +- "Now that our Colab runtime is set up, we can start by installing the dependencies required to run ViZDoom on linux.\n", +- "\n", +- "If you are following on your machine on Mac, you will want to follow the installation instructions on the [github page](https://github.com/Farama-Foundation/ViZDoom/blob/master/doc/Quickstart.md#-quickstart-for-macos-and-anaconda3-python-36)." +- ] +- }, +- { +- "cell_type": "code", +- "execution_count": null, +- "metadata": { +- "id": "RJMxkaldwIVx" +- }, +- "outputs": [], +- "source": [ +- "%%capture\n", +- "%%bash\n", +- "# Install ViZDoom deps from\n", +- "# https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md#-linux\n", +- "\n", +- "apt-get install build-essential zlib1g-dev libsdl2-dev libjpeg-dev \\\n", +- "nasm tar libbz2-dev libgtk2.0-dev cmake git libfluidsynth-dev libgme-dev \\\n", +- "libopenal-dev timidity libwildmidi-dev unzip ffmpeg\n", +- "\n", +- "# Boost libraries\n", +- "apt-get install libboost-all-dev\n", +- "\n", +- "# Lua binding dependencies\n", +- "apt-get install liblua5.1-dev" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "JT4att2c57MW" +- }, +- "source": [ +- "## Then we can install Sample Factory and ViZDoom\n", +- "- This can take 7min" +- ] +- }, +- { +- "cell_type": "code", +- "execution_count": null, +- "metadata": { +- "id": "bbqfPZnIsvA6" +- }, +- "outputs": [], +- "source": [ +- "# install python libraries\n", +- "# thanks toinsson\n", +- "!pip install faster-fifo==1.4.2\n", +- "!pip install vizdoom" +- ] +- }, +- { +- "cell_type": "code", +- "source": [ +- "!pip install sample-factory==2.1.1" +- ], +- "metadata": { +- "id": "alxUt7Au-O8e" +- }, +- "execution_count": null, +- "outputs": [] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "1jizouGpghUZ" +- }, +- "source": [ +- "## Setting up the Doom Environment in sample-factory" +- ] +- }, +- { +- "cell_type": "code", +- "execution_count": null, +- "metadata": { +- "id": "bCgZbeiavcDU" +- }, +- "outputs": [], +- "source": [ +- "import functools\n", +- "\n", +- "from sample_factory.algo.utils.context import global_model_factory\n", +- "from sample_factory.cfg.arguments import parse_full_cfg, parse_sf_args\n", +- "from sample_factory.envs.env_utils import register_env\n", +- "from sample_factory.train import run_rl\n", +- "\n", +- "from sf_examples.vizdoom.doom.doom_model import make_vizdoom_encoder\n", +- "from sf_examples.vizdoom.doom.doom_params import add_doom_env_args, doom_override_defaults\n", +- "from sf_examples.vizdoom.doom.doom_utils import DOOM_ENVS, make_doom_env_from_spec\n", +- "\n", +- "\n", +- "# Registers all the ViZDoom environments\n", +- "def register_vizdoom_envs():\n", +- " for env_spec in DOOM_ENVS:\n", +- " make_env_func = functools.partial(make_doom_env_from_spec, env_spec)\n", +- " register_env(env_spec.name, make_env_func)\n", +- "\n", +- "# Sample Factory allows the registration of a custom Neural Network architecture\n", +- "# See https://github.com/alex-petrenko/sample-factory/blob/master/sf_examples/vizdoom/doom/doom_model.py for more details\n", +- "def register_vizdoom_models():\n", +- " global_model_factory().register_encoder_factory(make_vizdoom_encoder)\n", +- "\n", +- "\n", +- "def register_vizdoom_components():\n", +- " register_vizdoom_envs()\n", +- " register_vizdoom_models()\n", +- "\n", +- "# parse the command line args and create a config\n", +- "def parse_vizdoom_cfg(argv=None, evaluation=False):\n", +- " parser, _ = parse_sf_args(argv=argv, evaluation=evaluation)\n", +- " # parameters specific to Doom envs\n", +- " add_doom_env_args(parser)\n", +- " # override Doom default values for algo parameters\n", +- " doom_override_defaults(parser)\n", +- " # second parsing pass yields the final configuration\n", +- " final_cfg = parse_full_cfg(parser, argv)\n", +- " return final_cfg" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "sgRy6wnrgnij" +- }, +- "source": [ +- "Now that the setup if complete, we can train the agent. We have chosen here to learn a ViZDoom task called `Health Gathering Supreme`.\n", +- "\n", +- "### The scenario: Health Gathering Supreme\n", +- "\n", +- "\"Health-Gathering-Supreme\"/\n", +- "\n", +- "\n", +- "\n", +- "The objective of this scenario is to **teach the agent how to survive without knowing what makes him survive**. Agent know only that **life is precious** and death is bad so **it must learn what prolongs his existence and that his health is connected with it**.\n", +- "\n", +- "Map is a rectangle containing walls and with a green, acidic floor which **hurts the player periodically**. Initially there are some medkits spread uniformly over the map. A new medkit falls from the skies every now and then. **Medkits heal some portions of player's health** - to survive agent needs to pick them up. Episode finishes after player's death or on timeout.\n", +- "\n", +- "Further configuration:\n", +- "- Living_reward = 1\n", +- "- 3 available buttons: turn left, turn right, move forward\n", +- "- 1 available game variable: HEALTH\n", +- "- death penalty = 100\n", +- "\n", +- "You can find out more about the scenarios available in ViZDoom [here](https://github.com/Farama-Foundation/ViZDoom/tree/master/scenarios).\n", +- "\n", +- "There are also a number of more complex scenarios that have been create for ViZDoom, such as the ones detailed on [this github page](https://github.com/edbeeching/3d_control_deep_rl).\n", +- "\n" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "siHZZ34DiZEp" +- }, +- "source": [ +- "## Training the agent\n", +- "- We're going to train the agent for 4000000 steps it will take approximately 20min" +- ] +- }, +- { +- "cell_type": "code", +- "execution_count": null, +- "metadata": { +- "id": "y_TeicMvyKHP" +- }, +- "outputs": [], +- "source": [ +- "## Start the training, this should take around 15 minutes\n", +- "register_vizdoom_components()\n", +- "\n", +- "# The scenario we train on today is health gathering\n", +- "# other scenarios include \"doom_basic\", \"doom_two_colors_easy\", \"doom_dm\", \"doom_dwango5\", \"doom_my_way_home\", \"doom_deadly_corridor\", \"doom_defend_the_center\", \"doom_defend_the_line\"\n", +- "env = \"doom_health_gathering_supreme\"\n", +- "cfg = parse_vizdoom_cfg(argv=[f\"--env={env}\", \"--num_workers=8\", \"--num_envs_per_worker=4\", \"--train_for_env_steps=4000000\"])\n", +- "\n", +- "status = run_rl(cfg)" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "5L0nBS9e_jqC" +- }, +- "source": [ +- "## Let's take a look at the performance of the trained policy and output a video of the agent." +- ] +- }, +- { +- "cell_type": "code", +- "execution_count": null, +- "metadata": { +- "id": "MGSA4Kg5_i0j" +- }, +- "outputs": [], +- "source": [ +- "from sample_factory.enjoy import enjoy\n", +- "cfg = parse_vizdoom_cfg(argv=[f\"--env={env}\", \"--num_workers=1\", \"--save_video\", \"--no_render\", \"--max_num_episodes=10\"], evaluation=True)\n", +- "status = enjoy(cfg)" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "Lj5L1x0WLxwB" +- }, +- "source": [ +- "## Now lets visualize the performance of the agent" +- ] +- }, +- { +- "cell_type": "code", +- "execution_count": null, +- "metadata": { +- "id": "WsXhBY7JNOdJ" +- }, +- "outputs": [], +- "source": [ +- "from base64 import b64encode\n", +- "from IPython.display import HTML\n", +- "\n", +- "mp4 = open('/content/train_dir/default_experiment/replay.mp4','rb').read()\n", +- "data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n", +- "HTML(\"\"\"\n", +- "\n", +- "\"\"\" % data_url)" +- ] +- }, +- { +- "cell_type": "markdown", +- "source": [ +- "The agent has learned something, but its performance could be better. We would clearly need to train for longer. But let's upload this model to the Hub." +- ], +- "metadata": { +- "id": "2A4pf_1VwPqR" +- } +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "CSQVWF0kNuy9" +- }, +- "source": [ +- "## Now lets upload your checkpoint and video to the Hugging Face Hub\n", +- "\n", +- "\n" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "JquRrWytA6eo" +- }, +- "source": [ +- "To be able to share your model with the community there are three more steps to follow:\n", +- "\n", +- "1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join\n", +- "\n", +- "2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website.\n", +- "- Create a new token (https://huggingface.co/settings/tokens) **with write role**\n", +- "\n", +- "\"Create\n", +- "\n", +- "- Copy the token\n", +- "- Run the cell below and paste the token" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "_tsf2uv0g_4p" +- }, +- "source": [ +- "If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`" +- ] +- }, +- { +- "cell_type": "code", +- "execution_count": null, +- "metadata": { +- "id": "GoQm_jYSOts0" +- }, +- "outputs": [], +- "source": [ +- "from huggingface_hub import notebook_login\n", +- "notebook_login()\n", +- "!git config --global credential.helper store" +- ] +- }, +- { +- "cell_type": "code", +- "execution_count": null, +- "metadata": { +- "id": "sEawW_i0OvJV" +- }, +- "outputs": [], +- "source": [ +- "from sample_factory.enjoy import enjoy\n", +- "\n", +- "hf_username = \"ThomasSimonini\" # insert your HuggingFace username here\n", +- "\n", +- "cfg = parse_vizdoom_cfg(argv=[f\"--env={env}\", \"--num_workers=1\", \"--save_video\", \"--no_render\", \"--max_num_episodes=10\", \"--max_num_frames=100000\", \"--push_to_hub\", f\"--hf_repository={hf_username}/rl_course_vizdoom_health_gathering_supreme\"], evaluation=True)\n", +- "status = enjoy(cfg)" +- ] +- }, +- { +- "cell_type": "markdown", +- "source": [ +- "## Let's load another model\n", +- "\n", +- "\n" +- ], +- "metadata": { +- "id": "9PzeXx-qxVvw" +- } +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "mHZAWSgL5F7P" +- }, +- "source": [ +- "This agent's performance was good, but can do better! Let's download and visualize an agent trained for 10B timesteps from the hub." +- ] +- }, +- { +- "cell_type": "code", +- "execution_count": null, +- "metadata": { +- "id": "Ud6DwAUl5S-l" +- }, +- "outputs": [], +- "source": [ +- "#download the agent from the hub\n", +- "!python -m sample_factory.huggingface.load_from_hub -r edbeeching/doom_health_gathering_supreme_2222 -d ./train_dir\n" +- ] +- }, +- { +- "cell_type": "code", +- "execution_count": null, +- "metadata": { +- "id": "qoUJhL6x6sY5" +- }, +- "outputs": [], +- "source": [ +- "!ls train_dir/doom_health_gathering_supreme_2222" +- ] +- }, +- { +- "cell_type": "code", +- "execution_count": null, +- "metadata": { +- "id": "lZskc8LG8qr8" +- }, +- "outputs": [], +- "source": [ +- "env = \"doom_health_gathering_supreme\"\n", +- "cfg = parse_vizdoom_cfg(argv=[f\"--env={env}\", \"--num_workers=1\", \"--save_video\", \"--no_render\", \"--max_num_episodes=10\", \"--experiment=doom_health_gathering_supreme_2222\", \"--train_dir=train_dir\"], evaluation=True)\n", +- "status = enjoy(cfg)" +- ] +- }, +- { +- "cell_type": "code", +- "execution_count": null, +- "metadata": { +- "id": "BtzXBoj65Wmq" +- }, +- "outputs": [], +- "source": [ +- "mp4 = open('/content/train_dir/doom_health_gathering_supreme_2222/replay.mp4','rb').read()\n", +- "data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n", +- "HTML(\"\"\"\n", +- "\n", +- "\"\"\" % data_url)" +- ] +- }, +- { +- "cell_type": "markdown", +- "source": [ +- "## Some additional challenges 🏆: Doom Deathmatch\n", +- "\n", +- "Training an agent to play a Doom deathmatch **takes many hours on a more beefy machine than is available in Colab**.\n", +- "\n", +- "Fortunately, we have have **already trained an agent in this scenario and it is available in the 🤗 Hub!** Let’s download the model and visualize the agent’s performance." +- ], +- "metadata": { +- "id": "ie5YWC3NyKO8" +- } +- }, +- { +- "cell_type": "code", +- "execution_count": null, +- "metadata": { +- "id": "fq3WFeus81iI" +- }, +- "outputs": [], +- "source": [ +- "# Download the agent from the hub\n", +- "!python -m sample_factory.huggingface.load_from_hub -r edbeeching/doom_deathmatch_bots_2222 -d ./train_dir" +- ] +- }, +- { +- "cell_type": "markdown", +- "source": [ +- "Given the agent plays for a long time the video generation can take **10 minutes**." +- ], +- "metadata": { +- "id": "7AX_LwxR2FQ0" +- } +- }, +- { +- "cell_type": "code", +- "execution_count": null, +- "metadata": { +- "id": "0hq6XL__85Bv" +- }, +- "outputs": [], +- "source": [ +- "\n", +- "from sample_factory.enjoy import enjoy\n", +- "register_vizdoom_components()\n", +- "env = \"doom_deathmatch_bots\"\n", +- "cfg = parse_vizdoom_cfg(argv=[f\"--env={env}\", \"--num_workers=1\", \"--save_video\", \"--no_render\", \"--max_num_episodes=1\", \"--experiment=doom_deathmatch_bots_2222\", \"--train_dir=train_dir\"], evaluation=True)\n", +- "status = enjoy(cfg)\n", +- "mp4 = open('/content/train_dir/doom_deathmatch_bots_2222/replay.mp4','rb').read()\n", +- "data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n", +- "HTML(\"\"\"\n", +- "\n", +- "\"\"\" % data_url)" +- ] +- }, +- { +- "cell_type": "markdown", +- "source": [ +- "\n", +- "You **can try to train your agent in this environment** using the code above, but not on colab.\n", +- "**Good luck 🤞**" +- ], +- "metadata": { +- "id": "N6mEC-4zyihx" +- } +- }, +- { +- "cell_type": "markdown", +- "source": [ +- "If you prefer an easier scenario, **why not try training in another ViZDoom scenario such as `doom_deadly_corridor` or `doom_defend_the_center`.**\n", +- "\n", +- "\n", +- "\n", +- "---\n", +- "\n", +- "\n", +- "This concludes the last unit. But we are not finished yet! 🤗 The following **bonus section include some of the most interesting, advanced and cutting edge work in Deep Reinforcement Learning**.\n", +- "\n", +- "## Keep learning, stay awesome 🤗" +- ], +- "metadata": { +- "id": "YnDAngN6zeeI" +- } +- } +- ], +- "metadata": { +- "accelerator": "GPU", +- "colab": { +- "provenance": [], +- "collapsed_sections": [ +- "PU4FVzaoM6fC", +- "nB68Eb9UgC94", +- "ez5UhUtYcWXF", +- "sgRy6wnrgnij" +- ], +- "private_outputs": true, +- "include_colab_link": true +- }, +- "gpuClass": "standard", +- "kernelspec": { +- "display_name": "Python 3", +- "name": "python3" +- }, +- "language_info": { +- "name": "python" +- } +- }, +- "nbformat": 4, +- "nbformat_minor": 0 +-} +\ No newline at end of file +diff --git a/notebooks/unit8/ppo.py b/notebooks/unit8/ppo.py +deleted file mode 100644 +index a11bc55..0000000 +--- a/notebooks/unit8/ppo.py ++++ /dev/null +@@ -1,532 +0,0 @@ +-import argparse +-import os +-import random +-import time +-from distutils.util import strtobool +- +-import gym +-import numpy as np +-import torch +-import torch.nn as nn +-import torch.optim as optim +-from torch.distributions.categorical import Categorical +-from torch.utils.tensorboard import SummaryWriter +- +-from huggingface_hub import HfApi, upload_folder +-from huggingface_hub.repocard import metadata_eval_result, metadata_save +- +-from pathlib import Path +-import datetime +-import tempfile +-import json +-import shutil +-import imageio +- +-from wasabi import Printer +-msg = Printer() +- +-def parse_args(): +- # fmt: off +- parser = argparse.ArgumentParser() +- parser.add_argument("--exp-name", type=str, default=os.path.basename(__file__).rstrip(".py"), +- help="the name of this experiment") +- parser.add_argument("--seed", type=int, default=1, +- help="seed of the experiment") +- parser.add_argument("--torch-deterministic", type=lambda x: bool(strtobool(x)), default=True, nargs="?", const=True, +- help="if toggled, `torch.backends.cudnn.deterministic=False`") +- parser.add_argument("--cuda", type=lambda x: bool(strtobool(x)), default=True, nargs="?", const=True, +- help="if toggled, cuda will be enabled by default") +- parser.add_argument("--track", type=lambda x: bool(strtobool(x)), default=False, nargs="?", const=True, +- help="if toggled, this experiment will be tracked with Weights and Biases") +- parser.add_argument("--wandb-project-name", type=str, default="cleanRL", +- help="the wandb's project name") +- parser.add_argument("--wandb-entity", type=str, default=None, +- help="the entity (team) of wandb's project") +- parser.add_argument("--capture-video", type=lambda x: bool(strtobool(x)), default=False, nargs="?", const=True, +- help="whether to capture videos of the agent performances (check out `videos` folder)") +- +- # Algorithm specific arguments +- parser.add_argument("--env-id", type=str, default="CartPole-v1", +- help="the id of the environment") +- parser.add_argument("--total-timesteps", type=int, default=50000, +- help="total timesteps of the experiments") +- parser.add_argument("--learning-rate", type=float, default=2.5e-4, +- help="the learning rate of the optimizer") +- parser.add_argument("--num-envs", type=int, default=4, +- help="the number of parallel game environments") +- parser.add_argument("--num-steps", type=int, default=128, +- help="the number of steps to run in each environment per policy rollout") +- parser.add_argument("--anneal-lr", type=lambda x: bool(strtobool(x)), default=True, nargs="?", const=True, +- help="Toggle learning rate annealing for policy and value networks") +- parser.add_argument("--gae", type=lambda x: bool(strtobool(x)), default=True, nargs="?", const=True, +- help="Use GAE for advantage computation") +- parser.add_argument("--gamma", type=float, default=0.99, +- help="the discount factor gamma") +- parser.add_argument("--gae-lambda", type=float, default=0.95, +- help="the lambda for the general advantage estimation") +- parser.add_argument("--num-minibatches", type=int, default=4, +- help="the number of mini-batches") +- parser.add_argument("--update-epochs", type=int, default=4, +- help="the K epochs to update the policy") +- parser.add_argument("--norm-adv", type=lambda x: bool(strtobool(x)), default=True, nargs="?", const=True, +- help="Toggles advantages normalization") +- parser.add_argument("--clip-coef", type=float, default=0.2, +- help="the surrogate clipping coefficient") +- parser.add_argument("--clip-vloss", type=lambda x: bool(strtobool(x)), default=True, nargs="?", const=True, +- help="Toggles whether or not to use a clipped loss for the value function, as per the paper.") +- parser.add_argument("--ent-coef", type=float, default=0.01, +- help="coefficient of the entropy") +- parser.add_argument("--vf-coef", type=float, default=0.5, +- help="coefficient of the value function") +- parser.add_argument("--max-grad-norm", type=float, default=0.5, +- help="the maximum norm for the gradient clipping") +- parser.add_argument("--target-kl", type=float, default=None, +- help="the target KL divergence threshold") +- +- # Adding HuggingFace argument +- parser.add_argument("--repo-id", type=str, default="LizardAPN/ppo-CartPole-v1", help="id of the model repository from the Hugging Face Hub {username/repo_name}") +- +- args = parser.parse_args() +- args.batch_size = int(args.num_envs * args.num_steps) +- args.minibatch_size = int(args.batch_size // args.num_minibatches) +- # fmt: on +- return args +- +-def package_to_hub(repo_id, +- model, +- hyperparameters, +- eval_env, +- video_fps=30, +- commit_message="Push agent to the Hub", +- token=None, +- logs=None): +- """ +- Evaluate, Generate a video and Upload a model to Hugging Face Hub. +- """ +- msg.info( +- "This function will save, evaluate, generate a video of your agent, " +- "create a model card and push everything to the hub." +- ) +- +- # Step 1: Create repo +- repo_url = HfApi().create_repo( +- repo_id=repo_id, +- token=token, +- private=False, +- exist_ok=True, +- ) +- +- with tempfile.TemporaryDirectory() as tmpdirname: +- tmpdirname = Path(tmpdirname) +- +- # Step 2: Save the model +- torch.save(model.state_dict(), tmpdirname / "model.pt") +- +- # Step 3: Evaluate the model +- mean_reward, std_reward = _evaluate_agent(eval_env, 10, model) +- +- # Prepare evaluation data +- eval_datetime = datetime.datetime.now() +- evaluate_data = { +- "env_id": hyperparameters.env_id, +- "mean_reward": mean_reward, +- "std_reward": std_reward, +- "n_evaluation_episodes": 10, +- "eval_datetime": eval_datetime.isoformat(), +- } +- +- # Save evaluation results +- with open(tmpdirname / "results.json", "w") as outfile: +- json.dump(evaluate_data, outfile) +- +- # Step 4: Generate video +- video_path = tmpdirname / "replay.mp4" +- record_video(eval_env, model, video_path, video_fps) +- +- # Step 5: Generate model card +- generated_model_card, metadata = _generate_model_card("PPO", hyperparameters.env_id, mean_reward, std_reward, hyperparameters) +- _save_model_card(tmpdirname, generated_model_card, metadata) +- +- # Step 6: Add logs if provided +- if logs: +- _add_logdir(tmpdirname, Path(logs)) +- +- # Step 7: Upload to Hub +- msg.info(f"Pushing repo {repo_id} to the Hugging Face Hub") +- repo_url = upload_folder( +- repo_id=repo_id, +- folder_path=tmpdirname, +- path_in_repo="", +- commit_message=commit_message, +- token=token, +- ) +- +- msg.info(f"Your model is pushed to the Hub. You can view your model here: {repo_url}") +- return repo_url +- +-def _evaluate_agent(env, n_eval_episodes, policy): +- """ +- Evaluate the agent for n_eval_episodes episodes. +- """ +- episode_rewards = [] +- for _ in range(n_eval_episodes): +- state, _ = env.reset() +- done = False +- total_rewards_ep = 0 +- +- while not done: +- state = torch.Tensor(state).to(device) +- with torch.no_grad(): +- action, _, _, _ = policy.get_action_and_value(state) +- state, reward, terminated, truncated, _ = env.step(action.cpu().numpy()) +- total_rewards_ep += reward +- done = terminated or truncated +- episode_rewards.append(total_rewards_ep) +- +- mean_reward = np.mean(episode_rewards) +- std_reward = np.std(episode_rewards) +- return mean_reward, std_reward +- +-def record_video(env, policy, out_directory, fps=30): +- """ +- Record a video of the agent's performance. +- """ +- images = [] +- state, _ = env.reset() +- img = env.render() +- images.append(img) +- +- done = False +- while not done: +- state = torch.Tensor(state).to(device) +- with torch.no_grad(): +- action, _, _, _ = policy.get_action_and_value(state) +- state, _, terminated, truncated, _ = env.step(action.cpu().numpy()) +- img = env.render() +- images.append(img) +- done = terminated or truncated +- +- imageio.mimsave(out_directory, [np.array(img) for img in images], fps=fps) +- +-def _generate_model_card(model_name, env_id, mean_reward, std_reward, hyperparameters): +- """ +- Generate the model card for the Hub. +- """ +- metadata = generate_metadata(model_name, env_id, mean_reward, std_reward) +- +- # Convert hyperparameters to string +- converted_dict = vars(hyperparameters) +- converted_str = '\n'.join([f"{k}: {v}" for k, v in converted_dict.items()]) +- +- model_card = f""" +- # PPO Agent Playing {env_id} +- +- This is a trained model of a PPO agent playing {env_id}. +- +- # Hyperparameters +- ```python +- {converted_str} +- ``` +- """ +- return model_card, metadata +- +-def generate_metadata(model_name, env_id, mean_reward, std_reward): +- """ +- Define the tags for the model card. +- """ +- metadata = { +- "tags": [ +- env_id, +- "ppo", +- "deep-reinforcement-learning", +- "reinforcement-learning", +- "custom-implementation", +- "deep-rl-course" +- ] +- } +- +- eval_metadata = metadata_eval_result( +- model_pretty_name=model_name, +- task_pretty_name="reinforcement-learning", +- task_id="reinforcement-learning", +- metrics_pretty_name="mean_reward", +- metrics_id="mean_reward", +- metrics_value=f"{mean_reward:.2f} +/- {std_reward:.2f}", +- dataset_pretty_name=env_id, +- dataset_id=env_id, +- ) +- +- metadata.update(eval_metadata) +- return metadata +- +-def _save_model_card(local_path, generated_model_card, metadata): +- """ +- Save the model card and metadata. +- """ +- readme_path = local_path / "README.md" +- with readme_path.open("w", encoding="utf-8") as f: +- f.write(generated_model_card) +- metadata_save(readme_path, metadata) +- +-def _add_logdir(local_path, logdir): +- """ +- Add log directory to the repository. +- """ +- if logdir.exists() and logdir.is_dir(): +- repo_logdir = local_path / "logs" +- if repo_logdir.exists(): +- shutil.rmtree(repo_logdir) +- shutil.copytree(logdir, repo_logdir) +- +-def make_env(env_id, seed, idx, capture_video, run_name): +- """ +- Create a wrapped environment. +- """ +- def thunk(): +- env = gym.make(env_id, render_mode="rgb_array") +- env = gym.wrappers.RecordEpisodeStatistics(env) +- if capture_video and idx == 0: +- env = gym.wrappers.RecordVideo(env, f"videos/{run_name}") +- env.reset(seed=seed) +- env.action_space.seed(seed) +- env.observation_space.seed(seed) +- return env +- return thunk +- +-def layer_init(layer, std=np.sqrt(2), bias_const=0.0): +- """ +- Initialize layer weights. +- """ +- torch.nn.init.orthogonal_(layer.weight, std) +- torch.nn.init.constant_(layer.bias, bias_const) +- return layer +- +-class Agent(nn.Module): +- def __init__(self, envs): +- super().__init__() +- self.critic = nn.Sequential( +- layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 64)), +- nn.Tanh(), +- layer_init(nn.Linear(64, 64)), +- nn.Tanh(), +- layer_init(nn.Linear(64, 1), std=1.0), +- ) +- self.actor = nn.Sequential( +- layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 64)), +- nn.Tanh(), +- layer_init(nn.Linear(64, 64)), +- nn.Tanh(), +- layer_init(nn.Linear(64, envs.single_action_space.n), std=0.01), +- ) +- +- def get_value(self, x): +- return self.critic(x) +- +- def get_action_and_value(self, x, action=None): +- logits = self.actor(x) +- probs = Categorical(logits=logits) +- if action is None: +- action = probs.sample() +- return action, probs.log_prob(action), probs.entropy(), self.critic(x) +- +-if __name__ == "__main__": +- args = parse_args() +- run_name = f"{args.env_id}__{args.exp_name}__{args.seed}__{int(time.time())}" +- +- if args.track: +- import wandb +- wandb.init( +- project=args.wandb_project_name, +- entity=args.wandb_entity, +- sync_tensorboard=True, +- config=vars(args), +- name=run_name, +- monitor_gym=True, +- save_code=True, +- ) +- +- writer = SummaryWriter(f"runs/{run_name}") +- writer.add_text( +- "hyperparameters", +- "|param|value|\n|-|-|\n%s" % ("\n".join([f"|{key}|{value}|" for key, value in vars(args).items()])), +- ) +- +- # Seeding +- random.seed(args.seed) +- np.random.seed(args.seed) +- torch.manual_seed(args.seed) +- torch.backends.cudnn.deterministic = args.torch_deterministic +- +- device = torch.device("cuda" if torch.cuda.is_available() and args.cuda else "cpu") +- +- # Environment setup +- envs = gym.vector.SyncVectorEnv( +- [make_env(args.env_id, args.seed + i, i, args.capture_video, run_name) for i in range(args.num_envs)] +- ) +- assert isinstance(envs.single_action_space, gym.spaces.Discrete), "only discrete action space is supported" +- +- agent = Agent(envs).to(device) +- optimizer = optim.Adam(agent.parameters(), lr=args.learning_rate, eps=1e-5) +- +- # Storage setup +- obs = torch.zeros((args.num_steps, args.num_envs) + envs.single_observation_space.shape).to(device) +- actions = torch.zeros((args.num_steps, args.num_envs) + envs.single_action_space.shape).to(device) +- logprobs = torch.zeros((args.num_steps, args.num_envs)).to(device) +- rewards = torch.zeros((args.num_steps, args.num_envs)).to(device) +- dones = torch.zeros((args.num_steps, args.num_envs)).to(device) +- values = torch.zeros((args.num_steps, args.num_envs)).to(device) +- +- # Training loop +- global_step = 0 +- start_time = time.time() +- next_obs, _ = envs.reset(seed=args.seed) +- next_obs = torch.Tensor(next_obs).to(device) +- next_done = torch.zeros(args.num_envs).to(device) +- num_updates = args.total_timesteps // args.batch_size +- +- for update in range(1, num_updates + 1): +- if args.anneal_lr: +- frac = 1.0 - (update - 1.0) / num_updates +- lrnow = frac * args.learning_rate +- optimizer.param_groups[0]["lr"] = lrnow +- +- for step in range(0, args.num_steps): +- global_step += 1 * args.num_envs +- obs[step] = next_obs +- dones[step] = next_done +- +- with torch.no_grad(): +- action, logprob, _, value = agent.get_action_and_value(next_obs) +- values[step] = value.flatten() +- actions[step] = action +- logprobs[step] = logprob +- +- next_obs, reward, terminated, truncated, infos = envs.step(action.cpu().numpy()) +- next_done = np.logical_or(terminated, truncated) +- rewards[step] = torch.tensor(reward).to(device).view(-1) +- next_obs, next_done = torch.Tensor(next_obs).to(device), torch.Tensor(next_done).to(device) +- +- if "final_info" in infos: +- for info in infos["final_info"]: +- if info and "episode" in info: +- print(f"global_step={global_step}, episodic_return={info['episode']['r']}") +- writer.add_scalar("charts/episodic_return", info["episode"]["r"], global_step) +- writer.add_scalar("charts/episodic_length", info["episode"]["l"], global_step) +- +- # Compute advantages and returns +- with torch.no_grad(): +- next_value = agent.get_value(next_obs).reshape(1, -1) +- if args.gae: +- advantages = torch.zeros_like(rewards).to(device) +- lastgaelam = 0 +- for t in reversed(range(args.num_steps)): +- if t == args.num_steps - 1: +- nextnonterminal = 1.0 - next_done +- nextvalues = next_value +- else: +- nextnonterminal = 1.0 - dones[t + 1] +- nextvalues = values[t + 1] +- delta = rewards[t] + args.gamma * nextvalues * nextnonterminal - values[t] +- advantages[t] = lastgaelam = delta + args.gamma * args.gae_lambda * nextnonterminal * lastgaelam +- returns = advantages + values +- else: +- returns = torch.zeros_like(rewards).to(device) +- for t in reversed(range(args.num_steps)): +- if t == args.num_steps - 1: +- nextnonterminal = 1.0 - next_done +- next_return = next_value +- else: +- nextnonterminal = 1.0 - dones[t + 1] +- next_return = returns[t + 1] +- returns[t] = rewards[t] + args.gamma * nextnonterminal * next_return +- advantages = returns - values +- +- # Flatten the batch +- b_obs = obs.reshape((-1,) + envs.single_observation_space.shape) +- b_logprobs = logprobs.reshape(-1) +- b_actions = actions.reshape((-1,) + envs.single_action_space.shape) +- b_advantages = advantages.reshape(-1) +- b_returns = returns.reshape(-1) +- b_values = values.reshape(-1) +- +- # Optimize policy and value network +- b_inds = np.arange(args.batch_size) +- clipfracs = [] +- for epoch in range(args.update_epochs): +- np.random.shuffle(b_inds) +- for start in range(0, args.batch_size, args.minibatch_size): +- end = start + args.minibatch_size +- mb_inds = b_inds[start:end] +- +- _, newlogprob, entropy, newvalue = agent.get_action_and_value(b_obs[mb_inds], b_actions.long()[mb_inds]) +- logratio = newlogprob - b_logprobs[mb_inds] +- ratio = logratio.exp() +- +- with torch.no_grad(): +- old_approx_kl = (-logratio).mean() +- approx_kl = ((ratio - 1) - logratio).mean() +- clipfracs += [((ratio - 1.0).abs() > args.clip_coef).float().mean().item()] +- +- mb_advantages = b_advantages[mb_inds] +- if args.norm_adv: +- mb_advantages = (mb_advantages - mb_advantages.mean()) / (mb_advantages.std() + 1e-8) +- +- # Policy loss +- pg_loss1 = -mb_advantages * ratio +- pg_loss2 = -mb_advantages * torch.clamp(ratio, 1 - args.clip_coef, 1 + args.clip_coef) +- pg_loss = torch.max(pg_loss1, pg_loss2).mean() +- +- # Value loss +- newvalue = newvalue.view(-1) +- if args.clip_vloss: +- v_loss_unclipped = (newvalue - b_returns[mb_inds]) ** 2 +- v_clipped = b_values[mb_inds] + torch.clamp( +- newvalue - b_values[mb_inds], +- -args.clip_coef, +- args.clip_coef, +- ) +- v_loss_clipped = (v_clipped - b_returns[mb_inds]) ** 2 +- v_loss_max = torch.max(v_loss_unclipped, v_loss_clipped) +- v_loss = 0.5 * v_loss_max.mean() +- else: +- v_loss = 0.5 * ((newvalue - b_returns[mb_inds]) ** 2).mean() +- +- entropy_loss = entropy.mean() +- loss = pg_loss - args.ent_coef * entropy_loss + v_loss * args.vf_coef +- +- optimizer.zero_grad() +- loss.backward() +- nn.utils.clip_grad_norm_(agent.parameters(), args.max_grad_norm) +- optimizer.step() +- +- if args.target_kl is not None and approx_kl > args.target_kl: +- break +- +- # Log training metrics +- y_pred, y_true = b_values.cpu().numpy(), b_returns.cpu().numpy() +- var_y = np.var(y_true) +- explained_var = np.nan if var_y == 0 else 1 - np.var(y_true - y_pred) / var_y +- +- writer.add_scalar("charts/learning_rate", optimizer.param_groups[0]["lr"], global_step) +- writer.add_scalar("losses/value_loss", v_loss.item(), global_step) +- writer.add_scalar("losses/policy_loss", pg_loss.item(), global_step) +- writer.add_scalar("losses/entropy", entropy_loss.item(), global_step) +- writer.add_scalar("losses/old_approx_kl", old_approx_kl.item(), global_step) +- writer.add_scalar("losses/approx_kl", approx_kl.item(), global_step) +- writer.add_scalar("losses/clipfrac", np.mean(clipfracs), global_step) +- writer.add_scalar("losses/explained_variance", explained_var, global_step) +- print("SPS:", int(global_step / (time.time() - start_time))) +- writer.add_scalar("charts/SPS", int(global_step / (time.time() - start_time)), global_step) +- +- envs.close() +- writer.close() +- +- # Push to Hub +- eval_env = gym.make(args.env_id, render_mode="rgb_array") +- package_to_hub( +- repo_id=args.repo_id, +- model=agent, +- hyperparameters=args, +- eval_env=eval_env, +- logs=f"runs/{run_name}", +- ) +\ No newline at end of file +diff --git a/notebooks/unit8/ppo_part1.ipynb b/notebooks/unit8/ppo_part1.ipynb +deleted file mode 100644 +index 96a2597..0000000 +--- a/notebooks/unit8/ppo_part1.ipynb ++++ /dev/null +@@ -1,1370 +0,0 @@ +-{ +- "cells": [ +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "view-in-github", +- "colab_type": "text" +- }, +- "source": [ +- "\"Open" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "-cf5-oDPjwf8" +- }, +- "source": [ +- "# Unit 8: Proximal Policy Gradient (PPO) with PyTorch 🤖\n", +- "\n", +- "\"Unit\n", +- "\n", +- "\n", +- "In this notebook, you'll learn to **code your PPO agent from scratch with PyTorch using CleanRL implementation as model**.\n", +- "\n", +- "To test its robustness, we're going to train it in:\n", +- "\n", +- "- [LunarLander-v2 🚀](https://www.gymlibrary.dev/environments/box2d/lunar_lander/)\n" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "2Fl6Rxt0lc0O" +- }, +- "source": [ +- "⬇️ Here is an example of what you will achieve. ⬇️" +- ] +- }, +- { +- "cell_type": "code", +- "execution_count": null, +- "metadata": { +- "id": "DbKfCj5ilgqT" +- }, +- "outputs": [], +- "source": [ +- "%%html\n", +- "" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "YcOFdWpnlxNf" +- }, +- "source": [ +- "We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the GitHub Repo](https://github.com/huggingface/deep-rl-class/issues)." +- ] +- }, +- { +- "cell_type": "markdown", +- "source": [ +- "## Objectives of this notebook 🏆\n", +- "\n", +- "At the end of the notebook, you will:\n", +- "\n", +- "- Be able to **code your PPO agent from scratch using PyTorch**.\n", +- "- Be able to **push your trained agent and the code to the Hub** with a nice video replay and an evaluation score 🔥.\n", +- "\n", +- "\n" +- ], +- "metadata": { +- "id": "T6lIPYFghhYL" +- } +- }, +- { +- "cell_type": "markdown", +- "source": [ +- "## This notebook is from the Deep Reinforcement Learning Course\n", +- "\"Deep\n", +- "\n", +- "In this free course, you will:\n", +- "\n", +- "- 📖 Study Deep Reinforcement Learning in **theory and practice**.\n", +- "- 🧑‍💻 Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, CleanRL and Sample Factory 2.0.\n", +- "- 🤖 Train **agents in unique environments**\n", +- "\n", +- "Don’t forget to **sign up to the course** (we are collecting your email to be able to **send you the links when each Unit is published and give you information about the challenges and updates).**\n", +- "\n", +- "\n", +- "The best way to keep in touch is to join our discord server to exchange with the community and with us 👉🏻 https://discord.gg/ydHrjt3WP5" +- ], +- "metadata": { +- "id": "Wp-rD6Fuhq31" +- } +- }, +- { +- "cell_type": "markdown", +- "source": [ +- "## Prerequisites 🏗️\n", +- "Before diving into the notebook, you need to:\n", +- "\n", +- "🔲 📚 Study [PPO by reading Unit 8](https://huggingface.co/deep-rl-course/unit8/introduction) 🤗 " +- ], +- "metadata": { +- "id": "rasqqGQlhujA" +- } +- }, +- { +- "cell_type": "markdown", +- "source": [ +- "To validate this hands-on for the [certification process](https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process), you need to push one model, we don't ask for a minimal result but we **advise you to try different hyperparameters settings to get better results**.\n", +- "\n", +- "If you don't find your model, **go to the bottom of the page and click on the refresh button**\n", +- "\n", +- "For more information about the certification process, check this section 👉 https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process" +- ], +- "metadata": { +- "id": "PUFfMGOih3CW" +- } +- }, +- { +- "cell_type": "markdown", +- "source": [ +- "## Set the GPU 💪\n", +- "- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`\n", +- "\n", +- "\"GPU" +- ], +- "metadata": { +- "id": "PU4FVzaoM6fC" +- } +- }, +- { +- "cell_type": "markdown", +- "source": [ +- "- `Hardware Accelerator > GPU`\n", +- "\n", +- "\"GPU" +- ], +- "metadata": { +- "id": "KV0NyFdQM9ZG" +- } +- }, +- { +- "cell_type": "markdown", +- "source": [ +- "## Create a virtual display 🔽\n", +- "\n", +- "During the notebook, we'll need to generate a replay video. To do so, with colab, **we need to have a virtual screen to be able to render the environment** (and thus record the frames).\n", +- "\n", +- "Hence the following cell will install the librairies and create and run a virtual screen 🖥" +- ], +- "metadata": { +- "id": "bTpYcVZVMzUI" +- } +- }, +- { +- "cell_type": "code", +- "source": [ +- "!pip install setuptools==65.5.0" +- ], +- "metadata": { +- "id": "Fd731S8-NuJA" +- }, +- "execution_count": null, +- "outputs": [] +- }, +- { +- "cell_type": "code", +- "execution_count": null, +- "metadata": { +- "id": "jV6wjQ7Be7p5" +- }, +- "outputs": [], +- "source": [ +- "%%capture\n", +- "!apt install python-opengl\n", +- "!apt install ffmpeg\n", +- "!apt install xvfb\n", +- "!apt install swig cmake\n", +- "!pip install pyglet==1.5\n", +- "!pip3 install pyvirtualdisplay" +- ] +- }, +- { +- "cell_type": "code", +- "source": [ +- "# Virtual display\n", +- "from pyvirtualdisplay import Display\n", +- "\n", +- "virtual_display = Display(visible=0, size=(1400, 900))\n", +- "virtual_display.start()" +- ], +- "metadata": { +- "id": "ww5PQH1gNLI4" +- }, +- "execution_count": null, +- "outputs": [] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "ncIgfNf3mOtc" +- }, +- "source": [ +- "## Install dependencies 🔽\n", +- "For this exercise, we use `gym==0.22`." +- ] +- }, +- { +- "cell_type": "code", +- "source": [ +- "!pip install gym==0.22\n", +- "!pip install imageio-ffmpeg\n", +- "!pip install huggingface_hub\n", +- "!pip install gym[box2d]==0.22" +- ], +- "metadata": { +- "id": "9xZQFTPcsKUK" +- }, +- "execution_count": null, +- "outputs": [] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "oDkUufewmq6v" +- }, +- "source": [ +- "## Let's code PPO from scratch with Costa Huang tutorial\n", +- "- For the core implementation of PPO we're going to use the excellent [Costa Huang](https://costa.sh/) tutorial.\n", +- "- In addition to the tutorial, to go deeper you can read the 37 core implementation details: https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/\n", +- "\n", +- "👉 The video tutorial: https://youtu.be/MEt6rrxH8W4" +- ] +- }, +- { +- "cell_type": "code", +- "execution_count": null, +- "metadata": { +- "id": "aNgEL1_uvhaq" +- }, +- "outputs": [], +- "source": [ +- "from IPython.display import HTML\n", +- "\n", +- "HTML('')" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "f34ILn7AvTbt" +- }, +- "source": [ +- "- The best is to code first on the cell below, this way, if you kill the machine **you don't loose the implementation**." +- ] +- }, +- { +- "cell_type": "code", +- "execution_count": null, +- "metadata": { +- "id": "_bE708C6mhE7" +- }, +- "outputs": [], +- "source": [ +- "### Your code here:" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "mk-a9CmNuS2W" +- }, +- "source": [ +- "## Add the Hugging Face Integration 🤗\n", +- "- In order to push our model to the Hub, we need to define a function `package_to_hub`" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "TPi1Nme-oGWd" +- }, +- "source": [ +- "- Add dependencies we need to push our model to the Hub" +- ] +- }, +- { +- "cell_type": "code", +- "execution_count": null, +- "metadata": { +- "id": "Sj8bz-AmoNVj" +- }, +- "outputs": [], +- "source": [ +- "from huggingface_hub import HfApi, upload_folder\n", +- "from huggingface_hub.repocard import metadata_eval_result, metadata_save\n", +- "\n", +- "from pathlib import Path\n", +- "import datetime\n", +- "import tempfile\n", +- "import json\n", +- "import shutil\n", +- "import imageio\n", +- "\n", +- "from wasabi import Printer\n", +- "msg = Printer()" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "5rDr8-lWn0zi" +- }, +- "source": [ +- "- Add new argument in `parse_args()` function to define the repo-id where we want to push the model." +- ] +- }, +- { +- "cell_type": "code", +- "execution_count": null, +- "metadata": { +- "id": "iHQiqQEFn0QH" +- }, +- "outputs": [], +- "source": [ +- "# Adding HuggingFace argument\n", +- "parser.add_argument(\"--repo-id\", type=str, default=\"ThomasSimonini/ppo-CartPole-v1\", help=\"id of the model repository from the Hugging Face Hub {username/repo_name}\")" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "blLZMiBAoUVT" +- }, +- "source": [ +- "- Next, we add the methods needed to push the model to the Hub\n", +- "\n", +- "- These methods will:\n", +- " - `_evalutate_agent()`: evaluate the agent.\n", +- " - `_generate_model_card()`: generate the model card of your agent.\n", +- " - `_record_video()`: record a video of your agent." +- ] +- }, +- { +- "cell_type": "code", +- "execution_count": null, +- "metadata": { +- "id": "WlLcz4L9odXs" +- }, +- "outputs": [], +- "source": [ +- "def package_to_hub(repo_id,\n", +- " model,\n", +- " hyperparameters,\n", +- " eval_env,\n", +- " video_fps=30,\n", +- " commit_message=\"Push agent to the Hub\",\n", +- " token= None,\n", +- " logs=None\n", +- " ):\n", +- " \"\"\"\n", +- " Evaluate, Generate a video and Upload a model to Hugging Face Hub.\n", +- " This method does the complete pipeline:\n", +- " - It evaluates the model\n", +- " - It generates the model card\n", +- " - It generates a replay video of the agent\n", +- " - It pushes everything to the hub\n", +- " :param repo_id: id of the model repository from the Hugging Face Hub\n", +- " :param model: trained model\n", +- " :param eval_env: environment used to evaluate the agent\n", +- " :param fps: number of fps for rendering the video\n", +- " :param commit_message: commit message\n", +- " :param logs: directory on local machine of tensorboard logs you'd like to upload\n", +- " \"\"\"\n", +- " msg.info(\n", +- " \"This function will save, evaluate, generate a video of your agent, \"\n", +- " \"create a model card and push everything to the hub. \"\n", +- " \"It might take up to 1min. \\n \"\n", +- " \"This is a work in progress: if you encounter a bug, please open an issue.\"\n", +- " )\n", +- " # Step 1: Clone or create the repo\n", +- " repo_url = HfApi().create_repo(\n", +- " repo_id=repo_id,\n", +- " token=token,\n", +- " private=False,\n", +- " exist_ok=True,\n", +- " )\n", +- "\n", +- " with tempfile.TemporaryDirectory() as tmpdirname:\n", +- " tmpdirname = Path(tmpdirname)\n", +- "\n", +- " # Step 2: Save the model\n", +- " torch.save(model.state_dict(), tmpdirname / \"model.pt\")\n", +- "\n", +- " # Step 3: Evaluate the model and build JSON\n", +- " mean_reward, std_reward = _evaluate_agent(eval_env,\n", +- " 10,\n", +- " model)\n", +- "\n", +- " # First get datetime\n", +- " eval_datetime = datetime.datetime.now()\n", +- " eval_form_datetime = eval_datetime.isoformat()\n", +- "\n", +- " evaluate_data = {\n", +- " \"env_id\": hyperparameters.env_id,\n", +- " \"mean_reward\": mean_reward,\n", +- " \"std_reward\": std_reward,\n", +- " \"n_evaluation_episodes\": 10,\n", +- " \"eval_datetime\": eval_form_datetime,\n", +- " }\n", +- "\n", +- " # Write a JSON file\n", +- " with open(tmpdirname / \"results.json\", \"w\") as outfile:\n", +- " json.dump(evaluate_data, outfile)\n", +- "\n", +- " # Step 4: Generate a video\n", +- " video_path = tmpdirname / \"replay.mp4\"\n", +- " record_video(eval_env, model, video_path, video_fps)\n", +- "\n", +- " # Step 5: Generate the model card\n", +- " generated_model_card, metadata = _generate_model_card(\"PPO\", hyperparameters.env_id, mean_reward, std_reward, hyperparameters)\n", +- " _save_model_card(tmpdirname, generated_model_card, metadata)\n", +- "\n", +- " # Step 6: Add logs if needed\n", +- " if logs:\n", +- " _add_logdir(tmpdirname, Path(logs))\n", +- "\n", +- " msg.info(f\"Pushing repo {repo_id} to the Hugging Face Hub\")\n", +- "\n", +- " repo_url = upload_folder(\n", +- " repo_id=repo_id,\n", +- " folder_path=tmpdirname,\n", +- " path_in_repo=\"\",\n", +- " commit_message=commit_message,\n", +- " token=token,\n", +- " )\n", +- "\n", +- " msg.info(f\"Your model is pushed to the Hub. You can view your model here: {repo_url}\")\n", +- " return repo_url\n", +- "\n", +- "\n", +- "def _evaluate_agent(env, n_eval_episodes, policy):\n", +- " \"\"\"\n", +- " Evaluate the agent for ``n_eval_episodes`` episodes and returns average reward and std of reward.\n", +- " :param env: The evaluation environment\n", +- " :param n_eval_episodes: Number of episode to evaluate the agent\n", +- " :param policy: The agent\n", +- " \"\"\"\n", +- " episode_rewards = []\n", +- " for episode in range(n_eval_episodes):\n", +- " state = env.reset()\n", +- " step = 0\n", +- " done = False\n", +- " total_rewards_ep = 0\n", +- "\n", +- " while done is False:\n", +- " state = torch.Tensor(state).to(device)\n", +- " action, _, _, _ = policy.get_action_and_value(state)\n", +- " new_state, reward, done, info = env.step(action.cpu().numpy())\n", +- " total_rewards_ep += reward\n", +- " if done:\n", +- " break\n", +- " state = new_state\n", +- " episode_rewards.append(total_rewards_ep)\n", +- " mean_reward = np.mean(episode_rewards)\n", +- " std_reward = np.std(episode_rewards)\n", +- "\n", +- " return mean_reward, std_reward\n", +- "\n", +- "\n", +- "def record_video(env, policy, out_directory, fps=30):\n", +- " images = []\n", +- " done = False\n", +- " state = env.reset()\n", +- " img = env.render(mode='rgb_array')\n", +- " images.append(img)\n", +- " while not done:\n", +- " state = torch.Tensor(state).to(device)\n", +- " # Take the action (index) that have the maximum expected future reward given that state\n", +- " action, _, _, _ = policy.get_action_and_value(state)\n", +- " state, reward, done, info = env.step(action.cpu().numpy()) # We directly put next_state = state for recording logic\n", +- " img = env.render(mode='rgb_array')\n", +- " images.append(img)\n", +- " imageio.mimsave(out_directory, [np.array(img) for i, img in enumerate(images)], fps=fps)\n", +- "\n", +- "\n", +- "def _generate_model_card(model_name, env_id, mean_reward, std_reward, hyperparameters):\n", +- " \"\"\"\n", +- " Generate the model card for the Hub\n", +- " :param model_name: name of the model\n", +- " :env_id: name of the environment\n", +- " :mean_reward: mean reward of the agent\n", +- " :std_reward: standard deviation of the mean reward of the agent\n", +- " :hyperparameters: training arguments\n", +- " \"\"\"\n", +- " # Step 1: Select the tags\n", +- " metadata = generate_metadata(model_name, env_id, mean_reward, std_reward)\n", +- "\n", +- " # Transform the hyperparams namespace to string\n", +- " converted_dict = vars(hyperparameters)\n", +- " converted_str = str(converted_dict)\n", +- " converted_str = converted_str.split(\", \")\n", +- " converted_str = '\\n'.join(converted_str)\n", +- "\n", +- " # Step 2: Generate the model card\n", +- " model_card = f\"\"\"\n", +- " # PPO Agent Playing {env_id}\n", +- "\n", +- " This is a trained model of a PPO agent playing {env_id}.\n", +- "\n", +- " # Hyperparameters\n", +- " ```python\n", +- " {converted_str}\n", +- " ```\n", +- " \"\"\"\n", +- " return model_card, metadata\n", +- "\n", +- "\n", +- "def generate_metadata(model_name, env_id, mean_reward, std_reward):\n", +- " \"\"\"\n", +- " Define the tags for the model card\n", +- " :param model_name: name of the model\n", +- " :param env_id: name of the environment\n", +- " :mean_reward: mean reward of the agent\n", +- " :std_reward: standard deviation of the mean reward of the agent\n", +- " \"\"\"\n", +- " metadata = {}\n", +- " metadata[\"tags\"] = [\n", +- " env_id,\n", +- " \"ppo\",\n", +- " \"deep-reinforcement-learning\",\n", +- " \"reinforcement-learning\",\n", +- " \"custom-implementation\",\n", +- " \"deep-rl-course\"\n", +- " ]\n", +- "\n", +- " # Add metrics\n", +- " eval = metadata_eval_result(\n", +- " model_pretty_name=model_name,\n", +- " task_pretty_name=\"reinforcement-learning\",\n", +- " task_id=\"reinforcement-learning\",\n", +- " metrics_pretty_name=\"mean_reward\",\n", +- " metrics_id=\"mean_reward\",\n", +- " metrics_value=f\"{mean_reward:.2f} +/- {std_reward:.2f}\",\n", +- " dataset_pretty_name=env_id,\n", +- " dataset_id=env_id,\n", +- " )\n", +- "\n", +- " # Merges both dictionaries\n", +- " metadata = {**metadata, **eval}\n", +- "\n", +- " return metadata\n", +- "\n", +- "\n", +- "def _save_model_card(local_path, generated_model_card, metadata):\n", +- " \"\"\"Saves a model card for the repository.\n", +- " :param local_path: repository directory\n", +- " :param generated_model_card: model card generated by _generate_model_card()\n", +- " :param metadata: metadata\n", +- " \"\"\"\n", +- " readme_path = local_path / \"README.md\"\n", +- " readme = \"\"\n", +- " if readme_path.exists():\n", +- " with readme_path.open(\"r\", encoding=\"utf8\") as f:\n", +- " readme = f.read()\n", +- " else:\n", +- " readme = generated_model_card\n", +- "\n", +- " with readme_path.open(\"w\", encoding=\"utf-8\") as f:\n", +- " f.write(readme)\n", +- "\n", +- " # Save our metrics to Readme metadata\n", +- " metadata_save(readme_path, metadata)\n", +- "\n", +- "\n", +- "def _add_logdir(local_path: Path, logdir: Path):\n", +- " \"\"\"Adds a logdir to the repository.\n", +- " :param local_path: repository directory\n", +- " :param logdir: logdir directory\n", +- " \"\"\"\n", +- " if logdir.exists() and logdir.is_dir():\n", +- " # Add the logdir to the repository under new dir called logs\n", +- " repo_logdir = local_path / \"logs\"\n", +- "\n", +- " # Delete current logs if they exist\n", +- " if repo_logdir.exists():\n", +- " shutil.rmtree(repo_logdir)\n", +- "\n", +- " # Copy logdir into repo logdir\n", +- " shutil.copytree(logdir, repo_logdir)" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "TqX8z8_rooD6" +- }, +- "source": [ +- "- Finally, we call this function at the end of the PPO training" +- ] +- }, +- { +- "cell_type": "code", +- "execution_count": null, +- "metadata": { +- "id": "I8V1vNiTo2hL" +- }, +- "outputs": [], +- "source": [ +- "# Create the evaluation environment\n", +- "eval_env = gym.make(args.env_id)\n", +- "\n", +- "package_to_hub(repo_id = args.repo_id,\n", +- " model = agent, # The model we want to save\n", +- " hyperparameters = args,\n", +- " eval_env = gym.make(args.env_id),\n", +- " logs= f\"runs/{run_name}\",\n", +- " )" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "muCCzed4o5TC" +- }, +- "source": [ +- "- Here's what look the ppo.py final file" +- ] +- }, +- { +- "cell_type": "code", +- "execution_count": null, +- "metadata": { +- "id": "LviRdtXgo7kF" +- }, +- "outputs": [], +- "source": [ +- "# docs and experiment results can be found at https://docs.cleanrl.dev/rl-algorithms/ppo/#ppopy\n", +- "\n", +- "import argparse\n", +- "import os\n", +- "import random\n", +- "import time\n", +- "from distutils.util import strtobool\n", +- "\n", +- "import gym\n", +- "import numpy as np\n", +- "import torch\n", +- "import torch.nn as nn\n", +- "import torch.optim as optim\n", +- "from torch.distributions.categorical import Categorical\n", +- "from torch.utils.tensorboard import SummaryWriter\n", +- "\n", +- "from huggingface_hub import HfApi, upload_folder\n", +- "from huggingface_hub.repocard import metadata_eval_result, metadata_save\n", +- "\n", +- "from pathlib import Path\n", +- "import datetime\n", +- "import tempfile\n", +- "import json\n", +- "import shutil\n", +- "import imageio\n", +- "\n", +- "from wasabi import Printer\n", +- "msg = Printer()\n", +- "\n", +- "def parse_args():\n", +- " # fmt: off\n", +- " parser = argparse.ArgumentParser()\n", +- " parser.add_argument(\"--exp-name\", type=str, default=os.path.basename(__file__).rstrip(\".py\"),\n", +- " help=\"the name of this experiment\")\n", +- " parser.add_argument(\"--seed\", type=int, default=1,\n", +- " help=\"seed of the experiment\")\n", +- " parser.add_argument(\"--torch-deterministic\", type=lambda x: bool(strtobool(x)), default=True, nargs=\"?\", const=True,\n", +- " help=\"if toggled, `torch.backends.cudnn.deterministic=False`\")\n", +- " parser.add_argument(\"--cuda\", type=lambda x: bool(strtobool(x)), default=True, nargs=\"?\", const=True,\n", +- " help=\"if toggled, cuda will be enabled by default\")\n", +- " parser.add_argument(\"--track\", type=lambda x: bool(strtobool(x)), default=False, nargs=\"?\", const=True,\n", +- " help=\"if toggled, this experiment will be tracked with Weights and Biases\")\n", +- " parser.add_argument(\"--wandb-project-name\", type=str, default=\"cleanRL\",\n", +- " help=\"the wandb's project name\")\n", +- " parser.add_argument(\"--wandb-entity\", type=str, default=None,\n", +- " help=\"the entity (team) of wandb's project\")\n", +- " parser.add_argument(\"--capture-video\", type=lambda x: bool(strtobool(x)), default=False, nargs=\"?\", const=True,\n", +- " help=\"weather to capture videos of the agent performances (check out `videos` folder)\")\n", +- "\n", +- " # Algorithm specific arguments\n", +- " parser.add_argument(\"--env-id\", type=str, default=\"CartPole-v1\",\n", +- " help=\"the id of the environment\")\n", +- " parser.add_argument(\"--total-timesteps\", type=int, default=50000,\n", +- " help=\"total timesteps of the experiments\")\n", +- " parser.add_argument(\"--learning-rate\", type=float, default=2.5e-4,\n", +- " help=\"the learning rate of the optimizer\")\n", +- " parser.add_argument(\"--num-envs\", type=int, default=4,\n", +- " help=\"the number of parallel game environments\")\n", +- " parser.add_argument(\"--num-steps\", type=int, default=128,\n", +- " help=\"the number of steps to run in each environment per policy rollout\")\n", +- " parser.add_argument(\"--anneal-lr\", type=lambda x: bool(strtobool(x)), default=True, nargs=\"?\", const=True,\n", +- " help=\"Toggle learning rate annealing for policy and value networks\")\n", +- " parser.add_argument(\"--gae\", type=lambda x: bool(strtobool(x)), default=True, nargs=\"?\", const=True,\n", +- " help=\"Use GAE for advantage computation\")\n", +- " parser.add_argument(\"--gamma\", type=float, default=0.99,\n", +- " help=\"the discount factor gamma\")\n", +- " parser.add_argument(\"--gae-lambda\", type=float, default=0.95,\n", +- " help=\"the lambda for the general advantage estimation\")\n", +- " parser.add_argument(\"--num-minibatches\", type=int, default=4,\n", +- " help=\"the number of mini-batches\")\n", +- " parser.add_argument(\"--update-epochs\", type=int, default=4,\n", +- " help=\"the K epochs to update the policy\")\n", +- " parser.add_argument(\"--norm-adv\", type=lambda x: bool(strtobool(x)), default=True, nargs=\"?\", const=True,\n", +- " help=\"Toggles advantages normalization\")\n", +- " parser.add_argument(\"--clip-coef\", type=float, default=0.2,\n", +- " help=\"the surrogate clipping coefficient\")\n", +- " parser.add_argument(\"--clip-vloss\", type=lambda x: bool(strtobool(x)), default=True, nargs=\"?\", const=True,\n", +- " help=\"Toggles whether or not to use a clipped loss for the value function, as per the paper.\")\n", +- " parser.add_argument(\"--ent-coef\", type=float, default=0.01,\n", +- " help=\"coefficient of the entropy\")\n", +- " parser.add_argument(\"--vf-coef\", type=float, default=0.5,\n", +- " help=\"coefficient of the value function\")\n", +- " parser.add_argument(\"--max-grad-norm\", type=float, default=0.5,\n", +- " help=\"the maximum norm for the gradient clipping\")\n", +- " parser.add_argument(\"--target-kl\", type=float, default=None,\n", +- " help=\"the target KL divergence threshold\")\n", +- "\n", +- " # Adding HuggingFace argument\n", +- " parser.add_argument(\"--repo-id\", type=str, default=\"ThomasSimonini/ppo-CartPole-v1\", help=\"id of the model repository from the Hugging Face Hub {username/repo_name}\")\n", +- "\n", +- " args = parser.parse_args()\n", +- " args.batch_size = int(args.num_envs * args.num_steps)\n", +- " args.minibatch_size = int(args.batch_size // args.num_minibatches)\n", +- " # fmt: on\n", +- " return args\n", +- "\n", +- "def package_to_hub(repo_id,\n", +- " model,\n", +- " hyperparameters,\n", +- " eval_env,\n", +- " video_fps=30,\n", +- " commit_message=\"Push agent to the Hub\",\n", +- " token= None,\n", +- " logs=None\n", +- " ):\n", +- " \"\"\"\n", +- " Evaluate, Generate a video and Upload a model to Hugging Face Hub.\n", +- " This method does the complete pipeline:\n", +- " - It evaluates the model\n", +- " - It generates the model card\n", +- " - It generates a replay video of the agent\n", +- " - It pushes everything to the hub\n", +- " :param repo_id: id of the model repository from the Hugging Face Hub\n", +- " :param model: trained model\n", +- " :param eval_env: environment used to evaluate the agent\n", +- " :param fps: number of fps for rendering the video\n", +- " :param commit_message: commit message\n", +- " :param logs: directory on local machine of tensorboard logs you'd like to upload\n", +- " \"\"\"\n", +- " msg.info(\n", +- " \"This function will save, evaluate, generate a video of your agent, \"\n", +- " \"create a model card and push everything to the hub. \"\n", +- " \"It might take up to 1min. \\n \"\n", +- " \"This is a work in progress: if you encounter a bug, please open an issue.\"\n", +- " )\n", +- " # Step 1: Clone or create the repo\n", +- " repo_url = HfApi().create_repo(\n", +- " repo_id=repo_id,\n", +- " token=token,\n", +- " private=False,\n", +- " exist_ok=True,\n", +- " )\n", +- "\n", +- " with tempfile.TemporaryDirectory() as tmpdirname:\n", +- " tmpdirname = Path(tmpdirname)\n", +- "\n", +- " # Step 2: Save the model\n", +- " torch.save(model.state_dict(), tmpdirname / \"model.pt\")\n", +- "\n", +- " # Step 3: Evaluate the model and build JSON\n", +- " mean_reward, std_reward = _evaluate_agent(eval_env,\n", +- " 10,\n", +- " model)\n", +- "\n", +- " # First get datetime\n", +- " eval_datetime = datetime.datetime.now()\n", +- " eval_form_datetime = eval_datetime.isoformat()\n", +- "\n", +- " evaluate_data = {\n", +- " \"env_id\": hyperparameters.env_id,\n", +- " \"mean_reward\": mean_reward,\n", +- " \"std_reward\": std_reward,\n", +- " \"n_evaluation_episodes\": 10,\n", +- " \"eval_datetime\": eval_form_datetime,\n", +- " }\n", +- "\n", +- " # Write a JSON file\n", +- " with open(tmpdirname / \"results.json\", \"w\") as outfile:\n", +- " json.dump(evaluate_data, outfile)\n", +- "\n", +- " # Step 4: Generate a video\n", +- " video_path = tmpdirname / \"replay.mp4\"\n", +- " record_video(eval_env, model, video_path, video_fps)\n", +- "\n", +- " # Step 5: Generate the model card\n", +- " generated_model_card, metadata = _generate_model_card(\"PPO\", hyperparameters.env_id, mean_reward, std_reward, hyperparameters)\n", +- " _save_model_card(tmpdirname, generated_model_card, metadata)\n", +- "\n", +- " # Step 6: Add logs if needed\n", +- " if logs:\n", +- " _add_logdir(tmpdirname, Path(logs))\n", +- "\n", +- " msg.info(f\"Pushing repo {repo_id} to the Hugging Face Hub\")\n", +- "\n", +- " repo_url = upload_folder(\n", +- " repo_id=repo_id,\n", +- " folder_path=tmpdirname,\n", +- " path_in_repo=\"\",\n", +- " commit_message=commit_message,\n", +- " token=token,\n", +- " )\n", +- "\n", +- " msg.info(f\"Your model is pushed to the Hub. You can view your model here: {repo_url}\")\n", +- " return repo_url\n", +- "\n", +- "def _evaluate_agent(env, n_eval_episodes, policy):\n", +- " \"\"\"\n", +- " Evaluate the agent for ``n_eval_episodes`` episodes and returns average reward and std of reward.\n", +- " :param env: The evaluation environment\n", +- " :param n_eval_episodes: Number of episode to evaluate the agent\n", +- " :param policy: The agent\n", +- " \"\"\"\n", +- " episode_rewards = []\n", +- " for episode in range(n_eval_episodes):\n", +- " state = env.reset()\n", +- " step = 0\n", +- " done = False\n", +- " total_rewards_ep = 0\n", +- "\n", +- " while done is False:\n", +- " state = torch.Tensor(state).to(device)\n", +- " action, _, _, _ = policy.get_action_and_value(state)\n", +- " new_state, reward, done, info = env.step(action.cpu().numpy())\n", +- " total_rewards_ep += reward\n", +- " if done:\n", +- " break\n", +- " state = new_state\n", +- " episode_rewards.append(total_rewards_ep)\n", +- " mean_reward = np.mean(episode_rewards)\n", +- " std_reward = np.std(episode_rewards)\n", +- "\n", +- " return mean_reward, std_reward\n", +- "\n", +- "\n", +- "def record_video(env, policy, out_directory, fps=30):\n", +- " images = []\n", +- " done = False\n", +- " state = env.reset()\n", +- " img = env.render(mode='rgb_array')\n", +- " images.append(img)\n", +- " while not done:\n", +- " state = torch.Tensor(state).to(device)\n", +- " # Take the action (index) that have the maximum expected future reward given that state\n", +- " action, _, _, _ = policy.get_action_and_value(state)\n", +- " state, reward, done, info = env.step(action.cpu().numpy()) # We directly put next_state = state for recording logic\n", +- " img = env.render(mode='rgb_array')\n", +- " images.append(img)\n", +- " imageio.mimsave(out_directory, [np.array(img) for i, img in enumerate(images)], fps=fps)\n", +- "\n", +- "\n", +- "def _generate_model_card(model_name, env_id, mean_reward, std_reward, hyperparameters):\n", +- " \"\"\"\n", +- " Generate the model card for the Hub\n", +- " :param model_name: name of the model\n", +- " :env_id: name of the environment\n", +- " :mean_reward: mean reward of the agent\n", +- " :std_reward: standard deviation of the mean reward of the agent\n", +- " :hyperparameters: training arguments\n", +- " \"\"\"\n", +- " # Step 1: Select the tags\n", +- " metadata = generate_metadata(model_name, env_id, mean_reward, std_reward)\n", +- "\n", +- " # Transform the hyperparams namespace to string\n", +- " converted_dict = vars(hyperparameters)\n", +- " converted_str = str(converted_dict)\n", +- " converted_str = converted_str.split(\", \")\n", +- " converted_str = '\\n'.join(converted_str)\n", +- "\n", +- " # Step 2: Generate the model card\n", +- " model_card = f\"\"\"\n", +- " # PPO Agent Playing {env_id}\n", +- "\n", +- " This is a trained model of a PPO agent playing {env_id}.\n", +- "\n", +- " # Hyperparameters\n", +- " ```python\n", +- " {converted_str}\n", +- " ```\n", +- " \"\"\"\n", +- " return model_card, metadata\n", +- "\n", +- "def generate_metadata(model_name, env_id, mean_reward, std_reward):\n", +- " \"\"\"\n", +- " Define the tags for the model card\n", +- " :param model_name: name of the model\n", +- " :param env_id: name of the environment\n", +- " :mean_reward: mean reward of the agent\n", +- " :std_reward: standard deviation of the mean reward of the agent\n", +- " \"\"\"\n", +- " metadata = {}\n", +- " metadata[\"tags\"] = [\n", +- " env_id,\n", +- " \"ppo\",\n", +- " \"deep-reinforcement-learning\",\n", +- " \"reinforcement-learning\",\n", +- " \"custom-implementation\",\n", +- " \"deep-rl-course\"\n", +- " ]\n", +- "\n", +- " # Add metrics\n", +- " eval = metadata_eval_result(\n", +- " model_pretty_name=model_name,\n", +- " task_pretty_name=\"reinforcement-learning\",\n", +- " task_id=\"reinforcement-learning\",\n", +- " metrics_pretty_name=\"mean_reward\",\n", +- " metrics_id=\"mean_reward\",\n", +- " metrics_value=f\"{mean_reward:.2f} +/- {std_reward:.2f}\",\n", +- " dataset_pretty_name=env_id,\n", +- " dataset_id=env_id,\n", +- " )\n", +- "\n", +- " # Merges both dictionaries\n", +- " metadata = {**metadata, **eval}\n", +- "\n", +- " return metadata\n", +- "\n", +- "def _save_model_card(local_path, generated_model_card, metadata):\n", +- " \"\"\"Saves a model card for the repository.\n", +- " :param local_path: repository directory\n", +- " :param generated_model_card: model card generated by _generate_model_card()\n", +- " :param metadata: metadata\n", +- " \"\"\"\n", +- " readme_path = local_path / \"README.md\"\n", +- " readme = \"\"\n", +- " if readme_path.exists():\n", +- " with readme_path.open(\"r\", encoding=\"utf8\") as f:\n", +- " readme = f.read()\n", +- " else:\n", +- " readme = generated_model_card\n", +- "\n", +- " with readme_path.open(\"w\", encoding=\"utf-8\") as f:\n", +- " f.write(readme)\n", +- "\n", +- " # Save our metrics to Readme metadata\n", +- " metadata_save(readme_path, metadata)\n", +- "\n", +- "def _add_logdir(local_path: Path, logdir: Path):\n", +- " \"\"\"Adds a logdir to the repository.\n", +- " :param local_path: repository directory\n", +- " :param logdir: logdir directory\n", +- " \"\"\"\n", +- " if logdir.exists() and logdir.is_dir():\n", +- " # Add the logdir to the repository under new dir called logs\n", +- " repo_logdir = local_path / \"logs\"\n", +- "\n", +- " # Delete current logs if they exist\n", +- " if repo_logdir.exists():\n", +- " shutil.rmtree(repo_logdir)\n", +- "\n", +- " # Copy logdir into repo logdir\n", +- " shutil.copytree(logdir, repo_logdir)\n", +- "\n", +- "def make_env(env_id, seed, idx, capture_video, run_name):\n", +- " def thunk():\n", +- " env = gym.make(env_id)\n", +- " env = gym.wrappers.RecordEpisodeStatistics(env)\n", +- " if capture_video:\n", +- " if idx == 0:\n", +- " env = gym.wrappers.RecordVideo(env, f\"videos/{run_name}\")\n", +- " env.seed(seed)\n", +- " env.action_space.seed(seed)\n", +- " env.observation_space.seed(seed)\n", +- " return env\n", +- "\n", +- " return thunk\n", +- "\n", +- "\n", +- "def layer_init(layer, std=np.sqrt(2), bias_const=0.0):\n", +- " torch.nn.init.orthogonal_(layer.weight, std)\n", +- " torch.nn.init.constant_(layer.bias, bias_const)\n", +- " return layer\n", +- "\n", +- "\n", +- "class Agent(nn.Module):\n", +- " def __init__(self, envs):\n", +- " super().__init__()\n", +- " self.critic = nn.Sequential(\n", +- " layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 64)),\n", +- " nn.Tanh(),\n", +- " layer_init(nn.Linear(64, 64)),\n", +- " nn.Tanh(),\n", +- " layer_init(nn.Linear(64, 1), std=1.0),\n", +- " )\n", +- " self.actor = nn.Sequential(\n", +- " layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 64)),\n", +- " nn.Tanh(),\n", +- " layer_init(nn.Linear(64, 64)),\n", +- " nn.Tanh(),\n", +- " layer_init(nn.Linear(64, envs.single_action_space.n), std=0.01),\n", +- " )\n", +- "\n", +- " def get_value(self, x):\n", +- " return self.critic(x)\n", +- "\n", +- " def get_action_and_value(self, x, action=None):\n", +- " logits = self.actor(x)\n", +- " probs = Categorical(logits=logits)\n", +- " if action is None:\n", +- " action = probs.sample()\n", +- " return action, probs.log_prob(action), probs.entropy(), self.critic(x)\n", +- "\n", +- "\n", +- "if __name__ == \"__main__\":\n", +- " args = parse_args()\n", +- " run_name = f\"{args.env_id}__{args.exp_name}__{args.seed}__{int(time.time())}\"\n", +- " if args.track:\n", +- " import wandb\n", +- "\n", +- " wandb.init(\n", +- " project=args.wandb_project_name,\n", +- " entity=args.wandb_entity,\n", +- " sync_tensorboard=True,\n", +- " config=vars(args),\n", +- " name=run_name,\n", +- " monitor_gym=True,\n", +- " save_code=True,\n", +- " )\n", +- " writer = SummaryWriter(f\"runs/{run_name}\")\n", +- " writer.add_text(\n", +- " \"hyperparameters\",\n", +- " \"|param|value|\\n|-|-|\\n%s\" % (\"\\n\".join([f\"|{key}|{value}|\" for key, value in vars(args).items()])),\n", +- " )\n", +- "\n", +- " # TRY NOT TO MODIFY: seeding\n", +- " random.seed(args.seed)\n", +- " np.random.seed(args.seed)\n", +- " torch.manual_seed(args.seed)\n", +- " torch.backends.cudnn.deterministic = args.torch_deterministic\n", +- "\n", +- " device = torch.device(\"cuda\" if torch.cuda.is_available() and args.cuda else \"cpu\")\n", +- "\n", +- " # env setup\n", +- " envs = gym.vector.SyncVectorEnv(\n", +- " [make_env(args.env_id, args.seed + i, i, args.capture_video, run_name) for i in range(args.num_envs)]\n", +- " )\n", +- " assert isinstance(envs.single_action_space, gym.spaces.Discrete), \"only discrete action space is supported\"\n", +- "\n", +- " agent = Agent(envs).to(device)\n", +- " optimizer = optim.Adam(agent.parameters(), lr=args.learning_rate, eps=1e-5)\n", +- "\n", +- " # ALGO Logic: Storage setup\n", +- " obs = torch.zeros((args.num_steps, args.num_envs) + envs.single_observation_space.shape).to(device)\n", +- " actions = torch.zeros((args.num_steps, args.num_envs) + envs.single_action_space.shape).to(device)\n", +- " logprobs = torch.zeros((args.num_steps, args.num_envs)).to(device)\n", +- " rewards = torch.zeros((args.num_steps, args.num_envs)).to(device)\n", +- " dones = torch.zeros((args.num_steps, args.num_envs)).to(device)\n", +- " values = torch.zeros((args.num_steps, args.num_envs)).to(device)\n", +- "\n", +- " # TRY NOT TO MODIFY: start the game\n", +- " global_step = 0\n", +- " start_time = time.time()\n", +- " next_obs = torch.Tensor(envs.reset()).to(device)\n", +- " next_done = torch.zeros(args.num_envs).to(device)\n", +- " num_updates = args.total_timesteps // args.batch_size\n", +- "\n", +- " for update in range(1, num_updates + 1):\n", +- " # Annealing the rate if instructed to do so.\n", +- " if args.anneal_lr:\n", +- " frac = 1.0 - (update - 1.0) / num_updates\n", +- " lrnow = frac * args.learning_rate\n", +- " optimizer.param_groups[0][\"lr\"] = lrnow\n", +- "\n", +- " for step in range(0, args.num_steps):\n", +- " global_step += 1 * args.num_envs\n", +- " obs[step] = next_obs\n", +- " dones[step] = next_done\n", +- "\n", +- " # ALGO LOGIC: action logic\n", +- " with torch.no_grad():\n", +- " action, logprob, _, value = agent.get_action_and_value(next_obs)\n", +- " values[step] = value.flatten()\n", +- " actions[step] = action\n", +- " logprobs[step] = logprob\n", +- "\n", +- " # TRY NOT TO MODIFY: execute the game and log data.\n", +- " next_obs, reward, done, info = envs.step(action.cpu().numpy())\n", +- " rewards[step] = torch.tensor(reward).to(device).view(-1)\n", +- " next_obs, next_done = torch.Tensor(next_obs).to(device), torch.Tensor(done).to(device)\n", +- "\n", +- " for item in info:\n", +- " if \"episode\" in item.keys():\n", +- " print(f\"global_step={global_step}, episodic_return={item['episode']['r']}\")\n", +- " writer.add_scalar(\"charts/episodic_return\", item[\"episode\"][\"r\"], global_step)\n", +- " writer.add_scalar(\"charts/episodic_length\", item[\"episode\"][\"l\"], global_step)\n", +- " break\n", +- "\n", +- " # bootstrap value if not done\n", +- " with torch.no_grad():\n", +- " next_value = agent.get_value(next_obs).reshape(1, -1)\n", +- " if args.gae:\n", +- " advantages = torch.zeros_like(rewards).to(device)\n", +- " lastgaelam = 0\n", +- " for t in reversed(range(args.num_steps)):\n", +- " if t == args.num_steps - 1:\n", +- " nextnonterminal = 1.0 - next_done\n", +- " nextvalues = next_value\n", +- " else:\n", +- " nextnonterminal = 1.0 - dones[t + 1]\n", +- " nextvalues = values[t + 1]\n", +- " delta = rewards[t] + args.gamma * nextvalues * nextnonterminal - values[t]\n", +- " advantages[t] = lastgaelam = delta + args.gamma * args.gae_lambda * nextnonterminal * lastgaelam\n", +- " returns = advantages + values\n", +- " else:\n", +- " returns = torch.zeros_like(rewards).to(device)\n", +- " for t in reversed(range(args.num_steps)):\n", +- " if t == args.num_steps - 1:\n", +- " nextnonterminal = 1.0 - next_done\n", +- " next_return = next_value\n", +- " else:\n", +- " nextnonterminal = 1.0 - dones[t + 1]\n", +- " next_return = returns[t + 1]\n", +- " returns[t] = rewards[t] + args.gamma * nextnonterminal * next_return\n", +- " advantages = returns - values\n", +- "\n", +- " # flatten the batch\n", +- " b_obs = obs.reshape((-1,) + envs.single_observation_space.shape)\n", +- " b_logprobs = logprobs.reshape(-1)\n", +- " b_actions = actions.reshape((-1,) + envs.single_action_space.shape)\n", +- " b_advantages = advantages.reshape(-1)\n", +- " b_returns = returns.reshape(-1)\n", +- " b_values = values.reshape(-1)\n", +- "\n", +- " # Optimizing the policy and value network\n", +- " b_inds = np.arange(args.batch_size)\n", +- " clipfracs = []\n", +- " for epoch in range(args.update_epochs):\n", +- " np.random.shuffle(b_inds)\n", +- " for start in range(0, args.batch_size, args.minibatch_size):\n", +- " end = start + args.minibatch_size\n", +- " mb_inds = b_inds[start:end]\n", +- "\n", +- " _, newlogprob, entropy, newvalue = agent.get_action_and_value(b_obs[mb_inds], b_actions.long()[mb_inds])\n", +- " logratio = newlogprob - b_logprobs[mb_inds]\n", +- " ratio = logratio.exp()\n", +- "\n", +- " with torch.no_grad():\n", +- " # calculate approx_kl http://joschu.net/blog/kl-approx.html\n", +- " old_approx_kl = (-logratio).mean()\n", +- " approx_kl = ((ratio - 1) - logratio).mean()\n", +- " clipfracs += [((ratio - 1.0).abs() > args.clip_coef).float().mean().item()]\n", +- "\n", +- " mb_advantages = b_advantages[mb_inds]\n", +- " if args.norm_adv:\n", +- " mb_advantages = (mb_advantages - mb_advantages.mean()) / (mb_advantages.std() + 1e-8)\n", +- "\n", +- " # Policy loss\n", +- " pg_loss1 = -mb_advantages * ratio\n", +- " pg_loss2 = -mb_advantages * torch.clamp(ratio, 1 - args.clip_coef, 1 + args.clip_coef)\n", +- " pg_loss = torch.max(pg_loss1, pg_loss2).mean()\n", +- "\n", +- " # Value loss\n", +- " newvalue = newvalue.view(-1)\n", +- " if args.clip_vloss:\n", +- " v_loss_unclipped = (newvalue - b_returns[mb_inds]) ** 2\n", +- " v_clipped = b_values[mb_inds] + torch.clamp(\n", +- " newvalue - b_values[mb_inds],\n", +- " -args.clip_coef,\n", +- " args.clip_coef,\n", +- " )\n", +- " v_loss_clipped = (v_clipped - b_returns[mb_inds]) ** 2\n", +- " v_loss_max = torch.max(v_loss_unclipped, v_loss_clipped)\n", +- " v_loss = 0.5 * v_loss_max.mean()\n", +- " else:\n", +- " v_loss = 0.5 * ((newvalue - b_returns[mb_inds]) ** 2).mean()\n", +- "\n", +- " entropy_loss = entropy.mean()\n", +- " loss = pg_loss - args.ent_coef * entropy_loss + v_loss * args.vf_coef\n", +- "\n", +- " optimizer.zero_grad()\n", +- " loss.backward()\n", +- " nn.utils.clip_grad_norm_(agent.parameters(), args.max_grad_norm)\n", +- " optimizer.step()\n", +- "\n", +- " if args.target_kl is not None:\n", +- " if approx_kl > args.target_kl:\n", +- " break\n", +- "\n", +- " y_pred, y_true = b_values.cpu().numpy(), b_returns.cpu().numpy()\n", +- " var_y = np.var(y_true)\n", +- " explained_var = np.nan if var_y == 0 else 1 - np.var(y_true - y_pred) / var_y\n", +- "\n", +- " # TRY NOT TO MODIFY: record rewards for plotting purposes\n", +- " writer.add_scalar(\"charts/learning_rate\", optimizer.param_groups[0][\"lr\"], global_step)\n", +- " writer.add_scalar(\"losses/value_loss\", v_loss.item(), global_step)\n", +- " writer.add_scalar(\"losses/policy_loss\", pg_loss.item(), global_step)\n", +- " writer.add_scalar(\"losses/entropy\", entropy_loss.item(), global_step)\n", +- " writer.add_scalar(\"losses/old_approx_kl\", old_approx_kl.item(), global_step)\n", +- " writer.add_scalar(\"losses/approx_kl\", approx_kl.item(), global_step)\n", +- " writer.add_scalar(\"losses/clipfrac\", np.mean(clipfracs), global_step)\n", +- " writer.add_scalar(\"losses/explained_variance\", explained_var, global_step)\n", +- " print(\"SPS:\", int(global_step / (time.time() - start_time)))\n", +- " writer.add_scalar(\"charts/SPS\", int(global_step / (time.time() - start_time)), global_step)\n", +- "\n", +- " envs.close()\n", +- " writer.close()\n", +- "\n", +- " # Create the evaluation environment\n", +- " eval_env = gym.make(args.env_id)\n", +- "\n", +- " package_to_hub(repo_id = args.repo_id,\n", +- " model = agent, # The model we want to save\n", +- " hyperparameters = args,\n", +- " eval_env = gym.make(args.env_id),\n", +- " logs= f\"runs/{run_name}\",\n", +- " )\n", +- "" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "JquRrWytA6eo" +- }, +- "source": [ +- "To be able to share your model with the community there are three more steps to follow:\n", +- "\n", +- "1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join\n", +- "\n", +- "2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website.\n", +- "- Create a new token (https://huggingface.co/settings/tokens) **with write role**\n", +- "\n", +- "\"Create\n", +- "\n", +- "- Copy the token\n", +- "- Run the cell below and paste the token" +- ] +- }, +- { +- "cell_type": "code", +- "execution_count": null, +- "metadata": { +- "id": "GZiFBBlzxzxY" +- }, +- "outputs": [], +- "source": [ +- "from huggingface_hub import notebook_login\n", +- "notebook_login()\n", +- "!git config --global credential.helper store" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "_tsf2uv0g_4p" +- }, +- "source": [ +- "If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "jRqkGvk7pFQ6" +- }, +- "source": [ +- "## Let's start the training 🔥\n", +- "- ⚠️ ⚠️ ⚠️ Don't use **the same repo id with the one you used for the Unit 1**\n", +- "- Now that you've coded from scratch PPO and added the Hugging Face Integration, we're ready to start the training 🔥" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "0tmEArP8ug2l" +- }, +- "source": [ +- "- First, you need to copy all your code to a file you create called `ppo.py`" +- ] +- }, +- { +- "cell_type": "markdown", +- "source": [ +- "\"PPO\"/" +- ], +- "metadata": { +- "id": "Sq0My0LOjPYR" +- } +- }, +- { +- "cell_type": "markdown", +- "source": [ +- "\"PPO\"/" +- ], +- "metadata": { +- "id": "A8C-Q5ZyjUe3" +- } +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "VrS80GmMu_j5" +- }, +- "source": [ +- "- Now we just need to run this python script using `python .py` with the additional parameters we defined with `argparse`\n", +- "\n", +- "- You should modify more hyperparameters otherwise the training will not be super stable." +- ] +- }, +- { +- "cell_type": "code", +- "source": [ +- "!python ppo.py --env-id=\"LunarLander-v2\" --repo-id=\"YOUR_REPO_ID\" --total-timesteps=50000" +- ], +- "metadata": { +- "id": "KXLih6mKseBs" +- }, +- "execution_count": null, +- "outputs": [] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "eVsVJ5AdqLE7" +- }, +- "source": [ +- "## Some additional challenges 🏆\n", +- "The best way to learn **is to try things by your own**! Why not trying another environment?\n" +- ] +- }, +- { +- "cell_type": "markdown", +- "metadata": { +- "id": "nYdl758GqLXT" +- }, +- "source": [ +- "See you on Unit 8, part 2 where we going to train agents to play Doom 🔥\n", +- "## Keep learning, stay awesome 🤗" +- ] +- } +- ], +- "metadata": { +- "colab": { +- "private_outputs": true, +- "provenance": [], +- "include_colab_link": true +- }, +- "gpuClass": "standard", +- "kernelspec": { +- "display_name": "Python 3", +- "name": "python3" +- }, +- "language_info": { +- "name": "python" +- }, +- "accelerator": "GPU" +- }, +- "nbformat": 4, +- "nbformat_minor": 0 +-} +\ No newline at end of file +diff --git a/notebooks/unit8/requirement.txt b/notebooks/unit8/requirement.txt +deleted file mode 100644 +index 920e673..0000000 +--- a/notebooks/unit8/requirement.txt ++++ /dev/null +@@ -1,119 +0,0 @@ +-absl-py==2.3.1 +-ale-py==0.11.2 +-annotated-types==0.7.0 +-asttokens==3.0.0 +-attrs==25.3.0 +-box2d-py==2.3.5 +-cattrs==1.5.0 +-certifi==2025.8.3 +-charset-normalizer==3.4.3 +-click==8.2.1 +-cloudpickle==3.1.1 +-comm==0.2.3 +-debugpy==1.8.16 +-decorator==5.2.1 +-exceptiongroup==1.3.0 +-executing==2.2.0 +-Farama-Notifications==0.0.4 +-filelock==3.18.0 +-fsspec==2025.7.0 +-future==1.0.0 +-gitdb==4.0.12 +-GitPython==3.1.45 +-grpcio==1.48.2 +-gym==0.26.2 +-gym-notices==0.1.0 +-gymnasium==1.2.0 +-h5py==3.14.0 +-hf-xet==1.1.7 +-huggingface-hub==0.34.4 +-idna==3.10 +-imageio==2.37.0 +-imageio-ffmpeg==0.6.0 +-ipykernel==6.30.1 +-ipython==8.37.0 +-ipywidgets==8.1.7 +-jedi==0.19.2 +-Jinja2==3.1.6 +-jupyter_client==8.6.3 +-jupyter_core==5.8.1 +-jupyterlab_widgets==3.0.15 +-Markdown==3.8.2 +-MarkupSafe==3.0.2 +-matplotlib-inline==0.1.7 +--e git+https://github.com/Unity-Technologies/ml-agents@7db884323f8619b578fc1c8327d57fa087df27e7#egg=mlagents&subdirectory=ml-agents +--e git+https://github.com/Unity-Technologies/ml-agents@7db884323f8619b578fc1c8327d57fa087df27e7#egg=mlagents_envs&subdirectory=ml-agents-envs +-mpmath==1.3.0 +-nest-asyncio==1.6.0 +-networkx==3.4.2 +-numpy==1.23.5 +-nvidia-cublas-cu11==11.11.3.6 +-nvidia-cublas-cu12==12.8.4.1 +-nvidia-cuda-cupti-cu11==11.8.87 +-nvidia-cuda-cupti-cu12==12.8.90 +-nvidia-cuda-nvrtc-cu11==11.8.89 +-nvidia-cuda-nvrtc-cu12==12.8.93 +-nvidia-cuda-runtime-cu11==11.8.89 +-nvidia-cuda-runtime-cu12==12.8.90 +-nvidia-cudnn-cu11==9.1.0.70 +-nvidia-cudnn-cu12==9.10.2.21 +-nvidia-cufft-cu11==10.9.0.58 +-nvidia-cufft-cu12==11.3.3.83 +-nvidia-cufile-cu12==1.13.1.3 +-nvidia-curand-cu11==10.3.0.86 +-nvidia-curand-cu12==10.3.9.90 +-nvidia-cusolver-cu11==11.4.1.48 +-nvidia-cusolver-cu12==11.7.3.90 +-nvidia-cusparse-cu11==11.7.5.86 +-nvidia-cusparse-cu12==12.5.8.93 +-nvidia-cusparselt-cu12==0.7.1 +-nvidia-nccl-cu11==2.21.5 +-nvidia-nccl-cu12==2.27.3 +-nvidia-nvjitlink-cu12==12.8.93 +-nvidia-nvtx-cu11==11.8.86 +-nvidia-nvtx-cu12==12.8.90 +-onnx==1.15.0 +-packaging==25.0 +-parso==0.8.4 +-PettingZoo==1.15.0 +-pexpect==4.9.0 +-pillow==11.3.0 +-platformdirs==4.3.8 +-prompt_toolkit==3.0.51 +-protobuf==3.20.3 +-psutil==7.0.0 +-ptyprocess==0.7.0 +-pure_eval==0.2.3 +-pydantic==2.11.7 +-pydantic_core==2.33.2 +-pygame==2.6.1 +-pyglet==1.5.0 +-Pygments==2.19.2 +-python-dateutil==2.9.0.post0 +-PyVirtualDisplay==3.0 +-PyYAML==6.0.2 +-pyzmq==27.0.1 +-requests==2.32.4 +-sentry-sdk==2.35.0 +-six==1.17.0 +-smmap==5.0.2 +-stack-data==0.6.3 +-swig==4.3.1 +-sympy==1.14.0 +-tensorboard==2.20.0 +-tensorboard-data-server==0.7.2 +-torch==2.7.1+cu118 +-torchaudio==2.7.1+cu118 +-torchvision==0.22.1+cu118 +-tornado==6.5.2 +-tqdm==4.67.1 +-traitlets==5.14.3 +-triton==3.3.1 +-typing-inspection==0.4.1 +-typing_extensions==4.14.1 +-urllib3==2.5.0 +-wandb==0.21.1 +-wasabi==1.1.3 +-wcwidth==0.2.13 +-Werkzeug==3.1.3 +-widgetsnbextension==4.0.14