diff --git a/notebooks/unit7/ml-agents b/notebooks/unit7/ml-agents --- a/notebooks/unit7/ml-agents +++ b/notebooks/unit7/ml-agents @@ -1 +1 @@ -Subproject commit 7db884323f8619b578fc1c8327d57fa087df27e7 +Subproject commit 7db884323f8619b578fc1c8327d57fa087df27e7-dirty diff --git a/notebooks/unit8/doom.ipynb b/notebooks/unit8/doom.ipynb deleted file mode 100644 index 86e29df..0000000 --- a/notebooks/unit8/doom.ipynb +++ /dev/null @@ -1,690 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "id": "view-in-github", - "colab_type": "text" - }, - "source": [ - "\"Open" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "OVx1gdg9wt9t" - }, - "source": [ - "# Unit 8 Part 2: Advanced Deep Reinforcement Learning. Using Sample Factory to play Doom from pixels\n", - "\n", - "\"Thumbnail\"/\n", - "\n", - "In this notebook, we will learn how to train a Deep Neural Network to collect objects in a 3D environment based on the game of Doom, a video of the resulting policy is shown below. We train this policy using [Sample Factory](https://www.samplefactory.dev/), an asynchronous implementation of the PPO algorithm.\n", - "\n", - "Please note the following points:\n", - "\n", - "* [Sample Factory](https://www.samplefactory.dev/) is an advanced RL framework and **only functions on Linux and Mac** (not Windows).\n", - "\n", - "* The framework performs best on a **GPU machine with many CPU cores**, where it can achieve speeds of 100k interactions per second. The resources available on a standard Colab notebook **limit the performance of this library**. So the speed in this setting **does not reflect the real-world performance**.\n", - "* Benchmarks for Sample Factory are available in a number of settings, check out the [examples](https://github.com/alex-petrenko/sample-factory/tree/master/sf_examples) if you want to find out more.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "I6_67HfI1CKg" - }, - "outputs": [], - "source": [ - "from IPython.display import HTML\n", - "\n", - "HTML(''''''\n", - ")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "DgHRAsYEXdyw" - }, - "source": [ - "To validate this hands-on for the [certification process](https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process), you need to push one model:\n", - "\n", - "- `doom_health_gathering_supreme` get a result of >= 5.\n", - "\n", - "To find your result, go to the [leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) and find your model, **the result = mean_reward - std of reward**\n", - "\n", - "If you don't find your model, **go to the bottom of the page and click on the refresh button**\n", - "\n", - "For more information about the certification process, check this section 👉 https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "PU4FVzaoM6fC" - }, - "source": [ - "## Set the GPU 💪\n", - "- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`\n", - "\n", - "\"GPU" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "KV0NyFdQM9ZG" - }, - "source": [ - "- `Hardware Accelerator > GPU`\n", - "\n", - "\"GPU" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "-fSy5HzUcMWB" - }, - "source": [ - "Before starting to train our agent, let's **study the library and environments we're going to use**.\n", - "\n", - "## Sample Factory\n", - "\n", - "[Sample Factory](https://www.samplefactory.dev/) is one of the **fastest RL libraries focused on very efficient synchronous and asynchronous implementations of policy gradients (PPO)**.\n", - "\n", - "Sample Factory is thoroughly **tested, used by many researchers and practitioners**, and is actively maintained. Our implementation is known to **reach SOTA performance in a variety of domains while minimizing RL experiment training time and hardware requirements**.\n", - "\n", - "\"Sample\n", - "\n", - "\n", - "\n", - "### Key features\n", - "\n", - "- Highly optimized algorithm [architecture](https://www.samplefactory.dev/06-architecture/overview/) for maximum learning throughput\n", - "- [Synchronous and asynchronous](https://www.samplefactory.dev/07-advanced-topics/sync-async/) training regimes\n", - "- [Serial (single-process) mode](https://www.samplefactory.dev/07-advanced-topics/serial-mode/) for easy debugging\n", - "- Optimal performance in both CPU-based and [GPU-accelerated environments](https://www.samplefactory.dev/09-environment-integrations/isaacgym/)\n", - "- Single- & multi-agent training, self-play, supports [training multiple policies](https://www.samplefactory.dev/07-advanced-topics/multi-policy-training/) at once on one or many GPUs\n", - "- Population-Based Training ([PBT](https://www.samplefactory.dev/07-advanced-topics/pbt/))\n", - "- Discrete, continuous, hybrid action spaces\n", - "- Vector-based, image-based, dictionary observation spaces\n", - "- Automatically creates a model architecture by parsing action/observation space specification. Supports [custom model architectures](https://www.samplefactory.dev/03-customization/custom-models/)\n", - "- Designed to be imported into other projects, [custom environments](https://www.samplefactory.dev/03-customization/custom-environments/) are first-class citizens\n", - "- Detailed [WandB and Tensorboard summaries](https://www.samplefactory.dev/05-monitoring/metrics-reference/), [custom metrics](https://www.samplefactory.dev/05-monitoring/custom-metrics/)\n", - "- [HuggingFace 🤗 integration](https://www.samplefactory.dev/10-huggingface/huggingface/) (upload trained models and metrics to the Hub)\n", - "- [Multiple](https://www.samplefactory.dev/09-environment-integrations/mujoco/) [example](https://www.samplefactory.dev/09-environment-integrations/atari/) [environment](https://www.samplefactory.dev/09-environment-integrations/vizdoom/) [integrations](https://www.samplefactory.dev/09-environment-integrations/dmlab/) with tuned parameters and trained models\n", - "\n", - "All of the above policies are available on the 🤗 hub. Search for the tag [sample-factory](https://huggingface.co/models?library=sample-factory&sort=downloads)\n", - "\n", - "### How sample-factory works\n", - "\n", - "Sample-factory is one of the **most highly optimized RL implementations available to the community**.\n", - "\n", - "It works by **spawning multiple processes that run rollout workers, inference workers and a learner worker**.\n", - "\n", - "The *workers* **communicate through shared memory, which lowers the communication cost between processes**.\n", - "\n", - "The *rollout workers* interact with the environment and send observations to the *inference workers*.\n", - "\n", - "The *inferences workers* query a fixed version of the policy and **send actions back to the rollout worker**.\n", - "\n", - "After *k* steps the rollout works send a trajectory of experience to the learner worker, **which it uses to update the agent’s policy network**.\n", - "\n", - "\"Sample" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "nB68Eb9UgC94" - }, - "source": [ - "### Actor Critic models in Sample-factory\n", - "\n", - "Actor Critic models in Sample Factory are composed of three components:\n", - "\n", - "- **Encoder** - Process input observations (images, vectors) and map them to a vector. This is the part of the model you will most likely want to customize.\n", - "- **Core** - Intergrate vectors from one or more encoders, can optionally include a single- or multi-layer LSTM/GRU in a memory-based agent.\n", - "- **Decoder** - Apply additional layers to the output of the model core before computing the policy and value outputs.\n", - "\n", - "The library has been designed to automatically support any observation and action spaces. Users can easily add their custom models. You can find out more in the [documentation](https://www.samplefactory.dev/03-customization/custom-models/#actor-critic-models-in-sample-factory)." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "ez5UhUtYcWXF" - }, - "source": [ - "## ViZDoom\n", - "\n", - "[ViZDoom](https://vizdoom.cs.put.edu.pl/) is an **open-source python interface for the Doom Engine**.\n", - "\n", - "The library was created in 2016 by Marek Wydmuch, Michal Kempka at the Institute of Computing Science, Poznan University of Technology, Poland.\n", - "\n", - "The library enables the **training of agents directly from the screen pixels in a number of scenarios**, including team deathmatch, shown in the video below. Because the ViZDoom environment is based on a game the was created in the 90s, it can be run on modern hardware at accelerated speeds, **allowing us to learn complex AI behaviors fairly quickly**.\n", - "\n", - "The library includes feature such as:\n", - "\n", - "- Multi-platform (Linux, macOS, Windows),\n", - "- API for Python and C++,\n", - "- [OpenAI Gym](https://www.gymlibrary.dev/) environment wrappers\n", - "- Easy-to-create custom scenarios (visual editors, scripting language, and examples available),\n", - "- Async and sync single-player and multiplayer modes,\n", - "- Lightweight (few MBs) and fast (up to 7000 fps in sync mode, single-threaded),\n", - "- Customizable resolution and rendering parameters,\n", - "- Access to the depth buffer (3D vision),\n", - "- Automatic labeling of game objects visible in the frame,\n", - "- Access to the audio buffer\n", - "- Access to the list of actors/objects and map geometry,\n", - "- Off-screen rendering and episode recording,\n", - "- Time scaling in async mode." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "wAMwza0d5QVj" - }, - "source": [ - "## We first need to install some dependencies that are required for the ViZDoom environment\n", - "\n", - "Now that our Colab runtime is set up, we can start by installing the dependencies required to run ViZDoom on linux.\n", - "\n", - "If you are following on your machine on Mac, you will want to follow the installation instructions on the [github page](https://github.com/Farama-Foundation/ViZDoom/blob/master/doc/Quickstart.md#-quickstart-for-macos-and-anaconda3-python-36)." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "RJMxkaldwIVx" - }, - "outputs": [], - "source": [ - "%%capture\n", - "%%bash\n", - "# Install ViZDoom deps from\n", - "# https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md#-linux\n", - "\n", - "apt-get install build-essential zlib1g-dev libsdl2-dev libjpeg-dev \\\n", - "nasm tar libbz2-dev libgtk2.0-dev cmake git libfluidsynth-dev libgme-dev \\\n", - "libopenal-dev timidity libwildmidi-dev unzip ffmpeg\n", - "\n", - "# Boost libraries\n", - "apt-get install libboost-all-dev\n", - "\n", - "# Lua binding dependencies\n", - "apt-get install liblua5.1-dev" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "JT4att2c57MW" - }, - "source": [ - "## Then we can install Sample Factory and ViZDoom\n", - "- This can take 7min" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "bbqfPZnIsvA6" - }, - "outputs": [], - "source": [ - "# install python libraries\n", - "# thanks toinsson\n", - "!pip install faster-fifo==1.4.2\n", - "!pip install vizdoom" - ] - }, - { - "cell_type": "code", - "source": [ - "!pip install sample-factory==2.1.1" - ], - "metadata": { - "id": "alxUt7Au-O8e" - }, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "1jizouGpghUZ" - }, - "source": [ - "## Setting up the Doom Environment in sample-factory" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "bCgZbeiavcDU" - }, - "outputs": [], - "source": [ - "import functools\n", - "\n", - "from sample_factory.algo.utils.context import global_model_factory\n", - "from sample_factory.cfg.arguments import parse_full_cfg, parse_sf_args\n", - "from sample_factory.envs.env_utils import register_env\n", - "from sample_factory.train import run_rl\n", - "\n", - "from sf_examples.vizdoom.doom.doom_model import make_vizdoom_encoder\n", - "from sf_examples.vizdoom.doom.doom_params import add_doom_env_args, doom_override_defaults\n", - "from sf_examples.vizdoom.doom.doom_utils import DOOM_ENVS, make_doom_env_from_spec\n", - "\n", - "\n", - "# Registers all the ViZDoom environments\n", - "def register_vizdoom_envs():\n", - " for env_spec in DOOM_ENVS:\n", - " make_env_func = functools.partial(make_doom_env_from_spec, env_spec)\n", - " register_env(env_spec.name, make_env_func)\n", - "\n", - "# Sample Factory allows the registration of a custom Neural Network architecture\n", - "# See https://github.com/alex-petrenko/sample-factory/blob/master/sf_examples/vizdoom/doom/doom_model.py for more details\n", - "def register_vizdoom_models():\n", - " global_model_factory().register_encoder_factory(make_vizdoom_encoder)\n", - "\n", - "\n", - "def register_vizdoom_components():\n", - " register_vizdoom_envs()\n", - " register_vizdoom_models()\n", - "\n", - "# parse the command line args and create a config\n", - "def parse_vizdoom_cfg(argv=None, evaluation=False):\n", - " parser, _ = parse_sf_args(argv=argv, evaluation=evaluation)\n", - " # parameters specific to Doom envs\n", - " add_doom_env_args(parser)\n", - " # override Doom default values for algo parameters\n", - " doom_override_defaults(parser)\n", - " # second parsing pass yields the final configuration\n", - " final_cfg = parse_full_cfg(parser, argv)\n", - " return final_cfg" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "sgRy6wnrgnij" - }, - "source": [ - "Now that the setup if complete, we can train the agent. We have chosen here to learn a ViZDoom task called `Health Gathering Supreme`.\n", - "\n", - "### The scenario: Health Gathering Supreme\n", - "\n", - "\"Health-Gathering-Supreme\"/\n", - "\n", - "\n", - "\n", - "The objective of this scenario is to **teach the agent how to survive without knowing what makes him survive**. Agent know only that **life is precious** and death is bad so **it must learn what prolongs his existence and that his health is connected with it**.\n", - "\n", - "Map is a rectangle containing walls and with a green, acidic floor which **hurts the player periodically**. Initially there are some medkits spread uniformly over the map. A new medkit falls from the skies every now and then. **Medkits heal some portions of player's health** - to survive agent needs to pick them up. Episode finishes after player's death or on timeout.\n", - "\n", - "Further configuration:\n", - "- Living_reward = 1\n", - "- 3 available buttons: turn left, turn right, move forward\n", - "- 1 available game variable: HEALTH\n", - "- death penalty = 100\n", - "\n", - "You can find out more about the scenarios available in ViZDoom [here](https://github.com/Farama-Foundation/ViZDoom/tree/master/scenarios).\n", - "\n", - "There are also a number of more complex scenarios that have been create for ViZDoom, such as the ones detailed on [this github page](https://github.com/edbeeching/3d_control_deep_rl).\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "siHZZ34DiZEp" - }, - "source": [ - "## Training the agent\n", - "- We're going to train the agent for 4000000 steps it will take approximately 20min" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "y_TeicMvyKHP" - }, - "outputs": [], - "source": [ - "## Start the training, this should take around 15 minutes\n", - "register_vizdoom_components()\n", - "\n", - "# The scenario we train on today is health gathering\n", - "# other scenarios include \"doom_basic\", \"doom_two_colors_easy\", \"doom_dm\", \"doom_dwango5\", \"doom_my_way_home\", \"doom_deadly_corridor\", \"doom_defend_the_center\", \"doom_defend_the_line\"\n", - "env = \"doom_health_gathering_supreme\"\n", - "cfg = parse_vizdoom_cfg(argv=[f\"--env={env}\", \"--num_workers=8\", \"--num_envs_per_worker=4\", \"--train_for_env_steps=4000000\"])\n", - "\n", - "status = run_rl(cfg)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "5L0nBS9e_jqC" - }, - "source": [ - "## Let's take a look at the performance of the trained policy and output a video of the agent." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "MGSA4Kg5_i0j" - }, - "outputs": [], - "source": [ - "from sample_factory.enjoy import enjoy\n", - "cfg = parse_vizdoom_cfg(argv=[f\"--env={env}\", \"--num_workers=1\", \"--save_video\", \"--no_render\", \"--max_num_episodes=10\"], evaluation=True)\n", - "status = enjoy(cfg)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Lj5L1x0WLxwB" - }, - "source": [ - "## Now lets visualize the performance of the agent" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "WsXhBY7JNOdJ" - }, - "outputs": [], - "source": [ - "from base64 import b64encode\n", - "from IPython.display import HTML\n", - "\n", - "mp4 = open('/content/train_dir/default_experiment/replay.mp4','rb').read()\n", - "data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n", - "HTML(\"\"\"\n", - "\n", - "\"\"\" % data_url)" - ] - }, - { - "cell_type": "markdown", - "source": [ - "The agent has learned something, but its performance could be better. We would clearly need to train for longer. But let's upload this model to the Hub." - ], - "metadata": { - "id": "2A4pf_1VwPqR" - } - }, - { - "cell_type": "markdown", - "metadata": { - "id": "CSQVWF0kNuy9" - }, - "source": [ - "## Now lets upload your checkpoint and video to the Hugging Face Hub\n", - "\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "JquRrWytA6eo" - }, - "source": [ - "To be able to share your model with the community there are three more steps to follow:\n", - "\n", - "1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join\n", - "\n", - "2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website.\n", - "- Create a new token (https://huggingface.co/settings/tokens) **with write role**\n", - "\n", - "\"Create\n", - "\n", - "- Copy the token\n", - "- Run the cell below and paste the token" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "_tsf2uv0g_4p" - }, - "source": [ - "If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "GoQm_jYSOts0" - }, - "outputs": [], - "source": [ - "from huggingface_hub import notebook_login\n", - "notebook_login()\n", - "!git config --global credential.helper store" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "sEawW_i0OvJV" - }, - "outputs": [], - "source": [ - "from sample_factory.enjoy import enjoy\n", - "\n", - "hf_username = \"ThomasSimonini\" # insert your HuggingFace username here\n", - "\n", - "cfg = parse_vizdoom_cfg(argv=[f\"--env={env}\", \"--num_workers=1\", \"--save_video\", \"--no_render\", \"--max_num_episodes=10\", \"--max_num_frames=100000\", \"--push_to_hub\", f\"--hf_repository={hf_username}/rl_course_vizdoom_health_gathering_supreme\"], evaluation=True)\n", - "status = enjoy(cfg)" - ] - }, - { - "cell_type": "markdown", - "source": [ - "## Let's load another model\n", - "\n", - "\n" - ], - "metadata": { - "id": "9PzeXx-qxVvw" - } - }, - { - "cell_type": "markdown", - "metadata": { - "id": "mHZAWSgL5F7P" - }, - "source": [ - "This agent's performance was good, but can do better! Let's download and visualize an agent trained for 10B timesteps from the hub." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "Ud6DwAUl5S-l" - }, - "outputs": [], - "source": [ - "#download the agent from the hub\n", - "!python -m sample_factory.huggingface.load_from_hub -r edbeeching/doom_health_gathering_supreme_2222 -d ./train_dir\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "qoUJhL6x6sY5" - }, - "outputs": [], - "source": [ - "!ls train_dir/doom_health_gathering_supreme_2222" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "lZskc8LG8qr8" - }, - "outputs": [], - "source": [ - "env = \"doom_health_gathering_supreme\"\n", - "cfg = parse_vizdoom_cfg(argv=[f\"--env={env}\", \"--num_workers=1\", \"--save_video\", \"--no_render\", \"--max_num_episodes=10\", \"--experiment=doom_health_gathering_supreme_2222\", \"--train_dir=train_dir\"], evaluation=True)\n", - "status = enjoy(cfg)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "BtzXBoj65Wmq" - }, - "outputs": [], - "source": [ - "mp4 = open('/content/train_dir/doom_health_gathering_supreme_2222/replay.mp4','rb').read()\n", - "data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n", - "HTML(\"\"\"\n", - "\n", - "\"\"\" % data_url)" - ] - }, - { - "cell_type": "markdown", - "source": [ - "## Some additional challenges 🏆: Doom Deathmatch\n", - "\n", - "Training an agent to play a Doom deathmatch **takes many hours on a more beefy machine than is available in Colab**.\n", - "\n", - "Fortunately, we have have **already trained an agent in this scenario and it is available in the 🤗 Hub!** Let’s download the model and visualize the agent’s performance." - ], - "metadata": { - "id": "ie5YWC3NyKO8" - } - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "fq3WFeus81iI" - }, - "outputs": [], - "source": [ - "# Download the agent from the hub\n", - "!python -m sample_factory.huggingface.load_from_hub -r edbeeching/doom_deathmatch_bots_2222 -d ./train_dir" - ] - }, - { - "cell_type": "markdown", - "source": [ - "Given the agent plays for a long time the video generation can take **10 minutes**." - ], - "metadata": { - "id": "7AX_LwxR2FQ0" - } - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "0hq6XL__85Bv" - }, - "outputs": [], - "source": [ - "\n", - "from sample_factory.enjoy import enjoy\n", - "register_vizdoom_components()\n", - "env = \"doom_deathmatch_bots\"\n", - "cfg = parse_vizdoom_cfg(argv=[f\"--env={env}\", \"--num_workers=1\", \"--save_video\", \"--no_render\", \"--max_num_episodes=1\", \"--experiment=doom_deathmatch_bots_2222\", \"--train_dir=train_dir\"], evaluation=True)\n", - "status = enjoy(cfg)\n", - "mp4 = open('/content/train_dir/doom_deathmatch_bots_2222/replay.mp4','rb').read()\n", - "data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n", - "HTML(\"\"\"\n", - "\n", - "\"\"\" % data_url)" - ] - }, - { - "cell_type": "markdown", - "source": [ - "\n", - "You **can try to train your agent in this environment** using the code above, but not on colab.\n", - "**Good luck 🤞**" - ], - "metadata": { - "id": "N6mEC-4zyihx" - } - }, - { - "cell_type": "markdown", - "source": [ - "If you prefer an easier scenario, **why not try training in another ViZDoom scenario such as `doom_deadly_corridor` or `doom_defend_the_center`.**\n", - "\n", - "\n", - "\n", - "---\n", - "\n", - "\n", - "This concludes the last unit. But we are not finished yet! 🤗 The following **bonus section include some of the most interesting, advanced and cutting edge work in Deep Reinforcement Learning**.\n", - "\n", - "## Keep learning, stay awesome 🤗" - ], - "metadata": { - "id": "YnDAngN6zeeI" - } - } - ], - "metadata": { - "accelerator": "GPU", - "colab": { - "provenance": [], - "collapsed_sections": [ - "PU4FVzaoM6fC", - "nB68Eb9UgC94", - "ez5UhUtYcWXF", - "sgRy6wnrgnij" - ], - "private_outputs": true, - "include_colab_link": true - }, - "gpuClass": "standard", - "kernelspec": { - "display_name": "Python 3", - "name": "python3" - }, - "language_info": { - "name": "python" - } - }, - "nbformat": 4, - "nbformat_minor": 0 -} \ No newline at end of file diff --git a/notebooks/unit8/ppo.py b/notebooks/unit8/ppo.py deleted file mode 100644 index a11bc55..0000000 --- a/notebooks/unit8/ppo.py +++ /dev/null @@ -1,532 +0,0 @@ -import argparse -import os -import random -import time -from distutils.util import strtobool - -import gym -import numpy as np -import torch -import torch.nn as nn -import torch.optim as optim -from torch.distributions.categorical import Categorical -from torch.utils.tensorboard import SummaryWriter - -from huggingface_hub import HfApi, upload_folder -from huggingface_hub.repocard import metadata_eval_result, metadata_save - -from pathlib import Path -import datetime -import tempfile -import json -import shutil -import imageio - -from wasabi import Printer -msg = Printer() - -def parse_args(): - # fmt: off - parser = argparse.ArgumentParser() - parser.add_argument("--exp-name", type=str, default=os.path.basename(__file__).rstrip(".py"), - help="the name of this experiment") - parser.add_argument("--seed", type=int, default=1, - help="seed of the experiment") - parser.add_argument("--torch-deterministic", type=lambda x: bool(strtobool(x)), default=True, nargs="?", const=True, - help="if toggled, `torch.backends.cudnn.deterministic=False`") - parser.add_argument("--cuda", type=lambda x: bool(strtobool(x)), default=True, nargs="?", const=True, - help="if toggled, cuda will be enabled by default") - parser.add_argument("--track", type=lambda x: bool(strtobool(x)), default=False, nargs="?", const=True, - help="if toggled, this experiment will be tracked with Weights and Biases") - parser.add_argument("--wandb-project-name", type=str, default="cleanRL", - help="the wandb's project name") - parser.add_argument("--wandb-entity", type=str, default=None, - help="the entity (team) of wandb's project") - parser.add_argument("--capture-video", type=lambda x: bool(strtobool(x)), default=False, nargs="?", const=True, - help="whether to capture videos of the agent performances (check out `videos` folder)") - - # Algorithm specific arguments - parser.add_argument("--env-id", type=str, default="CartPole-v1", - help="the id of the environment") - parser.add_argument("--total-timesteps", type=int, default=50000, - help="total timesteps of the experiments") - parser.add_argument("--learning-rate", type=float, default=2.5e-4, - help="the learning rate of the optimizer") - parser.add_argument("--num-envs", type=int, default=4, - help="the number of parallel game environments") - parser.add_argument("--num-steps", type=int, default=128, - help="the number of steps to run in each environment per policy rollout") - parser.add_argument("--anneal-lr", type=lambda x: bool(strtobool(x)), default=True, nargs="?", const=True, - help="Toggle learning rate annealing for policy and value networks") - parser.add_argument("--gae", type=lambda x: bool(strtobool(x)), default=True, nargs="?", const=True, - help="Use GAE for advantage computation") - parser.add_argument("--gamma", type=float, default=0.99, - help="the discount factor gamma") - parser.add_argument("--gae-lambda", type=float, default=0.95, - help="the lambda for the general advantage estimation") - parser.add_argument("--num-minibatches", type=int, default=4, - help="the number of mini-batches") - parser.add_argument("--update-epochs", type=int, default=4, - help="the K epochs to update the policy") - parser.add_argument("--norm-adv", type=lambda x: bool(strtobool(x)), default=True, nargs="?", const=True, - help="Toggles advantages normalization") - parser.add_argument("--clip-coef", type=float, default=0.2, - help="the surrogate clipping coefficient") - parser.add_argument("--clip-vloss", type=lambda x: bool(strtobool(x)), default=True, nargs="?", const=True, - help="Toggles whether or not to use a clipped loss for the value function, as per the paper.") - parser.add_argument("--ent-coef", type=float, default=0.01, - help="coefficient of the entropy") - parser.add_argument("--vf-coef", type=float, default=0.5, - help="coefficient of the value function") - parser.add_argument("--max-grad-norm", type=float, default=0.5, - help="the maximum norm for the gradient clipping") - parser.add_argument("--target-kl", type=float, default=None, - help="the target KL divergence threshold") - - # Adding HuggingFace argument - parser.add_argument("--repo-id", type=str, default="LizardAPN/ppo-CartPole-v1", help="id of the model repository from the Hugging Face Hub {username/repo_name}") - - args = parser.parse_args() - args.batch_size = int(args.num_envs * args.num_steps) - args.minibatch_size = int(args.batch_size // args.num_minibatches) - # fmt: on - return args - -def package_to_hub(repo_id, - model, - hyperparameters, - eval_env, - video_fps=30, - commit_message="Push agent to the Hub", - token=None, - logs=None): - """ - Evaluate, Generate a video and Upload a model to Hugging Face Hub. - """ - msg.info( - "This function will save, evaluate, generate a video of your agent, " - "create a model card and push everything to the hub." - ) - - # Step 1: Create repo - repo_url = HfApi().create_repo( - repo_id=repo_id, - token=token, - private=False, - exist_ok=True, - ) - - with tempfile.TemporaryDirectory() as tmpdirname: - tmpdirname = Path(tmpdirname) - - # Step 2: Save the model - torch.save(model.state_dict(), tmpdirname / "model.pt") - - # Step 3: Evaluate the model - mean_reward, std_reward = _evaluate_agent(eval_env, 10, model) - - # Prepare evaluation data - eval_datetime = datetime.datetime.now() - evaluate_data = { - "env_id": hyperparameters.env_id, - "mean_reward": mean_reward, - "std_reward": std_reward, - "n_evaluation_episodes": 10, - "eval_datetime": eval_datetime.isoformat(), - } - - # Save evaluation results - with open(tmpdirname / "results.json", "w") as outfile: - json.dump(evaluate_data, outfile) - - # Step 4: Generate video - video_path = tmpdirname / "replay.mp4" - record_video(eval_env, model, video_path, video_fps) - - # Step 5: Generate model card - generated_model_card, metadata = _generate_model_card("PPO", hyperparameters.env_id, mean_reward, std_reward, hyperparameters) - _save_model_card(tmpdirname, generated_model_card, metadata) - - # Step 6: Add logs if provided - if logs: - _add_logdir(tmpdirname, Path(logs)) - - # Step 7: Upload to Hub - msg.info(f"Pushing repo {repo_id} to the Hugging Face Hub") - repo_url = upload_folder( - repo_id=repo_id, - folder_path=tmpdirname, - path_in_repo="", - commit_message=commit_message, - token=token, - ) - - msg.info(f"Your model is pushed to the Hub. You can view your model here: {repo_url}") - return repo_url - -def _evaluate_agent(env, n_eval_episodes, policy): - """ - Evaluate the agent for n_eval_episodes episodes. - """ - episode_rewards = [] - for _ in range(n_eval_episodes): - state, _ = env.reset() - done = False - total_rewards_ep = 0 - - while not done: - state = torch.Tensor(state).to(device) - with torch.no_grad(): - action, _, _, _ = policy.get_action_and_value(state) - state, reward, terminated, truncated, _ = env.step(action.cpu().numpy()) - total_rewards_ep += reward - done = terminated or truncated - episode_rewards.append(total_rewards_ep) - - mean_reward = np.mean(episode_rewards) - std_reward = np.std(episode_rewards) - return mean_reward, std_reward - -def record_video(env, policy, out_directory, fps=30): - """ - Record a video of the agent's performance. - """ - images = [] - state, _ = env.reset() - img = env.render() - images.append(img) - - done = False - while not done: - state = torch.Tensor(state).to(device) - with torch.no_grad(): - action, _, _, _ = policy.get_action_and_value(state) - state, _, terminated, truncated, _ = env.step(action.cpu().numpy()) - img = env.render() - images.append(img) - done = terminated or truncated - - imageio.mimsave(out_directory, [np.array(img) for img in images], fps=fps) - -def _generate_model_card(model_name, env_id, mean_reward, std_reward, hyperparameters): - """ - Generate the model card for the Hub. - """ - metadata = generate_metadata(model_name, env_id, mean_reward, std_reward) - - # Convert hyperparameters to string - converted_dict = vars(hyperparameters) - converted_str = '\n'.join([f"{k}: {v}" for k, v in converted_dict.items()]) - - model_card = f""" - # PPO Agent Playing {env_id} - - This is a trained model of a PPO agent playing {env_id}. - - # Hyperparameters - ```python - {converted_str} - ``` - """ - return model_card, metadata - -def generate_metadata(model_name, env_id, mean_reward, std_reward): - """ - Define the tags for the model card. - """ - metadata = { - "tags": [ - env_id, - "ppo", - "deep-reinforcement-learning", - "reinforcement-learning", - "custom-implementation", - "deep-rl-course" - ] - } - - eval_metadata = metadata_eval_result( - model_pretty_name=model_name, - task_pretty_name="reinforcement-learning", - task_id="reinforcement-learning", - metrics_pretty_name="mean_reward", - metrics_id="mean_reward", - metrics_value=f"{mean_reward:.2f} +/- {std_reward:.2f}", - dataset_pretty_name=env_id, - dataset_id=env_id, - ) - - metadata.update(eval_metadata) - return metadata - -def _save_model_card(local_path, generated_model_card, metadata): - """ - Save the model card and metadata. - """ - readme_path = local_path / "README.md" - with readme_path.open("w", encoding="utf-8") as f: - f.write(generated_model_card) - metadata_save(readme_path, metadata) - -def _add_logdir(local_path, logdir): - """ - Add log directory to the repository. - """ - if logdir.exists() and logdir.is_dir(): - repo_logdir = local_path / "logs" - if repo_logdir.exists(): - shutil.rmtree(repo_logdir) - shutil.copytree(logdir, repo_logdir) - -def make_env(env_id, seed, idx, capture_video, run_name): - """ - Create a wrapped environment. - """ - def thunk(): - env = gym.make(env_id, render_mode="rgb_array") - env = gym.wrappers.RecordEpisodeStatistics(env) - if capture_video and idx == 0: - env = gym.wrappers.RecordVideo(env, f"videos/{run_name}") - env.reset(seed=seed) - env.action_space.seed(seed) - env.observation_space.seed(seed) - return env - return thunk - -def layer_init(layer, std=np.sqrt(2), bias_const=0.0): - """ - Initialize layer weights. - """ - torch.nn.init.orthogonal_(layer.weight, std) - torch.nn.init.constant_(layer.bias, bias_const) - return layer - -class Agent(nn.Module): - def __init__(self, envs): - super().__init__() - self.critic = nn.Sequential( - layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 64)), - nn.Tanh(), - layer_init(nn.Linear(64, 64)), - nn.Tanh(), - layer_init(nn.Linear(64, 1), std=1.0), - ) - self.actor = nn.Sequential( - layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 64)), - nn.Tanh(), - layer_init(nn.Linear(64, 64)), - nn.Tanh(), - layer_init(nn.Linear(64, envs.single_action_space.n), std=0.01), - ) - - def get_value(self, x): - return self.critic(x) - - def get_action_and_value(self, x, action=None): - logits = self.actor(x) - probs = Categorical(logits=logits) - if action is None: - action = probs.sample() - return action, probs.log_prob(action), probs.entropy(), self.critic(x) - -if __name__ == "__main__": - args = parse_args() - run_name = f"{args.env_id}__{args.exp_name}__{args.seed}__{int(time.time())}" - - if args.track: - import wandb - wandb.init( - project=args.wandb_project_name, - entity=args.wandb_entity, - sync_tensorboard=True, - config=vars(args), - name=run_name, - monitor_gym=True, - save_code=True, - ) - - writer = SummaryWriter(f"runs/{run_name}") - writer.add_text( - "hyperparameters", - "|param|value|\n|-|-|\n%s" % ("\n".join([f"|{key}|{value}|" for key, value in vars(args).items()])), - ) - - # Seeding - random.seed(args.seed) - np.random.seed(args.seed) - torch.manual_seed(args.seed) - torch.backends.cudnn.deterministic = args.torch_deterministic - - device = torch.device("cuda" if torch.cuda.is_available() and args.cuda else "cpu") - - # Environment setup - envs = gym.vector.SyncVectorEnv( - [make_env(args.env_id, args.seed + i, i, args.capture_video, run_name) for i in range(args.num_envs)] - ) - assert isinstance(envs.single_action_space, gym.spaces.Discrete), "only discrete action space is supported" - - agent = Agent(envs).to(device) - optimizer = optim.Adam(agent.parameters(), lr=args.learning_rate, eps=1e-5) - - # Storage setup - obs = torch.zeros((args.num_steps, args.num_envs) + envs.single_observation_space.shape).to(device) - actions = torch.zeros((args.num_steps, args.num_envs) + envs.single_action_space.shape).to(device) - logprobs = torch.zeros((args.num_steps, args.num_envs)).to(device) - rewards = torch.zeros((args.num_steps, args.num_envs)).to(device) - dones = torch.zeros((args.num_steps, args.num_envs)).to(device) - values = torch.zeros((args.num_steps, args.num_envs)).to(device) - - # Training loop - global_step = 0 - start_time = time.time() - next_obs, _ = envs.reset(seed=args.seed) - next_obs = torch.Tensor(next_obs).to(device) - next_done = torch.zeros(args.num_envs).to(device) - num_updates = args.total_timesteps // args.batch_size - - for update in range(1, num_updates + 1): - if args.anneal_lr: - frac = 1.0 - (update - 1.0) / num_updates - lrnow = frac * args.learning_rate - optimizer.param_groups[0]["lr"] = lrnow - - for step in range(0, args.num_steps): - global_step += 1 * args.num_envs - obs[step] = next_obs - dones[step] = next_done - - with torch.no_grad(): - action, logprob, _, value = agent.get_action_and_value(next_obs) - values[step] = value.flatten() - actions[step] = action - logprobs[step] = logprob - - next_obs, reward, terminated, truncated, infos = envs.step(action.cpu().numpy()) - next_done = np.logical_or(terminated, truncated) - rewards[step] = torch.tensor(reward).to(device).view(-1) - next_obs, next_done = torch.Tensor(next_obs).to(device), torch.Tensor(next_done).to(device) - - if "final_info" in infos: - for info in infos["final_info"]: - if info and "episode" in info: - print(f"global_step={global_step}, episodic_return={info['episode']['r']}") - writer.add_scalar("charts/episodic_return", info["episode"]["r"], global_step) - writer.add_scalar("charts/episodic_length", info["episode"]["l"], global_step) - - # Compute advantages and returns - with torch.no_grad(): - next_value = agent.get_value(next_obs).reshape(1, -1) - if args.gae: - advantages = torch.zeros_like(rewards).to(device) - lastgaelam = 0 - for t in reversed(range(args.num_steps)): - if t == args.num_steps - 1: - nextnonterminal = 1.0 - next_done - nextvalues = next_value - else: - nextnonterminal = 1.0 - dones[t + 1] - nextvalues = values[t + 1] - delta = rewards[t] + args.gamma * nextvalues * nextnonterminal - values[t] - advantages[t] = lastgaelam = delta + args.gamma * args.gae_lambda * nextnonterminal * lastgaelam - returns = advantages + values - else: - returns = torch.zeros_like(rewards).to(device) - for t in reversed(range(args.num_steps)): - if t == args.num_steps - 1: - nextnonterminal = 1.0 - next_done - next_return = next_value - else: - nextnonterminal = 1.0 - dones[t + 1] - next_return = returns[t + 1] - returns[t] = rewards[t] + args.gamma * nextnonterminal * next_return - advantages = returns - values - - # Flatten the batch - b_obs = obs.reshape((-1,) + envs.single_observation_space.shape) - b_logprobs = logprobs.reshape(-1) - b_actions = actions.reshape((-1,) + envs.single_action_space.shape) - b_advantages = advantages.reshape(-1) - b_returns = returns.reshape(-1) - b_values = values.reshape(-1) - - # Optimize policy and value network - b_inds = np.arange(args.batch_size) - clipfracs = [] - for epoch in range(args.update_epochs): - np.random.shuffle(b_inds) - for start in range(0, args.batch_size, args.minibatch_size): - end = start + args.minibatch_size - mb_inds = b_inds[start:end] - - _, newlogprob, entropy, newvalue = agent.get_action_and_value(b_obs[mb_inds], b_actions.long()[mb_inds]) - logratio = newlogprob - b_logprobs[mb_inds] - ratio = logratio.exp() - - with torch.no_grad(): - old_approx_kl = (-logratio).mean() - approx_kl = ((ratio - 1) - logratio).mean() - clipfracs += [((ratio - 1.0).abs() > args.clip_coef).float().mean().item()] - - mb_advantages = b_advantages[mb_inds] - if args.norm_adv: - mb_advantages = (mb_advantages - mb_advantages.mean()) / (mb_advantages.std() + 1e-8) - - # Policy loss - pg_loss1 = -mb_advantages * ratio - pg_loss2 = -mb_advantages * torch.clamp(ratio, 1 - args.clip_coef, 1 + args.clip_coef) - pg_loss = torch.max(pg_loss1, pg_loss2).mean() - - # Value loss - newvalue = newvalue.view(-1) - if args.clip_vloss: - v_loss_unclipped = (newvalue - b_returns[mb_inds]) ** 2 - v_clipped = b_values[mb_inds] + torch.clamp( - newvalue - b_values[mb_inds], - -args.clip_coef, - args.clip_coef, - ) - v_loss_clipped = (v_clipped - b_returns[mb_inds]) ** 2 - v_loss_max = torch.max(v_loss_unclipped, v_loss_clipped) - v_loss = 0.5 * v_loss_max.mean() - else: - v_loss = 0.5 * ((newvalue - b_returns[mb_inds]) ** 2).mean() - - entropy_loss = entropy.mean() - loss = pg_loss - args.ent_coef * entropy_loss + v_loss * args.vf_coef - - optimizer.zero_grad() - loss.backward() - nn.utils.clip_grad_norm_(agent.parameters(), args.max_grad_norm) - optimizer.step() - - if args.target_kl is not None and approx_kl > args.target_kl: - break - - # Log training metrics - y_pred, y_true = b_values.cpu().numpy(), b_returns.cpu().numpy() - var_y = np.var(y_true) - explained_var = np.nan if var_y == 0 else 1 - np.var(y_true - y_pred) / var_y - - writer.add_scalar("charts/learning_rate", optimizer.param_groups[0]["lr"], global_step) - writer.add_scalar("losses/value_loss", v_loss.item(), global_step) - writer.add_scalar("losses/policy_loss", pg_loss.item(), global_step) - writer.add_scalar("losses/entropy", entropy_loss.item(), global_step) - writer.add_scalar("losses/old_approx_kl", old_approx_kl.item(), global_step) - writer.add_scalar("losses/approx_kl", approx_kl.item(), global_step) - writer.add_scalar("losses/clipfrac", np.mean(clipfracs), global_step) - writer.add_scalar("losses/explained_variance", explained_var, global_step) - print("SPS:", int(global_step / (time.time() - start_time))) - writer.add_scalar("charts/SPS", int(global_step / (time.time() - start_time)), global_step) - - envs.close() - writer.close() - - # Push to Hub - eval_env = gym.make(args.env_id, render_mode="rgb_array") - package_to_hub( - repo_id=args.repo_id, - model=agent, - hyperparameters=args, - eval_env=eval_env, - logs=f"runs/{run_name}", - ) \ No newline at end of file diff --git a/notebooks/unit8/ppo_part1.ipynb b/notebooks/unit8/ppo_part1.ipynb deleted file mode 100644 index 96a2597..0000000 --- a/notebooks/unit8/ppo_part1.ipynb +++ /dev/null @@ -1,1370 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "id": "view-in-github", - "colab_type": "text" - }, - "source": [ - "\"Open" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "-cf5-oDPjwf8" - }, - "source": [ - "# Unit 8: Proximal Policy Gradient (PPO) with PyTorch 🤖\n", - "\n", - "\"Unit\n", - "\n", - "\n", - "In this notebook, you'll learn to **code your PPO agent from scratch with PyTorch using CleanRL implementation as model**.\n", - "\n", - "To test its robustness, we're going to train it in:\n", - "\n", - "- [LunarLander-v2 🚀](https://www.gymlibrary.dev/environments/box2d/lunar_lander/)\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "2Fl6Rxt0lc0O" - }, - "source": [ - "⬇️ Here is an example of what you will achieve. ⬇️" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "DbKfCj5ilgqT" - }, - "outputs": [], - "source": [ - "%%html\n", - "" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "YcOFdWpnlxNf" - }, - "source": [ - "We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the GitHub Repo](https://github.com/huggingface/deep-rl-class/issues)." - ] - }, - { - "cell_type": "markdown", - "source": [ - "## Objectives of this notebook 🏆\n", - "\n", - "At the end of the notebook, you will:\n", - "\n", - "- Be able to **code your PPO agent from scratch using PyTorch**.\n", - "- Be able to **push your trained agent and the code to the Hub** with a nice video replay and an evaluation score 🔥.\n", - "\n", - "\n" - ], - "metadata": { - "id": "T6lIPYFghhYL" - } - }, - { - "cell_type": "markdown", - "source": [ - "## This notebook is from the Deep Reinforcement Learning Course\n", - "\"Deep\n", - "\n", - "In this free course, you will:\n", - "\n", - "- 📖 Study Deep Reinforcement Learning in **theory and practice**.\n", - "- 🧑‍💻 Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, CleanRL and Sample Factory 2.0.\n", - "- 🤖 Train **agents in unique environments**\n", - "\n", - "Don’t forget to **sign up to the course** (we are collecting your email to be able to **send you the links when each Unit is published and give you information about the challenges and updates).**\n", - "\n", - "\n", - "The best way to keep in touch is to join our discord server to exchange with the community and with us 👉🏻 https://discord.gg/ydHrjt3WP5" - ], - "metadata": { - "id": "Wp-rD6Fuhq31" - } - }, - { - "cell_type": "markdown", - "source": [ - "## Prerequisites 🏗️\n", - "Before diving into the notebook, you need to:\n", - "\n", - "🔲 📚 Study [PPO by reading Unit 8](https://huggingface.co/deep-rl-course/unit8/introduction) 🤗 " - ], - "metadata": { - "id": "rasqqGQlhujA" - } - }, - { - "cell_type": "markdown", - "source": [ - "To validate this hands-on for the [certification process](https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process), you need to push one model, we don't ask for a minimal result but we **advise you to try different hyperparameters settings to get better results**.\n", - "\n", - "If you don't find your model, **go to the bottom of the page and click on the refresh button**\n", - "\n", - "For more information about the certification process, check this section 👉 https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process" - ], - "metadata": { - "id": "PUFfMGOih3CW" - } - }, - { - "cell_type": "markdown", - "source": [ - "## Set the GPU 💪\n", - "- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`\n", - "\n", - "\"GPU" - ], - "metadata": { - "id": "PU4FVzaoM6fC" - } - }, - { - "cell_type": "markdown", - "source": [ - "- `Hardware Accelerator > GPU`\n", - "\n", - "\"GPU" - ], - "metadata": { - "id": "KV0NyFdQM9ZG" - } - }, - { - "cell_type": "markdown", - "source": [ - "## Create a virtual display 🔽\n", - "\n", - "During the notebook, we'll need to generate a replay video. To do so, with colab, **we need to have a virtual screen to be able to render the environment** (and thus record the frames).\n", - "\n", - "Hence the following cell will install the librairies and create and run a virtual screen 🖥" - ], - "metadata": { - "id": "bTpYcVZVMzUI" - } - }, - { - "cell_type": "code", - "source": [ - "!pip install setuptools==65.5.0" - ], - "metadata": { - "id": "Fd731S8-NuJA" - }, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "jV6wjQ7Be7p5" - }, - "outputs": [], - "source": [ - "%%capture\n", - "!apt install python-opengl\n", - "!apt install ffmpeg\n", - "!apt install xvfb\n", - "!apt install swig cmake\n", - "!pip install pyglet==1.5\n", - "!pip3 install pyvirtualdisplay" - ] - }, - { - "cell_type": "code", - "source": [ - "# Virtual display\n", - "from pyvirtualdisplay import Display\n", - "\n", - "virtual_display = Display(visible=0, size=(1400, 900))\n", - "virtual_display.start()" - ], - "metadata": { - "id": "ww5PQH1gNLI4" - }, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "ncIgfNf3mOtc" - }, - "source": [ - "## Install dependencies 🔽\n", - "For this exercise, we use `gym==0.22`." - ] - }, - { - "cell_type": "code", - "source": [ - "!pip install gym==0.22\n", - "!pip install imageio-ffmpeg\n", - "!pip install huggingface_hub\n", - "!pip install gym[box2d]==0.22" - ], - "metadata": { - "id": "9xZQFTPcsKUK" - }, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "oDkUufewmq6v" - }, - "source": [ - "## Let's code PPO from scratch with Costa Huang tutorial\n", - "- For the core implementation of PPO we're going to use the excellent [Costa Huang](https://costa.sh/) tutorial.\n", - "- In addition to the tutorial, to go deeper you can read the 37 core implementation details: https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/\n", - "\n", - "👉 The video tutorial: https://youtu.be/MEt6rrxH8W4" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "aNgEL1_uvhaq" - }, - "outputs": [], - "source": [ - "from IPython.display import HTML\n", - "\n", - "HTML('')" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "f34ILn7AvTbt" - }, - "source": [ - "- The best is to code first on the cell below, this way, if you kill the machine **you don't loose the implementation**." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "_bE708C6mhE7" - }, - "outputs": [], - "source": [ - "### Your code here:" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "mk-a9CmNuS2W" - }, - "source": [ - "## Add the Hugging Face Integration 🤗\n", - "- In order to push our model to the Hub, we need to define a function `package_to_hub`" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "TPi1Nme-oGWd" - }, - "source": [ - "- Add dependencies we need to push our model to the Hub" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "Sj8bz-AmoNVj" - }, - "outputs": [], - "source": [ - "from huggingface_hub import HfApi, upload_folder\n", - "from huggingface_hub.repocard import metadata_eval_result, metadata_save\n", - "\n", - "from pathlib import Path\n", - "import datetime\n", - "import tempfile\n", - "import json\n", - "import shutil\n", - "import imageio\n", - "\n", - "from wasabi import Printer\n", - "msg = Printer()" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "5rDr8-lWn0zi" - }, - "source": [ - "- Add new argument in `parse_args()` function to define the repo-id where we want to push the model." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "iHQiqQEFn0QH" - }, - "outputs": [], - "source": [ - "# Adding HuggingFace argument\n", - "parser.add_argument(\"--repo-id\", type=str, default=\"ThomasSimonini/ppo-CartPole-v1\", help=\"id of the model repository from the Hugging Face Hub {username/repo_name}\")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "blLZMiBAoUVT" - }, - "source": [ - "- Next, we add the methods needed to push the model to the Hub\n", - "\n", - "- These methods will:\n", - " - `_evalutate_agent()`: evaluate the agent.\n", - " - `_generate_model_card()`: generate the model card of your agent.\n", - " - `_record_video()`: record a video of your agent." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "WlLcz4L9odXs" - }, - "outputs": [], - "source": [ - "def package_to_hub(repo_id,\n", - " model,\n", - " hyperparameters,\n", - " eval_env,\n", - " video_fps=30,\n", - " commit_message=\"Push agent to the Hub\",\n", - " token= None,\n", - " logs=None\n", - " ):\n", - " \"\"\"\n", - " Evaluate, Generate a video and Upload a model to Hugging Face Hub.\n", - " This method does the complete pipeline:\n", - " - It evaluates the model\n", - " - It generates the model card\n", - " - It generates a replay video of the agent\n", - " - It pushes everything to the hub\n", - " :param repo_id: id of the model repository from the Hugging Face Hub\n", - " :param model: trained model\n", - " :param eval_env: environment used to evaluate the agent\n", - " :param fps: number of fps for rendering the video\n", - " :param commit_message: commit message\n", - " :param logs: directory on local machine of tensorboard logs you'd like to upload\n", - " \"\"\"\n", - " msg.info(\n", - " \"This function will save, evaluate, generate a video of your agent, \"\n", - " \"create a model card and push everything to the hub. \"\n", - " \"It might take up to 1min. \\n \"\n", - " \"This is a work in progress: if you encounter a bug, please open an issue.\"\n", - " )\n", - " # Step 1: Clone or create the repo\n", - " repo_url = HfApi().create_repo(\n", - " repo_id=repo_id,\n", - " token=token,\n", - " private=False,\n", - " exist_ok=True,\n", - " )\n", - "\n", - " with tempfile.TemporaryDirectory() as tmpdirname:\n", - " tmpdirname = Path(tmpdirname)\n", - "\n", - " # Step 2: Save the model\n", - " torch.save(model.state_dict(), tmpdirname / \"model.pt\")\n", - "\n", - " # Step 3: Evaluate the model and build JSON\n", - " mean_reward, std_reward = _evaluate_agent(eval_env,\n", - " 10,\n", - " model)\n", - "\n", - " # First get datetime\n", - " eval_datetime = datetime.datetime.now()\n", - " eval_form_datetime = eval_datetime.isoformat()\n", - "\n", - " evaluate_data = {\n", - " \"env_id\": hyperparameters.env_id,\n", - " \"mean_reward\": mean_reward,\n", - " \"std_reward\": std_reward,\n", - " \"n_evaluation_episodes\": 10,\n", - " \"eval_datetime\": eval_form_datetime,\n", - " }\n", - "\n", - " # Write a JSON file\n", - " with open(tmpdirname / \"results.json\", \"w\") as outfile:\n", - " json.dump(evaluate_data, outfile)\n", - "\n", - " # Step 4: Generate a video\n", - " video_path = tmpdirname / \"replay.mp4\"\n", - " record_video(eval_env, model, video_path, video_fps)\n", - "\n", - " # Step 5: Generate the model card\n", - " generated_model_card, metadata = _generate_model_card(\"PPO\", hyperparameters.env_id, mean_reward, std_reward, hyperparameters)\n", - " _save_model_card(tmpdirname, generated_model_card, metadata)\n", - "\n", - " # Step 6: Add logs if needed\n", - " if logs:\n", - " _add_logdir(tmpdirname, Path(logs))\n", - "\n", - " msg.info(f\"Pushing repo {repo_id} to the Hugging Face Hub\")\n", - "\n", - " repo_url = upload_folder(\n", - " repo_id=repo_id,\n", - " folder_path=tmpdirname,\n", - " path_in_repo=\"\",\n", - " commit_message=commit_message,\n", - " token=token,\n", - " )\n", - "\n", - " msg.info(f\"Your model is pushed to the Hub. You can view your model here: {repo_url}\")\n", - " return repo_url\n", - "\n", - "\n", - "def _evaluate_agent(env, n_eval_episodes, policy):\n", - " \"\"\"\n", - " Evaluate the agent for ``n_eval_episodes`` episodes and returns average reward and std of reward.\n", - " :param env: The evaluation environment\n", - " :param n_eval_episodes: Number of episode to evaluate the agent\n", - " :param policy: The agent\n", - " \"\"\"\n", - " episode_rewards = []\n", - " for episode in range(n_eval_episodes):\n", - " state = env.reset()\n", - " step = 0\n", - " done = False\n", - " total_rewards_ep = 0\n", - "\n", - " while done is False:\n", - " state = torch.Tensor(state).to(device)\n", - " action, _, _, _ = policy.get_action_and_value(state)\n", - " new_state, reward, done, info = env.step(action.cpu().numpy())\n", - " total_rewards_ep += reward\n", - " if done:\n", - " break\n", - " state = new_state\n", - " episode_rewards.append(total_rewards_ep)\n", - " mean_reward = np.mean(episode_rewards)\n", - " std_reward = np.std(episode_rewards)\n", - "\n", - " return mean_reward, std_reward\n", - "\n", - "\n", - "def record_video(env, policy, out_directory, fps=30):\n", - " images = []\n", - " done = False\n", - " state = env.reset()\n", - " img = env.render(mode='rgb_array')\n", - " images.append(img)\n", - " while not done:\n", - " state = torch.Tensor(state).to(device)\n", - " # Take the action (index) that have the maximum expected future reward given that state\n", - " action, _, _, _ = policy.get_action_and_value(state)\n", - " state, reward, done, info = env.step(action.cpu().numpy()) # We directly put next_state = state for recording logic\n", - " img = env.render(mode='rgb_array')\n", - " images.append(img)\n", - " imageio.mimsave(out_directory, [np.array(img) for i, img in enumerate(images)], fps=fps)\n", - "\n", - "\n", - "def _generate_model_card(model_name, env_id, mean_reward, std_reward, hyperparameters):\n", - " \"\"\"\n", - " Generate the model card for the Hub\n", - " :param model_name: name of the model\n", - " :env_id: name of the environment\n", - " :mean_reward: mean reward of the agent\n", - " :std_reward: standard deviation of the mean reward of the agent\n", - " :hyperparameters: training arguments\n", - " \"\"\"\n", - " # Step 1: Select the tags\n", - " metadata = generate_metadata(model_name, env_id, mean_reward, std_reward)\n", - "\n", - " # Transform the hyperparams namespace to string\n", - " converted_dict = vars(hyperparameters)\n", - " converted_str = str(converted_dict)\n", - " converted_str = converted_str.split(\", \")\n", - " converted_str = '\\n'.join(converted_str)\n", - "\n", - " # Step 2: Generate the model card\n", - " model_card = f\"\"\"\n", - " # PPO Agent Playing {env_id}\n", - "\n", - " This is a trained model of a PPO agent playing {env_id}.\n", - "\n", - " # Hyperparameters\n", - " ```python\n", - " {converted_str}\n", - " ```\n", - " \"\"\"\n", - " return model_card, metadata\n", - "\n", - "\n", - "def generate_metadata(model_name, env_id, mean_reward, std_reward):\n", - " \"\"\"\n", - " Define the tags for the model card\n", - " :param model_name: name of the model\n", - " :param env_id: name of the environment\n", - " :mean_reward: mean reward of the agent\n", - " :std_reward: standard deviation of the mean reward of the agent\n", - " \"\"\"\n", - " metadata = {}\n", - " metadata[\"tags\"] = [\n", - " env_id,\n", - " \"ppo\",\n", - " \"deep-reinforcement-learning\",\n", - " \"reinforcement-learning\",\n", - " \"custom-implementation\",\n", - " \"deep-rl-course\"\n", - " ]\n", - "\n", - " # Add metrics\n", - " eval = metadata_eval_result(\n", - " model_pretty_name=model_name,\n", - " task_pretty_name=\"reinforcement-learning\",\n", - " task_id=\"reinforcement-learning\",\n", - " metrics_pretty_name=\"mean_reward\",\n", - " metrics_id=\"mean_reward\",\n", - " metrics_value=f\"{mean_reward:.2f} +/- {std_reward:.2f}\",\n", - " dataset_pretty_name=env_id,\n", - " dataset_id=env_id,\n", - " )\n", - "\n", - " # Merges both dictionaries\n", - " metadata = {**metadata, **eval}\n", - "\n", - " return metadata\n", - "\n", - "\n", - "def _save_model_card(local_path, generated_model_card, metadata):\n", - " \"\"\"Saves a model card for the repository.\n", - " :param local_path: repository directory\n", - " :param generated_model_card: model card generated by _generate_model_card()\n", - " :param metadata: metadata\n", - " \"\"\"\n", - " readme_path = local_path / \"README.md\"\n", - " readme = \"\"\n", - " if readme_path.exists():\n", - " with readme_path.open(\"r\", encoding=\"utf8\") as f:\n", - " readme = f.read()\n", - " else:\n", - " readme = generated_model_card\n", - "\n", - " with readme_path.open(\"w\", encoding=\"utf-8\") as f:\n", - " f.write(readme)\n", - "\n", - " # Save our metrics to Readme metadata\n", - " metadata_save(readme_path, metadata)\n", - "\n", - "\n", - "def _add_logdir(local_path: Path, logdir: Path):\n", - " \"\"\"Adds a logdir to the repository.\n", - " :param local_path: repository directory\n", - " :param logdir: logdir directory\n", - " \"\"\"\n", - " if logdir.exists() and logdir.is_dir():\n", - " # Add the logdir to the repository under new dir called logs\n", - " repo_logdir = local_path / \"logs\"\n", - "\n", - " # Delete current logs if they exist\n", - " if repo_logdir.exists():\n", - " shutil.rmtree(repo_logdir)\n", - "\n", - " # Copy logdir into repo logdir\n", - " shutil.copytree(logdir, repo_logdir)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "TqX8z8_rooD6" - }, - "source": [ - "- Finally, we call this function at the end of the PPO training" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "I8V1vNiTo2hL" - }, - "outputs": [], - "source": [ - "# Create the evaluation environment\n", - "eval_env = gym.make(args.env_id)\n", - "\n", - "package_to_hub(repo_id = args.repo_id,\n", - " model = agent, # The model we want to save\n", - " hyperparameters = args,\n", - " eval_env = gym.make(args.env_id),\n", - " logs= f\"runs/{run_name}\",\n", - " )" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "muCCzed4o5TC" - }, - "source": [ - "- Here's what look the ppo.py final file" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "LviRdtXgo7kF" - }, - "outputs": [], - "source": [ - "# docs and experiment results can be found at https://docs.cleanrl.dev/rl-algorithms/ppo/#ppopy\n", - "\n", - "import argparse\n", - "import os\n", - "import random\n", - "import time\n", - "from distutils.util import strtobool\n", - "\n", - "import gym\n", - "import numpy as np\n", - "import torch\n", - "import torch.nn as nn\n", - "import torch.optim as optim\n", - "from torch.distributions.categorical import Categorical\n", - "from torch.utils.tensorboard import SummaryWriter\n", - "\n", - "from huggingface_hub import HfApi, upload_folder\n", - "from huggingface_hub.repocard import metadata_eval_result, metadata_save\n", - "\n", - "from pathlib import Path\n", - "import datetime\n", - "import tempfile\n", - "import json\n", - "import shutil\n", - "import imageio\n", - "\n", - "from wasabi import Printer\n", - "msg = Printer()\n", - "\n", - "def parse_args():\n", - " # fmt: off\n", - " parser = argparse.ArgumentParser()\n", - " parser.add_argument(\"--exp-name\", type=str, default=os.path.basename(__file__).rstrip(\".py\"),\n", - " help=\"the name of this experiment\")\n", - " parser.add_argument(\"--seed\", type=int, default=1,\n", - " help=\"seed of the experiment\")\n", - " parser.add_argument(\"--torch-deterministic\", type=lambda x: bool(strtobool(x)), default=True, nargs=\"?\", const=True,\n", - " help=\"if toggled, `torch.backends.cudnn.deterministic=False`\")\n", - " parser.add_argument(\"--cuda\", type=lambda x: bool(strtobool(x)), default=True, nargs=\"?\", const=True,\n", - " help=\"if toggled, cuda will be enabled by default\")\n", - " parser.add_argument(\"--track\", type=lambda x: bool(strtobool(x)), default=False, nargs=\"?\", const=True,\n", - " help=\"if toggled, this experiment will be tracked with Weights and Biases\")\n", - " parser.add_argument(\"--wandb-project-name\", type=str, default=\"cleanRL\",\n", - " help=\"the wandb's project name\")\n", - " parser.add_argument(\"--wandb-entity\", type=str, default=None,\n", - " help=\"the entity (team) of wandb's project\")\n", - " parser.add_argument(\"--capture-video\", type=lambda x: bool(strtobool(x)), default=False, nargs=\"?\", const=True,\n", - " help=\"weather to capture videos of the agent performances (check out `videos` folder)\")\n", - "\n", - " # Algorithm specific arguments\n", - " parser.add_argument(\"--env-id\", type=str, default=\"CartPole-v1\",\n", - " help=\"the id of the environment\")\n", - " parser.add_argument(\"--total-timesteps\", type=int, default=50000,\n", - " help=\"total timesteps of the experiments\")\n", - " parser.add_argument(\"--learning-rate\", type=float, default=2.5e-4,\n", - " help=\"the learning rate of the optimizer\")\n", - " parser.add_argument(\"--num-envs\", type=int, default=4,\n", - " help=\"the number of parallel game environments\")\n", - " parser.add_argument(\"--num-steps\", type=int, default=128,\n", - " help=\"the number of steps to run in each environment per policy rollout\")\n", - " parser.add_argument(\"--anneal-lr\", type=lambda x: bool(strtobool(x)), default=True, nargs=\"?\", const=True,\n", - " help=\"Toggle learning rate annealing for policy and value networks\")\n", - " parser.add_argument(\"--gae\", type=lambda x: bool(strtobool(x)), default=True, nargs=\"?\", const=True,\n", - " help=\"Use GAE for advantage computation\")\n", - " parser.add_argument(\"--gamma\", type=float, default=0.99,\n", - " help=\"the discount factor gamma\")\n", - " parser.add_argument(\"--gae-lambda\", type=float, default=0.95,\n", - " help=\"the lambda for the general advantage estimation\")\n", - " parser.add_argument(\"--num-minibatches\", type=int, default=4,\n", - " help=\"the number of mini-batches\")\n", - " parser.add_argument(\"--update-epochs\", type=int, default=4,\n", - " help=\"the K epochs to update the policy\")\n", - " parser.add_argument(\"--norm-adv\", type=lambda x: bool(strtobool(x)), default=True, nargs=\"?\", const=True,\n", - " help=\"Toggles advantages normalization\")\n", - " parser.add_argument(\"--clip-coef\", type=float, default=0.2,\n", - " help=\"the surrogate clipping coefficient\")\n", - " parser.add_argument(\"--clip-vloss\", type=lambda x: bool(strtobool(x)), default=True, nargs=\"?\", const=True,\n", - " help=\"Toggles whether or not to use a clipped loss for the value function, as per the paper.\")\n", - " parser.add_argument(\"--ent-coef\", type=float, default=0.01,\n", - " help=\"coefficient of the entropy\")\n", - " parser.add_argument(\"--vf-coef\", type=float, default=0.5,\n", - " help=\"coefficient of the value function\")\n", - " parser.add_argument(\"--max-grad-norm\", type=float, default=0.5,\n", - " help=\"the maximum norm for the gradient clipping\")\n", - " parser.add_argument(\"--target-kl\", type=float, default=None,\n", - " help=\"the target KL divergence threshold\")\n", - "\n", - " # Adding HuggingFace argument\n", - " parser.add_argument(\"--repo-id\", type=str, default=\"ThomasSimonini/ppo-CartPole-v1\", help=\"id of the model repository from the Hugging Face Hub {username/repo_name}\")\n", - "\n", - " args = parser.parse_args()\n", - " args.batch_size = int(args.num_envs * args.num_steps)\n", - " args.minibatch_size = int(args.batch_size // args.num_minibatches)\n", - " # fmt: on\n", - " return args\n", - "\n", - "def package_to_hub(repo_id,\n", - " model,\n", - " hyperparameters,\n", - " eval_env,\n", - " video_fps=30,\n", - " commit_message=\"Push agent to the Hub\",\n", - " token= None,\n", - " logs=None\n", - " ):\n", - " \"\"\"\n", - " Evaluate, Generate a video and Upload a model to Hugging Face Hub.\n", - " This method does the complete pipeline:\n", - " - It evaluates the model\n", - " - It generates the model card\n", - " - It generates a replay video of the agent\n", - " - It pushes everything to the hub\n", - " :param repo_id: id of the model repository from the Hugging Face Hub\n", - " :param model: trained model\n", - " :param eval_env: environment used to evaluate the agent\n", - " :param fps: number of fps for rendering the video\n", - " :param commit_message: commit message\n", - " :param logs: directory on local machine of tensorboard logs you'd like to upload\n", - " \"\"\"\n", - " msg.info(\n", - " \"This function will save, evaluate, generate a video of your agent, \"\n", - " \"create a model card and push everything to the hub. \"\n", - " \"It might take up to 1min. \\n \"\n", - " \"This is a work in progress: if you encounter a bug, please open an issue.\"\n", - " )\n", - " # Step 1: Clone or create the repo\n", - " repo_url = HfApi().create_repo(\n", - " repo_id=repo_id,\n", - " token=token,\n", - " private=False,\n", - " exist_ok=True,\n", - " )\n", - "\n", - " with tempfile.TemporaryDirectory() as tmpdirname:\n", - " tmpdirname = Path(tmpdirname)\n", - "\n", - " # Step 2: Save the model\n", - " torch.save(model.state_dict(), tmpdirname / \"model.pt\")\n", - "\n", - " # Step 3: Evaluate the model and build JSON\n", - " mean_reward, std_reward = _evaluate_agent(eval_env,\n", - " 10,\n", - " model)\n", - "\n", - " # First get datetime\n", - " eval_datetime = datetime.datetime.now()\n", - " eval_form_datetime = eval_datetime.isoformat()\n", - "\n", - " evaluate_data = {\n", - " \"env_id\": hyperparameters.env_id,\n", - " \"mean_reward\": mean_reward,\n", - " \"std_reward\": std_reward,\n", - " \"n_evaluation_episodes\": 10,\n", - " \"eval_datetime\": eval_form_datetime,\n", - " }\n", - "\n", - " # Write a JSON file\n", - " with open(tmpdirname / \"results.json\", \"w\") as outfile:\n", - " json.dump(evaluate_data, outfile)\n", - "\n", - " # Step 4: Generate a video\n", - " video_path = tmpdirname / \"replay.mp4\"\n", - " record_video(eval_env, model, video_path, video_fps)\n", - "\n", - " # Step 5: Generate the model card\n", - " generated_model_card, metadata = _generate_model_card(\"PPO\", hyperparameters.env_id, mean_reward, std_reward, hyperparameters)\n", - " _save_model_card(tmpdirname, generated_model_card, metadata)\n", - "\n", - " # Step 6: Add logs if needed\n", - " if logs:\n", - " _add_logdir(tmpdirname, Path(logs))\n", - "\n", - " msg.info(f\"Pushing repo {repo_id} to the Hugging Face Hub\")\n", - "\n", - " repo_url = upload_folder(\n", - " repo_id=repo_id,\n", - " folder_path=tmpdirname,\n", - " path_in_repo=\"\",\n", - " commit_message=commit_message,\n", - " token=token,\n", - " )\n", - "\n", - " msg.info(f\"Your model is pushed to the Hub. You can view your model here: {repo_url}\")\n", - " return repo_url\n", - "\n", - "def _evaluate_agent(env, n_eval_episodes, policy):\n", - " \"\"\"\n", - " Evaluate the agent for ``n_eval_episodes`` episodes and returns average reward and std of reward.\n", - " :param env: The evaluation environment\n", - " :param n_eval_episodes: Number of episode to evaluate the agent\n", - " :param policy: The agent\n", - " \"\"\"\n", - " episode_rewards = []\n", - " for episode in range(n_eval_episodes):\n", - " state = env.reset()\n", - " step = 0\n", - " done = False\n", - " total_rewards_ep = 0\n", - "\n", - " while done is False:\n", - " state = torch.Tensor(state).to(device)\n", - " action, _, _, _ = policy.get_action_and_value(state)\n", - " new_state, reward, done, info = env.step(action.cpu().numpy())\n", - " total_rewards_ep += reward\n", - " if done:\n", - " break\n", - " state = new_state\n", - " episode_rewards.append(total_rewards_ep)\n", - " mean_reward = np.mean(episode_rewards)\n", - " std_reward = np.std(episode_rewards)\n", - "\n", - " return mean_reward, std_reward\n", - "\n", - "\n", - "def record_video(env, policy, out_directory, fps=30):\n", - " images = []\n", - " done = False\n", - " state = env.reset()\n", - " img = env.render(mode='rgb_array')\n", - " images.append(img)\n", - " while not done:\n", - " state = torch.Tensor(state).to(device)\n", - " # Take the action (index) that have the maximum expected future reward given that state\n", - " action, _, _, _ = policy.get_action_and_value(state)\n", - " state, reward, done, info = env.step(action.cpu().numpy()) # We directly put next_state = state for recording logic\n", - " img = env.render(mode='rgb_array')\n", - " images.append(img)\n", - " imageio.mimsave(out_directory, [np.array(img) for i, img in enumerate(images)], fps=fps)\n", - "\n", - "\n", - "def _generate_model_card(model_name, env_id, mean_reward, std_reward, hyperparameters):\n", - " \"\"\"\n", - " Generate the model card for the Hub\n", - " :param model_name: name of the model\n", - " :env_id: name of the environment\n", - " :mean_reward: mean reward of the agent\n", - " :std_reward: standard deviation of the mean reward of the agent\n", - " :hyperparameters: training arguments\n", - " \"\"\"\n", - " # Step 1: Select the tags\n", - " metadata = generate_metadata(model_name, env_id, mean_reward, std_reward)\n", - "\n", - " # Transform the hyperparams namespace to string\n", - " converted_dict = vars(hyperparameters)\n", - " converted_str = str(converted_dict)\n", - " converted_str = converted_str.split(\", \")\n", - " converted_str = '\\n'.join(converted_str)\n", - "\n", - " # Step 2: Generate the model card\n", - " model_card = f\"\"\"\n", - " # PPO Agent Playing {env_id}\n", - "\n", - " This is a trained model of a PPO agent playing {env_id}.\n", - "\n", - " # Hyperparameters\n", - " ```python\n", - " {converted_str}\n", - " ```\n", - " \"\"\"\n", - " return model_card, metadata\n", - "\n", - "def generate_metadata(model_name, env_id, mean_reward, std_reward):\n", - " \"\"\"\n", - " Define the tags for the model card\n", - " :param model_name: name of the model\n", - " :param env_id: name of the environment\n", - " :mean_reward: mean reward of the agent\n", - " :std_reward: standard deviation of the mean reward of the agent\n", - " \"\"\"\n", - " metadata = {}\n", - " metadata[\"tags\"] = [\n", - " env_id,\n", - " \"ppo\",\n", - " \"deep-reinforcement-learning\",\n", - " \"reinforcement-learning\",\n", - " \"custom-implementation\",\n", - " \"deep-rl-course\"\n", - " ]\n", - "\n", - " # Add metrics\n", - " eval = metadata_eval_result(\n", - " model_pretty_name=model_name,\n", - " task_pretty_name=\"reinforcement-learning\",\n", - " task_id=\"reinforcement-learning\",\n", - " metrics_pretty_name=\"mean_reward\",\n", - " metrics_id=\"mean_reward\",\n", - " metrics_value=f\"{mean_reward:.2f} +/- {std_reward:.2f}\",\n", - " dataset_pretty_name=env_id,\n", - " dataset_id=env_id,\n", - " )\n", - "\n", - " # Merges both dictionaries\n", - " metadata = {**metadata, **eval}\n", - "\n", - " return metadata\n", - "\n", - "def _save_model_card(local_path, generated_model_card, metadata):\n", - " \"\"\"Saves a model card for the repository.\n", - " :param local_path: repository directory\n", - " :param generated_model_card: model card generated by _generate_model_card()\n", - " :param metadata: metadata\n", - " \"\"\"\n", - " readme_path = local_path / \"README.md\"\n", - " readme = \"\"\n", - " if readme_path.exists():\n", - " with readme_path.open(\"r\", encoding=\"utf8\") as f:\n", - " readme = f.read()\n", - " else:\n", - " readme = generated_model_card\n", - "\n", - " with readme_path.open(\"w\", encoding=\"utf-8\") as f:\n", - " f.write(readme)\n", - "\n", - " # Save our metrics to Readme metadata\n", - " metadata_save(readme_path, metadata)\n", - "\n", - "def _add_logdir(local_path: Path, logdir: Path):\n", - " \"\"\"Adds a logdir to the repository.\n", - " :param local_path: repository directory\n", - " :param logdir: logdir directory\n", - " \"\"\"\n", - " if logdir.exists() and logdir.is_dir():\n", - " # Add the logdir to the repository under new dir called logs\n", - " repo_logdir = local_path / \"logs\"\n", - "\n", - " # Delete current logs if they exist\n", - " if repo_logdir.exists():\n", - " shutil.rmtree(repo_logdir)\n", - "\n", - " # Copy logdir into repo logdir\n", - " shutil.copytree(logdir, repo_logdir)\n", - "\n", - "def make_env(env_id, seed, idx, capture_video, run_name):\n", - " def thunk():\n", - " env = gym.make(env_id)\n", - " env = gym.wrappers.RecordEpisodeStatistics(env)\n", - " if capture_video:\n", - " if idx == 0:\n", - " env = gym.wrappers.RecordVideo(env, f\"videos/{run_name}\")\n", - " env.seed(seed)\n", - " env.action_space.seed(seed)\n", - " env.observation_space.seed(seed)\n", - " return env\n", - "\n", - " return thunk\n", - "\n", - "\n", - "def layer_init(layer, std=np.sqrt(2), bias_const=0.0):\n", - " torch.nn.init.orthogonal_(layer.weight, std)\n", - " torch.nn.init.constant_(layer.bias, bias_const)\n", - " return layer\n", - "\n", - "\n", - "class Agent(nn.Module):\n", - " def __init__(self, envs):\n", - " super().__init__()\n", - " self.critic = nn.Sequential(\n", - " layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 64)),\n", - " nn.Tanh(),\n", - " layer_init(nn.Linear(64, 64)),\n", - " nn.Tanh(),\n", - " layer_init(nn.Linear(64, 1), std=1.0),\n", - " )\n", - " self.actor = nn.Sequential(\n", - " layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 64)),\n", - " nn.Tanh(),\n", - " layer_init(nn.Linear(64, 64)),\n", - " nn.Tanh(),\n", - " layer_init(nn.Linear(64, envs.single_action_space.n), std=0.01),\n", - " )\n", - "\n", - " def get_value(self, x):\n", - " return self.critic(x)\n", - "\n", - " def get_action_and_value(self, x, action=None):\n", - " logits = self.actor(x)\n", - " probs = Categorical(logits=logits)\n", - " if action is None:\n", - " action = probs.sample()\n", - " return action, probs.log_prob(action), probs.entropy(), self.critic(x)\n", - "\n", - "\n", - "if __name__ == \"__main__\":\n", - " args = parse_args()\n", - " run_name = f\"{args.env_id}__{args.exp_name}__{args.seed}__{int(time.time())}\"\n", - " if args.track:\n", - " import wandb\n", - "\n", - " wandb.init(\n", - " project=args.wandb_project_name,\n", - " entity=args.wandb_entity,\n", - " sync_tensorboard=True,\n", - " config=vars(args),\n", - " name=run_name,\n", - " monitor_gym=True,\n", - " save_code=True,\n", - " )\n", - " writer = SummaryWriter(f\"runs/{run_name}\")\n", - " writer.add_text(\n", - " \"hyperparameters\",\n", - " \"|param|value|\\n|-|-|\\n%s\" % (\"\\n\".join([f\"|{key}|{value}|\" for key, value in vars(args).items()])),\n", - " )\n", - "\n", - " # TRY NOT TO MODIFY: seeding\n", - " random.seed(args.seed)\n", - " np.random.seed(args.seed)\n", - " torch.manual_seed(args.seed)\n", - " torch.backends.cudnn.deterministic = args.torch_deterministic\n", - "\n", - " device = torch.device(\"cuda\" if torch.cuda.is_available() and args.cuda else \"cpu\")\n", - "\n", - " # env setup\n", - " envs = gym.vector.SyncVectorEnv(\n", - " [make_env(args.env_id, args.seed + i, i, args.capture_video, run_name) for i in range(args.num_envs)]\n", - " )\n", - " assert isinstance(envs.single_action_space, gym.spaces.Discrete), \"only discrete action space is supported\"\n", - "\n", - " agent = Agent(envs).to(device)\n", - " optimizer = optim.Adam(agent.parameters(), lr=args.learning_rate, eps=1e-5)\n", - "\n", - " # ALGO Logic: Storage setup\n", - " obs = torch.zeros((args.num_steps, args.num_envs) + envs.single_observation_space.shape).to(device)\n", - " actions = torch.zeros((args.num_steps, args.num_envs) + envs.single_action_space.shape).to(device)\n", - " logprobs = torch.zeros((args.num_steps, args.num_envs)).to(device)\n", - " rewards = torch.zeros((args.num_steps, args.num_envs)).to(device)\n", - " dones = torch.zeros((args.num_steps, args.num_envs)).to(device)\n", - " values = torch.zeros((args.num_steps, args.num_envs)).to(device)\n", - "\n", - " # TRY NOT TO MODIFY: start the game\n", - " global_step = 0\n", - " start_time = time.time()\n", - " next_obs = torch.Tensor(envs.reset()).to(device)\n", - " next_done = torch.zeros(args.num_envs).to(device)\n", - " num_updates = args.total_timesteps // args.batch_size\n", - "\n", - " for update in range(1, num_updates + 1):\n", - " # Annealing the rate if instructed to do so.\n", - " if args.anneal_lr:\n", - " frac = 1.0 - (update - 1.0) / num_updates\n", - " lrnow = frac * args.learning_rate\n", - " optimizer.param_groups[0][\"lr\"] = lrnow\n", - "\n", - " for step in range(0, args.num_steps):\n", - " global_step += 1 * args.num_envs\n", - " obs[step] = next_obs\n", - " dones[step] = next_done\n", - "\n", - " # ALGO LOGIC: action logic\n", - " with torch.no_grad():\n", - " action, logprob, _, value = agent.get_action_and_value(next_obs)\n", - " values[step] = value.flatten()\n", - " actions[step] = action\n", - " logprobs[step] = logprob\n", - "\n", - " # TRY NOT TO MODIFY: execute the game and log data.\n", - " next_obs, reward, done, info = envs.step(action.cpu().numpy())\n", - " rewards[step] = torch.tensor(reward).to(device).view(-1)\n", - " next_obs, next_done = torch.Tensor(next_obs).to(device), torch.Tensor(done).to(device)\n", - "\n", - " for item in info:\n", - " if \"episode\" in item.keys():\n", - " print(f\"global_step={global_step}, episodic_return={item['episode']['r']}\")\n", - " writer.add_scalar(\"charts/episodic_return\", item[\"episode\"][\"r\"], global_step)\n", - " writer.add_scalar(\"charts/episodic_length\", item[\"episode\"][\"l\"], global_step)\n", - " break\n", - "\n", - " # bootstrap value if not done\n", - " with torch.no_grad():\n", - " next_value = agent.get_value(next_obs).reshape(1, -1)\n", - " if args.gae:\n", - " advantages = torch.zeros_like(rewards).to(device)\n", - " lastgaelam = 0\n", - " for t in reversed(range(args.num_steps)):\n", - " if t == args.num_steps - 1:\n", - " nextnonterminal = 1.0 - next_done\n", - " nextvalues = next_value\n", - " else:\n", - " nextnonterminal = 1.0 - dones[t + 1]\n", - " nextvalues = values[t + 1]\n", - " delta = rewards[t] + args.gamma * nextvalues * nextnonterminal - values[t]\n", - " advantages[t] = lastgaelam = delta + args.gamma * args.gae_lambda * nextnonterminal * lastgaelam\n", - " returns = advantages + values\n", - " else:\n", - " returns = torch.zeros_like(rewards).to(device)\n", - " for t in reversed(range(args.num_steps)):\n", - " if t == args.num_steps - 1:\n", - " nextnonterminal = 1.0 - next_done\n", - " next_return = next_value\n", - " else:\n", - " nextnonterminal = 1.0 - dones[t + 1]\n", - " next_return = returns[t + 1]\n", - " returns[t] = rewards[t] + args.gamma * nextnonterminal * next_return\n", - " advantages = returns - values\n", - "\n", - " # flatten the batch\n", - " b_obs = obs.reshape((-1,) + envs.single_observation_space.shape)\n", - " b_logprobs = logprobs.reshape(-1)\n", - " b_actions = actions.reshape((-1,) + envs.single_action_space.shape)\n", - " b_advantages = advantages.reshape(-1)\n", - " b_returns = returns.reshape(-1)\n", - " b_values = values.reshape(-1)\n", - "\n", - " # Optimizing the policy and value network\n", - " b_inds = np.arange(args.batch_size)\n", - " clipfracs = []\n", - " for epoch in range(args.update_epochs):\n", - " np.random.shuffle(b_inds)\n", - " for start in range(0, args.batch_size, args.minibatch_size):\n", - " end = start + args.minibatch_size\n", - " mb_inds = b_inds[start:end]\n", - "\n", - " _, newlogprob, entropy, newvalue = agent.get_action_and_value(b_obs[mb_inds], b_actions.long()[mb_inds])\n", - " logratio = newlogprob - b_logprobs[mb_inds]\n", - " ratio = logratio.exp()\n", - "\n", - " with torch.no_grad():\n", - " # calculate approx_kl http://joschu.net/blog/kl-approx.html\n", - " old_approx_kl = (-logratio).mean()\n", - " approx_kl = ((ratio - 1) - logratio).mean()\n", - " clipfracs += [((ratio - 1.0).abs() > args.clip_coef).float().mean().item()]\n", - "\n", - " mb_advantages = b_advantages[mb_inds]\n", - " if args.norm_adv:\n", - " mb_advantages = (mb_advantages - mb_advantages.mean()) / (mb_advantages.std() + 1e-8)\n", - "\n", - " # Policy loss\n", - " pg_loss1 = -mb_advantages * ratio\n", - " pg_loss2 = -mb_advantages * torch.clamp(ratio, 1 - args.clip_coef, 1 + args.clip_coef)\n", - " pg_loss = torch.max(pg_loss1, pg_loss2).mean()\n", - "\n", - " # Value loss\n", - " newvalue = newvalue.view(-1)\n", - " if args.clip_vloss:\n", - " v_loss_unclipped = (newvalue - b_returns[mb_inds]) ** 2\n", - " v_clipped = b_values[mb_inds] + torch.clamp(\n", - " newvalue - b_values[mb_inds],\n", - " -args.clip_coef,\n", - " args.clip_coef,\n", - " )\n", - " v_loss_clipped = (v_clipped - b_returns[mb_inds]) ** 2\n", - " v_loss_max = torch.max(v_loss_unclipped, v_loss_clipped)\n", - " v_loss = 0.5 * v_loss_max.mean()\n", - " else:\n", - " v_loss = 0.5 * ((newvalue - b_returns[mb_inds]) ** 2).mean()\n", - "\n", - " entropy_loss = entropy.mean()\n", - " loss = pg_loss - args.ent_coef * entropy_loss + v_loss * args.vf_coef\n", - "\n", - " optimizer.zero_grad()\n", - " loss.backward()\n", - " nn.utils.clip_grad_norm_(agent.parameters(), args.max_grad_norm)\n", - " optimizer.step()\n", - "\n", - " if args.target_kl is not None:\n", - " if approx_kl > args.target_kl:\n", - " break\n", - "\n", - " y_pred, y_true = b_values.cpu().numpy(), b_returns.cpu().numpy()\n", - " var_y = np.var(y_true)\n", - " explained_var = np.nan if var_y == 0 else 1 - np.var(y_true - y_pred) / var_y\n", - "\n", - " # TRY NOT TO MODIFY: record rewards for plotting purposes\n", - " writer.add_scalar(\"charts/learning_rate\", optimizer.param_groups[0][\"lr\"], global_step)\n", - " writer.add_scalar(\"losses/value_loss\", v_loss.item(), global_step)\n", - " writer.add_scalar(\"losses/policy_loss\", pg_loss.item(), global_step)\n", - " writer.add_scalar(\"losses/entropy\", entropy_loss.item(), global_step)\n", - " writer.add_scalar(\"losses/old_approx_kl\", old_approx_kl.item(), global_step)\n", - " writer.add_scalar(\"losses/approx_kl\", approx_kl.item(), global_step)\n", - " writer.add_scalar(\"losses/clipfrac\", np.mean(clipfracs), global_step)\n", - " writer.add_scalar(\"losses/explained_variance\", explained_var, global_step)\n", - " print(\"SPS:\", int(global_step / (time.time() - start_time)))\n", - " writer.add_scalar(\"charts/SPS\", int(global_step / (time.time() - start_time)), global_step)\n", - "\n", - " envs.close()\n", - " writer.close()\n", - "\n", - " # Create the evaluation environment\n", - " eval_env = gym.make(args.env_id)\n", - "\n", - " package_to_hub(repo_id = args.repo_id,\n", - " model = agent, # The model we want to save\n", - " hyperparameters = args,\n", - " eval_env = gym.make(args.env_id),\n", - " logs= f\"runs/{run_name}\",\n", - " )\n", - "" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "JquRrWytA6eo" - }, - "source": [ - "To be able to share your model with the community there are three more steps to follow:\n", - "\n", - "1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join\n", - "\n", - "2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website.\n", - "- Create a new token (https://huggingface.co/settings/tokens) **with write role**\n", - "\n", - "\"Create\n", - "\n", - "- Copy the token\n", - "- Run the cell below and paste the token" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "GZiFBBlzxzxY" - }, - "outputs": [], - "source": [ - "from huggingface_hub import notebook_login\n", - "notebook_login()\n", - "!git config --global credential.helper store" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "_tsf2uv0g_4p" - }, - "source": [ - "If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "jRqkGvk7pFQ6" - }, - "source": [ - "## Let's start the training 🔥\n", - "- ⚠️ ⚠️ ⚠️ Don't use **the same repo id with the one you used for the Unit 1**\n", - "- Now that you've coded from scratch PPO and added the Hugging Face Integration, we're ready to start the training 🔥" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "0tmEArP8ug2l" - }, - "source": [ - "- First, you need to copy all your code to a file you create called `ppo.py`" - ] - }, - { - "cell_type": "markdown", - "source": [ - "\"PPO\"/" - ], - "metadata": { - "id": "Sq0My0LOjPYR" - } - }, - { - "cell_type": "markdown", - "source": [ - "\"PPO\"/" - ], - "metadata": { - "id": "A8C-Q5ZyjUe3" - } - }, - { - "cell_type": "markdown", - "metadata": { - "id": "VrS80GmMu_j5" - }, - "source": [ - "- Now we just need to run this python script using `python .py` with the additional parameters we defined with `argparse`\n", - "\n", - "- You should modify more hyperparameters otherwise the training will not be super stable." - ] - }, - { - "cell_type": "code", - "source": [ - "!python ppo.py --env-id=\"LunarLander-v2\" --repo-id=\"YOUR_REPO_ID\" --total-timesteps=50000" - ], - "metadata": { - "id": "KXLih6mKseBs" - }, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "eVsVJ5AdqLE7" - }, - "source": [ - "## Some additional challenges 🏆\n", - "The best way to learn **is to try things by your own**! Why not trying another environment?\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "nYdl758GqLXT" - }, - "source": [ - "See you on Unit 8, part 2 where we going to train agents to play Doom 🔥\n", - "## Keep learning, stay awesome 🤗" - ] - } - ], - "metadata": { - "colab": { - "private_outputs": true, - "provenance": [], - "include_colab_link": true - }, - "gpuClass": "standard", - "kernelspec": { - "display_name": "Python 3", - "name": "python3" - }, - "language_info": { - "name": "python" - }, - "accelerator": "GPU" - }, - "nbformat": 4, - "nbformat_minor": 0 -} \ No newline at end of file diff --git a/notebooks/unit8/requirement.txt b/notebooks/unit8/requirement.txt deleted file mode 100644 index 920e673..0000000 --- a/notebooks/unit8/requirement.txt +++ /dev/null @@ -1,119 +0,0 @@ -absl-py==2.3.1 -ale-py==0.11.2 -annotated-types==0.7.0 -asttokens==3.0.0 -attrs==25.3.0 -box2d-py==2.3.5 -cattrs==1.5.0 -certifi==2025.8.3 -charset-normalizer==3.4.3 -click==8.2.1 -cloudpickle==3.1.1 -comm==0.2.3 -debugpy==1.8.16 -decorator==5.2.1 -exceptiongroup==1.3.0 -executing==2.2.0 -Farama-Notifications==0.0.4 -filelock==3.18.0 -fsspec==2025.7.0 -future==1.0.0 -gitdb==4.0.12 -GitPython==3.1.45 -grpcio==1.48.2 -gym==0.26.2 -gym-notices==0.1.0 -gymnasium==1.2.0 -h5py==3.14.0 -hf-xet==1.1.7 -huggingface-hub==0.34.4 -idna==3.10 -imageio==2.37.0 -imageio-ffmpeg==0.6.0 -ipykernel==6.30.1 -ipython==8.37.0 -ipywidgets==8.1.7 -jedi==0.19.2 -Jinja2==3.1.6 -jupyter_client==8.6.3 -jupyter_core==5.8.1 -jupyterlab_widgets==3.0.15 -Markdown==3.8.2 -MarkupSafe==3.0.2 -matplotlib-inline==0.1.7 --e git+https://github.com/Unity-Technologies/ml-agents@7db884323f8619b578fc1c8327d57fa087df27e7#egg=mlagents&subdirectory=ml-agents --e git+https://github.com/Unity-Technologies/ml-agents@7db884323f8619b578fc1c8327d57fa087df27e7#egg=mlagents_envs&subdirectory=ml-agents-envs -mpmath==1.3.0 -nest-asyncio==1.6.0 -networkx==3.4.2 -numpy==1.23.5 -nvidia-cublas-cu11==11.11.3.6 -nvidia-cublas-cu12==12.8.4.1 -nvidia-cuda-cupti-cu11==11.8.87 -nvidia-cuda-cupti-cu12==12.8.90 -nvidia-cuda-nvrtc-cu11==11.8.89 -nvidia-cuda-nvrtc-cu12==12.8.93 -nvidia-cuda-runtime-cu11==11.8.89 -nvidia-cuda-runtime-cu12==12.8.90 -nvidia-cudnn-cu11==9.1.0.70 -nvidia-cudnn-cu12==9.10.2.21 -nvidia-cufft-cu11==10.9.0.58 -nvidia-cufft-cu12==11.3.3.83 -nvidia-cufile-cu12==1.13.1.3 -nvidia-curand-cu11==10.3.0.86 -nvidia-curand-cu12==10.3.9.90 -nvidia-cusolver-cu11==11.4.1.48 -nvidia-cusolver-cu12==11.7.3.90 -nvidia-cusparse-cu11==11.7.5.86 -nvidia-cusparse-cu12==12.5.8.93 -nvidia-cusparselt-cu12==0.7.1 -nvidia-nccl-cu11==2.21.5 -nvidia-nccl-cu12==2.27.3 -nvidia-nvjitlink-cu12==12.8.93 -nvidia-nvtx-cu11==11.8.86 -nvidia-nvtx-cu12==12.8.90 -onnx==1.15.0 -packaging==25.0 -parso==0.8.4 -PettingZoo==1.15.0 -pexpect==4.9.0 -pillow==11.3.0 -platformdirs==4.3.8 -prompt_toolkit==3.0.51 -protobuf==3.20.3 -psutil==7.0.0 -ptyprocess==0.7.0 -pure_eval==0.2.3 -pydantic==2.11.7 -pydantic_core==2.33.2 -pygame==2.6.1 -pyglet==1.5.0 -Pygments==2.19.2 -python-dateutil==2.9.0.post0 -PyVirtualDisplay==3.0 -PyYAML==6.0.2 -pyzmq==27.0.1 -requests==2.32.4 -sentry-sdk==2.35.0 -six==1.17.0 -smmap==5.0.2 -stack-data==0.6.3 -swig==4.3.1 -sympy==1.14.0 -tensorboard==2.20.0 -tensorboard-data-server==0.7.2 -torch==2.7.1+cu118 -torchaudio==2.7.1+cu118 -torchvision==0.22.1+cu118 -tornado==6.5.2 -tqdm==4.67.1 -traitlets==5.14.3 -triton==3.3.1 -typing-inspection==0.4.1 -typing_extensions==4.14.1 -urllib3==2.5.0 -wandb==0.21.1 -wasabi==1.1.3 -wcwidth==0.2.13 -Werkzeug==3.1.3 -widgetsnbextension==4.0.14