{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nc0g2NLpUSGr"
   },
   "source": [
    "# Fine-tune SmolVLM on Visual Question Answering using Consumer GPU with QLoRA\n",
    "\n",
    "In this notebook we will fine-tune SmolVLM VQAv2 dataset. With this notebook you can also fine-tune Idefics3, since both models have the same model class/architecture.\n",
    "\n",
    "We will use some techniques in this notebook that will let you fine-tune the model on L4 with batch size of 4 only using around 16.4 GB of VRAM. We ran this notebook in that setup to test, but because we were able to afford A100 this notebook was last ran on an A100."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "WIhA1lQ7j0kw",
    "outputId": "d152531d-8a63-459f-d0b5-f61a47b268d2"
   },
   "outputs": [],
   "source": [
    "!pip install -q accelerate datasets peft bitsandbytes tensorboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "XyJaqZZ3uYYl",
    "outputId": "eff31ad7-7a77-4391-a1ed-6a871e667be5"
   },
   "outputs": [],
   "source": [
    "!pip install -q flash-attn --no-build-isolation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wAeMA0heVBjT"
   },
   "source": [
    "We will push out model to Hub so we need to authenticate ourselves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 17,
     "referenced_widgets": [
      "261a3abc28d74e4ca5af6f9df8cea3e5",
      "b6284cfacfd642278a7809a154463d69",
      "62c12672f59349b9ade248bee799fa5a",
      "9af532f878ab491096358d3bc83250d8",
      "599303d9f1204c85bca500c859dd0d87",
      "00617a46b15d45648c4796a91c96ec57",
      "5492da586f594365afc30ee6da1bf67c",
      "86aa1abb905346bf8956754a9704f250",
      "eeb2fbfd6cd54c4aa3983dc334a5377d",
      "ed34441fca164b389dfea1eabdba6e4a",
      "99f5b0432c1849128fa181b88925c77b",
      "5e529d6d6c4e40b4863961ea63bf259a",
      "ebfcd83e42ec46afb772d53ad7f35d43",
      "94958be916d6439d87dcd45c59178bec",
      "31a0c4a7fcff4744be56adf4125ef4e6",
      "2c975a8158bf49b389d47a5c4e40c97b",
      "b474bf8f464d40d8865665e4c7f0a411",
      "f8a75ac273fc408f923bf9d7f7263db8",
      "dd08ce6386184df38f47348e547738d8",
      "3aef5e8d5d9e4bd29bd3790ad139c02c"
     ]
    },
    "id": "yKd5xtSGj7cm",
    "outputId": "63b352c0-3f7d-4945-add2-52102246d7b2"
   },
   "outputs": [],
   "source": [
    "from huggingface_hub import notebook_login\n",
    "\n",
    "notebook_login()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WRq8ve-LVAzU"
   },
   "source": [
    "In this notebook we will not do full fine-tuning but use QLoRA method, which loads an adapter to the quantized version of the model, saving space. If you want to do full fine-tuning, set `USE_LORA` and `USE_QLORA` to False. If you want to do LoRA, set `USE_QLORA` to False and `USE_LORA` to True."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ[\"CUDA_DEVICE_ORDER\"] = \"PCI_BUS_ID\"\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"] = \"1, 2\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "b9CDMq0duYYn",
    "outputId": "65a4a5fa-fe4d-4243-b2d7-405a8aa81c04"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "23d3d175e6e642c7abc2bce09b73cf4d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "processor_config.json:   0%|          | 0.00/68.0 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "db6ca8f47f274464b135909c907c946a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "chat_template.json:   0%|          | 0.00/434 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d05822c6293c424fbf9df6ec0f6b532b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "preprocessor_config.json:   0%|          | 0.00/486 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "05582fca18f443d6965776a721e30e9f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/4.04k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3d8974fd1ba9415c8070c1eab8ad75cb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "vocab.json:   0%|          | 0.00/801k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "648257c1b1c24e25a26355bddf75aa41",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "merges.txt:   0%|          | 0.00/466k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "afa9a31c6b7f45e082ae07dea4a2600e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/3.52M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "92232af543a4446cac53e4fcf3f4b6e1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "added_tokens.json:   0%|          | 0.00/92.0 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a5f06e59634f4edf9f3d9409846a2b31",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/1.07k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some kwargs in processor config are unused and will not have any effect: image_seq_len. \n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7ddfa8718bc24882ba2b50a899656107",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/7.08k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5983728a1c1e43edb4d16bee6ad40171",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/4.49G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dff574197f1f4466abb0eb46d36b8378",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "generation_config.json:   0%|          | 0.00/132 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10536960, 2256809840)\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from peft import LoraConfig, prepare_model_for_kbit_training, get_peft_model\n",
    "from transformers import AutoProcessor, BitsAndBytesConfig, Idefics3ForConditionalGeneration\n",
    "\n",
    "USE_LORA = False\n",
    "USE_QLORA = True\n",
    "SMOL = True\n",
    "\n",
    "model_id = \"HuggingFaceTB/SmolVLM-Base\" if SMOL else \"HuggingFaceM4/Idefics3-8B-Llama3\"\n",
    "\n",
    "processor = AutoProcessor.from_pretrained(\n",
    "    model_id\n",
    ")\n",
    "\n",
    "if USE_QLORA or USE_LORA:\n",
    "    lora_config = LoraConfig(\n",
    "        r=8,\n",
    "        lora_alpha=8,\n",
    "        lora_dropout=0.1,\n",
    "        target_modules=['down_proj','o_proj','k_proj','q_proj','gate_proj','up_proj','v_proj'],\n",
    "        use_dora=False if USE_QLORA else True,\n",
    "        init_lora_weights=\"gaussian\"\n",
    "    )\n",
    "    lora_config.inference_mode = False\n",
    "    if USE_QLORA:\n",
    "        bnb_config = BitsAndBytesConfig(\n",
    "            load_in_4bit=True,\n",
    "            bnb_4bit_use_double_quant=True,\n",
    "            bnb_4bit_quant_type=\"nf4\",\n",
    "            bnb_4bit_compute_dtype=torch.bfloat16\n",
    "        )\n",
    "\n",
    "    model = Idefics3ForConditionalGeneration.from_pretrained(\n",
    "        model_id,\n",
    "        quantization_config=bnb_config if USE_QLORA else None,\n",
    "        _attn_implementation=\"flash_attention_2\",\n",
    "        device_map=\"auto\"\n",
    "    )\n",
    "    model.add_adapter(lora_config)\n",
    "    model.enable_adapters()\n",
    "    model = prepare_model_for_kbit_training(model)\n",
    "    model = get_peft_model(model, lora_config)\n",
    "    print(model.get_nb_trainable_parameters())\n",
    "else:\n",
    "    model = Idefics3ForConditionalGeneration.from_pretrained(\n",
    "        model_id,\n",
    "        torch_dtype=torch.bfloat16,\n",
    "        _attn_implementation=\"flash_attention_2\",\n",
    "    ).to(DEVICE)\n",
    "\n",
    "    # if you'd like to only fine-tune LLM\n",
    "    for param in model.model.vision_model.parameters():\n",
    "        param.requires_grad = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WIVhpp0EyZO2"
   },
   "source": [
    "The model as is is holding 2.7 GB of GPU RAM 💗"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LMTtg3dl3NX2"
   },
   "source": [
    "## Loading the dataset and Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pWHMWTSZ3Pyr"
   },
   "source": [
    "We will load a small portion of the VQAv2 dataset. We are loading a small portion of the model for education purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "POOqKqYRka5O"
   },
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "ds = load_dataset('merve/vqav2-small', trust_remote_code=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "Znf9vMo5rnSd"
   },
   "outputs": [],
   "source": [
    "split_ds = ds[\"validation\"].train_test_split(test_size=0.5)\n",
    "train_ds = split_ds[\"train\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FIDioFlRuYYn",
    "outputId": "79b697a7-d245-4fdc-b0e8-d9ffa8627953"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['multiple_choice_answer', 'question', 'image'],\n",
       "    num_rows: 10717\n",
       "})"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5nwMO3n0X7Hv"
   },
   "source": [
    "Let's write our data collating function. We will apply prompt template to have questions and answers together so model can learn to answer. Then we pass the formatted prompts and images to the processor which processes both."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "e0krVLZ-wNMl"
   },
   "outputs": [],
   "source": [
    "image_token_id = processor.tokenizer.additional_special_tokens_ids[\n",
    "            processor.tokenizer.additional_special_tokens.index(\"<image>\")]\n",
    "\n",
    "def collate_fn(examples):\n",
    "  texts = []\n",
    "  images = []\n",
    "  for example in examples:\n",
    "      image = example[\"image\"]\n",
    "      if image.mode != 'RGB':\n",
    "        image = image.convert('RGB')\n",
    "      question = example[\"question\"]\n",
    "      answer = example[\"multiple_choice_answer\"]\n",
    "      messages = [\n",
    "          {\n",
    "              \"role\": \"user\",\n",
    "              \"content\": [\n",
    "                  {\"type\": \"text\", \"text\": \"Answer briefly.\"},\n",
    "                  {\"type\": \"image\"},\n",
    "                  {\"type\": \"text\", \"text\": question}\n",
    "              ]\n",
    "          },\n",
    "          {\n",
    "              \"role\": \"assistant\",\n",
    "              \"content\": [\n",
    "                  {\"type\": \"text\", \"text\": answer}\n",
    "              ]\n",
    "          }\n",
    "      ]\n",
    "      text = processor.apply_chat_template(messages, add_generation_prompt=False)\n",
    "      texts.append(text.strip())\n",
    "      images.append([image])\n",
    "\n",
    "  batch = processor(text=texts, images=images, return_tensors=\"pt\", padding=True)\n",
    "  labels = batch[\"input_ids\"].clone()\n",
    "  labels[labels == processor.tokenizer.pad_token_id] = -100\n",
    "  labels[labels == image_token_id] = -100\n",
    "  batch[\"labels\"] = labels\n",
    "\n",
    "  return batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kEYDjWpE3LD5"
   },
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QvAs896cdwg8"
   },
   "source": [
    "We can now initialize `Trainer` and initialize `TrainingArguments` to pass to `Trainer`.\n",
    "\n",
    "Some notes:\n",
    "- If you use 8-bit QLoRA with the below setup it uses around 16.4 GB VRAM (beautiful, fits comfortably inside L4, Colab free tier)\n",
    "- We use gradient accumulation to simulate a larger batch size.\n",
    "- We also save up on memory from intermediate activations by using gradient checkpointing.\n",
    "\n",
    "**Disclaimer:** \n",
    "The techniques here aren't free lunch. The latter two will add additional compute to the training, thus slow down a bit (for reference on two A100s with bsz of 16, we were able to train for 2 hrs 43 mins with the gradient accumulation steps of 4, disabling it reduced it with 2 hr 35 mins). \n",
    "If you want to speed-up, you might play around, reduce to 4-bit precision and have a higher batch size. Note that 4-bit might result in model learning less."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "QNE2yWAYrAhD"
   },
   "outputs": [],
   "source": [
    "from transformers import TrainingArguments, Trainer\n",
    "\n",
    "model_name = model_id.split(\"/\")[-1]\n",
    "\n",
    "training_args = TrainingArguments(\n",
    "    num_train_epochs=1,\n",
    "    per_device_train_batch_size=16,\n",
    "    gradient_accumulation_steps=4,\n",
    "    warmup_steps=50,\n",
    "    learning_rate=1e-4,\n",
    "    weight_decay=0.01,\n",
    "    logging_steps=25,\n",
    "    save_strategy=\"steps\",\n",
    "    save_steps=250,\n",
    "    save_total_limit=1,\n",
    "    optim=\"paged_adamw_8bit\", # for 8-bit, keep this, else adamw_hf\n",
    "    bf16=True, # underlying precision for 8bit\n",
    "    output_dir=f\"./{model_name}-vqav2\",\n",
    "    hub_model_id=f\"{model_name}-vqav2\",\n",
    "    report_to=\"tensorboard\",\n",
    "    remove_unused_columns=False,\n",
    "    gradient_checkpointing=True\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "oBBSDpBhreJd"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.\n"
     ]
    }
   ],
   "source": [
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    args=training_args,\n",
    "    data_collator=collate_fn,\n",
    "    train_dataset=train_ds,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_QOCpw_-uYYo"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='9' max='670' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [  9/670 01:41 < 2:39:41, 0.07 it/s, Epoch 0.01/1]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "trainer.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0hN0QD9_uYYo"
   },
   "outputs": [],
   "source": [
    "trainer.push_to_hub()"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "A100",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "00617a46b15d45648c4796a91c96ec57": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2c975a8158bf49b389d47a5c4e40c97b",
      "placeholder": "​",
      "style": "IPY_MODEL_b474bf8f464d40d8865665e4c7f0a411",
      "value": "\n<b>Pro Tip:</b> If you don't already have one, you can create a dedicated\n'notebooks' token with 'write' access, that you can then easily reuse for all\nnotebooks. </center>"
     }
    },
    "261a3abc28d74e4ca5af6f9df8cea3e5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "VBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "VBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "VBoxView",
      "box_style": "",
      "children": [],
      "layout": "IPY_MODEL_5492da586f594365afc30ee6da1bf67c"
     }
    },
    "2c975a8158bf49b389d47a5c4e40c97b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "31a0c4a7fcff4744be56adf4125ef4e6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ButtonStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ButtonStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "button_color": null,
      "font_weight": ""
     }
    },
    "3aef5e8d5d9e4bd29bd3790ad139c02c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "5492da586f594365afc30ee6da1bf67c": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": "center",
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": "flex",
      "flex": null,
      "flex_flow": "column",
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": "50%"
     }
    },
    "599303d9f1204c85bca500c859dd0d87": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ButtonModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ButtonModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ButtonView",
      "button_style": "",
      "description": "Login",
      "disabled": false,
      "icon": "",
      "layout": "IPY_MODEL_94958be916d6439d87dcd45c59178bec",
      "style": "IPY_MODEL_31a0c4a7fcff4744be56adf4125ef4e6",
      "tooltip": ""
     }
    },
    "5e529d6d6c4e40b4863961ea63bf259a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "62c12672f59349b9ade248bee799fa5a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "PasswordModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "PasswordModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "PasswordView",
      "continuous_update": true,
      "description": "Token:",
      "description_tooltip": null,
      "disabled": false,
      "layout": "IPY_MODEL_ed34441fca164b389dfea1eabdba6e4a",
      "placeholder": "​",
      "style": "IPY_MODEL_99f5b0432c1849128fa181b88925c77b",
      "value": ""
     }
    },
    "86aa1abb905346bf8956754a9704f250": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "94958be916d6439d87dcd45c59178bec": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "99f5b0432c1849128fa181b88925c77b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "9af532f878ab491096358d3bc83250d8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "CheckboxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "CheckboxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "CheckboxView",
      "description": "Add token as git credential?",
      "description_tooltip": null,
      "disabled": false,
      "indent": true,
      "layout": "IPY_MODEL_5e529d6d6c4e40b4863961ea63bf259a",
      "style": "IPY_MODEL_ebfcd83e42ec46afb772d53ad7f35d43",
      "value": true
     }
    },
    "b474bf8f464d40d8865665e4c7f0a411": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b6284cfacfd642278a7809a154463d69": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_86aa1abb905346bf8956754a9704f250",
      "placeholder": "​",
      "style": "IPY_MODEL_eeb2fbfd6cd54c4aa3983dc334a5377d",
      "value": "<center> <img\nsrc=https://huggingface.co/front/assets/huggingface_logo-noborder.svg\nalt='Hugging Face'> <br> Copy a token from <a\nhref=\"https://huggingface.co/settings/tokens\" target=\"_blank\">your Hugging Face\ntokens page</a> and paste it below. <br> Immediately click login after copying\nyour token or it might be stored in plain text in this notebook file. </center>"
     }
    },
    "dd08ce6386184df38f47348e547738d8": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ebfcd83e42ec46afb772d53ad7f35d43": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "ed34441fca164b389dfea1eabdba6e4a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "eeb2fbfd6cd54c4aa3983dc334a5377d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "f8a75ac273fc408f923bf9d7f7263db8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "LabelModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "LabelModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "LabelView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_dd08ce6386184df38f47348e547738d8",
      "placeholder": "​",
      "style": "IPY_MODEL_3aef5e8d5d9e4bd29bd3790ad139c02c",
      "value": "Connecting..."
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}