{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "### Dataset Background & Loading\n", "\n", "The training dataset was sourced from the publicly available BIS central bank speeches, downloaded using the `gingado` package:\n", "\n", "```python\n", "from gingado.datasets import load_CB_speeches\n", "all_speeches = load_CB_speeches()\n", "all_speeches.to_csv(\"central_bank_speeches.csv\", index=False)\n", "```\n", "\n", "A preprocessing script was applied to clean the text, lowercase it, split speeches into well-formed sentences, and filter out short/noisy segments. This generated over **2 million sentence-level samples**, saved as `speeches_data_preprocessed.csv`.\n", "\n", "For training on Kaggle, the preprocessed dataset was uploaded as an external file and loaded.\n", "\n", "This ensures clean and consistent input for masked language modeling (MLM)." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19", "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5", "execution": { "iopub.execute_input": "2025-07-19T17:12:03.395329Z", "iopub.status.busy": "2025-07-19T17:12:03.395050Z", "iopub.status.idle": "2025-07-19T17:12:16.719665Z", "shell.execute_reply": "2025-07-19T17:12:16.719049Z", "shell.execute_reply.started": "2025-07-19T17:12:03.395302Z" }, "trusted": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(19609, 8)\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
urltitledescriptiondatetextauthorcountryprocessed_text
0https://www.bis.org/review/r970211c.pdfMr. Chen discusses monetary relations between ...Speech by the Deputy Governor of the People's ...1996-09-10 00:00:00Mr. Chen discusses monetary relations between ...Chen YuanChina[\"mr. chen discusses monetary relations betwee...
1https://www.bis.org/review/r970211b.pdfMr. Dai looks at the possibilities of strength...Speech by the Governor of the People's Bank of...1996-11-13 00:00:00Mr. Dai looks at the possibilities of strength...Dai XianglongChina[\"mr. dai looks at the possibilities of streng...
2https://www.bis.org/review/r970211a.pdfMr. Dai assesses the outlook for Hong Kong as ...Speech by the Governor of the People's Bank of...1996-09-30 00:00:00Mr. Dai assesses the outlook for Hong Kong as ...Dai XianglongChina[\"mr. dai assesses the outlook for hong kong a...
3https://www.bis.org/review/r970203b.pdfMr. Rangarajan examines the objectives of mone...Address by the Governor of the Reserve Bank of...1996-12-28 00:00:00Mr. Rangarajan examines the objectives of mone...Bimal JalanIndia[\"mr. rangarajan examines the objectives of mo...
4https://www.bis.org/review/r970115a.pdfM. Trichet presents the monetary policy guidel...BANK OF FRANCE, PRESS RELEASE, 17/12/96.1996-12-17 00:00:00M. Trichet presents the monetary policy guidel...Bank of FranceFrance['m. trichet presents the monetary policy guid...
\n", "
" ], "text/plain": [ " url \\\n", "0 https://www.bis.org/review/r970211c.pdf \n", "1 https://www.bis.org/review/r970211b.pdf \n", "2 https://www.bis.org/review/r970211a.pdf \n", "3 https://www.bis.org/review/r970203b.pdf \n", "4 https://www.bis.org/review/r970115a.pdf \n", "\n", " title \\\n", "0 Mr. Chen discusses monetary relations between ... \n", "1 Mr. Dai looks at the possibilities of strength... \n", "2 Mr. Dai assesses the outlook for Hong Kong as ... \n", "3 Mr. Rangarajan examines the objectives of mone... \n", "4 M. Trichet presents the monetary policy guidel... \n", "\n", " description date \\\n", "0 Speech by the Deputy Governor of the People's ... 1996-09-10 00:00:00 \n", "1 Speech by the Governor of the People's Bank of... 1996-11-13 00:00:00 \n", "2 Speech by the Governor of the People's Bank of... 1996-09-30 00:00:00 \n", "3 Address by the Governor of the Reserve Bank of... 1996-12-28 00:00:00 \n", "4 BANK OF FRANCE, PRESS RELEASE, 17/12/96. 1996-12-17 00:00:00 \n", "\n", " text author country \\\n", "0 Mr. Chen discusses monetary relations between ... Chen Yuan China \n", "1 Mr. Dai looks at the possibilities of strength... Dai Xianglong China \n", "2 Mr. Dai assesses the outlook for Hong Kong as ... Dai Xianglong China \n", "3 Mr. Rangarajan examines the objectives of mone... Bimal Jalan India \n", "4 M. Trichet presents the monetary policy guidel... Bank of France France \n", "\n", " processed_text \n", "0 [\"mr. chen discusses monetary relations betwee... \n", "1 [\"mr. dai looks at the possibilities of streng... \n", "2 [\"mr. dai assesses the outlook for hong kong a... \n", "3 [\"mr. rangarajan examines the objectives of mo... \n", "4 ['m. trichet presents the monetary policy guid... " ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "\n", "df = pd.read_csv('/kaggle/input/bis-speeches/speeches_data_preprocessed.csv')\n", "print(df.shape)\n", "df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Tokenize BIS Sentences for MLM Training\n", "\n", "This section prepares the preprocessed central bank speech sentences for masked language modeling (MLM) by:\n", "\n", "- Flattening over 2 million cleaned sentences into a single list.\n", "- Converting them into a Hugging Face `Dataset` object.\n", "- Tokenizing using the `bert-base-uncased` tokenizer with:\n", " - `max_length=128` (chosen based on sentence length distribution: ~99% of sentences fall within this limit),\n", " - truncation and padding enabled.\n", "- Applying tokenization in parallel using `num_proc=4` for efficiency.\n", "- Saving the tokenized dataset locally for later training use.\n", "\n", "The tokenized dataset is saved to:\n", "\n", "```\n", "/kaggle/working/tokenized_bis_dataset\n", "```\n", "\n", "This ensures the input is consistently preprocessed and optimally sized for efficient MLM training.\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2025-07-19T17:12:20.157315Z", "iopub.status.busy": "2025-07-19T17:12:20.157050Z", "iopub.status.idle": "2025-07-19T17:16:28.711161Z", "shell.execute_reply": "2025-07-19T17:16:28.710348Z", "shell.execute_reply.started": "2025-07-19T17:12:20.157296Z" }, "trusted": true }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "a51d2c2858e644a585bd2c6e07b2d618", "version_major": 2, "version_minor": 0 }, "text/plain": [ "tokenizer_config.json: 0%| | 0.00/48.0 [00:00\n", " \n", " \n", " [65238/65238 8:18:46, Epoch 1/1]\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
StepTraining Loss
2002.361400
4002.249100
6002.250100
8002.226300
10002.183100
12002.198800
14002.145000
16002.174700
18002.134100
20002.133000
22002.108300
24002.088800
26002.109000
28002.093700
30002.081200
32002.078700
34002.106100
36002.068000
38002.068400
40002.053400
42002.038000
44002.060000
46002.049400
48002.038100
50002.023700
52002.031100
54002.050300
56002.021300
58002.022600
60001.988000
62002.005500
64002.015700
66001.987600
68001.994200
70001.997800
72001.981400
74001.982900
76002.018900
78001.989700
80001.985400
82001.960200
84001.977300
86001.970100
88001.979800
90001.978400
92001.942400
94001.962300
96001.941700
98001.931800
100001.940800
102001.919900
104001.899700
106001.963000
108001.986000
110001.922700
112001.938600
114001.940800
116001.950300
118001.939500
120001.938500
122001.934400
124001.933600
126001.891400
128001.962800
130001.916100
132001.912500
134001.880500
136001.910800
138001.907600
140001.913600
142001.898900
144001.934200
146001.896200
148001.936400
150001.903800
152001.881700
154001.889800
156001.887100
158001.881200
160001.871600
162001.870300
164001.884200
166001.884400
168001.841900
170001.875600
172001.849700
174001.854700
176001.868200
178001.853200
180001.857000
182001.924400
184001.885000
186001.873600
188001.873100
190001.868300
192001.873800
194001.870100
196001.868400
198001.834900
200001.840700
202001.846400
204001.856500
206001.859000
208001.873700
210001.820800
212001.849200
214001.839000
216001.833900
218001.841400
220001.827000
222001.858900
224001.825700
226001.845400
228001.820800
230001.829700
232001.834400
234001.822500
236001.812600
238001.803200
240001.817000
242001.829300
244001.821900
246001.829200
248001.838800
250001.846000
252001.810500
254001.791800
256001.832400
258001.806300
260001.815600
262001.783200
264001.796400
266001.800400
268001.775000
270001.795000
272001.821800
274001.818200
276001.821500
278001.823700
280001.784400
282001.802300
284001.793400
286001.818000
288001.759300
290001.765300
292001.781000
294001.787900
296001.801300
298001.778400
300001.703900
302001.808600
304001.798500
306001.774700
308001.769300
310001.812800
312001.815200
314001.763000
316001.770900
318001.755600
320001.774800
322001.792300
324001.748700
326001.764200
328001.770000
330001.785100
332001.772400
334001.742800
336001.779800
338001.722400
340001.758500
342001.754000
344001.787000
346001.758700
348001.738800
350001.734000
352001.755200
354001.745000
356001.737300
358001.736600
360001.739600
362001.718000
364001.755300
366001.749200
368001.757300
370001.730600
372001.768200
374001.735300
376001.731500
378001.733600
380001.712900
382001.727000
384001.736600
386001.710300
388001.728400
390001.734800
392001.726600
394001.681900
396001.752200
398001.702100
400001.731000
402001.713800
404001.719200
406001.714400
408001.694700
410001.747300
412001.747600
414001.703500
416001.723200
418001.707700
420001.693900
422001.703700
424001.732700
426001.665700
428001.710400
430001.708900
432001.720000
434001.690400
436001.696600
438001.671700
440001.705700
442001.725100
444001.726000
446001.700000
448001.718800
450001.666500
452001.715900
454001.704800
456001.675300
458001.718500
460001.710300
462001.705200
464001.675400
466001.676400
468001.683600
470001.669400
472001.701700
474001.693300
476001.707200
478001.666400
480001.665500
482001.668200
484001.688100
486001.714800
488001.653800
490001.679800
492001.676300
494001.709800
496001.667600
498001.667900
500001.656900
502001.686600
504001.679800
506001.667100
508001.675700
510001.689400
512001.682400
514001.663600
516001.669500
518001.653500
520001.673900
522001.653600
524001.650300
526001.646600
528001.657700
530001.665000
532001.661700
534001.670700
536001.643200
538001.613200
540001.644600
542001.667900
544001.662500
546001.669900
548001.677700
550001.631500
552001.663500
554001.656300
556001.654600
558001.648000
560001.657400
562001.648000
564001.669800
566001.642000
568001.654600
570001.666300
572001.646200
574001.614200
576001.639200
578001.660000
580001.649900
582001.664000
584001.638000
586001.607500
588001.636300
590001.652900
592001.620800
594001.634200
596001.628300
598001.659300
600001.622400
602001.660000
604001.627900
606001.645900
608001.647800
610001.605900
612001.628400
614001.623200
616001.649100
618001.646600
620001.642000
622001.632000
624001.626200
626001.653700
628001.641200
630001.646200
632001.617100
634001.629300
636001.644500
638001.665600
640001.619800
642001.648800
644001.601500
646001.627900
648001.624000
650001.648100
652001.639600

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "โœ… Training completed at: 2025-07-20 01:35:50.877293\n", "๐ŸŽ‰ Final model saved to /kaggle/working/bert-mlm-bis\n" ] } ], "source": [ "# 1. Install required packages\n", "# !pip install -U transformers datasets --quiet\n", "\n", "# 2. Imports\n", "from transformers import (\n", " BertTokenizerFast,\n", " BertForMaskedLM,\n", " Trainer,\n", " TrainingArguments,\n", " DataCollatorForLanguageModeling\n", ")\n", "from datasets import load_from_disk\n", "from datetime import datetime\n", "import torch\n", "import os\n", "\n", "# 3. Force use of single GPU (for P100)\n", "os.environ[\"CUDA_VISIBLE_DEVICES\"] = \"0\"\n", "\n", "# 4. Load tokenizer and dataset\n", "tokenizer = BertTokenizerFast.from_pretrained(\"bert-base-uncased\")\n", "dataset = load_from_disk(\"/kaggle/working/tokenized_bis_dataset\")\n", "print(f\"โœ… Tokenized dataset loaded with {len(dataset)} samples.\")\n", "\n", "# 5. Load model\n", "model = BertForMaskedLM.from_pretrained(\"bert-base-uncased\")\n", "\n", "# 6. Data collator for MLM\n", "data_collator = DataCollatorForLanguageModeling(\n", " tokenizer=tokenizer,\n", " mlm=True,\n", " mlm_probability=0.15\n", ")\n", "\n", "# 7. Training arguments (gradient accumulation + smaller per-device batch)\n", "training_args = TrainingArguments(\n", " output_dir=\"/kaggle/working/bert-mlm-bis\",\n", " overwrite_output_dir=True,\n", " num_train_epochs=1, # โœ… Full dataset, 1 pass\n", " per_device_train_batch_size=16, # โœ… Lower memory per device\n", " gradient_accumulation_steps=2, # โœ… Effective batch size = 32\n", " eval_strategy=\"no\", # โœ… No eval during training\n", " save_strategy=\"epoch\", # โœ… Save once at end\n", " logging_dir=\"/kaggle/working/logs\",\n", " logging_steps=200,\n", " fp16=torch.cuda.is_available(), # โœ… Mixed precision\n", " dataloader_num_workers=4,\n", " save_total_limit=1,\n", " report_to=\"none\"\n", ")\n", "\n", "# 8. Initialize Trainer\n", "trainer = Trainer(\n", " model=model,\n", " args=training_args,\n", " train_dataset=dataset,\n", " tokenizer=tokenizer,\n", " data_collator=data_collator,\n", ")\n", "\n", "# 9. Train\n", "print(\"โฑ๏ธ Training started at:\", datetime.now())\n", "trainer.train()\n", "print(\"โœ… Training completed at:\", datetime.now())\n", "\n", "# 10. Save final model\n", "trainer.save_model(\"/kaggle/working/bert-mlm-bis\")\n", "print(\"๐ŸŽ‰ Final model saved to /kaggle/working/bert-mlm-bis\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Evaluate Trained Model and Compute Perplexity\n", "\n", "To assess the quality of the pretrained CB-BERT-MLM model, evaluated it on a randomly sampled subset of 10,000 sentences from the tokenized dataset. This step computes:\n", "\n", "- **Evaluation loss** on masked language modeling (MLM)\n", "- **Perplexity**, a standard metric indicating how confidently the model predicts masked tokens (lower is better)\n", "\n", "```python\n", "from datasets import load_from_disk\n", "from transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling\n", "import math\n", "\n", "# Load trained model and tokenizer\n", "model = AutoModelForMaskedLM.from_pretrained(...)\n", "tokenizer = AutoTokenizer.from_pretrained(...)\n", "\n", "# Select a subset of 10,000 sentences for quick evaluation\n", "eval_dataset = dataset.shuffle(seed=42).select(range(10000))\n", "\n", "# Evaluate\n", "metrics = trainer.evaluate()\n", "eval_loss = metrics[\"eval_loss\"]\n", "perplexity = math.exp(eval_loss)\n", "```\n", "\n", "> **Perplexity Score** is printed at the end of the cell. A lower perplexity indicates stronger masked token prediction performance and better fit to the domain-specific language.\n", "\n", "This provides a quantitative baseline for how well the model understands and reconstructs financial and monetary policy language.\n" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "execution": { "iopub.execute_input": "2025-07-20T02:00:40.964023Z", "iopub.status.busy": "2025-07-20T02:00:40.963322Z", "iopub.status.idle": "2025-07-20T02:01:32.119053Z", "shell.execute_reply": "2025-07-20T02:01:32.118331Z", "shell.execute_reply.started": "2025-07-20T02:00:40.963997Z" }, "trusted": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐Ÿš€ Using device: cuda\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/tmp/ipykernel_36/4227637877.py:39: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `Trainer.__init__`. Use `processing_class` instead.\n", " trainer = Trainer(\n" ] }, { "data": { "text/html": [ "\n", "

\n", " \n", " \n", " [625/625 00:50]\n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "๐Ÿ“‰ Evaluation Loss: 1.5392\n", "๐Ÿ“Š Perplexity Score (subset of 10000): 4.66\n" ] } ], "source": [ "# ๐Ÿ“ฆ Imports\n", "from transformers import (\n", " AutoModelForMaskedLM,\n", " AutoTokenizer,\n", " DataCollatorForLanguageModeling,\n", " Trainer,\n", " TrainingArguments\n", ")\n", "from datasets import load_from_disk\n", "import torch\n", "import math\n", "\n", "# ๐Ÿง  Ensure GPU is used\n", "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", "print(f\"๐Ÿš€ Using device: {device}\")\n", "\n", "# ๐Ÿ”„ Load model and tokenizer from saved path\n", "model = AutoModelForMaskedLM.from_pretrained(\"/kaggle/working/bert-mlm-bis\").to(device)\n", "tokenizer = AutoTokenizer.from_pretrained(\"/kaggle/working/bert-mlm-bis\")\n", "\n", "# ๐Ÿ“‚ Load tokenized dataset and sample subset\n", "dataset = load_from_disk(\"/kaggle/working/tokenized_bis_dataset\")\n", "eval_dataset = dataset.shuffle(seed=42).select(range(10000)) # ๐Ÿ”ฝ reduce for speed\n", "\n", "# ๐Ÿ” Data collator for masked LM\n", "data_collator = DataCollatorForLanguageModeling(\n", " tokenizer=tokenizer,\n", " mlm=True,\n", " mlm_probability=0.15\n", ")\n", "\n", "# โš™๏ธ Trainer setup\n", "training_args = TrainingArguments(\n", " output_dir=\"/kaggle/working/tmp_eval\",\n", " per_device_eval_batch_size=16,\n", " report_to=\"none\"\n", ")\n", "\n", "trainer = Trainer(\n", " model=model,\n", " args=training_args,\n", " data_collator=data_collator,\n", " eval_dataset=eval_dataset,\n", " tokenizer=tokenizer,\n", ")\n", "\n", "# ๐Ÿ“Š Evaluate and compute perplexity\n", "metrics = trainer.evaluate()\n", "eval_loss = metrics[\"eval_loss\"]\n", "perplexity = math.exp(eval_loss)\n", "\n", "print(f\"๐Ÿ“‰ Evaluation Loss: {eval_loss:.4f}\")\n", "print(f\"๐Ÿ“Š Perplexity Score (subset of 10000): {perplexity:.2f}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Compare Perplexity: BERT-Base vs CB-BERT-MLM\n", "\n", "This section evaluates and compares the perplexity of the original `bert-base-uncased` model and the domain-adapted `cb-bert-mlm` on a subset of 10,000 masked sentences from the BIS corpus.\n", "\n", "#### Evaluation Setup:\n", "- Both models use the same evaluation subset and masking strategy (MLM probability = 15%)\n", "- Performed on GPU (P100) with batch size 16\n", "- Perplexity is calculated from the evaluation loss: `perplexity = exp(loss)`\n", "\n", "#### Output:\n", "- Perplexity scores are printed for both models\n", "- Lower perplexity indicates better performance in masked token prediction on financial text\n", "\n", "This comparison highlights the impact of domain adaptation through MLM pretraining on central bank communication data." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "execution": { "iopub.execute_input": "2025-07-20T02:02:58.622861Z", "iopub.status.busy": "2025-07-20T02:02:58.622182Z", "iopub.status.idle": "2025-07-20T02:04:40.524560Z", "shell.execute_reply": "2025-07-20T02:04:40.523804Z", "shell.execute_reply.started": "2025-07-20T02:02:58.622839Z" }, "trusted": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "๐Ÿ“Š Evaluating: BERT-Base\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForMaskedLM: ['bert.pooler.dense.bias', 'bert.pooler.dense.weight', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight']\n", "- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n", "- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n", "/tmp/ipykernel_36/810192027.py:37: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `Trainer.__init__`. Use `processing_class` instead.\n", " trainer = Trainer(\n" ] }, { "data": { "text/html": [ "\n", "
\n", " \n", " \n", " [625/625 00:50]\n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "๐Ÿ“‰ Eval Loss: 2.5698\n", "๐Ÿ“ Perplexity: 13.06\n", "\n", "๐Ÿ“Š Evaluating: BIS-BERT-MLM\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/tmp/ipykernel_36/810192027.py:37: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `Trainer.__init__`. Use `processing_class` instead.\n", " trainer = Trainer(\n" ] }, { "data": { "text/html": [ "\n", "
\n", " \n", " \n", " [625/625 00:50]\n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "๐Ÿ“‰ Eval Loss: 1.5392\n", "๐Ÿ“ Perplexity: 4.66\n", "\n", "๐Ÿงพ Summary:\n", "โžก๏ธ BERT-Base Perplexity : 13.06\n", "โžก๏ธ BIS-BERT-MLM Perplexity : 4.66\n" ] } ], "source": [ "from transformers import (\n", " AutoModelForMaskedLM,\n", " AutoTokenizer,\n", " DataCollatorForLanguageModeling,\n", " Trainer,\n", " TrainingArguments\n", ")\n", "from datasets import load_from_disk\n", "import math\n", "import torch\n", "\n", "# โœ… Load the tokenized dataset (use a subset for fast eval)\n", "dataset = load_from_disk(\"/kaggle/working/tokenized_bis_dataset\")\n", "eval_dataset = dataset.shuffle(seed=42).select(range(10000)) # adjust size if needed\n", "\n", "# โœ… Common data collator for both models\n", "def get_data_collator(tokenizer):\n", " return DataCollatorForLanguageModeling(\n", " tokenizer=tokenizer,\n", " mlm=True,\n", " mlm_probability=0.15\n", " )\n", "\n", "# ๐Ÿ” Evaluation function\n", "def evaluate_perplexity(model_path, label):\n", " print(f\"\\n๐Ÿ“Š Evaluating: {label}\")\n", " tokenizer = AutoTokenizer.from_pretrained(model_path)\n", " model = AutoModelForMaskedLM.from_pretrained(model_path).to(\"cuda\")\n", "\n", " collator = get_data_collator(tokenizer)\n", " args = TrainingArguments(\n", " output_dir=\"/kaggle/working/tmp_eval_\" + label.replace(\"-\", \"_\"),\n", " per_device_eval_batch_size=16,\n", " report_to=\"none\"\n", " )\n", "\n", " trainer = Trainer(\n", " model=model,\n", " args=args,\n", " eval_dataset=eval_dataset,\n", " data_collator=collator,\n", " tokenizer=tokenizer\n", " )\n", "\n", " metrics = trainer.evaluate()\n", " loss = metrics[\"eval_loss\"]\n", " perplexity = math.exp(loss)\n", "\n", " print(f\"๐Ÿ“‰ Eval Loss: {loss:.4f}\")\n", " print(f\"๐Ÿ“ Perplexity: {perplexity:.2f}\")\n", " return perplexity\n", "\n", "# โš–๏ธ Compare both models\n", "p1 = evaluate_perplexity(\"bert-base-uncased\", \"BERT-Base\")\n", "p2 = evaluate_perplexity(\"/kaggle/working/bert-mlm-bis\", \"BIS-BERT-MLM\")\n", "\n", "# ๐Ÿ“ˆ Summary\n", "print(\"\\n๐Ÿงพ Summary:\")\n", "print(f\"โžก๏ธ BERT-Base Perplexity : {p1:.2f}\")\n", "print(f\"โžก๏ธ BIS-BERT-MLM Perplexity : {p2:.2f}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Manual Masked Sentence Evaluation\n", "\n", "This section tests the `cb-bert-mlm` model on 20 manually constructed masked sentences based on real central banking and financial policy language.\n", "\n", "Each sentence contains a single `[MASK]` token, and is evaluated for whether the model correctly predicts the expected token.\n", "\n", "#### Evaluation Highlights:\n", "- Sentences represent realistic use cases in financial regulation, digital currency, and monetary policy\n", "- Most mismatches were plausible paraphrases (e.g., synonyms or domain-relevant alternates)\n", "\n", "The test demonstrates the model's strong contextual understanding of domain-specific language, particularly in predicting terminology used in central bank communication.\n", "Results are displayed in a tabular format showing the masked sentence, expected token, predicted token, and whether it matched exactly.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": { "iopub.execute_input": "2025-07-20T02:15:43.730657Z", "iopub.status.busy": "2025-07-20T02:15:43.730330Z", "iopub.status.idle": "2025-07-20T02:15:45.482523Z", "shell.execute_reply": "2025-07-20T02:15:45.481827Z", "shell.execute_reply.started": "2025-07-20T02:15:43.730635Z" }, "trusted": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
SentenceExpectedPredictedMatch?
0Central banks are exploring the potential of d...currencies##isationโŒ
1The governor highlighted the importance of mon...policypolicyโœ…
2Inflation expectations remain [MASK] anchored ...wellwellโœ…
3Cross-border [MASK] are still slow and expensive.paymentspaymentsโœ…
4Financial [MASK] is a key objective for many c...inclusionstabilityโŒ
5Stablecoins pose new [MASK] for regulators and...challengeschallengesโœ…
6Monetary [MASK] must adapt to technological in...policypolicyโœ…
7The BIS supports the development of secure dig...paymentpaymentโœ…
8Central banks need to coordinate on [MASK] fra...regulatorytheseโŒ
9Emerging markets are experiencing strong capit...inflowsflowsโŒ
10The committee emphasized the need for macropru...oversightpoliciesโŒ
11Tokenization of [MASK] could transform financi...assetsriskโŒ
12Interoperability between payment [MASK] is cru...systemssystemsโœ…
13Cybersecurity [MASK] increase with digital fin...risksrisksโœ…
14Central banks must ensure [MASK] in digital in...resiliencetrustโŒ
15The future of [MASK] may involve public and pr...moneyfinanceโŒ
16Pilot [MASK] help central banks understand new...projectsexercisesโŒ
17Legal frameworks need to [MASK] for modern fin...evolveevolveโœ…
18Foreign exchange [MASK] have remained relative...marketsreservesโŒ
19The central bank raised its key interest [MASK...raterateโœ…
\n", "
" ], "text/plain": [ " Sentence Expected Predicted \\\n", "0 Central banks are exploring the potential of d... currencies ##isation \n", "1 The governor highlighted the importance of mon... policy policy \n", "2 Inflation expectations remain [MASK] anchored ... well well \n", "3 Cross-border [MASK] are still slow and expensive. payments payments \n", "4 Financial [MASK] is a key objective for many c... inclusion stability \n", "5 Stablecoins pose new [MASK] for regulators and... challenges challenges \n", "6 Monetary [MASK] must adapt to technological in... policy policy \n", "7 The BIS supports the development of secure dig... payment payment \n", "8 Central banks need to coordinate on [MASK] fra... regulatory these \n", "9 Emerging markets are experiencing strong capit... inflows flows \n", "10 The committee emphasized the need for macropru... oversight policies \n", "11 Tokenization of [MASK] could transform financi... assets risk \n", "12 Interoperability between payment [MASK] is cru... systems systems \n", "13 Cybersecurity [MASK] increase with digital fin... risks risks \n", "14 Central banks must ensure [MASK] in digital in... resilience trust \n", "15 The future of [MASK] may involve public and pr... money finance \n", "16 Pilot [MASK] help central banks understand new... projects exercises \n", "17 Legal frameworks need to [MASK] for modern fin... evolve evolve \n", "18 Foreign exchange [MASK] have remained relative... markets reserves \n", "19 The central bank raised its key interest [MASK... rate rate \n", "\n", " Match? \n", "0 โŒ \n", "1 โœ… \n", "2 โœ… \n", "3 โœ… \n", "4 โŒ \n", "5 โœ… \n", "6 โœ… \n", "7 โœ… \n", "8 โŒ \n", "9 โŒ \n", "10 โŒ \n", "11 โŒ \n", "12 โœ… \n", "13 โœ… \n", "14 โŒ \n", "15 โŒ \n", "16 โŒ \n", "17 โœ… \n", "18 โŒ \n", "19 โœ… " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from transformers import BertForMaskedLM, BertTokenizerFast\n", "import torch\n", "import pandas as pd\n", "from IPython.display import display\n", "\n", "# 1) Load trained MLM\n", "model_path = \"/kaggle/working/bert-mlm-bis\"\n", "tokenizer = BertTokenizerFast.from_pretrained(model_path)\n", "model = BertForMaskedLM.from_pretrained(model_path)\n", "model.eval()\n", "\n", "# 2) Manual maskedโ€‘sentence test set\n", "masked_data = [\n", " (\"Central banks are exploring the potential of digital [MASK].\", \"currencies\"),\n", " (\"The governor highlighted the importance of monetary [MASK] transparency.\", \"policy\"),\n", " (\"Inflation expectations remain [MASK] anchored across most economies.\", \"well\"),\n", " (\"Cross-border [MASK] are still slow and expensive.\", \"payments\"),\n", " (\"Financial [MASK] is a key objective for many central banks.\", \"inclusion\"),\n", " (\"Stablecoins pose new [MASK] for regulators and policymakers.\", \"challenges\"),\n", " (\"Monetary [MASK] must adapt to technological innovation.\", \"policy\"),\n", " (\"The BIS supports the development of secure digital [MASK] systems.\", \"payment\"),\n", " (\"Central banks need to coordinate on [MASK] frameworks.\", \"regulatory\"),\n", " (\"Emerging markets are experiencing strong capital [MASK].\", \"inflows\"),\n", " (\"The committee emphasized the need for macroprudential [MASK].\", \"oversight\"),\n", " (\"Tokenization of [MASK] could transform financial markets.\", \"assets\"),\n", " (\"Interoperability between payment [MASK] is crucial.\", \"systems\"),\n", " (\"Cybersecurity [MASK] increase with digital financial services.\", \"risks\"),\n", " (\"Central banks must ensure [MASK] in digital infrastructure.\", \"resilience\"),\n", " (\"The future of [MASK] may involve public and private sector collaboration.\", \"money\"),\n", " (\"Pilot [MASK] help central banks understand new financial instruments.\", \"projects\"),\n", " (\"Legal frameworks need to [MASK] for modern financial technology.\", \"evolve\"),\n", " (\"Foreign exchange [MASK] have remained relatively stable.\", \"markets\"),\n", " (\"The central bank raised its key interest [MASK] by 25 basis points.\", \"rate\"),\n", "]\n", "\n", "# 3) Run predictions\n", "results = []\n", "for sent, true_word in masked_data:\n", " # encode + mask\n", " inputs = tokenizer(sent, return_tensors=\"pt\")\n", " mask_index = torch.where(inputs.input_ids[0] == tokenizer.mask_token_id)[0]\n", "\n", " # forward pass\n", " with torch.no_grad():\n", " logits = model(**inputs).logits\n", "\n", " # pick top-1\n", " token_id = logits[0, mask_index, :].argmax(dim=-1).item()\n", " pred = tokenizer.decode([token_id]).strip()\n", "\n", " results.append({\n", " \"Sentence\": sent,\n", " \"Expected\": true_word,\n", " \"Predicted\": pred,\n", " \"Match?\": \"โœ…\" if pred.lower() == true_word.lower() else \"โŒ\"\n", " })\n", "\n", "# 4) Show as DataFrame\n", "df = pd.DataFrame(results)\n", "display(df)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Top-K Accuracy Evaluation on 100,000 Randomly Masked Sentences\n", "\n", "This section evaluates the `cb-bert-mlm` model's ability to recover randomly masked words in context across **100,000 test sentences**. The procedure involves:\n", "\n", "#### Procedure:\n", "\n", "1. **Sentence Sampling** \n", " 100,000 random sentences were sampled from the BIS preprocessed dataset.\n", "\n", "2. **Masking Strategy** \n", " One random eligible word (min sentence length = 5, alphabetic tokens only) was replaced with `[MASK]` in each sentence.\n", "\n", "3. **Prediction** \n", " The model generated **Top-K token predictions** for the masked position, with `k` ranging from 1 to 20.\n", "\n", "4. **Accuracy Computation** \n", " A prediction is considered correct if the original word appears in the top-K list. The accuracy is computed as: \n", " \\[\n", " \\text{Top-k Accuracy} = \\frac{\\text{\\# correct predictions}}{\\text{total samples}} \\times 100\n", " \\]\n", "\n", "\n", "#### Results:\n", "\n", "> *Exact values are printed at the end of the cell and visualized in the curve below.*\n", "\n", "\n", "#### Top-K Accuracy Curve\n", "\n", "A line plot visualizes model performance across increasing values of `k`, showing how quickly prediction confidence saturates.\n", "\n", "This benchmark confirms the model's strong ability to predict masked financial-domain tokens, with over **90% Top-20 accuracy**.\n" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "execution": { "iopub.execute_input": "2025-07-20T02:42:10.239306Z", "iopub.status.busy": "2025-07-20T02:42:10.239002Z", "iopub.status.idle": "2025-07-20T03:00:16.278533Z", "shell.execute_reply": "2025-07-20T03:00:16.277747Z", "shell.execute_reply.started": "2025-07-20T02:42:10.239285Z" }, "trusted": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "โš™๏ธ Using device: cuda\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "๐Ÿ” Evaluating (Topโ€‘k): 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 100000/100000 [17:43<00:00, 94.01it/s]\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1YAAAIoCAYAAABqA3puAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAABsjUlEQVR4nO3deVxU9eLG8WfYFwEFlcUFcd81y91rWeSSuaSVpqVpi5VWarfFrqW2W7cyy9vq0qJpeTO1fmnuy3XJUEvTTI3cxY1NERjg+/sDmUBABgacQT/v12teMGebZwYc5+Gc8z0WY4wRAAAAAKDE3JwdAAAAAADKO4oVAAAAADiIYgUAAAAADqJYAQAAAICDKFYAAAAA4CCKFQAAAAA4iGIFAAAAAA6iWAEAAACAgyhWAAAAAOAgihWAK9Zff/0li8Wie++919lRAIds27ZN7u7umjNnjrOjoJxavny5LBaL/u///s/ZUYArFsUKQIlYLJZi3a4U586dU2BgoCwWi0aOHOnsOFeVFStWaNCgQapVq5Z8fX3l7++vRo0aacSIEdq8ebOz45WpsWPHqmHDhho4cGCe6QX9W/P19VWDBg30xBNP6OTJk/m2de+998pisWjTpk15pmdkZOi9995T+/btFRQUJC8vL4WHh6tt27YaM2aMtm3bZnfeiRMn5svl7u6uypUrq2vXrlq4cGG+dXL+EHKpW61atfKsU6tWrXyPERISoptuuklff/11vuds723WrFlFPsfc23zvvfcKXW7AgAGFbrdWrVry8fEp8rGkv3/W3t7eOn36dIHLxMfHy9fXt8D33ejoaHXq1ElPPfWUMjMz7XpMAMXj4ewAAMqnCRMm5Js2ZcoUJSYmFjjvSvHVV18pOTlZFotFc+bM0Ztvvmn3ByOUzPnz5zV8+HDNnTtXfn5+io6OVv369SVJf/zxh2bPnq2PPvpIn332me655x4npy19K1eu1OrVqzV9+nS5ueX/e2hISIhGjRplu3/69GmtXr1ab731lhYuXKitW7cqMDDwko+RmZmpHj16aPny5YqIiNAdd9yh0NBQJSQkaOvWrZo6dar8/f11zTXXFCt7//791bRpU0lSenq69u/fr0WLFmnZsmV67733CvzjRJ06dXT33XcXuL2KFSvmm+bu7q7x48dLkqxWq/bt26cFCxZo5cqVeuWVVzRu3Dj17ds3XylbvXq11qxZoz59+qhly5Z55l18/1I8PDw0Y8aMPD+DHGfOnNHChQvl4eGhjIwMu7d5qcdKT0/X7Nmz9dhjj+WbP3v2bKWmphb6eE899ZR69+6tuXPnavDgwQ7nAXARAwClJDIy0rjS20psbKyRZIYOHVpq2+zYsaPx8PAwo0ePNpLM7NmzS23bKNhdd91lJJmbb77ZHD9+PN/8+Ph489RTT5mpU6c6IV3Zu/32242vr69JTEzMN0+SadCgQb7pWVlZpmfPnkaSmT59ep55Q4cONZLMxo0bbdM+++wzI8l0797dpKen59vesWPHTExMjN2ZJ0yYYCSZL7/8Mt+8n376yUgyNWrUyDM9599rt27d7H6cyMhI4+3tnW/6+vXrjZubm/H19TXnzp27ZMaZM2fa/Xi55byOvXr1MpLM9u3b8y3zzjvvGEmmd+/eBT5WYfkLkvOzrl+/vmnZsmWBy1xzzTWmQYMGpkGDBgW+F6enp5vKlSubTp062fWYAIqHQwEBlLlTp05p9OjRioqKkre3t6pWrao777xTO3fuzLdszuE1f/75p15//XXVq1dPPj4+ioqK0gsvvCCr1epwnvT0dN15552yWCx66qmnZIyxa709e/bof//7n7p3764xY8bIYrFo+vTpl3yct99+W61bt1ZAQIAqVKigxo0ba+zYsYqPj8+z7IkTJ/TEE0+oQYMG8vX1VXBwsNq2bat///vftmVWr14ti8WiiRMn5nusws4nq1WrlmrVqqWEhASNGjVKNWrUkIeHh+2QpJiYGI0aNUpNmzZVUFCQfH191axZM7322muFvtZFZd27d6/c3Nx0yy23FLh+cnKyKlSooIYNGxb62uVYtWqVvvzyS9WvX1/ffvutQkND8y1TsWJFTZ48WQ8++GC+512QG264Id9hUjmHrq1evVqzZs1Sq1at5OfnpxtuuEGff/65LBaLXnjhhQK3t3XrVlkslnx7AE6cOKExY8aobt268vb2VuXKldW/f/8Cf+8LEx8fr4ULF6pbt25F7nXKzWKxqFu3bpKy//0VZePGjZKkESNGyNPTM9/8sLAwtWrVyu7Hv5TWrVsrODjYrlwl1bFjRzVs2FDnz5/Xrl27yuxxJGno0KFyd3cv8L1g5syZatSokdq3b19qjzds2DBt375dW7duzTP9l19+0bZt2zRs2LBC1/X09FTfvn21fv167du3r9QyAchGsQJQpk6ePKl27drpnXfeUa1atTR27FjdeOON+uabb9S2bVutX7++wPVGjx6tyZMnKzo6Wo8++qi8vb01YcIE3XXXXQ7lSU5OVo8ePTR//ny9+eabev311+0+Byzng9OQIUNUs2ZN3XDDDVq1apViY2PzLXv+/HndeOONGjt2rBITEzVs2DA9/PDDql+/vj788EMdOHDAtuyePXvUsmVLvfXWW6pataoee+wxDRo0SH5+fnrllVccer6SlJaWphtvvFE//vijevfurZEjR9oKyscff6wFCxaoWbNmGjFihO677z4ZYzRu3Lh85/PYm7VevXrq0qWLli5dqkOHDuXbxpw5c3Tu3Dndf//9RWbPec3/+c9/ys/P75LLent7F7m9orzxxht65JFH1KBBAz322GPq2LGj+vXrJ39/f82ePbvAdT7//HNJynMY4v79+3XttddqypQpqlOnjh599FHdcsstWrJkidq1a2f3OWFr166V1WpVu3btiv1cli1bJkl2FaKQkBBJ2YdWlrWYmBidOXOm1IpaUTw8yvash2rVqqlr166aM2eO0tPTbdO3bt2q7du3X7LolEROkZs5c2ae6dOnT5e7u7uGDBlyyfVzSt7KlStLNRcAudAxOwDKvYIOBRw2bJiRZMaNG5dn+vfff28kmbp165rMzEzb9JzDa6pUqWIOHTpkm56WlmY6d+5sJJn58+fblefiQwGPHz9urrnmGuPp6Wk+//zzYj03q9VqQkNDTcWKFc358+eNMcbMmDHDSDLjx4/Pt/wTTzxhJJl77rnHZGRk5JmXkJBgkpOTbfevu+46I8l89NFH+baT+zVYtWqVkWQmTJhQ5HPNkfMz6datm0lJScm33oEDB/Lly8rKMsOHDzeSzPr16/PMszfrvHnzjCQzceLEfMtdd911xsvLy5w4cSLfvIvVqlXLSDL79u0rctncIiMjTWRkZIHzrr/++ny/pzmHhfn7+5tff/013zp33323kWQ2b96cZ3pGRoYJDQ01YWFheV7HDh06GHd3d7NkyZI8y+/Zs8cEBASYZs2a2fU8nnzySSPJLFu2rMD5kkxISIiZMGGC7fbYY4+Z5s2bGw8PD/P444/nW6egQwFjYmKMh4eH8fLyMiNGjDCLFi0yR48etStjQXJez/79+9tyjRs3zgwcOND4+fmZ2rVr5zt0Lud3uE6dOnmeT+7bDz/8kGedog4FDAkJsf17LSyjo4cCbty40cyfP99IMl999ZVt/iOPPGI8PDzM8ePHzauvvlpqhwIaY8ytt95qgoODTWpqqjHGmNTUVBMcHGx69epljDGFHgpojDG//PKLkWSGDBlS3KcMoAgUKwCl5uJilZaWZnx8fExISEiB5zncfPPNRpJZu3atbVrOh5WXXnop3/Lr1q0zksytt95qV57cZWPfvn2mTp06xs/PL9+HM3ssWLDASDIPPPCAbVpSUpLx8/Mz1atXz1MOrVarCQgIMEFBQebMmTOX3O7mzZuNJNO5c+ciMzhSrH755Zcit59bTExMvmJUnKzp6ekmNDTUREZG5nltcj7U3XHHHXbl8PHxMZJsHyDtVdJiNWbMmALXWbp0qZFkHn300TzT/+///s9IMqNHj7ZN27p1q5Fkhg8fXuC2xo4daySZHTt2FPk8cs4vK6jsGZP9YbuwW6dOncyqVavyrVNQsTLGmNmzZ5vKlSvn2Ub16tXNvffea37++ecis+aW83oWdPP39zf/+te/zNmzZ/Osk/M7fKnbxUUxMjLSuLu724rXs88+a+68807j6elpPDw8zLx584rMWBrFKufcpe7duxtjjDl//rypVKmS6dOnjzHGlHqx+uabb4wkM3fuXGOMMXPnzjWSzIIFC4wxly5Wx48fN5LMjTfeWMxnDKAojAoIoMz8/vvvSk1NVZcuXQo8jKtLly5atmyZtm/frn/84x955l18X8o+hMXDwyPPsM8FnW80evToPKOH/f777+rYsaMyMjK0cuVKtW3bttjP5ZNPPpGkPIfZBAQEqG/fvpozZ46WLl2qHj162B4vOTlZ0dHRqlSp0iW3+9NPP0mSunbtWuxM9vLx8VGzZs0KnJeenq733ntPc+fO1e+//66zZ8/mOefs6NGjJcrq6empYcOG6bXXXtOPP/6o7t27S8o+9FCSHnjggRI/n7LUpk2bAqffdNNNCg8P19y5c/XWW2/ZDi/74osvJOU9DDBnGPO4uLgCfz9///1329ecEfMKkzOsdkGj4eVo0KCBbZuSbCP5jR07VtHR0fr666912223XfJxJGnQoEHq16+fli1bpvXr1ysmJkYbNmzQrFmz9Nlnn2natGl66KGHJGWf77d69eo867ds2VJ9+/bNM+3LL7+0HVKakZGhI0eOaNasWZo0aZKWLVum//3vf/kO1evWrZuWLFlSZN4cmZmZmjRpUp5pHh4e+vrrr/PlKY5vv/1W27dvzzPthhtu0A033JBvWU9PT919992aOnWqjhw5orVr1yo+Pl7Dhw8v8eNfyq233qqqVatqxowZGjBggGbMmKGqVavq1ltvLXLd4OBgSfadewegeChWAMpMUlKSJBU44IAkhYeH51kut4LWyblGTWJiom3axR+opOwBMHJ/EP3jjz8UHx+vDh06FPlBtiBHjx7VkiVLVLt2bXXq1CnPvCFDhmjOnDmaMWOGrVjl5KtWrVqR2y7OsiVVtWrVQs8ju/3227V48WLVr19fAwYMUNWqVeXp6amEhAS98847SktLK3HWBx98UJMnT9Ynn3yi7t27KzU1VbNnz1ZUVJSio6Pt2kZYWJj++usvHTlyRLVr17ZrHUcU9rvq7u6uQYMG6c0339TSpUvVs2dPnT17Vt9++60aN26c53yhM2fOSJK+//57ff/994U+1rlz54rM4+vrK0lKTU21+zlUrFhRN954o+bPn6969erpqaeesqtYSdklvFevXurVq5ftcf/973/rueee0+OPP66+ffsqLCxMq1evzvdvb+jQoZcsMh4eHoqMjNSECRO0d+9ezZ49W/PmzXN42G9vb2/b63P27FmtXLlSw4cP1z333KP169erRYsWJdrut99+q08//TTf9IKKlSQNHz5cU6ZM0axZs7R69WqFhYUVOoCLo3KK3JQpU7RhwwYtX75cY8aMset8svPnz0tSkecsAig+Bq8AUGZyRjGLi4srcP7x48fzLJdbQetkZmbq9OnTCgoKsk0z2Yc057ldPBpc7969NXHiRG3YsEG33HKLXR9oc5s1a5YyMzP1559/5ruQaM6emEWLFtn+ApxT6o4cOVLktouzbM41jAq6Pk3usnmxwkrVli1btHjxYnXr1k27du3Sxx9/rJdfflkTJ04scOCK4mSVpKioKHXt2lWLFi3SiRMn9N///lfx8fG677777B4wpGPHjpKyLw5cHG5uboVeN6gkr5X0916pnL1U//3vf5WSkpLv2lk5v8/vvvtugb+fObehQ4cW+TyqVKki6e+yVhx169ZVcHCw9u3bp4SEhGKvL2UXrfHjx6tz585KT0/X//73P0nZe4ovfj72XFQ3R85e4y1btpQoV2EqVKig3r17a968eTp79qyGDRtm96ifF5s1a1a+51jQHsgczZo1U+vWrTVt2jStXLlSQ4YMKdOBM+677z5lZWXpzjvvVFZWlu677z671sv5Xcr53QJQeihWAMpMw4YN5ePjoy1btiglJSXf/JxDiQq6GOe6devyTdu4caMyMjKKfZFSKfuCxi+++KLWrl2rHj166OzZs3atZ4zRjBkzJGXvCbvvvvvy3Tp06KD09HTb6HANGjRQYGCgtmzZkm9Y9YvlHHr2448/Fpkl57DCgopN7sMj7bV//35JUs+ePeXu7p5nXkGvf3Gy5hgxYoSsVqs+/fRTffLJJ3J3dy/WKGk5HxbffPNN21/aC5N771qlSpV04sSJfOXq3Llz2rt3r92Pn1uLFi3UrFkzLVy4UMnJyfriiy8KHGY9pzTkDGHuiJxDOPfs2VPsdTMyMpScnCxJysrKcihHhQoVHFr/Yjn/LhzNVZibbrpJffv21bZt2/Tll1+WyWMUZPjw4Tp27JiysrLK7DDAHI0bN1bbtm115MgRtWvXTo0aNbJrvZzfpcIODwZQchQrAGXGy8tLd911l06dOqVXX301z7wlS5Zo6dKlqlu3rm2vRG7vvPOODh8+bLufnp6uf/3rX5KU71pN9ho/frxefvllrVu3zu5ytWbNGu3fv1+dO3fWzJkz9cknn+S75RSvnKHBPTw8NGLECCUmJurxxx9XZmZmnm0mJibaHrt169Zq3bq11q5dazv/KLfcJapBgwYKCAjQokWL8uzBiIuL00svvVTs1yMyMlKS8g15/9tvv+X7eRU3a45evXopIiJCb7/9ttasWaOePXsqIiLC7oxdunTRXXfdpT179qhfv346ceJEvmWSkpL07LPP6qOPPsqT1Wq15hki3VwYRr64eyxzu+eee3T+/HlNnTpVK1eu1PXXX68aNWrkWaZNmzZq27atvvzyS82bNy/fNrKysrRmzRq7Hu/666+XJLuHZ8/tvffek9VqVZMmTWzn1RRm7ty5WrlyZYF7dzZt2qRVq1bJw8OjRMO+Xyw+Pt42VHjnzp0d3l5hcq5NNmnSpHz/BsvK3XffrQULFuiHH35QgwYNyvzxZsyYoQULFlzyenoXy/ldyvndAlB6OMcKQJmaPHmy1qxZo5deekkbNmxQ27Zt9ddff+nrr7+Wn5+fZs6caTvELbd27dqpRYsWGjBggPz9/bV48WLbh+v+/fuXOM+zzz4rNzc3jRs3Tt27d9eSJUsu+df4nA8sl9rL0qBBA3Xo0EEbNmzQ5s2b1bZtW73wwgvatGmTPv/8c23atEk9evSQt7e3/vzzTy1ZskTr16+37ambPXu2brjhBj344IP6/PPP1b59e6Wmpuq3337Ttm3bbAMYeHl56dFHH9Urr7yiVq1aqU+fPkpOTtbixYt1/fXX2/ZA2atNmzZq06aNvvrqKx07dkzt2rXTwYMHtWjRIvXs2VPz58/Pt469WXN4eHjovvvu04svviipZINWTJ8+XcYYzZ0713Z4Yf369WWM0d69e7VixQolJyfb9hhK0qhRozRz5kzdf//9WrZsmapUqaJ169YpISFBLVq00C+//FLsHFL2AA/PPPOMJk2apKysrHyHAeb48ssv1aVLFw0cOFBTpkxRq1at5Ovrq4MHD2rjxo06efKkXedNNW/eXLVr17Zdk6ogp06dynOIWmJiorZu3aq1a9fK29tb7777bpGPs2nTJr3zzjuqVq2aOnfurJo1ayo9PV27d+/Wjz/+qKysLL322mvFPhdw/vz5toE1MjMzdfjwYdsfBrp3765+/frlW2ffvn2XPOTumWeekY+PT5GP3aJFC91222365ptv9MUXX9h16KWjKlSoUOwBM6xW6yX/WHSpQywbN26sxo0bF+vxli1bpkqVKpVpqQWuWpdp9EEAV4GCrmNljDEnT540jz32mImMjDSenp6mcuXK5vbbby9wuOmcIYz3799vXnvtNVO3bl3j5eVlIiMjzcSJE01aWprdeQobgtwYYyZPnmwkmQ4dOpikpKQC109ISDC+vr7G398/z3WnCvLxxx/nG449NTXV/Pvf/zYtW7Y0vr6+pkKFCqZx48bmiSeeMPHx8XnWP378uHn88cdN7dq1jZeXlwkODjZt27Y1b731Vp7lMjMzzcSJE02NGjWMl5eXqV+/vnnnnXfMn3/+Wehw64UNO26MMSdOnDDDhw83ERERxsfHxzRr1sxMmzat0O0VJ2uOffv2GUmmWrVq+a6ZVRzLli0zd911l4mMjDQ+Pj7Gx8fH1KtXz9x///35ri9ljDErV640bdu2Nd7e3iYkJMTcc889Ji4u7pLDrRc0PPnFoqOjjSTj4+NjEhMTC13uzJkzZvz48aZp06a2n3+9evXMoEGDzDfffGP38875XS3oOaqAIck9PT1NzZo1zT333GN27tyZb52Chls/ePCgeffdd02vXr1M3bp1jb+/v/Hy8jI1a9Y0d9xxh1mxYoXdeY0pfLj1gIAA065dOzN16lRjtVrzrGPPcOuS8vzbKWq48l9++cVYLBZTu3btfI9XmsOtF+VSw60X9XxzKNdw60UpbLj12NhYY7FY8lweAEDpsRhTwrM6AaAM3Hvvvfr0008VGxubbxAKlE/z58/XHXfcoeeee04vvPCCs+OUO2fOnFHt2rV1xx13FHgIJmCv8ePH6/XXX9fu3btVp04dZ8cBrjicYwUAKDPGGL355pvy8PBw2WtXubrg4GCNGzdOn376qQ4cOODsOCin4uPj9e677+rhhx+mVAFlhHOsAAClbseOHfruu++0YcMGbdq0SSNGjMg3yAPs9/jjjystLU0HDx60DToCFEdsbKzGjBmjRx991NlRgCsWxQoAUOpiYmL07LPPKigoSPfcc4/+/e9/OztSuebj46Pnn3/e2TFQjrVq1SrPhawBlD6XOhRw7dq1tqF5LRaLvv322zzzjTF6/vnnFR4eLl9fX0VHR+e7HsmZM2c0ePBgBQYGqmLFirrvvvvsvl4NAOfLuSgn51eVb/fee6+MMUpISNBnn31W6tdBAgDA1bhUsTp37pxatGihadOmFTj/9ddf19SpU/XBBx9o8+bN8vf3V7du3fIMWTt48GD99ttvWrZsmb777jutXbtWDz744OV6CgAAAACuQi47KqDFYtGCBQts14MwxigiIkJPPPGE/vnPf0rKvlZHaGioZs2apYEDB2r37t1q3LixtmzZouuuu05S9kVIb7nlFh0+fLhYF6UEAAAAAHuVm3OsYmNjdfz4cUVHR9umBQUFqW3bttq4caMGDhyojRs3qmLFirZSJUnR0dFyc3PT5s2bddtttxW47bS0NKWlpdnuZ2Vl6cyZMwoJCZHFYim7JwUAAADApRljlJycrIiICLm5FX7AX7kpVsePH5ckhYaG5pkeGhpqm3f8+HFVrVo1z3wPDw8FBwfblinIq6++qkmTJpVyYgAAAABXikOHDql69eqFzi83xaosjRs3TmPHjrXdT0xMVM2aNRUbG6uAgAAnJpOsVqtWrVqlLl26yNPT06lZcpDJPmSyD5ns44qZJNfMRSb7kMk+ZLIPmexDJvu4Wqbk5GRFRUUV2QvKTbEKCwuTJMXFxSk8PNw2PS4uTi1btrQtc+LEiTzrZWRk6MyZM7b1C+Lt7S1vb+9804ODgxUYGFgK6UvOarXKz89PISEhLvGLJZHJXmSyD5ns44qZJNfMRSb7kMk+ZLIPmexDJvu4WqacDEWdIuRSowJeSlRUlMLCwrRixQrbtKSkJG3evFnt27eXJLVv314JCQmKiYmxLbNy5UplZWWpbdu2lz0zAAAAgKuDS+2xOnv2rPbt22e7Hxsbq+3btys4OFg1a9bU6NGj9dJLL6levXqKiorSc889p4iICNvIgY0aNVL37t31wAMP6IMPPpDVatWoUaM0cOBARgQEAAAAUGZcqlj9/PPP6tKli+1+znlPQ4cO1axZs/TUU0/p3LlzevDBB5WQkKBOnTppyZIl8vHxsa0ze/ZsjRo1SjfddJPc3NzUv39/TZ069bI/FwAAAABXD5cqVjfccIMudVkti8WiF154QS+88EKhywQHB2vOnDllEQ8AAAAAClRuzrECAAAAAFdFsQIAAAAAB1GsAAAAAMBBFCsAAAAAcBDFCgAAAAAcRLECAAAAAAdRrAAAAADAQRQrAAAAAHAQxQoAAAAAHESxAgAAAAAHUawAAAAAwEEUKwAAAABwEMUKAAAAABxEsQIAAAAAB1GsAAAAAMBBFCsAAAAAcBDFCgAAAAAcRLECAAAAAAdRrAAAAADAQRQrAAAAAHAQxQoAAAAAHESxAgAAAAAHUawAAAAAwEEUKwAAAABwEMUKAAAAABxEsQIAAAAAB1GsAAAAAMBBFCsAAAAAcBDFCgAAAAAcRLECAAAAAAdRrAAAAADAQRQrAAAAAHAQxQoAAACAS8jMMtoce0YxpyzaHHtGmVnG2ZHs5uHsAAAAAAAuv9wlJiT2jNrXrSp3N4vT8izZeUyTFu/SscRUSe76bO/PCg/y0YRejdW9abjTctmLYgUAAACUMUpM0Xke/mKrLt4/dTwxVQ9/sVXv393K5csVxQoAAABXFEpM0XmcUWIys4ysmVkXbtnfp2dkKS0jS+O/3ZkvjyQZSRZJkxbv0s2Nw5z6cywKxQoAAAAlRokpOk9ZlxhjjNIysmxFJf3CV2tmdmlJz8i5b5RqzdAz3+wotMRI0j+//lUxB+KVmSVbEUrPzFJGpsl13ygj1/fWC4+XkWVsj229sE76he9LerqUkXQsMVU/xZ5R+zohJXyVyh7FCgAAoJygxBSdp6xLTEbm38UlZ29L7iJjm56ZpdT0TI0rosQ88dUvWrf3lDIvFJK0AraVflFhyvnemvM1s3QHeDiblqGP18WW6jYL4u5mkad79u9vqjWryOVPJKeWdSSHUKwAAAAKQIkpOs/lOpzMGJO9tyUjU2nWLKVlZCr1wte0jCylWjN1Pj2zyD0xOSUmZy9K3mKUadurk7vEZO/xybTdL+1B6s6lZ2r25oOluk1Pd4u83N3k6eEmL3c3eXlcuLm76Vx6hg6dOV/kNro0qKJG4YHydHeTp7vlwtfsbXq6WWzfe7lb5OF2YXrO47q7ySPX9xev4+lukaebm9wu/HvauP+07vp4U5GZqgb4OPzalCWKFQAAwEWu5hJTGGOMMrKyDzlLScvQ8wt/u2SJeea/O3QiOe3C4WfZBSgtVzFKs2blKkpZeZa5uDSlZWTJlEKhKe0SY7HIVly8Ly4xHm46m5qhv06nFLmdbo1D1ax6kK385C5E3he25emevyTl/t62jvvfhaUg9paYBzvXuWyH3bWJClZ4kI+OJ6YW+DtlkRQW5KM2UcGXJU9JUawAAIDTudLeIVcoMbllZhlNWrzrkiVm/Lc7FezvrYys7JKSu7zkFJTs6ZkXlZjcy+UtN7m3k1N0irO3JuG8Vc8v/K00XoJ8vD3c5OPpLm8PN3l7usnHw13nrZk6HF/0nphuTcLUvHrQhb0pFnl5uOcpKN4ehZeX3PM83d3k4WaRxeJ4ibm3Y9RVXWLc3Sya0KuxHv5iqyxSnlw5r+6EXo1deuAKiWIFAMBVx5VKjORae4eKKjE5o5Pd2DBUGVlZf+9ZybX3JbWA8pJ/Wv5D2grcVkamks9nKOG89ZK5T51N150fbiyLl8QhzasHqlZIBfl4usnbwz1/Icr53sM9zzLenjnLXpjm+fcyXu5uBZYZu0tMh1qUGBcsMd2bhuv9u1vlei/IFsZ1rAAAgCtypRKTk6c09w5l5Ryqlp6h89ZMpVozlZKeff5NijVTqemZOn9hmm3ehfNzzqdn6lB8Sp4PdRfLGZ2s/vgfSvaEy1CIv5eC/b3ylBBbUfH4u6D8XWZyTbOVl9zlpqDtZC+39UC8Bn2yuchM43o0psRQYoqV6+bGYdq474R+XLdZXf/R1ul/+CkOihUAAGXIlfYOucohbukXik9SasYlr10jZQ/7vDn2jNIysmzlx1aErHm/T0nPsGtksdLm6W7JW0DyFJK85aSgPTc+ngWUm1zz9hxP0rhvdhaZ471BrS5biWlbO4QSYydKTPG4u1nUNipYp3cbtY0Kdnqe4qBYAQBQRlxp75C9h7jlvgBnzrVxUtIzdS4tI/treoZS0jJ1Ni1DKekZOpeeqZS0i76mZ+hcWs46+ecVZ2jos2kZmvm/v0r0nL093OTr5S4/T3f5eLnL19Ndfl7u8rnw1dfTXb5e7vL19JCvl5t8Pd0Vl5SmzzcdKHLbH99zrTrWqyxvD/cy/+DXonpFTV2xjxJjB0pM8ZTnEuOKKFYAAJQBZ+0dSs/I0tm0DJ1Ly1Byaobt+60H4+06xK3T5JUyRtkFKj1TmaU9tnQu7m4Wu7Z/U6Oqal6tYnb58fLILkQXl6Q8RSl7ekk+JGZmGS3fHVdkibmxUehl+xBKiSl+LkoMnIFiBQC4YrjKYXfF3TuUmWV0Ni2jwEJ0NjVDyTnfX5iX8/3ZC8vlvqVnOHYoXGHly9fTXf7e7vLz8pCfl7v8vS989fKQn/dFX73cVcHbQ37eHvL3yl4nZ93c2/j5r3i7Bhu4v1Pty3aIGyWm+LkoMUA2ihUA4IrgjMPu0jOylJxqtRWenK/bD9m3d6jVi8uUnpGl89bMUs/m6+muCj4eCvD2kL+3hzKzjHYdSypyvedvbaQ2USF5ypOfl0eZfDB1xcEGJEpMcVFigGwUKwBAibjK3iGp+Ifd5Rwul5xqtRWi7D1DVtseorO5ilL2MrkK1IVlHN07lHjRENqe7hYF+HjK39tdFbw9L5Qid1Xw8VQFbw8F+HjI38sjT2Gq4OOhCt4Xbhe+9/dyl4e7W55tZ2YZdZq8ssgSM7RD1FV/iJtEiQFQfBQrAECxOXtQhozMLCWetyo+xarTZ9M07psdlxxZ7tEvtykiaLfOpmWWSiG6WM6hbxV8PBTg46nMzCztPFr03qFX+zVVxzpVVMEnu0B5e7iXaq7cXLXEuOreIYkSA6B4KFYAgGIpzUEZjMk+tyghxaqEFKviU9IVn5Ju+z57erric32NT0lXcmpGsTJbM40OnDmfb3qeQmT76mnb8xPgk32rcGFawIVptvne2XuXSrp36M7ral7WD+uuWmJcde8QABQHxQoAXJwrHXJX1KAMkvTct78p0MdTSak5ZSmnFP1dkHKmJ55PL9bQ2xcL8PGQl7ubTp9LL3LZx2+qpx7Nwi5ZiEqLq+4dkly3xLB3CEB5R7ECABfmrEPusrKMEs5bdeZc+oVbms6csxY5KIMknTybpkGfbC7W43l5uKmSn6cq+Xmpou1rzveequjnpUp+Xhe+z74f5OspT3c3bdx/2q6R5drVDlHDsMBi5XKEq+4dkigxAFAWKFYA4KJK85C7tIxMxZ+z6vS5tFxlKe/t9Ll0xV/4Pj4lXY5cvqhqgLeqV/LNVY688hSkihcKUqUL9329Sn5ukauOLCe57t4hAEDpo1gBgAuy5zpIzy/8TZUreCvxvFWncwrRhYJ0cWk6m1a8c5JyBPh4KMTfS8EXbhmZWVr9x6ki13tn4DVX/XWHcrB3CACuDhQrAMjFmeczZWYZnTqbprikVK3942SR10E6kZym2z/YaPf23d0squTnpRB/L1Xy91SIv7etMOXcsudlf63o5yUvj5INysB1hwAAVxuKFQBcUFbnMxljFJ9i1fHEVMUlp+pEUqrikrILVFxSmk4kpyouKVUnk9OKffhdJT9PVa/kZytFwbmKUd7C5K0AHw+5OVgSXXnvEIfdAQCciWIFACrZ+UzGGCWnZdiK0t/FKac0ZU8/mZym9Ez7rpvk7mZRlQre8vdy1/5T54pc/j+Dr71sh9zlcOW9Qxx2BwBwFooVgKuePUOIPzX/V/0Ue0Ynz6YrLunvvU7nrZl2P06Iv5dCA30UGuit0EAfVc35PsDHNj2kgrfc3Swue8hdDvYOAQCQF8UKwFUp8bxVh+NTdCT+vNbvPVXkEOJJqRma8b+/CpwX6OOh0EAfhQX5qGrA38UpNND7QnnyUZUK3vnOV7oUVz7kLgd7hwAA+BvFCoDTlNVAEcYYnTmXriMJ53U4/ryOxJ/PLlG57ieXYJS8Lg2qqGPdyhdK04XiFODj0FDhl+LKh9wBAIC8KFYAnMKRgSKysoxOnk3T4VyFKbs8nbd9b88heiH+XqpWyVe+nm7aHBtf5PIPdq7jlPOZOOQOAADXR7ECcNkVNVDEe4OuUcualXT4TN69TNnfp+hoQqpdg0HkXKS2WiW/7K8VfVW9UvYtoqKv/Lyy3wJd/XwmDrkDAMD1UawAXFb2DBQxcs62IrfjZpHCg3xVrZKvqle88LWSr6pVzC5R4RV95O1h3yF65eF8JgAA4NooVgDKVKo1U3+ePKe9J5K1N+6sNv15usiBIiTJ3U2qXsnPtpepWkW/XOXJV2FBPvJ0t38wiKJwPhMAAHAExQpAqUjLyFTsqXP6I+6s9sYl64+47CL11+lzxb7orST9+46Wuu2aaqUf9BI4nwkAAJQUxQq4SpTWCHzpGVkXClTyhQJ1Vn+cSNaB0ynKLKRBBfp4qH5ogOqFBsjTzaLPNh0o8nHCAn2Kna00cD4TAAAoCYoVcBUoyQh81szcBeqs9p7ILlF/nTqnjEIKVEBOgapaQfVCA1Q/tILqhwaoaoC3LJbsgpKZZbRsd5zLDhQBAABQEhQr4ApX1Ah87w66Rg3DArL3PF0oUX/EJSv2EgWqgreH6oVWUP2qAdlfQwNUPzRAoYF/F6jCMFAEAAC4ElGsgCuYPSPwjbrECHwVvD1Ut2oF256nehf2RoUH+RRZoC6FgSIAAMCVhmIFXKHOnEvXvC0H7RqBz8fDTQ3CA1W/agXVC805jC9AEQ4WqEthoAgAAHAloVgBV4BUa6Z2HknU9kMJ+uVwon45lKCDZ1LsXv+1/s3V9zKPwCcxUAQAALhyUKyAciYzy2jfibP65VCCth9O0C+HEvT78eQCR+QLD/Kxa49VqJNG4AMAALhSUKwAF2aM0bHEVFuJ2n4wQTuOJColPTPfslUCvNWyRkW1rFFRLapXVLPqQarg7aFOk1cyAh8AAEAZo1gBZaCk14xKPG/VjsOJ2n4oXtsPJeqXwwk6mZyWbzl/L3c1qx6kFjUqqmX1impRo2KhA0owAh8AAEDZo1gBpczea0alZWRq97Fk/XIowbZH6s+T5/Jtz93NooZhAXlKVN2qFewuQ4zABwAAUPYoVkAputQ1ox76Yqvu7RApY6TthxK061iSrJn5D9CrGeyXXaJqVFTLGkFqHB4kXy93h3IxAh8AAEDZolgBpcSea0bN2nAgz/RKfp7Z50Tl3KpXVLC/V5nkYwQ+AACAskOxAkrJhn2n7BqB75amYereLFwtq1dUjWDfMrtOFAAAAC4fihXggLSMTK3fe0r/t+O4/m/HUbvW6dY0TL1bRJRxMgAAAFxOFCugmFKtmVrzx0n9sOOYlu8+obNpGcVav2oA14wCAAC40rg5O0BxZGZm6rnnnlNUVJR8fX1Vp04dvfjiizLm77NajDF6/vnnFR4eLl9fX0VHR2vv3r1OTI0rQUp6hv5vxzGNmrNVrV5cphGfx+jb7Ud1Ni1DoYHeurdDLX15f1uFBfqosAP7LMq+YC/XjAIAALjylKs9VpMnT9b777+vTz/9VE2aNNHPP/+sYcOGKSgoSI899pgk6fXXX9fUqVP16aefKioqSs8995y6deumXbt2yceHPQWw39m0DK38/YR+2HFMq/acUKo1yzavWkVf9Wgaph7NwnRNjUpyuzAQxMTeXDMKAADgalSuitWGDRvUp08f9ezZU5JUq1Ytffnll/rpp58kZe+tmjJlisaPH68+ffpIkj777DOFhobq22+/1cCBA52WHeVDUqpVK3bH6f92HNeaP04qPePvMlUj2Fe3NAvXLU3D1bx6UIGDTnDNKAAAgKtTuSpWHTp00EcffaQ//vhD9evX1y+//KL169frrbfekiTFxsbq+PHjio6Otq0TFBSktm3bauPGjYUWq7S0NKWlpdnuJyUlSZKsVqusVmsZPqOi5Ty+s3PkdqVlSkixasXvJ7Tktzj9b//pPNeWqhXipx5NQtWtSagahwfYylRGRuHnVd3UoLJuqPcPbdp/Uis3xujG9teqXZ0qcnezOP01u9J+dmWFTPZzxVxksg+Z7EMm+5DJPmSyj6tlsjeHxeQ+QcnFZWVl6dlnn9Xrr78ud3d3ZWZm6uWXX9a4ceMkZe/R6tixo44eParw8L/3DNx5552yWCyaN29egdudOHGiJk2alG/6nDlz5OfnVzZPBk511irtOGPR9tMW/ZFkUZb5e+9TqK9RyxCjlsFZCveTGA0dAADg6pWSkqJBgwYpMTFRgYGBhS5XrvZYffXVV5o9e7bmzJmjJk2aaPv27Ro9erQiIiI0dOjQEm933LhxGjt2rO1+UlKSatSooa5du17yxbscrFarli1bpptvvlmenp5OzZLD1TJlZpkC9w5d7NTZNP2464SW/hanzX/FKzPr778pNAitoG5NQtW9SajqVa1QKrlc7XWSyGQvMtnPFXORyT5ksg+Z7EMm+5DJPq6WKedotqKUq2L15JNP6plnnrEd0tesWTMdOHBAr776qoYOHaqwsDBJUlxcXJ49VnFxcWrZsmWh2/X29pa3t3e+6Z6eni7xw5RcK0sOV8i0ZOexXOczueuzvdsVnut8prikVC3ZeVz/t+OYfvrrjHLvn21aLVA9moarR9Mw1a5SOmWqIK7wOl2MTPYhk/1cMReZ7EMm+5DJPmSyD5ns4yqZ7M1QropVSkqK3NzyjhDv7u6urKzsAQaioqIUFhamFStW2IpUUlKSNm/erIcffvhyx0UZW7LzmB7+YqsuPpb1WGKqHvpiq+pU8df+k+fyzGtRo6JuaRqmHk3DVTOEwzwBAABQOspVserVq5defvll1axZU02aNNG2bdv01ltvafjw4ZIki8Wi0aNH66WXXlK9evVsw61HRESob9++zg2PUpWZZTRp8a58pSq3nFJ1bWQl9Wgapu5Nw1S9EmUKAAAApa9cFat3331Xzz33nB555BGdOHFCERERGjFihJ5//nnbMk899ZTOnTunBx98UAkJCerUqZOWLFnCNayuMD/FnskznHlhpg26Rj2bR1yGRAAAALialatiFRAQoClTpmjKlCmFLmOxWPTCCy/ohRdeuHzBcNkdTzxv13IZWeVm0EsAAACUY25FLwK4lrV/nNRby/6wa9mqAeypBAAAQNkrV3uscHXbdyJZL32/W6v3nJSUfX2pwq7CZpEUFuSjNlHBly8gAAAArloUK7i8M+fSNWX5H5q9+aAys4w83S0a0r6WGocH6p9f/yJJeQaxyLmC1YRejQu8nhUAAABQ2ihWcFlpGZn6dMNfenflPiWnZkiSujUJ1TM9Gimqsr8kyd/bPdd1rLKF5bqOFQAAAHA5UKzgcowxWrLzuF794XcdPJMiSWoSEajxPRurfZ2QPMt2bxqumxuHaeO+E/px3WZ1/Udbta9blT1VAAAAuKwoVnApvx5O0Evf7dZPf52RJFUN8NaT3RqoX6vqhZYldzeL2kYF6/Ruo7ZRwZQqAAAAXHYUK7iEY4nn9cbSPfpm6xFJko+nmx7sXEcjOteWvze/pgAAAHBtfGKFU6WkZ+jDNX/qw7X7lWrNkiT1u6aa/tmtgSIq+jo5HQAAAGAfihWcIivL6JttR/TG0t8Vl5QmSWpdq5Keu7Wxmlev6NxwAAAAQDFRrHDZbfrztF76fpd2HkmSJNUI9tWzPRqpe9MwWSycHwUAAIDyh2KFy+avU+f06g+7tfS3OElSgLeHHr2proZ2qCVvD3cnpwMAAABKjmKFMpd43qp3V+zVpxv/kjXTyM0iDWpbU2Oi6yukgrez4wEAAAAOo1ihzFgzszRn80FNWf6H4lOskqQbGlTRs7c0Uv3QACenAwAAAEoPxQqlzhijVXtO6OXvd2v/yXOSpPqhFfSvno11ff0qTk4HAAAAlD6KFUrV78eT9PL3u7Vu7ylJUoi/l8bcXF8DW9eQh7ubk9MBAAAAZYNihWLJzDLaHHtGMacsCok9o/Z1q8rdzaKTyWl6a9kfmrfloLKM5OXupmGdamlkl7oK9PF0dmwAAACgTFGsYLclO49p0uJdOpaYKsldn+39WWGB3mpXO0TLd5/Q2bQMSVLPZuF6pkdD1Qj2c25gAAAA4DKhWMEuS3Ye08NfbJW5aPrxpDR9u/2oJKlF9SA9d2tjXVcr+PIHBAAAAJyIYoUiZWYZTVq8K1+pyq2ir6fmP9RBnh6cRwUAAICrD5+CUaSfYs9cOPyvcAnnrfr5QPxlSgQAAAC4FooVinQi+dKlqrjLAQAAAFcaihWKVDXAp1SXAwAAAK40FCsUqU1UsCp4F346nkVSeJCP2kQxaAUAAACuThQrFGnd3pO2odQvZrnwdUKvxnJ3sxS4DAAAAHClo1jhkg7Hp2j0vO2SpH/Uq6zwoLyH+4UF+ej9u1upe9NwJ6QDAAAAXAPDraNQaRmZemT2ViWkWNW8epA+GXqdPNzctHHfCf24brO6/qOt2tetyp4qAAAAXPUoVijUi9/t0q+HE1XRz1P/GdxK3h7ukqS2UcE6vduobVQwpQoAAAAQhwKiEAu2HdYXmw7KYpGmDGip6pX8nB0JAAAAcFkUK+Tz+/EkjftmhyTpsRvr6YYGVZ2cCAAAAHBtFCvkkZxq1cNfbFWqNUv/qFdZj91Uz9mRAAAAAJdHsYKNMUZPfv2rYk+dU0SQj94ZeA3nUAEAAAB2oFjBZvr6WC357bg83S36z93XKtjfy9mRAAAAgHKBYgVJ0k+xZ/TqD79Lkp6/tbFa1qjo3EAAAABAOUKxgk4kp2rknK3KzDLq2zJCd7eLdHYkAAAAoFyhWF3lMjKz9OicbTqZnKb6oRX0Sr9mslg4rwoAAAAoDorVVe6NH/doc+wZVfD20Pt3Xys/L64ZDQAAABQXxeoqtvS34/pwzZ+SpNdvb646VSo4OREAAABQPlGsrlJ/nTqnf371iyTpvk5RuqVZuJMTAQAAAOUXxeoqdD49Uw99EaPktAxdF1lJz/Ro6OxIAAAAQLlGsbrKGGM0/tud+v14sipX8NK0wa3k6c6vAQAAAOAIPlFfZeZuOaT/bj0sN4v07l2tFBro4+xIAAAAQLlHsbqK7DicqAkLf5MkPdmtodrXCXFyIgAAAODKQLG6SiSkpOvh2TFKz8xSdKNQPXR9bWdHAgAAAK4YFKurQFaW0Zh523U4/rxqBvvpzTtbcBFgAAAAoBRRrK4C/1m9T6v2nJS3h5vev7uVgnw9nR0JAAAAuKJQrK5w6/ae1JvL/pAkvdi3qZpEBDk5EQAAAHDloVhdwY4mnNfjc7fLGGlg6xq687oazo4EAAAAXJEoVleo9IwsjZyzVWfOpatptUBN7N3E2ZEAAACAKxbF6gr1yv/t1raDCQr08dD7g6+Vj6e7syMBAAAAVyyK1RVo0S9HNWvDX5Kktwe0VI1gP+cGAgAAAK5wFKsrzN64ZD3z318lSSO71NFNjUKdnAgAAAC48lGsriBn0zL00BcxSknPVMe6IRp7cwNnRwIAAACuChSrK4QxRs/891ftP3lOYYE+emfgNXJ34yLAAAAAwOVAsbpCzNrwl7779Zg83CyaNriVKlfwdnYkAAAA4KpBsboCxBw4o5e/3y1J+lfPRro2spKTEwEAAABXF4pVOXfqbJpGzt6mjCyjns3DdW+HWs6OBAAAAFx1KFblWGaW0WNfbtPxpFTVqeKvyf2by2LhvCoAAADgcqNYlWNvLdujDftPy8/LXR/cfa0qeHs4OxIAAABwVaJYlVMrdsdp2qr9kqTX+jdXvdAAJycCAAAArl4Uq3Lo4OkUjZm3XZJ0b4da6t0iwrmBAAAAgKscxaqcSbVm6uHZMUpKzdA1NSvq2VsaOTsSAAAAcNWjWJUzExf9pt+OJinY30v/GdxKXh78CAEAAABn41N5OfLVz4c0d8shWSzS1IHXKDzI19mRAAAAAEhiGDkXlplltDn2jGJOWZT88yG98N3vkqSx0fXVqV5lJ6cDAAAAkINi5aKW7DymSYt36VhiqiR3ae9uSVLTiECN7FLXueEAAAAA5MGhgC5oyc5jeviLrRdKVV6/HU3Sj7uOOyEVAAAAgMJQrFxMZpbRpMW7ZC6xzKTFu5SZdaklAAAAAFxOFCsX81PsmQL3VOUwko4lpuqn2DOXLxQAAACAS6JYuZgTyYWXqpIsBwAAAKDsUaxcTNUAn1JdDgAAAEDZo1i5mDZRwQoP8pGlkPkWSeFBPmoTFXw5YwEAAAC4BIqVi3F3s2hCr8aSlK9c5dyf0Kux3N0Kq14AAAAALjeKlQvq3jRc79/dSmFBeQ/3Cwvy0ft3t1L3puFOSgYAAACgIFwg2EV1bxqumxuHaeO+E/px3WZ1/Udbta9blT1VAAAAgAuiWLkwdzeL2kYF6/Ruo7ZRwZQqAAAAwEVxKCAAAAAAOIhiBQAAAAAOolgBAAAAgIMoVgAAAADgIIcGrzh16pROnToli8WiypUrKyQkpLRyAQAAAEC5Uaxide7cOX399ddauHChNmzYoFOnTuWZX7lyZbVv3159+/bVHXfcIX9//1INCwAAAACuyK5idfr0ab366qv68MMPlZqaqubNm6tPnz6qXbu2KlWqJGOM4uPjFRsbq5iYGD3wwAN69NFHNWLECD3zzDOqXLlyWT8PAAAAAHAau4pVrVq1VLduXb3xxhvq37+/qlSpcsnlT548qf/+97/66KOP9NFHHykpKalUwkrSkSNH9PTTT+uHH35QSkqK6tatq5kzZ+q6666TJBljNGHCBH388cdKSEhQx44d9f7776tevXqllgEAAAAAcrNr8Ir58+dr27Zteuihh4osVZJUpUoVPfTQQ9q6dau+/vprh0PmiI+PV8eOHeXp6akffvhBu3bt0ptvvqlKlSrZlnn99dc1depUffDBB9q8ebP8/f3VrVs3paamlloOAAAAAMjNrj1W3bp1K/EDOLLuxSZPnqwaNWpo5syZtmlRUVG2740xmjJlisaPH68+ffpIkj777DOFhobq22+/1cCBA0stCwAAAADkcGhUwNyOHj2qI0eOKCwsTDVq1CitzeaxaNEidevWTXfccYfWrFmjatWq6ZFHHtEDDzwgSYqNjdXx48cVHR1tWycoKEht27bVxo0bCy1WaWlpSktLs93POXTRarXKarWWyXOxV87jOztHbmSyD5nsQyb7uGImyTVzkck+ZLIPmexDJvuQyT6ulsneHBZjjHHkgY4dO6ZBgwZpzZo12Ru0WNSuXTvNnj1btWrVcmTT+fj4+EiSxo4dqzvuuENbtmzR448/rg8++EBDhw7Vhg0b1LFjRx09elTh4eG29e68805ZLBbNmzevwO1OnDhRkyZNyjd9zpw58vPzK9XnAAAAAKD8SElJ0aBBg5SYmKjAwMBCl3O4WPXp00fe3t56/fXXFRERoV27dmn48OGqWLGiVq5c6cim8/Hy8tJ1112nDRs22KY99thj2rJlizZu3FjiYlXQHqsaNWro1KlTl3zxLger1aply5bp5ptvlqenp1Oz5CCTfchkHzLZxxUzSa6Zi0z2IZN9yGQfMtmHTPZxtUxJSUmqXLlykcXK7kMBX3vtNT3xxBP5ntzPP/+s7777zrZ3qmXLlrr//vs1bty4kiW/hPDwcDVu3DjPtEaNGum///2vJCksLEySFBcXl6dYxcXFqWXLloVu19vbW97e3vmme3p6usQPU3KtLDnIZB8y2YdM9nHFTJJr5iKTfchkHzLZh0z2IZN9XCWTvRnsGhVQkr766is1atRICxcuzDP92muv1eTJk3Xo0CFlZGRo586dmj59ulq1alW8xHbo2LGj9uzZk2faH3/8ocjISEnZA1mEhYVpxYoVtvlJSUnavHmz2rdvX+p5AAAAAEAqRrGKiYnRk08+qQceeEDR0dH67bffJEkffPCBjhw5osjISHl7e6t58+Zyd3fXjBkzSj3smDFjtGnTJr3yyivat2+f5syZo48++kgjR46UlH1+1+jRo/XSSy9p0aJF2rFjh4YMGaKIiAj17du31PMAAAAAgFSMYmWxWDRixAjt3btXTZs21XXXXadRo0bJ19dX69at04EDB7Rx40bFxsbqp59+yjMMemlp3bq1FixYoC+//FJNmzbViy++qClTpmjw4MG2ZZ566ik9+uijevDBB9W6dWudPXtWS5YssQ18AQAAAAClze5ilSMoKEhTpkxRTEyM9u7dq7p16+rdd99VtWrV1KZNG9theWXl1ltv1Y4dO5Samqrdu3fbhlrPYbFY9MILL+j48eNKTU3V8uXLVb9+/TLNBAAAAODqVuxilaNx48ZaunSpZs6cqXfffVfNmjXTsmXLSjMbAAAAAJQLdhers2fP6uGHH1a1atVUqVIlde/eXbt27VLv3r3122+/aciQIerfv7969+6t/fv3l2VmAAAAAHApdherRx55RIsWLdIrr7yiTz/9VOfPn9ctt9yi9PR0eXp66umnn9aePXtUqVIlNWvWTE899VRZ5gYAAAAAl2F3sfr+++81btw4DR06VL1799Ynn3yigwcP2kYHlLKvM/Xpp59q9erVWrduXZkEBgAAAABXY3exCgoKUmxsrO3+X3/9JYvFoqCgoHzLtmnTRhs3biydhAAAAADg4jzsXfDpp5/WI488ol9++UWVKlXSDz/8oH79+ql27dplmQ8AAAAAXJ7dxWrEiBFq0qSJvv/+e50/f14ffvih7rrrrrLMBgAAAADlgt3FSpI6deqkTp06lVUWAAAAACiX7DrHKiUlpcQP4Mi6AAAAAFAe2FWsatSooRdeeEHHjh2ze8NHjhzR888/r5o1a5Y4HAAAAACUB3YdCvj+++9r4sSJeuGFF9SxY0dFR0erVatWioqKUqVKlWSMUXx8vGJjY/Xzzz9r+fLl2rRpk+rVq6f//Oc/Zf0cAAAAAMCp7CpWd955p26//XYtWrRIs2bN0ssvv6z09HRZLJY8yxlj5OXlpa5du2r+/Pnq3bu33NzsHtEdAAAAAMoluwevcHNzU9++fdW3b1+lpaUpJiZGv//+u06fPi1JCgkJUcOGDXXttdfK29u7zAIDAAAAgKsp1qiAOby9vdWhQwd16NChtPMAAAAAQLnDcXoAAAAA4CCKFQAAAAA4iGIFAAAAAA6iWAEAAACAgyhWAAAAAOCgEhWrzZs3l3YOAAAAACi3SlSs2rdvr/r16+vFF1/Un3/+WdqZAAAAAKBcKVGx+uKLL1SvXj29+OKLqlevnjp27KgPPvhAZ86cKe18AAAAAODySlSsBg0apO+//15Hjx7VO++8I2OMHnnkEUVERKhv376aP3++0tPTSzsrAAAAALgkhwavqFy5skaNGqUNGzZo7969+te//qXff/9dAwYMUFhYmB588EGtX7++tLICAAAAgEsqtVEBfX195efnJx8fHxljZLFYtHDhQl1//fVq3bq1du3aVVoPBQAAAAAuxaFilZycrJkzZyo6OlqRkZF69tlnVatWLc2fP1/Hjx/X0aNHNW/ePJ04cULDhg0rrcwAAAAA4FI8SrLSwoULNXv2bH333XdKTU1V69atNWXKFA0cOFAhISF5lr399tsVHx+vkSNHlkpgAAAAAHA1JSpWt912m2rUqKExY8ZoyJAhatCgwSWXb9GihQYPHlyigAAAAADg6kpUrFauXKkbbrjB7uXbtGmjNm3alOShAAAAAMDllegcq+KUKgAAAAC40pWoWI0fP14tW7YsdP4111yjSZMmlTQTAAAAAJQrJSpW8+fPV48ePQqdf8stt2jevHklDgUAAAAA5UmJitXBgwdVp06dQudHRUXpwIEDJQ4FAAAAAOVJiYpVhQoVLlmcYmNj5ePjU+JQAAAAAFCelHjwig8//FBHjhzJN+/QoUP66KOP1KVLF4fDAQAAAEB5UKLh1l988UW1adNGTZo00X333acmTZpIknbu3KkZM2bIGKMXX3yxVIMCAAAAgKsqUbFq0KCB1q1bp0cffVRvv/12nnmdO3fW1KlT1ahRo1IJCAAAAACurkTFSpKaN2+uNWvW6NSpU/rzzz8lSbVr11blypVLLRwAAAAAlAclLlY5KleuTJkCAAAAcFVzqFgdPnxY27ZtU2JiorKysvLNHzJkiCObBwAAAIByoUTFKjU1VUOHDtV///tfZWVlyWKxyBgjSbJYLLblKFYAAAAArgYlGm792Wef1TfffKOXX35Zq1evljFGn376qX788Uf16NFDLVq00C+//FLaWQEAAADAJZWoWM2fP1/Dhg3T008/bRtqvVq1aoqOjtZ3332nihUratq0aaUaFAAAAABcVYmK1YkTJ9SmTRtJkq+vryTp3Llztvn9+/fXN998UwrxAAAAAMD1lahYhYaG6vTp05IkPz8/VapUSXv27LHNT0pKUmpqaukkBAAAAAAXV6LBK9q2bav169fr6aefliT16tVLb7zxhsLDw5WVlaW3335b7dq1K9WgAAAAAOCqSrTH6rHHHlPt2rWVlpYmSXrxxRdVsWJF3XPPPRo6dKiCgoI0derUUg0KAAAAAK6qRHusOnXqpE6dOtnu16hRQ7t379aOHTvk7u6uhg0bysPD4WsPAwAAAEC5UOw9VikpKerXr59mz56dd0NubmrRooWaNm1KqQIAAABwVSl2sfLz89Py5cuVkpJSFnkAAAAAoNwp0TlWnTp10saNG0s7CwAAAACUSyUqVu+9957WrVun8ePH6/Dhw6WdCQAAAADKlRIVqxYtWujw4cN69dVXFRkZKW9vbwUGBua5BQUFlXZWAAAAAHBJJRplon///rJYLKWdBQAAAADKpRIVq1mzZpVyDAAAAAAov0p0KCAAAAAA4G8l2mP12Wef2bXckCFDSrJ5AAAAAChXSlSs7r333kLn5T73imIFAAAA4GpQomIVGxubb1pmZqb++usv/ec//9HBgwf16aefOhwOAAAAAMqDEhWryMjIAqfXrl1bN954o3r27Kn33ntP06ZNcygcAAAAAJQHZTJ4xa233qp58+aVxaYBAAAAwOWUSbHav3+/0tLSymLTAAAAAOBySnQo4Nq1awucnpCQoLVr12rq1Knq27evI7kAAAAAoNwoUbG64YYb8oz+l8MYI3d3d91xxx169913HQ4HAAAAAOVBiYrVqlWr8k2zWCyqVKmSIiMjFRgY6HAwAAAAACgvSlSsrr/++tLOAQAAAADlVokGr4iNjdXixYsLnb948WL99ddfJc0EAAAAAOVKifZY/fOf/1RSUpJ69epV4Pxp06apYsWKmjt3rkPhAAAAAKA8KNEeq40bN+rmm28udP5NN92kdevWlTgUAAAAAJQnJSpW8fHxCggIKHR+hQoVdPr06RKHAgAAAIDypETFqmbNmvrf//5X6Px169apevXqJQ4FAAAAAOVJiYrVXXfdpS+//FJTp05VVlaWbXpmZqbeeecdzZs3T4MGDSq1kAAAAADgyko0eMW4ceO0fv16jR49Wi+//LIaNGggSdqzZ49OnjypG264Qf/6179KNSgAAAAAuKoS7bHy9vbWjz/+qOnTp6tNmzY6deqUTp06pTZt2mjGjBlavny5vL29SzsrAAAAALikEu2xkiQ3NzcNGzZMw4YNK808AAAAAFDulGiP1ZkzZ/Trr78WOn/Hjh2Kj48vcSgAAAAAKE9KVKzGjBmjBx98sND5I0aM0D//+c8ShwIAAACA8qRExWrlypXq3bt3ofN79eql5cuXlzgUAAAAAJQnJSpWJ0+eVOXKlQudHxISohMnTpQ4FAAAAACUJyUqVuHh4dq2bVuh82NiYlSlSpUShwIAAACA8qRExapv376aPn26Fi1alG/ewoULNXPmTN12220OhwMAAACA8qBExWrixIlq0KCBbrvtNrVq1UpDhgzRkCFD1KpVK/Xr10/169fXpEmTSjtrPq+99posFotGjx5tm5aamqqRI0cqJCREFSpUUP/+/RUXF1fmWQAAAABcvUpUrIKCgrRp0yaNHz9eVqtV8+fP1/z582W1WvXcc89p8+bNqlixYilHzWvLli368MMP1bx58zzTx4wZo8WLF+vrr7/WmjVrdPToUfXr169MswAAAAC4upX4AsH+/v6aNGlSoXum4uPjValSpRIHu5SzZ89q8ODB+vjjj/XSSy/ZpicmJmr69OmaM2eObrzxRknSzJkz1ahRI23atEnt2rUrkzwAAAAArm4lLlYFSUtL06JFizR79mwtWbJEqamppbl5m5EjR6pnz56Kjo7OU6xiYmJktVoVHR1tm9awYUPVrFlTGzduLLRYpaWlKS0tzXY/KSlJkmS1WmW1WsvkOdgr5/GdnSM3MtmHTPYhk31cMZPkmrnIZB8y2YdM9iGTfchkH1fLZG8OizHGOPJAxhitWLFCs2fP1oIFC5SUlKQqVaqoZ8+emjFjhiObLtDcuXP18ssva8uWLfLx8dENN9ygli1basqUKZozZ46GDRuWpyRJUps2bdSlSxdNnjy5wG1OnDixwD1vc+bMkZ+fX6k/BwAAAADlQ0pKigYNGqTExEQFBgYWulyJ91jFxMRo9uzZmjt3ro4fPy6LxaKBAwdq1KhRateunSwWS0k3XahDhw7p8ccf17Jly+Tj41Nq2x03bpzGjh1ru5+UlKQaNWqoa9eul3zxLger1aply5bp5ptvlqenp1Oz5CCTfchkHzLZxxUzSa6Zi0z2IZN9yGQfMtmHTPZxtUw5R7MVpVjF6s8//9Ts2bM1e/Zs7d27V9WqVdPgwYPVpk0bDRgwQP3791f79u1LFNgeMTExOnHihFq1amWblpmZqbVr1+q9997T0qVLlZ6eroSEhDyDZ8TFxSksLKzQ7Xp7e8vb2zvfdE9PT5f4YUqulSUHmexDJvuQyT6umElyzVxksg+Z7EMm+5DJPmSyj6tksjeD3cWqffv2+umnn1S5cmXdfvvt+uSTT9SpUydJ0v79+0uWsphuuukm7dixI8+0YcOGqWHDhnr66adVo0YNeXp6asWKFerfv78kac+ePTp48GCZFj4AAAAAVze7i9XmzZsVFRWlt956Sz179pSHR6mOe2GXgIAANW3aNM80f39/hYSE2Kbfd999Gjt2rIKDgxUYGKhHH31U7du3Z0RAAAAAAGXG7utYvffeewoPD9dtt92msLAwjRgxQqtWrZKDY1+Uurffflu33nqr+vfvr86dOyssLEzffPONs2MBAAAAuILZvdvpkUce0SOPPKLY2FjNnj1bc+bM0ccff6ywsDB16dJFFoulTAasKMrq1avz3Pfx8dG0adM0bdq0y54FAAAAwNXJ7j1WOaKiojR+/Hjt2rVLW7Zs0cCBA7V69WoZY/TII4/owQcf1HfffVdm17ACAAAAAFdT7GKV27XXXqu33npLhw4d0o8//qhu3bpp3rx56t27typXrlxaGQEAAADApTlUrGwbcXNTdHS0Zs2apbi4OH355Ze66aabSmPTAAAAAODySqVY5ebj46MBAwZo4cKFpb1pAAAAAHBJpV6sAAAAAOBqQ7ECAAAAAAdRrAAAAADAQRQrAAAAAHAQxQoAAAAAHESxAgAAAAAHUawAAAAAwEEUKwAAAABwEMUKAAAAABxEsQIAAAAAB1GsAAAAAMBBFCsAAAAAcBDFCgAAAAAcRLECAAAAAAdRrAAAAADAQRQrAAAAAHAQxQoAAAAAHESxAgAAAAAHUawAAAAAwEEUKwAAAABwEMUKAAAAABxEsQIAAAAAB1GsAAAAAMBBFCsAAAAAcBDFCgAAAAAcRLECAAAAAAdRrAAAAADAQRQrAAAAAHAQxQoAAAAAHESxAgAAAAAHUawAAAAAwEEUKwAAAABwEMUKAAAAABxEsQIAAAAAB1GsAAAAAMBBFCsAAAAAcBDFCgAAAAAcRLECAAAAAAdRrAAAAADAQRQrAAAAAHAQxQoAAAAAHESxAgAAAAAHUawAAAAAwEEUKwAAAABwEMUKAAAAABxEsQIAAAAAB1GsAAAAAMBBFCsAAAAAcBDFCgAAAAAcRLECAAAAAAdRrAAAAADAQRQrAAAAAHAQxQoAAAAAHESxAgAAAAAHUawAAAAAwEEUKwAAAABwEMUKAAAAABxEsQIAAAAAB1GsAAAAAMBBFCsAAAAAcBDFCgAAAAAcRLECAAAAAAdRrAAAAADAQRQrAAAAAHAQxQoAAAAAHESxAgAAAAAHUawAAAAAwEEUKwAAAABwEMUKAAAAABxEsQIAAAAAB1GsAAAAAMBBFCsAAAAAcBDFCgAAAAAcRLECAAAAAAdRrAAAAADAQRQrAAAAAHAQxQoAAAAAHFSuitWrr76q1q1bKyAgQFWrVlXfvn21Z8+ePMukpqZq5MiRCgkJUYUKFdS/f3/FxcU5KTEAAACAq0G5KlZr1qzRyJEjtWnTJi1btkxWq1Vdu3bVuXPnbMuMGTNGixcv1tdff601a9bo6NGj6tevnxNTAwAAALjSeTg7QHEsWbIkz/1Zs2apatWqiomJUefOnZWYmKjp06drzpw5uvHGGyVJM2fOVKNGjbRp0ya1a9fOGbEBAAAAXOHKVbG6WGJioiQpODhYkhQTEyOr1aro6GjbMg0bNlTNmjW1cePGQotVWlqa0tLSbPeTkpIkSVarVVartazi2yXn8Z2dIzcy2YdM9iGTfVwxk+SauchkHzLZh0z2IZN9yGQfV8tkbw6LMcaUcZYykZWVpd69eyshIUHr16+XJM2ZM0fDhg3LU5IkqU2bNurSpYsmT55c4LYmTpyoSZMm5Zs+Z84c+fn5lX54AAAAAOVCSkqKBg0apMTERAUGBha6XLndYzVy5Ejt3LnTVqocMW7cOI0dO9Z2PykpSTVq1FDXrl0v+eJdDlarVcuWLdPNN98sT09Pp2bJQSb7kMk+ZLKPK2aSXDMXmexDJvuQyT5ksg+Z7ONqmXKOZitKuSxWo0aN0nfffae1a9eqevXqtulhYWFKT09XQkKCKlasaJseFxensLCwQrfn7e0tb2/vfNM9PT1d4ocpuVaWHGSyD5nsQyb7uGImyTVzkck+ZLIPmexDJvuQyT6uksneDOVqVEBjjEaNGqUFCxZo5cqVioqKyjP/2muvlaenp1asWGGbtmfPHh08eFDt27e/3HEBAAAAXCXK1R6rkSNHas6cOVq4cKECAgJ0/PhxSVJQUJB8fX0VFBSk++67T2PHjlVwcLACAwP16KOPqn379owICAAAAKDMlKti9f7770uSbrjhhjzTZ86cqXvvvVeS9Pbbb8vNzU39+/dXWlqaunXrpv/85z+XOSkAAACAq0m5Klb2DGDo4+OjadOmadq0aZchEQAAAACUs3OsAAAAAMAVUawAAAAAwEEUKwAAAABwEMUKAAAAABxEsQIAAAAAB1GsAAAAAMBBFCsAAAAAcBDFCgAAAAAcRLECAAAAAAdRrAAAAADAQRQrAAAAAHAQxQoAAAAAHESxAgAAAAAHUawAAAAAwEEUKwAAAABwEMUKAAAAABxEsQIAAAAAB1GsAAAAAMBBFCsAAAAAcBDFCgAAAAAcRLECAAAAAAdRrAAAAADAQRQrAAAAAHAQxQoAAAAAHESxAgAAAAAHUawAAAAAwEEUKwAAAABwEMUKAAAAABxEsQIAAAAAB1GsAAAAAMBBFCsAAAAAcBDFCgAAAAAcRLECAAAAAAdRrAAAAADAQRQrAAAAAHAQxQoAAAAAHESxAgAAAAAHUawAAAAAwEEUKwAAAABwEMUKAAAAABxEsQIAAAAAB1GsAAAAAMBBFCsAAAAAcBDFCgAAAAAcRLECAAAAAAdRrAAAAADAQRQrAAAAAHAQxQoAAAAAHESxAgAAAAAHUawAAAAAwEEUKwAAAABwEMUKAAAAABxEsQIAAAAAB1GsAAAAAMBBFCsAAAAAcBDFCgAAAAAcRLECAAAAAAdRrAAAAADAQRQrAAAAAHAQxQoAAAAAHESxAgAAAAAHUawAAAAAwEEUKwAAAABwEMUKAAAAABxEsQIAAAAAB1GsAAAAAMBBFCsAAAAAcBDFCgAAAAAcRLECAAAAAAdRrAAAAADAQRQrAAAAAHAQxQoAAAAAHESxAgAAAAAHUawAAAAAwEEUKwAAAABwEMUKAAAAABxEsQIAAAAAB1GsAAAAAMBBFCsAAAAAcBDFCgAAAAAcRLECAAAAAAdRrAAAAADAQVdssZo2bZpq1aolHx8ftW3bVj/99JOzIwEAAAC4Ql2RxWrevHkaO3asJkyYoK1bt6pFixbq1q2bTpw44exoAAAAAK5AV2Sxeuutt/TAAw9o2LBhaty4sT744AP5+flpxowZzo4GAAAA4Ark4ewApS09PV0xMTEaN26cbZqbm5uio6O1cePGAtdJS0tTWlqa7X5iYqIk6cyZM7JarWUbuAhWq1UpKSk6ffq0PD09nZolB5nsQyb7kMk+rphJcs1cZLIPmexDJvuQyT5kso+rZUpOTpYkGWMuudwVV6xOnTqlzMxMhYaG5pkeGhqq33//vcB1Xn31VU2aNCnf9KioqDLJCAAAAKB8SU5OVlBQUKHzr7hiVRLjxo3T2LFjbfezsrJ05swZhYSEyGKxODGZlJSUpBo1aujQoUMKDAx0apYcZLIPmexDJvu4YibJNXORyT5ksg+Z7EMm+5DJPq6WyRij5ORkRUREXHK5K65YVa5cWe7u7oqLi8szPS4uTmFhYQWu4+3tLW9v7zzTKlasWFYRSyQwMNAlfrFyI5N9yGQfMtnHFTNJrpmLTPYhk33IZB8y2YdM9nGlTJfaU5Xjihu8wsvLS9dee61WrFhhm5aVlaUVK1aoffv2TkwGAAAA4Ep1xe2xkqSxY8dq6NChuu6669SmTRtNmTJF586d07Bhw5wdDQAAAMAV6IosVgMGDNDJkyf1/PPP6/jx42rZsqWWLFmSb0CL8sDb21sTJkzId6iiM5HJPmSyD5ns44qZJNfMRSb7kMk+ZLIPmexDJvu4YiZ7WExR4wYCAAAAAC7pijvHCgAAAAAuN4oVAAAAADiIYgUAAAAADqJYAQAAAICDKFYuau3aterVq5ciIiJksVj07bffOjuSXn31VbVu3VoBAQGqWrWq+vbtqz179jg10/vvv6/mzZvbLiDXvn17/fDDD07NlNtrr70mi8Wi0aNHOzXHxIkTZbFY8twaNmzo1EySdOTIEd19990KCQmRr6+vmjVrpp9//tlpeWrVqpXvdbJYLBo5cqTTMmVmZuq5555TVFSUfH19VadOHb344oty9rhDycnJGj16tCIjI+Xr66sOHTpoy5Ytl+3xi3qPNMbo+eefV3h4uHx9fRUdHa29e/c6NdM333yjrl27KiQkRBaLRdu3by/TPPbkslqtevrpp9WsWTP5+/srIiJCQ4YM0dGjR52WScp+z2rYsKH8/f1VqVIlRUdHa/PmzU7NlNtDDz0ki8WiKVOmODXTvffem+/9qnv37k7NJEm7d+9W7969FRQUJH9/f7Vu3VoHDx50WqaC3tctFoveeOMNp2U6e/asRo0aperVq8vX11eNGzfWBx98UGZ57MkUFxene++9VxEREfLz81P37t3L/H3Tns+WqampGjlypEJCQlShQgX1799fcXFxZZqrpChWLurcuXNq0aKFpk2b5uwoNmvWrNHIkSO1adMmLVu2TFarVV27dtW5c+eclql69ep67bXXFBMTo59//lk33nij+vTpo99++81pmXJs2bJFH374oZo3b+7sKJKkJk2a6NixY7bb+vXrnZonPj5eHTt2lKenp3744Qft2rVLb775pipVquS0TFu2bMnzGi1btkySdMcddzgt0+TJk/X+++/rvffe0+7duzV58mS9/vrrevfdd52WSZLuv/9+LVu2TJ9//rl27Nihrl27Kjo6WkeOHLksj1/Ue+Trr7+uqVOn6oMPPtDmzZvl7++vbt26KTU11WmZzp07p06dOmny5MlllqG4uVJSUrR161Y999xz2rp1q7755hvt2bNHvXv3dlomSapfv77ee+897dixQ+vXr1etWrXUtWtXnTx50mmZcixYsECbNm1SREREmWUpTqbu3bvned/68ssvnZpp//796tSpkxo2bKjVq1fr119/1XPPPScfHx+nZcr9+hw7dkwzZsyQxWJR//79nZZp7NixWrJkib744gvt3r1bo0eP1qhRo7Ro0SKnZDLGqG/fvvrzzz+1cOFCbdu2TZGRkYqOji7Tz3n2fLYcM2aMFi9erK+//lpr1qzR0aNH1a9fvzLL5BADlyfJLFiwwNkx8jlx4oSRZNasWePsKHlUqlTJfPLJJ07NkJycbOrVq2eWLVtmrr/+evP44487Nc+ECRNMixYtnJrhYk8//bTp1KmTs2Nc0uOPP27q1KljsrKynJahZ8+eZvjw4Xmm9evXzwwePNhJiYxJSUkx7u7u5rvvvsszvVWrVuZf//rXZc9z8XtkVlaWCQsLM2+88YZtWkJCgvH29jZffvmlUzLlFhsbaySZbdu2XZYsudnz/8lPP/1kJJkDBw64TKbExEQjySxfvtypmQ4fPmyqVatmdu7caSIjI83bb799WfIUlmno0KGmT58+ly3DxQrKNGDAAHP33Xc7J5Cx7/epT58+5sYbb7w8gUzBmZo0aWJeeOGFPNMu53voxZn27NljJJmdO3fapmVmZpoqVaqYjz/++LJkMib/Z8uEhATj6elpvv76a9syu3fvNpLMxo0bL1sue7HHCiWWmJgoSQoODnZykmyZmZmaO3euzp07p/bt2zs1y8iRI9WzZ09FR0c7NUdue/fuVUREhGrXrq3BgweX6WEZ9li0aJGuu+463XHHHapataquueYaffzxx07NlFt6erq++OILDR8+XBaLxWk5OnTooBUrVuiPP/6QJP3yyy9av369evTo4bRMGRkZyszMzPcXaF9fX6fvCZWk2NhYHT9+PM+/v6CgILVt21YbN250YrLyITExURaLRRUrVnR2FEnZ/xY/+ugjBQUFqUWLFk7LkZWVpXvuuUdPPvmkmjRp4rQcF1u9erWqVq2qBg0a6OGHH9bp06edliUrK0vff/+96tevr27duqlq1apq27atS5zOkCMuLk7ff/+97rvvPqfm6NChgxYtWqQjR47IGKNVq1bpjz/+UNeuXZ2SJy0tTZLyvK+7ubnJ29v7sr6vX/zZMiYmRlarNc/7ecOGDVWzZk2XfD+nWKFEsrKyNHr0aHXs2FFNmzZ1apYdO3aoQoUK8vb21kMPPaQFCxaocePGTsszd+5cbd26Va+++qrTMlysbdu2mjVrlpYsWaL3339fsbGx+sc//qHk5GSnZfrzzz/1/vvvq169elq6dKkefvhhPfbYY/r000+dlim3b7/9VgkJCbr33nudmuOZZ57RwIED1bBhQ3l6euqaa67R6NGjNXjwYKdlCggIUPv27fXiiy/q6NGjyszM1BdffKGNGzfq2LFjTsuV4/jx45Kk0NDQPNNDQ0Nt81Cw1NRUPf3007rrrrsUGBjo1CzfffedKlSoIB8fH7399ttatmyZKleu7LQ8kydPloeHhx577DGnZbhY9+7d9dlnn2nFihWaPHmy1qxZox49eigzM9MpeU6cOKGzZ8/qtddeU/fu3fXjjz/qtttuU79+/bRmzRqnZLrYp59+qoCAAKcfSvbuu++qcePGql69ury8vNS9e3dNmzZNnTt3dkqenLIybtw4xcfHKz09XZMnT9bhw4cv2/t6QZ8tjx8/Li8vr3x/6HHV93MPZwdA+TRy5Ejt3LnTJf463aBBA23fvl2JiYmaP3++hg4dqjVr1jilXB06dEiPP/64li1bVqbHkxdX7r0bzZs3V9u2bRUZGamvvvrKaX+1y8rK0nXXXadXXnlFknTNNddo586d+uCDDzR06FCnZMpt+vTp6tGjx2U5j+JSvvrqK82ePVtz5sxRkyZNtH37do0ePVoRERFOfZ0+//xzDR8+XNWqVZO7u7tatWqlu+66SzExMU7LBMdYrVbdeeedMsbo/fffd3YcdenSRdu3b9epU6f08ccf684779TmzZtVtWrVy54lJiZG77zzjrZu3erUPdgXGzhwoO37Zs2aqXnz5qpTp45Wr16tm2666bLnycrKkiT16dNHY8aMkSS1bNlSGzZs0AcffKDrr7/+sme62IwZMzR48GCn/x/97rvvatOmTVq0aJEiIyO1du1ajRw5UhEREU452sXT01PffPON7rvvPgUHB8vd3V3R0dHq0aPHZRssyZU+W5YUe6xQbKNGjdJ3332nVatWqXr16s6OIy8vL9WtW1fXXnutXn31VbVo0ULvvPOOU7LExMToxIkTatWqlTw8POTh4aE1a9Zo6tSp8vDwcNpfES9WsWJF1a9fX/v27XNahvDw8Hzlt1GjRk4/RFGSDhw4oOXLl+v+++93dhQ9+eSTtr1WzZo10z333KMxY8Y4fY9onTp1tGbNGp09e1aHDh3STz/9JKvVqtq1azs1lySFhYVJUr5Ro+Li4mzzkFdOqTpw4ICWLVvm9L1VkuTv76+6deuqXbt2mj59ujw8PDR9+nSnZFm3bp1OnDihmjVr2t7bDxw4oCeeeEK1atVySqaC1K5dW5UrV3bae3vlypXl4eHhsu/t69at0549e5z+3n7+/Hk9++yzeuutt9SrVy81b95co0aN0oABA/Tvf//babmuvfZabd++XQkJCTp27JiWLFmi06dPX5b39cI+W4aFhSk9PV0JCQl5lnfV93OKFexmjNGoUaO0YMECrVy5UlFRUc6OVKCsrCzbscKX20033aQdO3Zo+/btttt1112nwYMHa/v27XJ3d3dKroudPXtW+/fvV3h4uNMydOzYMd+Qqn/88YciIyOdlOhvM2fOVNWqVdWzZ09nR1FKSorc3PK+Vbu7u9v+Muxs/v7+Cg8PV3x8vJYuXao+ffo4O5KioqIUFhamFStW2KYlJSVp8+bNTj//0hXllKq9e/dq+fLlCgkJcXakAjnzvf2ee+7Rr7/+mue9PSIiQk8++aSWLl3qlEwFOXz4sE6fPu2093YvLy+1bt3aZd/bp0+frmuvvdap5+pJ2f/mrFary763BwUFqUqVKtq7d69+/vnnMn1fL+qz5bXXXitPT8887+d79uzRwYMHXfL9nEMBXdTZs2fz/MUpNjZW27dvV3BwsGrWrOmUTCNHjtScOXO0cOFCBQQE2I5tDQoKkq+vr1MyjRs3Tj169FDNmjWVnJysOXPmaPXq1U77jy4gICDfOWf+/v4KCQlx6rlo//znP9WrVy9FRkbq6NGjmjBhgtzd3XXXXXc5LdOYMWPUoUMHvfLKK7rzzjv1008/6aOPPtJHH33ktExS9oe3mTNnaujQofLwcP5bZK9evfTyyy+rZs2aatKkibZt26a33npLw4cPd2qupUuXyhijBg0aaN++fXryySfVsGFDDRs27LI8flHvkaNHj9ZLL72kevXqKSoqSs8995wiIiLUt29fp2U6c+aMDh48aLtGVM6Hz7CwsDL9y+ulcoWHh+v222/X1q1b9d133ykzM9P23h4cHCwvL6/LnikkJEQvv/yyevfurfDwcJ06dUrTpk3TkSNHyvTSB0X9/C4unJ6engoLC1ODBg2ckik4OFiTJk1S//79FRYWpv379+upp55S3bp11a1bN6dkqlmzpp588kkNGDBAnTt3VpcuXbRkyRItXrxYq1evdlomKfuPK19//bXefPPNMstRnEzXX3+9nnzySfn6+ioyMlJr1qzRZ599prfeestpmb7++mtVqVJFNWvW1I4dO/T444+rb9++ZTqgRlGfLYOCgnTfffdp7NixCg4OVmBgoB599FG1b99e7dq1K7NcJebMIQlRuFWrVhlJ+W5Dhw51WqaC8kgyM2fOdFqm4cOHm8jISOPl5WWqVKlibrrpJvPjjz86LU9BXGG49QEDBpjw8HDj5eVlqlWrZgYMGGD27dvn1EzGGLN48WLTtGlT4+3tbRo2bGg++ugjZ0cyS5cuNZLMnj17nB3FGGNMUlKSefzxx03NmjWNj4+PqV27tvnXv/5l0tLSnJpr3rx5pnbt2sbLy8uEhYWZkSNHmoSEhMv2+EW9R2ZlZZnnnnvOhIaGGm9vb3PTTTeV+c+0qEwzZ84scP6ECROclitn6PeCbqtWrXJKpvPnz5vbbrvNREREGC8vLxMeHm569+5tfvrppzLLU1SmglyO4dYvlSklJcV07drVVKlSxXh6eprIyEjzwAMPmOPHjzstU47p06ebunXrGh8fH9OiRQvz7bffOj3Thx9+aHx9fS/b+1RRmY4dO2buvfdeExERYXx8fEyDBg3Mm2++WaaX9ygq0zvvvGOqV69uPD09Tc2aNc348ePL/P8aez5bnj9/3jzyyCOmUqVKxs/Pz9x2223m2LFjZZqrpCzGXKYz0gAAAADgCsU5VgAAAADgIIoVAAAAADiIYgUAAAAADqJYAQAAAICDKFYAAAAA4CCKFQAAAAA4iGIFAAAAAA6iWAEAAACAgyhWAADkMnHiRFksFp06dcrZUQAA5QjFCgAAAAAcRLECAAAAAAdRrAAAAADAQRQrAACKcODAAdWtW1dNmzZVXFycs+MAAFwQxQoAgEvYv3+/OnfurICAAK1evVqhoaHOjgQAcEEUKwAACvH777+rc+fOCg0N1cqVK1W5cmVnRwIAuCiKFQAABdi5c6euv/561apVS8uXL1elSpWcHQkA4MIoVgAAFKBXr14KCAjQ0qVLFRgY6Ow4AAAXR7ECAKAA/fv31/79+zV79mxnRwEAlAMezg4AAIAreuONN+Th4aFHHnlEAQEBGjRokLMjAQBcGMUKAIACWCwWffTRR0pOTtbQoUNVoUIF9e7d29mxAAAuikMBAQAohJubm7744gt17dpVd955p1auXOnsSAAAF0WxAgDgEjw9PTV//ny1a9dOffr00ebNm50dCQDggizGGOPsEAAAAABQnrHHCgAAAAAcRLECAAAAAAdRrAAAAADAQRQrAAAAAHAQxQoAAAAAHESxAgAAAAAHUawAAAAAwEEUKwAAAABwEMUKAAAAABxEsQIAAAAAB1GsAAAAAMBBFCsAAAAAcND/AwqkHDe2AsjKAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Top- 1 Accuracy: 63.84%\n", "Top- 2 Accuracy: 74.24%\n", "Top- 3 Accuracy: 78.77%\n", "Top- 4 Accuracy: 81.41%\n", "Top- 5 Accuracy: 83.10%\n", "Top- 6 Accuracy: 84.45%\n", "Top- 7 Accuracy: 85.43%\n", "Top- 8 Accuracy: 86.25%\n", "Top- 9 Accuracy: 86.90%\n", "Top-10 Accuracy: 87.49%\n", "Top-11 Accuracy: 87.94%\n", "Top-12 Accuracy: 88.37%\n", "Top-13 Accuracy: 88.75%\n", "Top-14 Accuracy: 89.07%\n", "Top-15 Accuracy: 89.33%\n", "Top-16 Accuracy: 89.59%\n", "Top-17 Accuracy: 89.85%\n", "Top-18 Accuracy: 90.07%\n", "Top-19 Accuracy: 90.28%\n", "Top-20 Accuracy: 90.46%\n" ] } ], "source": [ "import pandas as pd\n", "import random\n", "import torch\n", "from transformers import BertTokenizerFast, BertForMaskedLM\n", "from tqdm import tqdm\n", "import matplotlib.pyplot as plt\n", "\n", "# ===============================\n", "# ๐Ÿ”น Step 1: Load raw BIS sentences\n", "# ===============================\n", "df = pd.read_csv(\"/kaggle/input/bis-speeches/speeches_data_preprocessed.csv\")\n", "df = df[df[\"processed_text\"].notna()]\n", "df[\"processed_text\"] = df[\"processed_text\"].apply(eval)\n", "sentences = [sentence for sublist in df[\"processed_text\"] for sentence in sublist]\n", "\n", "# ===============================\n", "# ๐Ÿ”น Step 2: Setup device, model & tokenizer\n", "# ===============================\n", "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", "print(\"โš™๏ธ Using device:\", device)\n", "\n", "model_path = \"/kaggle/working/bert-mlm-bis\"\n", "tokenizer = BertTokenizerFast.from_pretrained(model_path)\n", "model = BertForMaskedLM.from_pretrained(model_path).to(device)\n", "model.eval()\n", "\n", "# ===============================\n", "# ๐Ÿ”น Step 3: Function to mask one word in a sentence\n", "# ===============================\n", "def mask_random_word(sentence):\n", " words = sentence.strip().split()\n", " if len(words) < 5:\n", " return None\n", " # choose only alphabetic tokens\n", " candidates = [i for i, w in enumerate(words) if w.isalpha()]\n", " if not candidates:\n", " return None\n", " idx = random.choice(candidates)\n", " true_word = words[idx]\n", " words[idx] = \"[MASK]\"\n", " return \" \".join(words), true_word\n", "\n", "# ===============================\n", "# ๐Ÿ”น Step 4: Generate 100,000 masked test samples\n", "# ===============================\n", "masked_samples = []\n", "for sent in random.sample(sentences, len(sentences)):\n", " pair = mask_random_word(sent)\n", " if pair:\n", " masked_samples.append(pair)\n", " if len(masked_samples) >= 100000:\n", " break\n", "\n", "df_masked = pd.DataFrame(masked_samples, columns=[\"Sentence with [MASK]\", \"Masked Word\"])\n", "\n", "# ===============================\n", "# ๐Ÿ”น Step 5: Evaluate Topโ€‘k Accuracy\n", "# ===============================\n", "results = []\n", "max_k = 20\n", "\n", "for _, row in tqdm(df_masked.iterrows(), total=len(df_masked), desc=\"๐Ÿ” Evaluating (Topโ€‘k)\"):\n", " masked_sentence = row[\"Sentence with [MASK]\"]\n", " true_word = row[\"Masked Word\"].lower().strip()\n", "\n", " # Tokenize with truncation & padding\n", " inputs = tokenizer(\n", " masked_sentence,\n", " return_tensors=\"pt\",\n", " truncation=True,\n", " max_length=128,\n", " padding=\"max_length\"\n", " ).to(device)\n", "\n", " mask_indices = torch.where(inputs.input_ids[0] == tokenizer.mask_token_id)[0]\n", " if len(mask_indices) != 1:\n", " continue\n", " mask_idx = mask_indices.item()\n", "\n", " # Forward pass\n", " with torch.no_grad():\n", " outputs = model(**inputs)\n", " logits = outputs.logits\n", "\n", " # Get topโ€‘k predictions\n", " mask_logits = logits[0, mask_idx]\n", " topk = torch.topk(mask_logits, k=max_k).indices.tolist()\n", " top_tokens = [tokenizer.decode([tid]).strip().lower() for tid in topk]\n", "\n", " results.append({\n", " \"Masked Word\": true_word,\n", " \"Top-k Predictions\": top_tokens\n", " })\n", "\n", "# ===============================\n", "# ๐Ÿ”น Step 6: Compute Topโ€‘k Accuracy Curve\n", "# ===============================\n", "k_range = list(range(1, max_k+1))\n", "accuracies = []\n", "total = len(results)\n", "\n", "for k in k_range:\n", " correct = sum(true in preds[:k] for true, preds in \n", " [(r[\"Masked Word\"], r[\"Top-k Predictions\"]) for r in results])\n", " accuracies.append(correct/total*100)\n", "\n", "# ===============================\n", "# ๐Ÿ”น Step 7: Plot Topโ€‘k Curve\n", "# ===============================\n", "plt.figure(figsize=(10,6))\n", "plt.plot(k_range, accuracies, marker='o')\n", "plt.title(\"Topโ€‘k Accuracy Curve (BISโ€‘BERTโ€‘MLM)\", fontsize=14)\n", "plt.xlabel(\"k\", fontsize=12)\n", "plt.ylabel(\"Accuracy (%)\", fontsize=12)\n", "plt.xticks(k_range)\n", "plt.grid(True)\n", "plt.ylim(0, 100)\n", "plt.show()\n", "\n", "# ===============================\n", "# ๐Ÿ”น Step 8: Print Summary\n", "# ===============================\n", "for k, acc in zip(k_range, accuracies):\n", " print(f\"Top-{k:2d} Accuracy: {acc:5.2f}%\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Corpus Statistics and Training Metadata Summary\n", "\n", "This section computes descriptive statistics for the corpus, tokenizer, and model, and documents training configurations used for pretraining `cb-bert-mlm`.\n", "\n", "These figures provide reproducibility and clarity for evaluating the scale and setup of the domain-adaptive masked language modeling process." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "โœ… Loaded tokenized dataset with 2087615 sentences.\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "9a15e03981b9432da3ea1226c3269018", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Map: 0%| | 0/2087615 [00:00