AbstractPhil/bert-beatrix-2048

This is the 26 categorical finetune of nomic-bert-2048's encoder.

130,000,000 - 4-30 masked token samples with 80% mask rate

253,952,000 - 77 token samples with 20% mask rate

775,000,000 - 144-256 token samples with 30% mask rate

453,800,000 - 385-512 token samples with 30% mask rate

Total samples; 1,228,800,000

The model has learned to categorize certain masked patterns with their categories and special tokens.

She's still cooking, but her accuracy is getting really up there. I plan to run the 256-512 token counts for a considerable amount of time and then we'll upgrade to 512-1024, and finally 1024-2048.

<subject>
<subject1>
<subject2>
<pose>
<emotion>
<surface>
<lighting>
<material>
<accessory>
<footwear>
<upper_body_clothing>
<hair_style>
<hair_length>
<headwear>
<texture>
<pattern>
<grid>
<zone>
<offset>
<object_left>
<object_right>
<relation>
<intent>
<style>
<fabric>
<jewelry>

With the categorical shunts;

[SHUNT_1000000]
[SHUNT_1000001]
[SHUNT_1000002]
[SHUNT_1000003]
[SHUNT_1000004]
[SHUNT_1000005]
[SHUNT_1000006]
[SHUNT_1000007]
[SHUNT_1000008]
[SHUNT_1000009]
[SHUNT_1000010]
[SHUNT_1000011]
[SHUNT_1000012]
[SHUNT_1000013]
[SHUNT_1000014]
[SHUNT_1000015]
[SHUNT_1000016]
[SHUNT_1000017]
[SHUNT_1000018]
[SHUNT_1000019]
[SHUNT_1000020]
[SHUNT_1000021]
[SHUNT_1000022]
[SHUNT_1000023]
[SHUNT_1000024]
[SHUNT_1000025]

Each shunt meant to activate cross-categorical conceptualization within their 77 token window.