Attempts to fill out the 1B3 model details that diverge from the main one.

Here I am using: https://github.com/bigscience-workshop/bigscience/blob/master/train/tr11-176B-ml/smaller_models/tr11b-1B3-ml.slurm to help flesh it out.

README.md

@@ -120,11 +120,11 @@ Please see [the BLOOM training README](https://github.com/bigscience-workshop/bi
 
 * ALiBI positional encodings (see [paper](https://arxiv.org/pdf/2108.12409.pdf)), with GeLU activation functions
 
-* 176 billion parameters:
+* 1.3 billion parameters:
 
-    * 70 layers, 112 attention heads
+    * 24 layers, 16 attention heads
 
-    * Hidden layers are 14336-dimensional
+    * Hidden layers are 2048-dimensional
 
     * Sequence length of 2048 tokens used (see [BLOOM tokenizer](https://huggingface.co/bigscience/tokenizer), [tokenizer description](#tokenization))
 
@@ -132,12 +132,14 @@ Please see [the BLOOM training README](https://github.com/bigscience-workshop/bi
     
 **Compute infrastructure:** Jean Zay Public Supercomputer, provided by the French government (see [announcement](https://www.enseignementsup-recherche.gouv.fr/fr/signature-du-marche-d-acquisition-de-l-un-des-supercalculateurs-les-plus-puissants-d-europe-46733)).
 
-* Hardware: 384 A100 80GB GPUs (48 nodes):
-    
-    * Additional 32 A100 80GB GPUs (4 nodes) in reserve
-
-    *  8 GPUs per node Using NVLink 4 inter-gpu connects, 4 OmniPath links
+* Hardware: 128 V100 80GB GPUs (16 nodes):
 
+    *   4 GPUs per node
+    
+    *   40 CPUs per task
+    
+    *   1 task per node
+    
     *   CPU: AMD
 
     *   CPU memory: 512GB per node
@@ -163,28 +165,21 @@ Please see [the BLOOM training README](https://github.com/bigscience-workshop/bi
 
 #### **Training**
 
- 
-_In progress._
-
-Current training logs: [Tensorboard link](https://huggingface.co/tensorboard/bigscience/tr11-176B-ml-logs/)
-
 - Checkpoint size:
     
-    - Bf16 weights: 329GB
+    - Fp16 weights: 2.6GB (# params * 2)
     
-    - Full checkpoint with optimizer states: 2.3TB
+    - Full checkpoint with optimizer states: --
 
-- Training throughput: About 150 TFLOP per GPU per second
+- Training throughput: --
 
-- Number of epochs: 1 (*current target*)
+- Number of epochs: 1
 
 - Dates:
     
-    - Started 11th March, 2022 11:42am PST
-
-    - Estimated end: 5th July, 2022
+    - Start: 11th March, 2022 11:42am PST
 
-- Estimated cost of training: Equivalent of $2-5M in cloud computing (including preliminary experiments)
+    - End: 20 May, 2022
 
 - Server training location: Île-de-France, France