Update README.md

README.md

@@ -59,7 +59,28 @@ print(tokenizer.decode(outputs[0], skip_special_tokens=True))
 # b (int): The second number.
 
 ```
+# Fine tuning
 
+In fine tuning the model, i used the special token `<tdec>`. According to CodeT5+'s paper:
+
+" Specifically, when the input is a text
+sample, we prepend a [CDec] token to the input
+sequence to the decoder. In this case, the decoder
+operates under code generation functionality. Alternatively, when the input is a code sample, we
+prepend a [TDec] token to the input sequence to
+the decoder. The decoder operates under text generation functionality in this case. This type of Causal
+LM has been shown to be an effective learning
+objective to close the pretrain-finetune gap for generative downstream tasks"
+
+Generally speaking, the `<tdec>` token was appended to the target (the docstring) to signal to the decoder that it is in a text generation functionality. A sample row looks like this:
+
+```
+<s><tdec> Creates a task that to retry a previously abandoned task.
+
+Returns:
+Task: a task that was abandoned but should be retried or None if there are
+no abandoned tasks that should be retried.</s>
+```
 # Hyperparameters
 
 MAX_SOURCE_LENGTH = 256 <br>