Update README.md

README.md

@@ -6,7 +6,7 @@ pipeline_tag: text-generation
 library_name: transformers
 ---
 
-# Granite 3.0 8B Instruct - Intrinsics LoRA v1
+# Granite 3.0 8B Instruct - Intrinsics LoRA v0.1
 
 Welcome to Granite Experiments!
 
@@ -15,7 +15,7 @@ Think of Experiments as a preview of what's to come. These projects are still un
 
 ## Model Summary
 
-**Granite Intrinsics 3.0 8b Instruct v1** is a LoRA adapter for [ibm-granite/granite-3.0-8b-instruct](https://huggingface.co/ibm-granite/granite-3.0-8b-instruct), 
+**Granite Intrinsics 3.0 8b Instruct v0.1** is a LoRA adapter for [ibm-granite/granite-3.0-8b-instruct](https://huggingface.co/ibm-granite/granite-3.0-8b-instruct), 
 providing access to the Uncertainty, Hallucination Detection, and Safety Exception intrinsics in addition to retaining the full abilities of the [ibm-granite/granite-3.0-8b-instruct](https://huggingface.co/ibm-granite/granite-3.0-8b-instruct) model. 
 
 - **Developer:** IBM Research
@@ -48,7 +48,7 @@ The Safety Exception intrinsic was designed as a binary classifier that analyses
 
 This is an experimental LoRA testing new functionality being developed for IBM's Granite LLM family.  We are welcoming the community to test it out and give us feedback, but we are NOT recommending this model be used for real deployments at this time.  Stay tuned for more updates on the Granite roadmap.
 
-**Granite Intrinsics 3.0 8b v1** is lightly tuned so that its behavior closely mimics that of [ibm-granite/granite-3.0-8b-instruct](https://huggingface.co/ibm-granite/granite-3.0-8b-instruct), 
+**Granite Intrinsics 3.0 8b v0.1** is lightly tuned so that its behavior closely mimics that of [ibm-granite/granite-3.0-8b-instruct](https://huggingface.co/ibm-granite/granite-3.0-8b-instruct), 
 with the added ability to generate the three specified intrinsics. 
 
 
@@ -88,7 +88,58 @@ we can evaluate the certainty and hallucination status of this reply by invoking
 
 
 ### Intrinsics Example with PDL
-Given a hosted instance of **Granite Intrinsics 3.0 8b Instruct v1** at `API_BASE` (insert the host address here), this uses the [PDL language](https://github.com/IBM/prompt-declaration-language) to implement the RAG intrinsic invocation scenario described above.
+Given a hosted instance of **Granite Intrinsics 3.0 8b Instruct v0.1** at `API_BASE` (insert the host address here), this uses the [PDL language](https://github.com/IBM/prompt-declaration-language) to implement the RAG intrinsic invocation scenario described above.
+
+First, create a file `intrinsics.pdl` with the following content.
+```python
+defs:
+  system_prompt: "You are an AI language model developed by IBM Research. You are a cautious assistant. You carefully follow instructions. You are helpful and harmless and you follow ethical guidelines and promote positive behavior."
+
+  rag_prompt: "Provide a short response to the user's question based on the information present in the documents. If the documents lack the necessary details, inform the user that the question cannot be answered."
+
+  document:
+      |
+      Disability housing grants for Veterans 
+      We offer housing grants for Veterans and service members with certain service - connected disabilities so they can buy or change a home to meet their needs and live more independently. Changing a home might involve installing ramps or widening doorways. Find out if you re eligible for a disability housing grant and how to apply. 
+
+      Can I get a Specially Adapted Housing (SAH) grant and how much funding does this grant offer? 
+      You may be able to get an SAH grant if you re using the grant money to buy, build, or change your permanent home a home you plan to live in for a long time and you meet both of the requirements listed below. Both of these must be true. You: Own or will own the home , and Have a qualifying service - connected disability Qualifying service - connected disabilities include : The loss or loss of use of more than one limb The loss or loss of use of a lower leg along with the residuals lasting effects of an organic natural disease or injury Blindness in both eyes having only light perception along with the loss or loss of use of a leg Certain severe burns The loss or loss of use of one or both lower extremities feet or legs after September 11 , 2001, that makes it so you can t balance or walk without the help of braces, crutches, canes, or a wheelchair Note : Only 30 Veterans and service members each fiscal year FY can qualify for a grant based on the loss of extremities after September 11 , 2001. If you qualify for but don t receive a grant in 2019 because the cap was reached , you may be able to use this benefit in FY 2020 or future years if the law continues to give us the authority to offer these grants and we don t go beyond the new FY cap. For FY 2019 , you may be able to get up to 3 grants for a total of up to $85,645 through the SAH grant program. Learn more about how to apply for a housing grant 
+
+      Can I get a Special Housing Adaptation (SHA) grant and how much funding does this grant offer? 
+      You may be able to get an SHA grant if you re using the grant money to buy, build, or change your permanent home a home you plan to live in for a long time and you meet both of the requirements listed below. Both of these must be true : You or a family member own or will own the home , and You have a qualifying service - connected disability Qualifying service - connected disabilities include : Blindness in both eyes with 20/200 visual acuity or less The loss or loss of use of both hands Certain severe burns Certain respiratory or breathing injuries For FY 2019 , you may be able to get up to 3 grants for a total of up to $17,130 through the SHA grant program. Learn more about how to apply for a housing grant \n\nWhat if I need money to make changes to a family member s home that I m living in for a short period of time? \nYou may be able to get a Temporary Residence Adaptation TRA grant if you meet both of the requirements listed below. Both of these must be true. You: Qualify for an SAH or SHA grant see above , and Are living temporarily in a family member s home that needs changes to meet your needs If you qualify for an SAH grant , you can get up to $37,597 through the TRA grant program for FY 2019. If you qualify for an SHA grant , you can get up to $6,713 through the TRA grant program for FY 2019. 
+
+      Apply for an SAH, SHA, or TRA grant 
+      You can apply online right now by going to our eBenefits website. You ll need to sign in to eBenefits with your DS Logon basic or premium account. If you don t have a DS Logon account , you can register for one on the site. Go to eBenefits to apply.
+
+
+  query: How much funding does the SAH grant offer?
+text:
+  - include: intrinsics-defs.pdl
+  - defs:
+      unsafe:
+        call: get_safety
+        args:
+          query: ${ query }
+  - role: system
+    text: ${ system_prompt }
+    contribute: [context]
+  - if: ${ unsafe == "N" }
+    then: 
+      text:
+        - "\n\nDocuments: ${ document }\n\n ${ query }\n"
+        - model: openai/granite-8b-intrinsics-v2-20241203
+          def: answer
+          parameters: {api_key: EMPTY, api_base: API_BASE, temperature: 0, stop: "\n"}
+        - defs:  ## Implicit fork of context
+            certainty:
+              call: get_certainty         
+            hallucination:  
+              call: get_hallucination
+        - "\nCertainty: ${ certainty }"
+        - "\nHallucination: ${ hallucination }"
+```
+Next, create a file `intrinsics-defs.pdl` with the following content.
+
 ```python
 defs:
   apply_template:
@@ -107,15 +158,14 @@ defs:
           with: "\n"
   get_intrinsic:
     function:
-      context: [{role: str, content: str}]
       intrinsic: str
     return:
       lastOf:
       - call: apply_template
         def: mycontext
         args: 
-          context: ${ context }
-      - model: granite-8b-intrinsics-v1
+          context: ${ pdl_context }
+      - model: granite-intrinsics-3.0-8b-instruct-v0.1
         parameters:
           api_key: EMPTY
           api_base: API_BASE
@@ -126,66 +176,127 @@ defs:
             |
             ${ mycontext }
             <|start_of_role|>${ intrinsic }<|end_of_role|>
+  
+  get_safety:
+    function:
+      query: str
+    return:
+      lastOf:
+      - ${ query }
+      - call: apply_template
+        def: mycontext
+        args: 
+          context: ${ pdl_context }
+      - call: get_intrinsic
+        args: 
+         intrinsic: safety
 
-  system_prompt: "You are an AI language model developed by IBM Research. You are a cautious assistant. You carefully follow instructions. You are helpful and harmless and you follow ethical guidelines and promote positive behavior."
+  get_hallucination:
+    function:
+    return:
+      call: get_intrinsic
+      args: 
+        intrinsic: hallucination
+  
+  get_certainty:
+    function:
+    return:
+      call: get_intrinsic
+      args: 
+        intrinsic: certainty
+```
 
-  rag_prompt: "Provide a short response to the user's question based on the information present in the documents. If the documents lack the necessary details, inform the user that the question cannot be answered."
+To run the example, in the command line run `pdl intrinsics.pdl` after installing the PDL CLI (`pip install prompt-declaration-language`).
 
-  document:
-    text:
-      |
-      Disability housing grants for Veterans 
-      We offer housing grants for Veterans and service members with certain service - connected disabilities so they can buy or change a home to meet their needs and live more independently. Changing a home might involve installing ramps or widening doorways. Find out if you re eligible for a disability housing grant and how to apply. 
+### Intrinsics Example with SGLang
+The below SGLang implementation uses the SGLang fork at [https://github.com/frreiss/sglang/tree/granite](https://github.com/frreiss/sglang/tree/granite) that supports Granite models. 
 
-      Can I get a Specially Adapted Housing (SAH) grant and how much funding does this grant offer? 
-      You may be able to get an SAH grant if you re using the grant money to buy, build, or change your permanent home a home you plan to live in for a long time and you meet both of the requirements listed below. Both of these must be true. You: Own or will own the home , and Have a qualifying service - connected disability Qualifying service - connected disabilities include : The loss or loss of use of more than one limb The loss or loss of use of a lower leg along with the residuals lasting effects of an organic natural disease or injury Blindness in both eyes having only light perception along with the loss or loss of use of a leg Certain severe burns The loss or loss of use of one or both lower extremities feet or legs after September 11 , 2001, that makes it so you can t balance or walk without the help of braces, crutches, canes, or a wheelchair Note : Only 30 Veterans and service members each fiscal year FY can qualify for a grant based on the loss of extremities after September 11 , 2001. If you qualify for but don t receive a grant in 2019 because the cap was reached , you may be able to use this benefit in FY 2020 or future years if the law continues to give us the authority to offer these grants and we don t go beyond the new FY cap. For FY 2019 , you may be able to get up to 3 grants for a total of up to $85,645 through the SAH grant program. Learn more about how to apply for a housing grant 
+```python
 
-      Can I get a Special Housing Adaptation (SHA) grant and how much funding does this grant offer? 
-      You may be able to get an SHA grant if you re using the grant money to buy, build, or change your permanent home a home you plan to live in for a long time and you meet both of the requirements listed below. Both of these must be true : You or a family member own or will own the home , and You have a qualifying service - connected disability Qualifying service - connected disabilities include : Blindness in both eyes with 20/200 visual acuity or less The loss or loss of use of both hands Certain severe burns Certain respiratory or breathing injuries For FY 2019 , you may be able to get up to 3 grants for a total of up to $17,130 through the SHA grant program. Learn more about how to apply for a housing grant \n\nWhat if I need money to make changes to a family member s home that I m living in for a short period of time? \nYou may be able to get a Temporary Residence Adaptation TRA grant if you meet both of the requirements listed below. Both of these must be true. You: Qualify for an SAH or SHA grant see above , and Are living temporarily in a family member s home that needs changes to meet your needs If you qualify for an SAH grant , you can get up to $37,597 through the TRA grant program for FY 2019. If you qualify for an SHA grant , you can get up to $6,713 through the TRA grant program for FY 2019. 
+import sglang as sgl 
+from sglang.lang.chat_template import get_chat_template
 
-      Apply for an SAH, SHA, or TRA grant 
-      You can apply online right now by going to our eBenefits website. You ll need to sign in to eBenefits with your DS Logon basic or premium account. If you don t have a DS Logon account , you can register for one on the site. Go to eBenefits to apply.
+@sgl.function
+def safety_check (s, question):
+    s += sgl.user(question)
+    s += "<|start_of_role|>safety<|end_of_role|>" + sgl.gen("safety", temperature=0, max_tokens=1)
+ 
+    # print ("\n====== Safety check state =======\n")
+    # print (s)
+    # print ("\n")
+
+
+# Input data
+system_prompt = "You are an AI language model developed by IBM Research. You are a cautious assistant. You carefully follow instructions. You are helpful and harmless and you follow ethical guidelines and promote positive behavior."
+
+rag_prompt = "Provide a short response to the user's question based on the information present in the documents. If the documents lack the necessary details, inform the user that the question cannot be answered."
+
+document = """Disability housing grants for Veterans 
+We offer housing grants for Veterans and service members with certain service - connected disabilities so they can buy or change a home to meet their needs and live more independently. Changing a home might involve installing ramps or widening doorways. Find out if you re eligible for a disability housing grant and how to apply. 
+
+Can I get a Specially Adapted Housing (SAH) grant and how much funding does this grant offer? 
+You may be able to get an SAH grant if you re using the grant money to buy, build, or change your permanent home a home you plan to live in for a long time and you meet both of the requirements listed below. Both of these must be true. You: Own or will own the home , and Have a qualifying service - connected disability Qualifying service - connected disabilities include : The loss or loss of use of more than one limb The loss or loss of use of a lower leg along with the residuals lasting effects of an organic natural disease or injury Blindness in both eyes having only light perception along with the loss or loss of use of a leg Certain severe burns The loss or loss of use of one or both lower extremities feet or legs after September 11 , 2001, that makes it so you can t balance or walk without the help of braces, crutches, canes, or a wheelchair Note : Only 30 Veterans and service members each fiscal year FY can qualify for a grant based on the loss of extremities after September 11 , 2001. If you qualify for but don t receive a grant in 2019 because the cap was reached , you may be able to use this benefit in FY 2020 or future years if the law continues to give us the authority to offer these grants and we don t go beyond the new FY cap. For FY 2019 , you may be able to get up to 3 grants for a total of up to $85,645 through the SAH grant program. Learn more about how to apply for a housing grant 
+
+Can I get a Special Housing Adaptation (SHA) grant and how much funding does this grant offer? 
+You may be able to get an SHA grant if you re using the grant money to buy, build, or change your permanent home a home you plan to live in for a long time and you meet both of the requirements listed below. Both of these must be true : You or a family member own or will own the home , and You have a qualifying service - connected disability Qualifying service - connected disabilities include : Blindness in both eyes with 20/200 visual acuity or less The loss or loss of use of both hands Certain severe burns Certain respiratory or breathing injuries For FY 2019 , you may be able to get up to 3 grants for a total of up to $17,130 through the SHA grant program. Learn more about how to apply for a housing grant \n\nWhat if I need money to make changes to a family member s home that I m living in for a short period of time? \nYou may be able to get a Temporary Residence Adaptation TRA grant if you meet both of the requirements listed below. Both of these must be true. You: Qualify for an SAH or SHA grant see above , and Are living temporarily in a family member s home that needs changes to meet your needs If you qualify for an SAH grant , you can get up to $37,597 through the TRA grant program for FY 2019. If you qualify for an SHA grant , you can get up to $6,713 through the TRA grant program for FY 2019. 
+
+Apply for an SAH, SHA, or TRA grant 
+You can apply online right now by going to our eBenefits website. You ll need to sign in to eBenefits with your DS Logon basic or premium account. If you don t have a DS Logon account , you can register for one on the site. Go to eBenefits to apply.
+"""
+
+query = "How much funding does the SAH grant offer?"
+
+
+# The following function processes a chat between a user and an assistant.
+# For simplicity, this assumes a fixed document, but in a true RAG setting, the
+# documents will be retrieved dynamically based on the user turns.  
+@sgl.function
+def main_chat_flow (s, doc, query):
+    s += sgl.system (system_prompt)
+
+    # Safety check on query
+    state = safety_check (question=query) 
+    print (f"Safety Output: {state["safety"]} for question: {query}\n")
+
+    # RAG answer generation
+    s += sgl.user(rag_prompt + "\n\nDocuments: " + doc + "\n\n" + query)
+    s += sgl.assistant (sgl.gen ("answer", stop="\n", temperature=0, max_tokens=200))
+    answer = s["answer"]
+    print (f"Assistant: {answer}\n")
+
+    # Hallucination check in parallel with uncertainty quantification for the generated answer
+    forks = s.fork(2)
+    for i, f in enumerate(forks):
+        if (i == 0): 
+            f += "<|start_of_role|>hallucination<|end_of_role|>"
+            f += sgl.gen("hallucination", temperature=0, max_tokens=1)   
+            # print ("\n====== Fork 0 state =======\n")
+            # print (f) 
+            # print ("\n")
+        else:    
+            f += "<|start_of_role|>certainty<|end_of_role|>"
+            f += sgl.gen("certainty", temperature=0, max_tokens=1)
+            # print ("\n====== Fork 1 state =======\n")
+            # print (f)
+            # print ("\n")
+    
+    print (f"Hallucination Output: {forks [0]["hallucination"]} for answer: {answer}\n")
+    print (f"Certainty Output: {forks [1]["certainty"]} for answer: {answer}\n")
 
+    
 
-  query: How much funding does the SAH grant offer?
-text:
-  - role: system
-    text: ${ system_prompt }
-  - text: ${ query }
-    contribute: [context]
-  - call: get_intrinsic
-    def: safety
-    contribute: []
-    args:
-      context: ${ pdl_context }
-      intrinsic: safety
-  - role: system
-    text: ${ system_prompt }   
-  - if: ${ safety != "Y" }
-    then: 
-      text:
-        - "\n\nDocuments: ${ document }\n\n ${ query }"
-        - model: openai/granite-8b-intrinsics-v1
-          def: answer
-          parameters: {api_key: EMPTY, api_base: API_BASE, temperature: 0, stop: "\n"}
-        - call: get_intrinsic
-          def: certainty
-          contribute: []
-          args:
-            context: ${ pdl_context }
-            intrinsic: certainty 
-        - call: get_intrinsic
-          def: hallucination
-          contribute: []
-          args:
-            context: ${ pdl_context }
-            intrinsic: hallucination
-        - "\nCertainty: ${ certainty }"
-        - "\nHallucination: ${ hallucination }"
-```
-### Intrinsics Example with SGLang
-The below SGLang implementation uses the SGLang fork at [https://github.com/frreiss/sglang/tree/granite](https://github.com/frreiss/sglang/tree/granite) that supports Granite models. 
+if __name__ == "__main__":
+    model_path = "ibm-granite/granite-intrinsics-3.0-8b-lora-v0.1"
+
+    # Setting the model_path to the granite model, and chat template to be the granite template
+    # This assumes "granite3-instruct" chat template has been registered in "sglang/lang/chat_template.py"
+    runtime = sgl.Runtime(model_path=model_path)
+    runtime.endpoint.chat_template = get_chat_template("granite3-instruct")
+    sgl.set_default_backend(runtime)
 
+    state = main_chat_flow (doc=document, query=query)
+
+```
 
 
 #### Notes
@@ -204,20 +315,20 @@ red-teamed examples.
 ## Evaluation
 We evaluate the performance of the intrinsics themselves and the RAG performance of the model. 
 
-We first find that the performance of the intrinsics in our shared model **Granite Instrinsics 3.0 8b v1** is not degraded
+We first find that the performance of the intrinsics in our shared model **Granite Instrinsics 3.0 8b v0.1** is not degraded
 versus the baseline procedure of maintaining 3 separate instrinsic models. Here, percent error is shown for the Hallucination Detection and Safety Exception intrinsics as they have
 binary output, and Mean Absolute Error (MAE) is shown for the Uncertainty Intrinsic as it outputs numbers 0 to 9. For all, lower is better. Performance is calculated on a randomly drawn 400 sample validation set from each intrinsic's dataset. 
 
 
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/6602ffd971410cf02bf42c06/NsvMpweFjmjIhWFaKtI-K.png)
 
-We then find that RAG performance of **Granite Instrinsics 3.0 8b v1** does not suffer with respect to the base model [ibm-granite/granite-3.0-8b-instruct](https://huggingface.co/ibm-granite/granite-3.0-8b-instruct). Here we evaluate the RAGBench benchmark on RAGAS faithfulness and correction metrics.
+We then find that RAG performance of **Granite Instrinsics 3.0 8b v0.1** does not suffer with respect to the base model [ibm-granite/granite-3.0-8b-instruct](https://huggingface.co/ibm-granite/granite-3.0-8b-instruct). Here we evaluate the RAGBench benchmark on RAGAS faithfulness and correction metrics.
 
 
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/6602ffd971410cf02bf42c06/hyOlQmXPirlCYeILLBXhc.png)
 
 ## Training Details
-The **Granite Instrinsics 3.0 8b v1** model is a LoRA adapter finetuned to provide 3 desired intrinsic outputs - Uncertainty Quantification, Hallucination Detection, and Safety.
+The **Granite Instrinsics 3.0 8b v0.1** model is a LoRA adapter finetuned to provide 3 desired intrinsic outputs - Uncertainty Quantification, Hallucination Detection, and Safety.