Update README.md

README.md

@@ -3,10 +3,153 @@ library_name: transformers
 tags: []
 ---
 
-# Model Card for Model ID
+# ServiceNow Environment Assistant
 
 <!-- Provide a quick summary of what the model is/does. -->
 
+## Introduction
+ServiceNow is a platform that helps businesses automate their processes and workflows. They offer several solutions such as ITSM. Currently, users of ServiceNow generally need to apply filters and/or build dashboards to observe data about tables in ServiceNow, such as incidents and problems. Building dashboards and reports often require the help of developers and may be a hassle just for quick information. Dashboards are useful for visual representation, but it would be useful to be able to ask questions about the data just to a chatbot. As I didn’t know how feasible it would be to integrate a ServiceNow instance with LLM, but I could instead just import example tables as csv files and use that as data. The task is to create a custom LLM chatbot that uses data from tables ServiceNow uses such as incident, change, and problem, which can then be used to respond to user queries in natural language. An LLM like chat-gpt probably wouldn’t work too poorly with this table data, but the costs of using those are probably higher as well. 
+
+## Training Data
+
+For this project, the training data was structured around ServiceNow ITSM tables, specifically Incident, Change, and Problem tables. I used a certain subset fields from Incident, Change, and Problem tables.  For example, Problem tables have a problem id, priority, status, root cause, and resolved at field. Since I can’t use official data from in-use ServiceNow instances, which contain private information, I generated a synthetic dataset with custom code. Then, I had to structure that code in sqa format, which is the best format for the model I was using, TAPAS. For this, I had to save each table in a CSV file. The final refined dataset that I would pass in would contain an id, uestion, table_file, answer_coordinates if the answer was in the table itself, the actual answer, and a float answer if the answer was a numeric value not in the data, such as a count. I do have an aggregation_label field as well, which I set right before the training process, but after the train_test_table split. I used the method train_test_split() to obtain the training, validation, and test data. I specifically used a seed of 42:
+
+```python
+train_val_data, test_data = train_test_split(data, test_size=0.1, random_state=42)
+# Then split train+validation into train and validation
+train_data, val_data = train_test_split(train_val_data, test_size=0.1, random_state=42)
+```
+
+## Training Method
+I used full-fine tuning. The model did not really need generalization abilities. Its primary purpose is to take ServiceNow Tables and answer queries based on those tables. Keeping some generalization ability would be nice, but isn't really that necessary. PEFT could work as well to prevent catastrophic overfitting, but generalization is not hugely important. The drawbacks I had expected was some generalization loss, but that wasn't really the case.
+
+These were the arguments/hyperparameters, I used. I tried using higher epochs, but those usually caused worse results:
+```python
+        num_train_epochs=1,  # Number of training epochs
+    	per_device_train_batch_size=32,  # Batch size per device during training
+    	per_device_eval_batch_size=64,   # Batch size per device during evaluation
+    	learning_rate=0.00001,
+    	warmup_steps=100,   # Number of warmup steps for learning rate scheduler
+    	weight_decay=0.01,  # Strength of weight decay
+    	evaluation_strategy="steps",  # Evaluate every 'eval_steps'
+    	eval_steps=50,  # Evaluation frequency in steps
+    	logging_steps=50,  # Log every eval_steps
+    	save_steps=150,  # Save model every 500 steps
+    	save_total_limit=2,
+    	load_best_model_at_end=True,  # Load the best model when finished training
+    	metric_for_best_model="eval_loss",  # Metric to use for best model selection
+```
+
+## Evaluation
+I had three benchmarks, the WikiTableQuestions dataset, the TabFact dataset, and the Synthetic Validation set. Fine-tuning did not harm the results of on the WTQ Validation Set and the TabFact Dataset, in which I got accuracies of .3405 and .5005, respectively for both the pre-trained and fine-tuned model. There were improvements in the validation and test results after training though. On Validation, there was a jump from 0.4000 to 0.4222. On the test set, there was quite a larger jump in accuracy from 0.2033 to 0.4667 after fine-tuning.
+
+| Model                                                | Benchmark 1 (WTQ Validation Set) | Benchmark 2 (TabFact) | Benchmark 3 (Synthetic Validation Set) | Test Set of Synthetic Dataset |
+|------------------------------------------------------|----------------------------------|-----------------------|----------------------------------------|-------------------------------|
+| google/tapas-base-finetuned-wtq (before Fine-tuning) | 0.3405                           | .5005                 | 0.4000                                 | 0.2933                        |
+| google/tapas-base-finetuned-wtq (Fine-tuned)         | 0.3405                           | .5005                 | 0.4222                                 | 0.4667                        |
+
+
+## Usage
+The prompt for the TAPAS model should be a natural language question paired with a structured table that can be passed in in dataframe format. The prompt should look like this:
+
+```python
+question = "How many Hardware Upgrade changes are still pending?"
+table_df = pd.DataFrame({
+    "change_id": [
+        "CHG3000",
+        "CHG3001",
+        "CHG3002",
+        "CHG3003"
+    ],
+    "category": [
+        "Security Patch",
+        "Software Update",
+        "Hardware Upgrade",
+        "Software Update"
+    ],
+    "status": [
+        "Rejected",
+        "In Progress",
+        "In Progress",
+        "Completed"
+    ],
+    "approved_by": [
+        "",
+        "Manager2",
+        "",
+        "Admin1"
+    ],
+    "implementation_date": [
+        "",
+        "",
+        "",
+        "2023-05-30"
+    ]
+})
+```
+
+Or you could define table in json format and then have table = pd.DataFrame(table) in your tokenizer.
+
+## Expected Output Format
+You tokenize the inputs and then perform a specific function to get outputs, which are the aggregation operation, answer, and predicted_cells. You can just grab the middle value which is the predicted answer. 
+```python
+# Tokenize both Question and Table together
+inputs = tokenizer(table=table_df, queries=[question], padding='max_length', return_tensors='pt')
+
+# Model prediction
+ ##--- Helper function ---
+ 
+def get_final_answer(model, tokenizer, inputs, table_df):
+    outputs = model(**inputs)
+
+    logits = outputs.logits
+    logits_agg = outputs.logits_aggregation
+
+    predicted_answer_coordinates, predicted_aggregation_indices = tokenizer.convert_logits_to_predictions(
+        inputs,
+        logits.detach(),
+        logits_agg=logits_agg.detach()
+    )
+
+    aggregation_operators = ["NONE", "SUM", "AVERAGE", "COUNT"]
+
+    agg_op_idx = predicted_aggregation_indices[0] if predicted_aggregation_indices else 0
+    agg_op = aggregation_operators[agg_op_idx]
+
+    predicted_cells = []
+    for coord in predicted_answer_coordinates[0]:
+        cell_value = table_df.iat[coord[0], coord[1]]
+        predicted_cells.append(cell_value)
+
+    if agg_op == "COUNT":
+        answer = len(predicted_cells)
+    elif agg_op == "SUM":
+        try:
+            answer = sum(float(cell) for cell in predicted_cells)
+        except ValueError:
+            answer = "Could not SUM non-numeric cells"
+    elif agg_op == "AVERAGE":
+        try:
+            answer = sum(float(cell) for cell in predicted_cells) / len(predicted_cells)
+        except ValueError:
+            answer = "Could not AVERAGE non-numeric cells"
+    else:  # NONE
+        answer = predicted_cells
+
+    return agg_op, answer, predicted_cells
+
+_, answer, _ = get_final_answer(model, tokenizer, inputs, table_df)
+print(answer)
+```
+
+## Limitations
+The model does still not come close to a 100% accuracy. Possiblly using a larger model could help. It does seem to only be able to take in a limited size for tables, larger than a whole system. Once again, possibly a larger model could help. Also this needs to take in a question and table in dataframe format, so more preocessing is necessary than just a regular prompt.
+
+## Prompt Format
+
+
+
+
 
 
 ## Model Details