Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+ModelComparisonNew.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -45,34 +45,28 @@ Solidity-Code-LLM is a specialized language model trained in two stages: pre-tra
 - **Dtype**: bfloat16
 
 ### Model Sources
-Model is deployed on huggingface space for inference
+For more details, please refer to,
+- **Paper [optional]:** {{ paper | default("[More Information Needed]", true)}}
 - **Demo:** [Demo On Hugging Face Space](https://huggingface.co/spaces/Chain-GPT/SolidityLLMDemo)
 
+
 # Model Comparison
 We have compared our model with the following models
-- GPT 4.5 Preview
+- Qwen/CodeQwen1.5-7B
+- deepseek-ai/deepseek-coder-1.3b-base
+- codellama/CodeLlama-7b-hf
 - GPT 4o mini
-- [Qwen 2.5-Coder-7B](https://huggingface.co/Qwen/Qwen2.5-Coder-7B)
-- [DeepSeek-Coder-7B-Instruct-v1.5](https://huggingface.co/deepseek-ai/deepseek-coder-7b-instruct-v1.5)
 
 On the following parameters
-- **Compilation(%)** - Percentage of generated contracts that compile successfully without modification.
-- **OpenZeppelin Compliance(%)** - Adherence to OpenZeppelin library usage and standards.
-- **Gas Efficiency(%)** - Degree of gas optimization based on Slither’s suggestions.
-- **Security(%)** - Percentage of code free from common vulnerabilities detected by Slither.
-- **Average Lines of Code** - Average number of non-empty, commented-included lines in generated contracts, indicating verbosity or conciseness
+- Compilation(%)--Percentage of generated contracts that compile successfully without modification.
+- OpenZeppelin Compliance(%)--Adherence to OpenZeppelin library usage and standards.
+- Gas Efficiency(%)--Degree of gas optimization based on Slither’s suggestions.
+- Security(%)--Percentage of code free from common vulnerabilities detected by Slither.
 
 ## Benchmark
-Below is a figure summarizing the performance of each model across the four evaluation metrics.
+The figure below presents a detailed comparison of the models across all evaluation criteria
 ![Benchmark](Benchmark.png)
 
-Following obvservation were made regarding Solidity LLM.
-- Highest Compilation Success Rate (~83%), demonstrating strong Solidity syntax and structure generation.
-- Good OpenZeppelin Compliance (~65%), indicating frequent use of standard libraries and contract patterns. While GPT-4.5, being a much larger model, naturally exhibits stronger adherence to OpenZeppelin standards due to its broader training data, Solidity LLM achieves commendable compliance given its smaller size.
-- Top Gas Efficiency (~72%), producing optimized code as evaluated by tools like Slither.
-- Moderate Security Score (~58%), showing acceptable security posture but room for improvement. GPT-4.5 benefits from its scale in handling more security cases.
-- Concise Code (~70% LOC score), generating relatively compact and efficient smart contracts.
-
 
 # Uses
 ### Direct Use
@@ -135,7 +129,7 @@ from threading import Thread
 from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer
 
 model = AutoModelForCausalLM.from_pretrained(
-    "ChainGPT/SolidityLLM",
+    "Chain-GPT/Solidity-LLM",
     torch_dtype=torch.bfloat16,
     device_map="cuda"
 )
@@ -278,7 +272,7 @@ contract DecentralizedLibrary is Ownable(msg.sender) {
 # Evaluation Matrics
 To evaluate the performance of our fine-tuned LLM specialized in Solidity smart contract generation, we used **[Slither](https://github.com/crytic/slither)**, a static analysis framework widely used for analyzing Solidity code.
 
-We focused on following key evaluation criteria:
+We focused on four key evaluation criteria:
 
 - **Compilation Success Rate**
 We measured the percentage of generated smart contracts that compile successfully without modification. This helps assess the syntactic and structural correctness of the model outputs.
@@ -292,17 +286,8 @@ Using Slither’s gas optimization analysis, we identified areas in the generate
 - **Security Vulnerabilities**
 We analyzed each contract for known security vulnerabilities using Slither’s built-in detectors. We recorded the number and severity of the vulnerabilities detected, providing a measure of the security quality of the model’s outputs.
 
-- **Average Lines of Code (LOC)**
-Captures the average number of lines per generated contract, excluding blank lines but including comments. This metric reflects code verbosity or conciseness, and helps gauge implementation completeness versus potential redundancy.
-
-These metrics collectively provide a multi-dimensional view of the model’s effectiveness, spanning correctness, efficiency, security, and usability. They are designed to reflect both automated benchmarks and real-world developer expectations.
+These evaluation metrics help quantify the practical usability and reliability of the generated smart contracts in real-world scenarios.
 
 
 # Summary
-Solidity LLM, despite its compact 2B parameter size, delivers standout performance in generating Solidity smart contracts. It achieved the highest compilation success rate (83%), showcasing robust syntactic and structural understanding. Its strong OpenZeppelin compliance (65%), though slightly behind very large models like GPT-4.5, is impressive given the scale difference, reflecting reliable use of industry-standard patterns and libraries.
-
-Further, Solidity LLM ranked highest in gas efficiency (72%), producing optimized code suitable for cost-sensitive deployments. While the security score (58%) indicates room for improvement, the model consistently generated secure-enough contracts for practical use. Its concise output (70% LOC score) also suggests an efficient coding style, balancing brevity with completeness.
-
-Overall, Solidity LLM proves to be a resource-efficient, reliable, and well-balanced model for Solidity code generation.
-
-Looking ahead, future releases will focus on improving support for newer versions of the Solidity language and OpenZeppelin libraries, enhancing user interaction by enabling contract modifications, expanding compatibility to other languages like Rust, and developing larger models capable of handling longer context windows.
+Model shows improved understanding and generation capabilities in Solidity when compared to baseline LLMs not trained on Solidity data.

image.py

@@ -0,0 +1,46 @@
+import matplotlib.pyplot as plt
+import numpy as np
+
+# Labels for x-axis
+criteria = [
+    "Code Quality", "Security Features", "Feature Completeness", "Gas Optimization",
+    "Error Handling", "Documentation", "Contract Structure", "Token Integration", 
+    "Event Implementation", "Success Rate"
+]
+
+# Data for each model (scores out of 10 for the first 10 criteria, and success rate as percentage of True/60)
+models = {
+    "Solidity LLM": [9, 9, 9, 9, 8, 9, 9, 9, 9, 55/60*10],
+    "GPT-4.5-preview": [9, 9, 8, 7, 8, 8, 8, 9, 8, 37/60*10],
+    "GPT-4o-mini": [4, 3, 4, 4, 5, 4, 5, 3, 5, 9/60*10],
+    # "gpt-4.o-preview": [8, 8, 8, 6, 6, 5, 7, 8, 6, 25/60*10],
+    # "gpt-4o": [3, 3, 4, 4, 4, 4, 5, 2, 4, 30/60*10],
+    "gpt-4.1": [8, 8, 8, 6, 6, 7, 7, 8, 6, 19/60*10],
+    # "gpt-4.1-mini": [5, 5, 5, 5, 6, 7, 6, 3, 5, 21/60*10],
+    # "gpt-4.1-nano": [9, 9, 7, 6, 7, 8, 8, 7, 7, 37/60*10],
+    # "GPT-o3": [3, 2, 4, 3, 4, 3, 4, 5, 4, 5/60*10],
+    # "llama-4-scout": [2, 2, 3, 2, 3, 2, 3, 4, 3, 3/60*10],
+    "llama-4-maverick": [4, 3, 5, 4, 5, 6, 6, 6, 5, 8/60*10]
+}
+
+# X-axis positions
+x = np.arange(len(criteria))
+width = 0.08
+
+# Plotting
+fig, ax = plt.subplots(figsize=(20, 8))
+
+for i, (model, values) in enumerate(models.items()):
+    ax.bar(x + i*width, values, width, label=model)
+
+# Labels and formatting
+ax.set_ylabel('Score (Out of 10)')
+ax.set_title('Comparison of LLMs on Solidity Smart Contract Generation')
+ax.set_xticks(x + width * len(models) / 2)
+ax.set_xticklabels(criteria, rotation=45, ha="right")
+ax.set_ylim(0, 10)
+ax.legend(loc='upper left', bbox_to_anchor=(1, 1))
+
+plt.tight_layout()
+plt.show()
+plt.savefig('model_comparison_new.png', dpi=500)