init

README.md

@@ -1,50 +1,212 @@
 ---
 title: VerifiableRewardsForScalableLogicalReasoning
-datasets:
--  
-tags:
-- evaluate
-- metric
-description: "TODO: add a description here"
+datasets: []
+colorFrom: blue
+colorTo: red
 sdk: gradio
-sdk_version: 3.19.1
+sdk_version: 5.34.2
 app_file: app.py
 pinned: false
+tags:
+  - evaluate
+  - reward
+  - reasoning
+  - metric
+description: >-
+    VerifiableRewardsForScalableLogicalReasoning is a metric for evaluating logical reasoning in AI systems by providing verifiable rewards. It computes rewards through symbolic execution of candidate solutions against validation programs, enabling automatic, transparent and reproducible evaluation in AI systems.
 ---
 
-# Metric Card for VerifiableRewardsForScalableLogicalReasoning
-
-***Module Card Instructions:*** *Fill out the following subsections. Feel free to take a look at existing metric cards if you'd like examples.*
+# Metric Card for Symbolic Judge: Verifiable Rewards for Scalable Logical Reasoning
 
 ## Metric Description
-*Give a brief overview of this metric, including what task(s) it is usually used for, if any.*
+Verifiable Rewards for Scalable Logical Reasoning (**SLR**) introduces a **symbolic judge** that provides verifiable rewards for logical reasoning tasks.
+To check whether a given task is solved, the symbolic judge evaluates a candidate solution (i.e., a logic rule, typically generated by a language model) and using an **executable validation program** that encodes the task's background knowledge and labeled examples.
+Evaluations performed by the symbolic judge are fully verifiable and grounded in formal logic, ensuring an automatic, transparent, and reproducible standard for evaluation and reward in both supervised and reinforcement learning settings.
+### How it Works
+- **Input:** The symbolic judge takes as input a candidate hypothesis (logic rule) and an executable validation program containing background knowledge and examples.
+- **Execution:** The candidate rule is executed against the validation program using a Prolog interpreter.
+- **Correctness Criteria:** The rule is considered correct if it entails all positive examples and rejects all negative examples.
+- **Metrics:** The symbolic judge computes a range of evaluation metrics (detailed below).
+**Note:** A local Prolog interpreter is required to execute validation programs.
+---
 
-## How to Use
-*Give general statement of how to use the metric*
+### Inputs
+- **predictions** (`list` of `str`): Each prediction should be a Prolog rule like "eastbound(T) :- Body."
+- **references** (`list` of `dict`): Each reference should contain:
+  - **validation_program** (`str`): Prolog program with background knowledge and examples
+  - **evaluation_config** (`dict`, optional): Configuration with:
+    - **positive_predicate** (`str`): Predicate identifying positive examples (default: "eastbound")
+    - **negative_predicate** (`str`): Predicate identifying negative examples (default: "westbound")
 
-*Provide simplest possible example for using the metric*
+### Metrics & Output Values
+- **accuracy** (`float`): Proportion of predictions that correctly classify all examples (0.0 to 1.0)
+- **partial_score** (`float`): Average proportion of correctly classified examples (0.0 to 1.0)
+- **syntax_score** (`float`): Proportion of rules with valid Prolog syntax (0.0 to 1.0)
+- **detailed_results** (`list` of `dict`): Per-example results with:
+  - **is_correct** (`bool`): Whether the rule correctly classifies all examples
+  - **partial_score** (`float`): Proportion of correctly classified examples
+  - **syntax_valid** (`bool`): Whether the rule has valid syntax
+  - **error** (`str`, optional): Any error messages from Prolog evaluation
+  - **exec_time** (`float`, optional): Execution time for evaluation
 
-### Inputs
-*List all input arguments in the format below*
-- **input_field** *(type): Definition of input, with explanation if necessary. State any default value(s).*
+---
+## How to Use with the datasets library
+```python
+from evaluate import load
+from datasets import load_dataset
+
+# Load the symbolic judge for logical reasoning
+symbolic_judge = load("AIML-TUDA/VerifiableRewardsForScalableLogicalReasoning")
+
+# load dataset AIML-TUDA/SLR-Bench
+ds = load_dataset('AIML-TUDA/SLR-Bench', 'v1-All')
+ds_test = ds['test'][:5]
+
+# Prepare the predictions and references
+rules = ds_test['ground-truth rule']
+references = [{'validation_program': p,
+                'evaluation_config': {
+                    "positive_predicate": "eastbound",
+                    "negative_predicate": "westbound"
+                }
+               } for p in ds_test['validation program']]
+# Compute the evaluation
+r2 = symbolic_judge.compute(predictions=rules, references=references)
+r2
+```
+
+### Outputs
+
+```python
+{'accuracy': 1.0,
+ 'partial_score': 1.0,
+ 'syntax_score': 1.0,
+ 'detailed_results': [{'is_correct': True,'partial_score': 1.0,'syntax_valid': True,'error': None,'exec_time1': 0.014362812042236328},
+                      {'is_correct': True,'partial_score': 1.0,'syntax_valid': True,'error': None,'exec_time1': 0.012364625930786133},
+                      {'is_correct': True,'partial_score': 1.0,'syntax_valid': True,'error': None,'exec_time1': 0.012273550033569336},
+                      {'is_correct': True,'partial_score': 1.0,'syntax_valid': True,'error': None,'exec_time1': 0.012486696243286133},
+                      {'is_correct': True,'partial_score': 1.0,'syntax_valid': True,'error': None,'exec_time1': 0.012131929397583008}]}
+```
+
+
+---
+
+## Examples
 
-### Output Values
+### Example 1: Evaluating a Single Rule
 
-*Explain what this metric outputs and provide an example of what the metric output looks like. Modules should return a dictionary with one or multiple key-value pairs, e.g. {"bleu" : 6.02}*
+```python
+from evaluate import load
 
-*State the range of possible values that the metric's output can take, as well as what in that range is considered good. For example: "This metric can take on any value between 0 and 100, inclusive. Higher scores are better."*
+symbolic_judge = load("AIML-TUDA/VerifiableRewardsForScalableLogicalReasoning")
 
-#### Values from Popular Papers
-*Give examples, preferrably with links to leaderboards or publications, to papers that have reported this metric, along with the values they have reported.*
+validation_program = """
+eastbound(train0).
+has_car(train0, car0_1).
+car_num(car0_1, 1).
+car_color(car0_1, white).
+car_len(car0_1, short).
+has_wall(car0_1, full).
 
-### Examples
-*Give code examples of the metric being used. Try to include examples that clear up any potential ambiguity left from the metric description above. If possible, provide a range of examples that show both typical and atypical results, as well as examples where a variety of input parameters are passed.*
+westbound(train1).
+has_car(train1, car1_1).
+car_num(car1_1, 1).
+car_color(car1_1, yellow).
+car_len(car1_1, short).
+has_wall(car1_1, full).
+"""
 
-## Limitations and Bias
-*Note any known limitations or biases that the metric has, with links and references if possible.*
+predicted_rule = "eastbound(Train):- has_car(Train, Car1), car_color(Car1, white)."
+
+results = symbolic_judge.compute(
+    predictions=[predicted_rule],
+    references=[{"validation_program": validation_program,
+                 "evaluation_config": {
+                     "positive_predicate": "eastbound",
+                     "negative_predicate": "westbound"
+                 }}]
+)
+
+print(results)
+```
+
+### Output Example 1
+
+```python
+{'accuracy': 1.0,
+ 'partial_score': 1.0,
+ 'syntax_score': 1.0,
+ 'detailed_results': [
+     {'is_correct': True,
+      'partial_score': 1.0,
+      'syntax_valid': True,
+      'error': None,
+      'exec_time1': 0.012056350708007812}]
+ }
+
+```
+
+### Example 2: Evaluating Multiple Rules
+
+```python
+correct_rule = "eastbound(Train):- has_car(Train, Car1), car_color(Car1, white)."
+incorrect_rule = "eastbound(Train):- has_car(Train, Car1), car_color(Car1, green)."
+
+results = symbolic_judge.compute(
+    predictions=[correct_rule, incorrect_rule],
+    references=[
+        {"validation_program": validation_program},
+        {"validation_program": validation_program}
+    ]
+)
+
+print(results)
+```
+
+### Example 3: Custom Evaluation Configuration
+
+```python
+validation_program = """
+% Custom problem
+parent(john, mary).
+parent(john, bob).
+parent(alice, bob).
+parent(susan, alice).
+
+% Examples
+grandparent(susan, bob).
+not_grandparent(john, alice).
+"""
+
+rule = "grandparent(X, Y) :- parent(X, Z), parent(Z, Y)."
+
+results = symbolic_judge.compute(
+    predictions=[rule],
+    references=[{
+        "validation_program": validation_program,
+        "evaluation_config": {
+            "positive_predicate": "grandparent",
+            "negative_predicate": "not_grandparent"
+        }
+    }]
+)
+```
 
 ## Citation
-*Cite the source where this metric was introduced.*
+
+```
+@misc{helff2025slrautomatedsynthesisframework,
+      title={SLR: An Automated Synthesis Framework for Scalable Logical Reasoning},
+      author={Lukas Helff and Ahmad Omar and Felix Friedrich and Wolfgang Stammer and Antonia Wüst and Tim Woydt and Rupert Mitchell and Patrick Schramowski and Kristian Kersting},
+      year={2025},
+      eprint={2506.15787},
+      archivePrefix={arXiv},
+      primaryClass={cs.AI},
+      url={https://arxiv.org/abs/2506.15787},
+}
+```
 
 ## Further References
-*Add any useful further references.*
+
+- [SLR-Bench Dataset](https://huggingface.co/datasets/AIML-TUDA/SLR-Bench)
+- [SLR-Bench GitHub Repository](https://github.com/AIML-TUDA/SLR)

app.py

@@ -1,6 +1,260 @@
 import evaluate
-from evaluate.utils import launch_gradio_widget
+import gradio as gr
+import os
 
 
-module = evaluate.load("LukasHug/verifiablerewardsforscalablelogicalreasoning")
-launch_gradio_widget(module)
+# Create your own Gradio interface instead of using the built-in widget
+def create_interface(module):
+    def evaluate_fn(prediction, references, pos_pred, neg_pred):
+        # Process a single prediction instead of multiple
+        pred = prediction.strip()
+
+        # Create reference with evaluation_config at the top level
+        ref = {
+            "validation_program": references.strip(),
+            "evaluation_config": {
+                "positive_predicate": pos_pred,
+                "negative_predicate": neg_pred
+            }
+        }
+
+        # Use a list with a single prediction and reference
+        results = module.compute(predictions=[pred], references=[ref])
+
+        # Extract the error message from detailed_results if it exists
+        error_msg = ""
+        if results["detailed_results"] and len(results["detailed_results"]) > 0:
+            error = results["detailed_results"][0].get("error")
+            if error:
+                error_msg = f"Error: {error}"
+
+        return (
+            f"Accuracy score: {results['accuracy']:.4f}",
+            f"Partial score: {results['partial_score']:.4f}",
+            f"Syntax score: {results['syntax_score']:.4f}",
+            error_msg
+        )
+
+    # Helper function to load example data
+    def load_example(example):
+        return (
+            example["rule"],
+            example["validation"],
+            example["pos_pred"],
+            example["neg_pred"]
+        )
+
+    # Read README.md content
+    readme_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "README.md")
+    with open(readme_path, 'r') as f:
+        readme_content = f.read()
+        readme_content = '# Metric Card ' + readme_content.split('# Metric Card ')[1]
+
+    # Define examples for quick use
+    example_train = {
+        "description": "Basic Train Problem",
+        "validation": """eastbound(train0).
+has_car(train0, car0_1).
+car_num(car0_1, 1).
+car_color(car0_1, white).
+car_len(car0_1, short).
+has_wall(car0_1, full).
+
+westbound(train1).
+has_car(train1, car1_1).
+car_num(car1_1, 1).
+car_color(car1_1, yellow).
+car_len(car1_1, short).
+has_wall(car1_1, full).
+""",
+        "rule": "eastbound(Train):- has_car(Train, Car1), car_color(Car1, white).",
+        "pos_pred": "eastbound",
+        "neg_pred": "westbound"
+    }
+
+    example_family = {
+        "description": "Family Relationships",
+        "validation": """% Custom problem
+parent(john, mary).
+parent(john, bob).
+parent(alice, bob).
+parent(susan, alice).
+
+% Examples
+grandparent(susan, bob).
+not_grandparent(john, alice).""",
+        "rule": "grandparent(X, Y) :- parent(X, Z), parent(Z, Y).",
+        "pos_pred": "grandparent",
+        "neg_pred": "not_grandparent"
+    }
+
+    with gr.Blocks(title="Symbolic Judge") as demo:
+        with gr.Tab("Evaluation"):
+            gr.Markdown("# Symbolic Judge: Verifiable Rewards for Scalable Logical Reasoning")
+            gr.Markdown("""
+Verifiable Rewards for Scalable Logical Reasoning (**SLR**) introduces a **symbolic judge** that provides verifiable rewards for logical reasoning tasks.
+To check whether a given task is solved, the symbolic judge evaluates a candidate solution (i.e., a logic rule, typically generated by a language model) and using an **executable validation program** that encodes the task's background knowledge and labeled examples.
+Evaluations performed by the symbolic judge are fully verifiable and grounded in formal logic, ensuring an automatic, transparent, and reproducible standard for evaluation and reward in both supervised and reinforcement learning settings.
+### How it Works
+- **Input:** The symbolic judge takes as input a candidate hypothesis (logic rule) and an executable validation program containing background knowledge and examples.
+- **Execution:** The candidate rule is executed against the validation program using a Prolog interpreter.
+- **Correctness Criteria:** The rule is considered correct if it entails all positive examples and rejects all negative examples.
+- **Metrics:** The symbolic judge computes a range of evaluation metrics (detailed below).
+- **Usage:** see **Documentation tab** for details on how to use the symbolic judge in your own projects.
+**Note:** A local Prolog interpreter is required to execute validation programs.
+""")
+
+            with gr.Row():
+                with gr.Column(scale=1):
+                    with gr.Group():
+                        gr.Markdown("### Model Output")
+                        prediction_input = gr.Textbox(
+                            label="Candidate Hypothesis to be evaluated(predicted rule by the model)",
+                            placeholder="eastbound(T) :- has_car(T, C), short(C), open(C).",
+                            lines=5
+                        )
+
+                        with gr.Group():
+                            gr.Markdown("### Validation Program")
+
+                            references_input = gr.Textbox(
+                                label="The validation program contains background knowledge and examples for testing",
+                                placeholder="% Background knowledge\ncar(car_1). car(car_2).\nlong(car_2). short(car_1).\nopen(car_1). closed(car_2).\n\n% Examples\neastbound(train_1).\nwestbound(train_2).\n\n% Train configurations\nhas_car(train_1, car_1).\nhas_car(train_2, car_2).",
+                                lines=12
+                            )
+
+                            with gr.Row():
+                                pos_pred_input = gr.Textbox(
+                                    label="Positive Validation Examples",
+                                    value="eastbound",
+                                    placeholder="eastbound",
+                                    info="The predicate name identifying positive examples in the validation program"
+                                )
+                                neg_pred_input = gr.Textbox(
+                                    label="Negative Validation Examples",
+                                    value="westbound",
+                                    placeholder="westbound",
+                                    info="The predicate name identifying negative examples in the validation program"
+                                )
+
+                        eval_button = gr.Button("Evaluate", variant="primary")
+
+                with gr.Column(scale=1):
+                    with gr.Group():
+                        gr.Markdown("### Evaluation Metrics")
+                        with gr.Group():
+                            accuracy_output = gr.Textbox(
+                                label="Overall Accuracy",
+                                info="Proportion of hypotheses that solve the tasks",
+                                container=True
+                            )
+                            partial_score_output = gr.Textbox(
+                                label="Partial Score",
+                                info="Proportion of examples that are correctly classified in the tasks",
+                                container=True
+                            )
+                            syntax_score_output = gr.Textbox(
+                                label="Syntax Score",
+                                info="Proportion of syntactically valid hypothesis",
+                                container=True
+                            )
+                            error_output = gr.Textbox(
+                                label="Syntax Details",
+                                info="Error messages for syntax errors or execution failures",
+                                container=True,
+                            )
+                            gr.Markdown("Note: This interface evaluates a single hypothesis at a time. Use Python API for batch processing")
+            # Add the examples section
+            # Example list
+            examples = [
+                ["Train Problem", example_train],
+                ["Family Relationships", example_family]
+            ]
+
+            with gr.Accordion("Example Logical Reasoning Tasks", open=True):
+                example_radio = gr.Radio([ex[0] for ex in examples], label="Select an example", value="Train Problem")
+                # Show preview of selected example
+                with gr.Row():
+                    with gr.Column():
+                        gr.Markdown("### Selected Example Preview")
+                        example_description = gr.Markdown("**Description**: " + example_train["description"])
+                        with gr.Row():
+                            with gr.Column():
+                                gr.Markdown("#### Candidate Hypothesis")
+                                example_rule = gr.Code(value=example_train["rule"])
+                            with gr.Column():
+                                gr.Markdown("#### Validation Program")
+                                example_validation = gr.Code(value=example_train["validation"])
+
+                        with gr.Row():
+                            with gr.Column():
+                                gr.Markdown("#### Validation Examples")
+                                example_predicates = gr.Markdown(f"""
+                                **Positive Examples**: `{example_train["pos_pred"]}`  
+                                **Negative Examples**: `{example_train["neg_pred"]}`
+                                """)
+
+                # Load button for the selected example
+                load_button = gr.Button("Load Selected Example", variant="secondary")
+            gr.Markdown("### Citation")
+
+            gr.Markdown("""
+            If you use Symbolic Judge in your work, please cite:
+            ```
+            @misc{helff2025slrautomatedsynthesisframework,
+                  title={SLR: An Automated Synthesis Framework for Scalable Logical Reasoning},
+                  author={Lukas Helff and Ahmad Omar and Felix Friedrich and Wolfgang Stammer and Antonia Wüst and Tim Woydt and Rupert Mitchell and Patrick Schramowski and Kristian Kersting},
+                  year={2025},
+                  eprint={2506.15787},
+                  archivePrefix={arXiv},
+                  primaryClass={cs.AI},
+                  url={https://arxiv.org/abs/2506.15787},
+            }
+            ```
+            """)
+
+            # Set up event handlers for the example selection
+            def update_example_preview(selection):
+                selected_example = next((ex[1] for ex in examples if ex[0] == selection), example_train)
+                return (
+                    "**Description**: " + selected_example["description"],
+                    selected_example["rule"],
+                    selected_example["validation"],
+                    f"""
+                    **Positive Examples**: `{selected_example["pos_pred"]}`  
+                    **Negative Examples**: `{selected_example["neg_pred"]}`
+                    """
+                )
+
+            example_radio.change(
+                fn=update_example_preview,
+                inputs=[example_radio],
+                outputs=[example_description, example_rule, example_validation, example_predicates]
+            )
+
+            # Event handler for the load button
+            def load_selected_example(selection):
+                selected_example = next((ex[1] for ex in examples if ex[0] == selection), example_train)
+                return load_example(selected_example)
+
+            load_button.click(
+                fn=load_selected_example,
+                inputs=[example_radio],
+                outputs=[prediction_input, references_input, pos_pred_input, neg_pred_input]
+            )
+
+            # Set up the evaluate button
+            eval_button.click(
+                fn=evaluate_fn,
+                inputs=[prediction_input, references_input, pos_pred_input, neg_pred_input],
+                outputs=[accuracy_output, partial_score_output, syntax_score_output, error_output]
+            )
+
+        with gr.Tab("Documentation"):
+            gr.Markdown(readme_content)
+
+    return demo
+
+# Use a local path instead of a module name
+module = evaluate.load("AIML-TUDA/VerifiableRewardsForScalableLogicalReasoning")
+create_interface(module).launch()

packages.txt

@@ -0,0 +1 @@
+swi-prolog

requirements.txt

@@ -1 +1,5 @@
-git+https://github.com/huggingface/evaluate@main
+git+https://github.com/huggingface/evaluate@main
+evaluate
+gradio>=5.34.2
+datasets
+tqdm

tests.py

@@ -1,17 +0,0 @@
-test_cases = [
-    {
-        "predictions": [0, 0],
-        "references": [1, 1],
-        "result": {"metric_score": 0}
-    },
-    {
-        "predictions": [1, 1],
-        "references": [1, 1],
-        "result": {"metric_score": 1}
-    },
-    {
-        "predictions": [1, 0],
-        "references": [1, 1],
-        "result": {"metric_score": 0.5}
-    }
-]

verifiablerewardsforscalablelogicalreasoning.py

@@ -11,85 +11,296 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-"""TODO: Add a description here."""
+""" Symbolic Judge: Verifiable Rewards for Logical Reasoning at Scale """
 
+import os
+import subprocess
+import tempfile
 import evaluate
+import logging
 import datasets
+from tqdm import tqdm
+import time
+import multiprocessing as mp
+import re
 
+logger = logging.getLogger(__name__)
 
-# TODO: Add BibTeX citation
 _CITATION = """\
-@InProceedings{huggingface:module,
-title = {A great new module},
-authors={huggingface, Inc.},
-year={2020}
+@misc{helff2025slrautomatedsynthesisframework,
+      title={SLR: An Automated Synthesis Framework for Scalable Logical Reasoning}, 
+      author={Lukas Helff and Ahmad Omar and Felix Friedrich and Wolfgang Stammer and Antonia Wüst and Tim Woydt and Rupert Mitchell and Patrick Schramowski and Kristian Kersting},
+      year={2025},
+      eprint={2506.15787},
+      archivePrefix={arXiv},
+      primaryClass={cs.AI},
+      url={https://arxiv.org/abs/2506.15787}, 
 }
 """
 
-# TODO: Add description of the module here
 _DESCRIPTION = """\
-This new module is designed to solve this great ML task and is crafted with a lot of care.
-"""
+Verifiable Rewards for Scalable Logical Reasoning (SLR) introduces a symbolic judge that provides verifiable rewards for logical reasoning tasks.
+To check whether a given task is solved, the symbolic judge evaluates a candidate solution (i.e., a logic rule, typically generated by a language model) using an executable validation program that encodes the task's background knowledge and labeled examples.
+Evaluations performed by the symbolic judge are fully verifiable and grounded in formal logic, ensuring an automatic, transparent, and reproducible standard for evaluation and reward in both supervised and reinforcement learning settings.
+
+How it Works:
+  - Input: The symbolic judge takes as input a candidate hypothesis (logic rule) and an executable validation program containing background knowledge and examples.
+  - Execution: The candidate rule is executed against the validation program using a Prolog interpreter.
+  - Correctness Criteria: The rule is considered correct if it entails all positive examples and rejects all negative examples.
+  - Metrics: The symbolic judge computes a range of evaluation metrics (detailed below).
+  - Usage: see **Documentation tab** for details on how to use the symbolic judge in your own projects.
+
+Example usage:
+  from evaluate import load
+
+  symbolic_judge = load("AIML-TUDA/VerifiableRewardsForScalableLogicalReasoning")
+
+  validation_program = \"\"\"
+  eastbound(train0).
+  has_car(train0, car0_1).
+  car_num(car0_1, 1).
+  car_color(car0_1, white).
+  car_len(car0_1, short).
+  has_wall(car0_1, full).
 
+  westbound(train1).
+  has_car(train1, car1_1).
+  car_num(car1_1, 1).
+  car_color(car1_1, yellow).
+  car_len(car1_1, short).
+  has_wall(car1_1, full).
+  \"\"\"
+
+  predicted_rule = "eastbound(Train):- has_car(Train, Car1), car_color(Car1, white)."
+
+  results = symbolic_judge.compute(
+      predictions=[predicted_rule],
+      references=[{"validation_program": validation_program,
+                   "evaluation_config": {
+                       "positive_predicate": "eastbound",
+                       "negative_predicate": "westbound"
+                   }}]
+  )
+
+Note: A local Prolog interpreter is required to execute validation programs.
+"""
 
-# TODO: Add description of the arguments of the module here
 _KWARGS_DESCRIPTION = """
-Calculates how good are predictions given some references, using certain scores
 Args:
-    predictions: list of predictions to score. Each predictions
-        should be a string with tokens separated by spaces.
-    references: list of reference for each prediction. Each
-        reference should be a string with tokens separated by spaces.
+    predictions (`list` of `str`): Each prediction should be a Prolog rule like "pred(T) :- Body."
+    references (`list` of `dict`): Each reference should contain:
+        - 'validation_program' (`str`): Background knowledge in Prolog syntax
+        - 'evaluation_config' (`dict`, optional): Configuration of predicates to use for evaluation.
+        Define: positive_predicate, and negative_predicate
 Returns:
-    accuracy: description of the first score,
-    another_score: description of the second score,
-Examples:
-    Examples should be written in doctest format, and should illustrate how
-    to use the function.
-
-    >>> my_new_module = evaluate.load("my_new_module")
-    >>> results = my_new_module.compute(references=[0, 1], predictions=[0, 1])
-    >>> print(results)
-    {'accuracy': 1.0}
+    accuracy (`float`): The proportion of predictions that correctly solve all examples. Value is between 0 and 1.
+    partial_score (`float`): Average proportion of correctly classified examples across all predictions. Value is between 0 and 1.
+    syntax_score (`float`): Proportion of rules with valid syntax. Value is between 0 and 1.
+    detailed_results (`list` of `dict`): Per-example results including correctness, partial score, execution time, and any errors encountered.
 """
 
-# TODO: Define external resources urls if needed
-BAD_WORDS_URL = "http://url/to/external/resource/bad_words.txt"
+
+def _evaluate_with_prolog(rule_to_evaluate, validation_program, eval_config, timeout=5):
+    """
+    Evaluates a predicted rule against the validation program using Prolog.
+    """
+    # Extract configuration
+    positive_pred = eval_config.get("positive_predicate", "eastbound")
+    negative_pred = eval_config.get("negative_predicate", "westbound")
+    # extract predicate from rule_to_evaluate
+    if positive_pred not in rule_to_evaluate:
+        logger.warning(f"Rule '{rule_to_evaluate}' does not contain positive predicate '{positive_pred}'")
+        return {
+            "is_correct": False,
+            "partial_score": 0.0,
+            "syntax_valid": False,
+            "error": f"Invalid Syntax: Rule does not contain positive predicate '{positive_pred}'"
+        }
+
+    pos_examples = re.findall(rf'{positive_pred}\(([^)]+)\)', validation_program)
+    neg_examples = re.findall(rf'{negative_pred}\(([^)]+)\)', validation_program)
+
+    # Determine arity by counting commas in first example plus 1
+    arity = 1  # default to unary
+    if pos_examples:
+        arity = pos_examples[0].count(',') + 1
+    elif neg_examples:
+        arity = neg_examples[0].count(',') + 1
+
+    # Create variables based on arity
+    vars = ", ".join([f"X{i}" for i in range(1, arity + 1)])
+
+    symbolic_judge = f"""
+% Dynamic evaluation predicates
+check({vars}) :- pos({vars}), {positive_pred}({vars}).      % positive covered
+check({vars}) :- neg({vars}), \\+ {positive_pred}({vars}).  % negative rejected
+
+% Count successful checks
+check_count(Count) :- 
+    setof(({vars}), ((pos({vars}); neg({vars})), check({vars})), CorrectExamples),
+    length(CorrectExamples, Count).
+
+check_all :- forall((pos({vars});neg({vars})), check({vars})).
+    """
+    # Add the rule to evaluate
+    validation_program = re.sub(rf'\b{positive_pred}\b', 'pos', validation_program)
+    validation_program = re.sub(rf'\b{negative_pred}\b', 'neg', validation_program)
+
+    pos_negs = validation_program.count("pos(") + validation_program.count("neg(")
+    validation_program = '\n'.join(sorted(validation_program.splitlines()))
+    full_program = validation_program + "\n\n" + symbolic_judge + "\n\n" + rule_to_evaluate + "\n\n"
+
+    with tempfile.NamedTemporaryFile(suffix='.pl', mode='w', delete=False) as f:
+        f.write(full_program)
+        temp_file = f.name
+
+    try:
+        eval_start_time = time.time()
+        # Execute the Prolog program
+        cmd = ['swipl', '-s', temp_file, '-g', 'check_count(Count), writeln(Count)', '-t', 'halt']
+        result = subprocess.run(
+            cmd,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+            timeout=timeout,
+            text=True
+        )
+        # Extract partial score from output
+        partial_score = int(result.stdout.strip()) / pos_negs if pos_negs > 0 else 0.0
+
+        is_correct = True if partial_score == 1.0 else False
+
+        error = result.stderr if result.stderr else None
+        t1 = time.time()
+
+        return {
+            "is_correct": is_correct,
+            "partial_score": partial_score,
+            "syntax_valid": True,
+            "error": error,
+            "exec_time1": t1 - eval_start_time,
+        }
+
+    except subprocess.TimeoutExpired:
+        logger.warning(f"Evaluation timed out after {timeout} seconds for rule: {rule_to_evaluate}...")
+        return {"is_correct": False, "partial_score": 0.0, "syntax_valid": False,
+                "error": f"Evaluation timed out after {timeout} seconds"}
+    except Exception as e:
+        logger.warning(f"Error evaluating rule '{rule_to_evaluate}' returns: '{result.stdout.strip() if result else 'No error message'}'")
+        error_message = 'Invalid Syntax: exit with ' + str(e)
+        return {"is_correct": False, "partial_score": 0.0, "syntax_valid": False, "error": error_message}
+    finally:
+        if os.path.exists(temp_file):
+            os.remove(temp_file)
+
 
 
 @evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class VerifiableRewardsForScalableLogicalReasoning(evaluate.Metric):
-    """TODO: Short description of my evaluation module."""
+    def __init__(self, config_name=None, **kwargs):
+        """
+        Initializes the PrologEval metric.
+
+        Args:
+            config_name (str, optional): Name of the configuration to use.
+            **kwargs: Additional keyword arguments.
+        """
+        super().__init__(config_name=config_name, **kwargs)
+        self.config_name = config_name or "default"
+        self._info = self._info()
+        self._download_and_prepare(dl_manager=None)
 
     def _info(self):
-        # TODO: Specifies the evaluate.EvaluationModuleInfo object
         return evaluate.MetricInfo(
-            # This is the description that will appear on the modules page.
-            module_type="metric",
             description=_DESCRIPTION,
             citation=_CITATION,
             inputs_description=_KWARGS_DESCRIPTION,
-            # This defines the format of each prediction and reference
             features=datasets.Features({
-                'predictions': datasets.Value('int64'),
-                'references': datasets.Value('int64'),
+                'predictions': datasets.Value('string'),
+                'references': {
+                    'validation_program': datasets.Value('string'),
+                    'evaluation_config': {
+                        'positive_predicate': datasets.Value('string'),
+                        'negative_predicate': datasets.Value('string')
+                    }
+                },
             }),
-            # Homepage of the module for documentation
-            homepage="http://module.homepage",
-            # Additional links to the codebase or references
-            codebase_urls=["http://github.com/path/to/codebase/of/new_module"],
-            reference_urls=["http://path.to.reference.url/new_module"]
+            codebase_urls=["https://github.com/AIML-TUDA/SLR-Bench"],
+            reference_urls=["https://huggingface.co/datasets/AIML-TUDA/SLR-Bench"]
         )
 
     def _download_and_prepare(self, dl_manager):
-        """Optional: download external resources useful to compute the scores"""
-        # TODO: Download external resources if needed
-        pass
-
-    def _compute(self, predictions, references):
-        """Returns the scores"""
-        # TODO: Compute the different scores of the module
-        accuracy = sum(i == j for i, j in zip(predictions, references)) / len(predictions)
+        """Checks if SWI-Prolog is installed or warns the user."""
+        try:
+            subprocess.run(
+                ["swipl", "--version"],
+                stdout=subprocess.PIPE,
+                stderr=subprocess.PIPE,
+                check=True
+            )
+        except (subprocess.CalledProcessError, FileNotFoundError):
+            logger.warning(
+                "SWI-Prolog not found. Please install it:\n"
+                "Ubuntu/Debian: sudo apt-get install swi-prolog\n"
+                "macOS: brew install swi-prolog\n"
+                "Windows: download from https://www.swi-prolog.org/download/stable"
+            )
+
+    def _compute(self, predictions: list, references: list):
+        """Calculates the accuracy of predictions using Prolog for evaluation with multiprocessing."""
+        if not isinstance(predictions, list):
+            predictions = [predictions]
+
+        if len(predictions) != len(references):
+            raise ValueError(
+                f"Number of predictions ({len(predictions)}) and references {len(references)}) don't match")
+
+        # Prepare evaluation inputs
+        eval_inputs = []
+        for i, (prediction, reference) in enumerate(zip(predictions, references)):
+            validation_program = reference.get("validation_program", reference.get("validation program"))
+
+            # Extract configuration parameters directly from reference
+            # This is the key fix: look for config values at the top level if evaluation_config doesn't exist
+            eval_config = reference.get("evaluation_config", {
+                    "positive_predicate": "eastbound",
+                    "negative_predicate": "westbound"
+                })
+
+            if not validation_program:
+                raise ValueError(f"Example {i} does not contain validation program field")
+
+            # Make sure the prediction is a proper rule format
+            if not prediction.strip().endswith('.'):
+                prediction = prediction.strip() + '.'
+
+            eval_inputs.append((prediction, validation_program, eval_config))
+
+        # Process evaluations in parallel
+        num_cpus = max(1, mp.cpu_count() - 1)  # Leave one CPU free
+        with mp.Pool(processes=num_cpus) as pool:
+            results = list(tqdm(
+                pool.starmap(_evaluate_with_prolog, eval_inputs),
+                total=len(eval_inputs),
+                desc="Evaluating rules (parallel)"
+            ))
+        # no multiprocessing in the main thread, so we can use tqdm directly
+        # results = []
+        # for prediction, validation_program, eval_config in tqdm(eval_inputs, total=len(predictions), desc="Evaluating rules"):
+        #     results.append(_evaluate_with_prolog(prediction, validation_program, eval_config))
+
+        # Calculate metrics
+        partial_scores = [result["partial_score"] for result in results]
+        correct_count = sum(1 for result in results if result["is_correct"])
+        syntax_valid_count = sum(1 for result in results if result["syntax_valid"])
+
+        accuracy = correct_count / len(predictions) if predictions else 0
+        partial_score = sum(partial_scores) / len(predictions) if partial_scores else 0
+        syntax_score = syntax_valid_count / len(predictions) if predictions else 0
+
         return {
             "accuracy": accuracy,
-        }
+            "partial_score": partial_score,
+            "syntax_score": syntax_score,
+            "detailed_results": results
+        }