ORCHESTRATOR_TEST_CASE.md

🎼 Comprehensive Orchestrator Test Case

Test Scenario: Complex Multi-Drug Resistant Infection

This test case demonstrates the orchestrator's ability to coordinate multiple specialist agents to solve a complex infectious disease case requiring expertise in multiple domains.

---

🎯 Setup: Create Your Agent Team

Step 1: Create Specialist Subagents

Create these 4 specialist agents first (they will be the orchestrator's team):

Agent 1: Stewardship Specialist

Agent Type: 🎯 Specialist
Agent Name: Stewardship Expert
Mission: Expert in antimicrobial stewardship, antibiotic selection, de-escalation strategies, and optimizing duration of therapy
Skills:
  ✓ recommend_deescalation
  ✓ alert_prolonged_antibiotic_use
  ✓ search_pubmed (if you want literature support)

Agent 2: ID Diagnostician

Agent Type: 🎯 Specialist
Agent Name: ID Diagnostician
Mission: Expert in infectious disease diagnosis, differential diagnosis, culture interpretation, and diagnostic workup
Skills:
  ✓ differential_diagnosis
  ✓ generate_ddx_mermaid
  ✓ search_pubmed

Agent 3: ICU Consultant

Agent Type: 🎯 Specialist  
Agent Name: ICU Sepsis Consultant
Mission: Expert in critical care infectious diseases, sepsis management, hemodynamic support, and ICU-specific considerations
Skills:
  ✓ Any relevant tools you want
  ✓ search_pubmed

Agent 4: Infection Prevention Specialist

Agent Type: 🎯 Specialist
Agent Name: IPC Specialist
Mission: Expert in infection prevention and control, isolation precautions, outbreak management, and transmission-based precautions
Skills:
  ✓ ipc_reporting (if available)
  ✓ Any other relevant tools

Step 2: Create the Orchestrator

Agent Type: 🎼 Orchestrator
Agent Name: ID Maestro
Mission: Coordinate multiple ID specialists to provide comprehensive infectious disease consultation for complex cases requiring multidisciplinary expertise
Skills:
  (Orchestrators have access to all your other agents automatically)
API Key: Your OpenAI API key

---

📋 Test Case: Complex Septic Patient

Patient Presentation:

Copy and paste this detailed case to the ID Maestro orchestrator:

I need help with a complex case:

Patient: 68-year-old male
PMH: Type 2 diabetes (A1C 9.2%), ESRD on hemodialysis (MWF), recent hospitalization 3 weeks ago for MRSA bacteremia from infected dialysis catheter

Current Presentation:
- Day 3 ICU admission for septic shock
- Fever 39.8°C, HR 118, BP 85/50 on norepinephrine 0.15 mcg/kg/min
- WBC 24,000 with 18% bands, procalcitonin 8.5
- Lactate 4.2 → 2.8 after 4L crystalloid

Cultures:
- Blood cultures (×2 sets): Gram-positive cocci in clusters at 14 hours
- Preliminary: MRSA (same strain as 3 weeks ago)
- Sensitivities pending (results in 24 hours)
- Urine culture: Negative
- CXR: Right lower lobe infiltrate

Current Antibiotics (started 72 hours ago):
- Vancomycin 1g IV q12h (trough pending)
- Piperacillin-tazobactam 3.375g IV q6h

Additional Info:
- New tunneled dialysis catheter placed 2 weeks ago
- Patient on contact precautions
- Last vancomycin trough (from previous admission): 18 mcg/mL
- CrCl: Not applicable (on dialysis)
- Patient improving clinically: BP improving, lactate trending down

Questions:
1. Is current antibiotic coverage appropriate?
2. Should we de-escalate or change therapy?
3. What's the optimal duration?
4. Any diagnostic workup needed?
5. Are isolation precautions correct?
6. What are the key stewardship considerations?

---

✅ Expected Orchestrator Behavior

Phase 1: Planning

The orchestrator should:

1. Analyze the request and identify multiple distinct tasks
2. Create an execution plan listing which agents to invoke:
   • ID Diagnostician (for differential diagnosis and culture interpretation)
   • Stewardship Expert (for antibiotic optimization and de-escalation)
   • ICU Sepsis Consultant (for critical care considerations)
   • IPC Specialist (for isolation precautions)
3. Display the plan with numbered steps
4. Wait for your confirmation ("proceed")

Phase 2: Execution (after you say "proceed")

The orchestrator should:

1. Invoke each specialist agent sequentially or in parallel
2. Show progress: "🚀 Invoking [Agent Name]..."
3. Collect responses from each specialist
4. Display intermediate results as they come in

Phase 3: Synthesis

The orchestrator should:

1. Synthesize all specialist inputs into a comprehensive recommendation
2. Address all 6 questions from the original query
3. Highlight agreements between specialists
4. Resolve conflicts if specialists disagree
5. Provide prioritized recommendations
6. Include specific details from each specialty area

---

📊 What to Look For

✅ Good Orchestrator Performance:

Planning Phase:

• Identifies 4-6 distinct tasks in the case
• Plans to invoke 3-4 specialist agents
• Explains rationale for each agent selection
• Asks for confirmation before proceeding

Execution Phase:

• Shows "Invoking [Agent Name]" messages
• Displays each agent's response as it arrives
• Maintains context between agent calls
• Handles any agent errors gracefully

Synthesis Phase:

• Comprehensive final answer addressing all questions
• Specific recommendations from each specialist:
  • Diagnostician: Culture interpretation, workup recommendations
  • Stewardship: De-escalation plan, duration recommendations
  • ICU Consultant: Hemodynamic considerations, monitoring
  • IPC: Isolation precautions, transmission prevention
• Prioritized action items (urgent vs routine)
• Clinical reasoning and evidence-based rationale

❌ Signs of Issues:

• Orchestrator doesn't invoke any subagents (just gives generic answer)
• Shows execution plan but doesn't actually call the agents
• Can't find subagents (error: "agent not found")
• Synthesis doesn't incorporate subagent responses
• Generic response without specialty-specific details

---

🔍 Detailed Expected Outputs

From ID Diagnostician:

Expected content:
- MRSA bacteremia recurrence vs new infection
- Differential diagnosis for persistent bacteremia
- Recommendations for:
  * Echo to rule out endocarditis
  * Consider removing/replacing dialysis catheter
  * Imaging for metastatic foci
- Discussion of complicated vs uncomplicated bacteremia

From Stewardship Expert:

Expected content:
- Vancomycin optimization (check trough, AUC/MIC)
- Piperacillin-tazobactam: likely unnecessary (can de-escalate)
- Alternative options: daptomycin, ceftaroline, linezolid
- Duration: 14 days for uncomplicated, longer if endocarditis
- Monitoring: Weekly vancomycin troughs, renal function
- De-escalation timeline: After sensitivities available

From ICU Sepsis Consultant:

Expected content:
- Hemodynamic status: Improving (decreasing vasopressor needs)
- Source control: Consider catheter removal
- Fluid resuscitation: Adequate (lactate improving)
- Monitoring: Daily blood cultures until clearance
- ICU-specific considerations: Dialysis timing with antibiotics
- Prognosis: Good if source controlled

From IPC Specialist:

Expected content:
- Contact precautions: Appropriate for MRSA
- Isolation duration: Until cultures negative
- Staff education: Hand hygiene, PPE compliance
- Cohorting considerations
- Decolonization protocols (if recurrent MRSA)
- Environmental cleaning protocols

Orchestrator's Synthesis:

Expected structure:
1. Summary of case (MRSA bacteremia, improving septic shock)
2. Answers to each question with specialist input:
   Q1: Coverage - vanc appropriate, pip-tazo can stop
   Q2: De-escalation - stop pip-tazo, optimize vanc dosing
   Q3: Duration - 14+ days depending on source control
   Q4: Workup - echo, consider catheter removal
   Q5: Isolation - contact precautions correct
   Q6: Stewardship - multiple opportunities to optimize
3. Prioritized action plan:
   - Urgent (today): Echo, check vanc trough, stop pip-tazo
   - Within 24h: Review sensitivities, daily blood cultures
   - Ongoing: Monitor clinical response, consider catheter removal
4. Key takeaways and follow-up plan

---

🧪 Alternative Test Cases

Quick Test (Simpler):

"Patient with pneumonia needing antibiotic choice and stewardship guidance. 
How should I treat and what's the optimal duration?"

Complex Multi-System Test:

"78F with UTI, pneumonia, and C. diff. Multiple antibiotics on board. 
Need help with antibiotic optimization, infection prevention, and diagnostic workup."

Outbreak Scenario:

"3 patients in ICU with carbapenem-resistant Enterobacterales. 
Need infection control measures, treatment options, and stewardship guidance."

---

📝 Testing Checklist

Before Testing:

• All 4 specialist agents created
• Orchestrator agent created
• NCBI_EMAIL and API keys set (for PubMed searches)
• Browser ready in builder panel

During Test:

• Paste complete case to orchestrator
• Wait for execution plan
• Type "proceed" to start execution
• Watch for agent invocation messages
• Note any errors or missing agents

After Test:

• Review final synthesis
• Verify all questions answered
• Check if recommendations are actionable
• Confirm specialty-specific details included
• Test passed: ✅ or needs debugging: ❌

---

🐛 Troubleshooting

Problem: Orchestrator doesn't invoke subagents

Solution: Verify you created the subagents first and they're visible in your "Active Agents" list

Problem: "Agent not found" errors

Solution: Check agent names match exactly (case-sensitive). Recreate agents if needed.

Problem: Generic response without specialist details

Solution: Orchestrator might not have access to subagents. Verify per-user isolation working correctly.

Problem: Execution plan created but agents not invoked

Solution: Make sure to type "proceed" after the plan is shown

Problem: Only one agent invoked instead of multiple

Solution: Case might not be complex enough. Use the detailed test case above.

---

🎓 Learning Objectives

This test demonstrates:

1. ✅ Multi-agent coordination - Orchestrator managing 4+ specialists
2. ✅ Complex reasoning - Breaking down multifaceted clinical case
3. ✅ Information synthesis - Combining multiple expert opinions
4. ✅ Conflict resolution - Handling differing recommendations
5. ✅ Prioritization - Urgent vs routine actions
6. ✅ Comprehensive coverage - Addressing all aspects of care
7. ✅ Clinical applicability - Actionable recommendations

---

📚 Documentation Reference

• Orchestrator Setup: See FIX_ORCHESTRATOR_SUBAGENTS.md
• Agent Isolation: See AGENT_ISOLATION_COMPLETE.md
• Testing Guide: See TEST_AGENT_ISOLATION.md

---

Ready to test? Start by creating your 4 specialist agents, then the orchestrator, then paste the complex case! 🎼🔬