File size: 2,887 Bytes
e54fd17 6f40440 e54fd17 6f40440 e54fd17 6f40440 e54fd17 6f40440 e54fd17 6f40440 e54fd17 6f40440 e54fd17 6f40440 e54fd17 6f40440 e54fd17 6f40440 e54fd17 6f40440 e54fd17 6f40440 e54fd17 6f40440 e54fd17 6f40440 e54fd17 6f40440 e54fd17 6f40440 e54fd17 6f40440 e54fd17 6f40440 e54fd17 6f40440 e54fd17 6f40440 e54fd17 6f40440 e54fd17 6f40440 e54fd17 6f40440 e54fd17 6f40440 e54fd17 6f40440 e54fd17 6f40440 e54fd17 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 |
# CLI Usage
LLMPromptKit provides a command-line interface (CLI) for managing prompts, versions, tests, and evaluations.
## Basic Commands
### Prompt Management
```bash
# Create a prompt
llmpromptkit prompt create "Weather Forecast" --content "Provide a weather forecast for {location} on {date}" --tags "weather,forecast"
# List all prompts
llmpromptkit prompt list
# Get prompt details
llmpromptkit prompt get <prompt_id>
# Update a prompt
llmpromptkit prompt update <prompt_id> --content "New content" --tags "new,tags"
# Delete a prompt
llmpromptkit prompt delete <prompt_id>
```
### Version Control
```bash
# Commit a version
llmpromptkit version commit <prompt_id> --message "Version description"
# List versions
llmpromptkit version list <prompt_id>
# Check out (revert to) a specific version
llmpromptkit version checkout <prompt_id> <version_number>
# Compare versions
llmpromptkit version diff <prompt_id> <version1> <version2>
```
### Testing
```bash
# Create a test case
llmpromptkit test create <prompt_id> --input '{"location": "New York", "date": "tomorrow"}' --expected "Expected output"
# List test cases
llmpromptkit test list <prompt_id>
# Run a specific test case
llmpromptkit test run <test_case_id> --llm openai
# Run all test cases for a prompt
llmpromptkit test run-all <prompt_id> --llm openai
# Run an A/B test between two prompts
llmpromptkit test ab <prompt_id_a> <prompt_id_b> --inputs '[{"var": "value1"}, {"var": "value2"}]' --llm openai
```
### Evaluation
```bash
# Evaluate a prompt
llmpromptkit eval run <prompt_id> --inputs '[{"var": "value1"}, {"var": "value2"}]' --llm openai
# List available metrics
llmpromptkit eval metrics
# Register a custom metric
llmpromptkit eval register-metric <metric_file.py>
```
## Environment Configuration
The CLI supports environment variables for configuration:
- `LLMPROMPTKIT_STORAGE`: Path to store prompts and related data
- `LLMPROMPTKIT_OPENAI_API_KEY`: OpenAI API key for built-in LLM support
- `LLMPROMPTKIT_DEFAULT_LLM`: Default LLM to use for testing and evaluation
You can also create a config file at `~/.llmpromptkit/config.json`:
```json
{
"storage_path": "/path/to/storage",
"default_llm": "openai",
"api_keys": {
"openai": "your-openai-key"
}
}
```
## Advanced Usage
### Multiple Storage Locations
```bash
# Specify a storage location for a command
llmpromptkit --storage /path/to/storage prompt list
# Export a prompt to another storage
llmpromptkit prompt export <prompt_id> --output /path/to/output.json
# Import a prompt from a file
llmpromptkit prompt import /path/to/prompt.json
```
### Automation and Scripting
```bash
# Get output in JSON format
llmpromptkit --json prompt list
# Use in shell scripts
PROMPT_ID=$(llmpromptkit --json prompt create "Script Prompt" --content "Content" | jq -r '.id')
echo "Created prompt with ID: $PROMPT_ID"
```
|