Update README.md
Browse files
README.md
CHANGED
|
@@ -95,9 +95,10 @@ lm_eval --model hf --model_args pretrained=jerryzh168/phi4-mini-float8dq --tasks
|
|
| 95 |
|
| 96 |
# Model Performance
|
| 97 |
|
| 98 |
-
#
|
| 99 |
```
|
| 100 |
-
|
|
|
|
| 101 |
```
|
| 102 |
|
| 103 |
# Download dataset
|
|
@@ -105,6 +106,9 @@ Download sharegpt dataset: `wget https://huggingface.co/datasets/anon8231489123/
|
|
| 105 |
|
| 106 |
Other datasets can be found in: https://github.com/vllm-project/vllm/tree/main/benchmarks
|
| 107 |
# benchmark_latency
|
|
|
|
|
|
|
|
|
|
| 108 |
## baseline
|
| 109 |
```
|
| 110 |
python benchmarks/benchmark_latency.py --input-len 256 --output-len 256 --model microsoft/Phi-4-mini-instruct --batch-size 1
|
|
@@ -119,6 +123,8 @@ python benchmarks/benchmark_latency.py --input-len 256 --output-len 256 --model
|
|
| 119 |
|
| 120 |
We also benchmarked the throughput in a serving environment.
|
| 121 |
|
|
|
|
|
|
|
| 122 |
## baseline
|
| 123 |
Server:
|
| 124 |
```
|
|
|
|
| 95 |
|
| 96 |
# Model Performance
|
| 97 |
|
| 98 |
+
# Download vllm source code and install vllm
|
| 99 |
```
|
| 100 |
+
git clone git@github.com:vllm-project/vllm.git
|
| 101 |
+
VLLM_USE_PRECOMPILED=1 pip install .
|
| 102 |
```
|
| 103 |
|
| 104 |
# Download dataset
|
|
|
|
| 106 |
|
| 107 |
Other datasets can be found in: https://github.com/vllm-project/vllm/tree/main/benchmarks
|
| 108 |
# benchmark_latency
|
| 109 |
+
|
| 110 |
+
Run the following under `vllm` source code root folder:
|
| 111 |
+
|
| 112 |
## baseline
|
| 113 |
```
|
| 114 |
python benchmarks/benchmark_latency.py --input-len 256 --output-len 256 --model microsoft/Phi-4-mini-instruct --batch-size 1
|
|
|
|
| 123 |
|
| 124 |
We also benchmarked the throughput in a serving environment.
|
| 125 |
|
| 126 |
+
Run the following under `vllm` source code root folder:
|
| 127 |
+
|
| 128 |
## baseline
|
| 129 |
Server:
|
| 130 |
```
|