feat(speculative-sampling): allow to specify a draft model in the model config (#1052)

Browse Source

**Description**

This PR fixes #1013.

It adds `draft_model` and `n_draft` to the model YAML config in order to
load models with speculative sampling. This should be compatible as well
with grammars.

example:

```yaml
backend: llama                                                                                                                                                                   
context_size: 1024                                                                                                                                                                        
name: my-model-name
parameters:
  model: foo-bar
n_draft: 16                                                                                                                                                                      
draft_model: model-name
```

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

...

This commit is contained in:

Ettore Di Giacinto

2023-09-14 17:44:16 +02:00

committed by GitHub

parent 247d85b523

commit 8ccf5b2044

12 changed files with 485 additions and 427 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

64

extra/grpc/autogptq/backend_pb2.py

File diff suppressed because one or more lines are too long

64

extra/grpc/bark/backend_pb2.py

File diff suppressed because one or more lines are too long

64

extra/grpc/diffusers/backend_pb2.py

File diff suppressed because one or more lines are too long

64

extra/grpc/exllama/backend_pb2.py

File diff suppressed because one or more lines are too long

64

extra/grpc/huggingface/backend_pb2.py

File diff suppressed because one or more lines are too long

64

extra/grpc/vall-e-x/backend_pb2.py

File diff suppressed because one or more lines are too long

64

extra/grpc/vllm/backend_pb2.py

File diff suppressed because one or more lines are too long