Skip to main content
POST
Deploy Create Llm
When called with an API key, the key needs the models scope (about scopes).

Authorizations

Authorization
string
header
required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Headers

xi-api-key
string | null
x-api-key
string | null

Body

application/json
model_name
string
required

model name for deepinfra (username/mode-name format)

gpu
enum<string>
required

The type of GPU the deployment is running on.

Available options:
L4-24GB,
L40S-48GB,
A100-80GB,
H100-80GB,
H200-141GB,
B200-180GB,
B300-270GB,
RTXPRO6000-96GB,
other
num_gpus
integer
default:1

Number of GPUs used by one instance

Required range: 1 <= x <= 8
hf
HFWeights · object | null
base_model
string | null

Base public model

container_image
string | null

Docker image for the deployment (e.g. vllm/vllm-openai:v0.8.4)

settings
ScaleSettings · object | null
extra_args
string[] | null

Extra command line arguments for custom deployments

standard_args
StandardArgs · object | null

Engine tuning knobs. Values are validated on submission; unsupported or out-of-range values are rejected. Unset knobs use the model/engine defaults.

preset_id
string | null

Apply a stored preset or vLLM recipe by id. A non-empty standard_args or extra_args in this request replaces the preset's whole matching field.

Response

Successful Response

deploy_id
string
required

Deploy Id

Example:

"fkj843kjh8"

model_name
string
required

Model Id from huggingface

Example:

"google/vit-base-patch16-224"

version
string
required

Model version

Example:

"d8b79b422843bd59d628bf25b01aded94a9ec1a9b917e69fe460df9ff39ec42b"

task
string
required

Task

Example:

"image-classification"

status
string
required

Status

Example:

"deployed"

fail_reason
string
required

Failure reason

Example:

"Initialization failed"

created_at
string
required

Created at

Example:

"2021-08-27T17:19:21+00:00"

updated_at
string
required

Updated at

Example:

"2021-08-27T17:19:21+00:00"

type
enum<string>
default:legacy
Available options:
legacy,
llm,
lora,
tts
instances
DeployInstances · object | null

Details about number of instances running right now

config
DeployLLMConfig · object | null

Immutable deploy configuration

settings
ScaleSettings · object | null

Scale Settings

standard_args
StandardArgs · object

Current engine tuning knobs

extra_args
string[] | null

Current extra engine-specific command-line args (custom-weight deploys only)

update_error
string | null

If the last config update was auto-reverted after an engine crash-loop, the error that caused it (status stays 'running')

rollout
DeployRollout · object | null

Per-instance rollout progress of the current config