# Authentication
Source: https://docs.deepinfra.com/account/authentication
API tokens and scoped JWT for secure, scope-limited inference access.
This page covers authenticating **API requests**. For signing in to the dashboard itself, see [Signing In](/account/signing-in).
DeepInfra supports two authentication methods:
1. **API keys** — full-access tokens for your own use
2. **Scoped JWTs** — short-lived, scope-limited tokens you can issue to third parties
## API keys
Get your API keys from the [Dashboard](https://deepinfra.com/dash/api_keys). Use them in the `Authorization` header:
```bash theme={null}
Authorization: Bearer $DEEPINFRA_TOKEN
```
## Scoped JWT
Scoped JWT tokens let you grant limited inference access to third parties without sharing your API key. You can restrict the token by:
* **Models allowed** — specific model(s) only
* **Expiration** — time-limited (up to 1 year)
* **Spending limit** — maximum USD spend
Usage is counted against the API key that was used to sign the JWT.
### Create a scoped JWT
```bash theme={null}
curl -X POST "https://api.deepinfra.com/v1/scoped-jwt" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_API_KEY" \
-d '{
"api_key_name": "auto",
"models": ["deepseek-ai/DeepSeek-R1"],
"expires_delta": 3600,
"spending_limit": 1.0
}'
```
Response:
```json theme={null}
{"token": "jwt:eyJhbGciOiJIUzI1NiIsImtpZCxxxxxxxxxxxxxxxxxx"}
```
This creates a token limited to `deepseek-ai/DeepSeek-R1`, expiring in 1 hour, with a \$1.00 spending limit.
**Optional fields** (omit to remove restriction):
* `models` — allow any model
* `expires_delta` — no expiration (defaults to 1 year)
* `spending_limit` — no spending limit
* Use `expires_at` (unix timestamp) instead of `expires_delta` if preferred
### Inspect a JWT
```bash theme={null}
curl "https://api.deepinfra.com/v1/scoped-jwt?jwtoken=XXXX" \
-H "Authorization: Bearer $DEEPINFRA_API_KEY"
```
```json theme={null}
{
"expires_at": 1738843515,
"models": ["deepseek-ai/DeepSeek-R1"],
"spending_limit": 1
}
```
### Use a scoped JWT
Use it exactly like a regular API key in the `Authorization` header:
```bash theme={null}
curl "https://api.deepinfra.com/v1/openai/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $SCOPED_JWT" \
-d '{
"model": "deepseek-ai/DeepSeek-R1",
"messages": [{"role": "user", "content": "Hello!"}]
}'
```
Requests using disallowed models, expired tokens, or over-budget tokens will be rejected.
## JWT format (advanced)
You can create and inspect scoped JWTs yourself using standard JWT libraries.
### Header
```json theme={null}
{
"alg": "HS256",
"kid": "di:1000000000000:YXV0bw==",
"typ": "JWT"
}
```
The `kid` field is `{user_id}:{base64(api_key_name)}` joined with colons. Only `HS256` (HMAC-SHA256) is supported.
### Payload
```json theme={null}
{
"sub": "di:1000000000000",
"model": "deepseek-ai/DeepSeek-R1",
"exp": 1734616903
}
```
### Signature
```
HMAC_SHA256(
api_key,
base64urlEncoding(header) + '.' + base64urlEncoding(payload)
)
```
### Token format
```
jwt:{base64url(header)}.{base64url(payload)}.{base64url(signature)}
```
# Data Privacy
Source: https://docs.deepinfra.com/account/data-privacy
How DeepInfra handles your data during inference — what's stored, what's not.
This page explains how DeepInfra handles data when you use our inference APIs.
## Summary
* **Input data is not stored to disk** during inference — it exists only in memory while the request is being processed
* **Output data is not stored** — it is sent to you and then deleted from memory
* **We do not train on your data** (except Google/Anthropic model exceptions below)
* **We do not share your data with third parties** (except Google/Anthropic model exceptions below)
## Data privacy
When using DeepInfra inference APIs, we do not store the data you submit to our APIs on disk. We only hold it in memory during the inference process. Once inference is complete, the data is deleted from memory.
The same applies to outputs — once sent back to you, they are deleted.
**Exception:** Outputs of image generation models are stored for a short period of time to allow easy access (e.g., for the model demo page).
### Google models
If you opt to use a Google model, Google will store the output as outlined in their [Privacy Notice](https://cloud.google.com/terms/cloud-privacy-notice).
### Anthropic models
If you opt to use an Anthropic model, Anthropic will store the output as outlined in their [Trust Center](https://trust.anthropic.com/).
## Bulk inference APIs
When using our bulk inference APIs (submitting multiple requests in a single call), we may need to store data for a longer period, potentially on disk in encrypted form. Once inference is complete and results are returned to you, the data is deleted from disk and memory after a short retention period.
## No training
We do not use data you submit to our APIs for training models, except when using Google or Anthropic models, where the receiving company's training policy applies.
## No sharing
We do not share data you submit to our APIs with any third party, except when using Google or Anthropic models, where we are required to transfer data to those endpoints to fulfill the request.
## Logs
We generally do not log the content of your requests. We log metadata useful for debugging: request ID, cost, sampling parameters. We reserve the right to log a small portion of requests when necessary for debugging or security purposes.
When using the Google model, Google logs prompts and responses for a limited period solely to detect violations of their [Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy).
## Personal information
Personal information and data provided when using certain models through our API may be shared with the relevant model API endpoints, as specified at the time of use.
# Okta SSO
Source: https://docs.deepinfra.com/account/okta-sso
Configure Okta as a Single Sign-On provider for your DeepInfra team.
DeepInfra supports Okta-based Single Sign-On (SSO) via OpenID Connect for teams that need centralized identity management. For individual sign-in options (Google, GitHub, email), see [Signing In](/account/signing-in).
## Supported features
* SSO initiated via Okta (IdP-initiated)
* SSO initiated via DeepInfra (SP-initiated)
* Automatic account creation on first sign-in
## Configuration steps
1. **Install the DeepInfra app** in your Okta instance
2. **Fill in the configuration**:
* **Team ID** — your Okta subdomain is a good starting point. For multi-tenancy (multiple teams in one Okta instance), use `subdomain-group`. Lowercase only, starting with subdomain, dashes for separators.
* **Use Stage** — leave blank
3. **Assign users or groups** who should have access to DeepInfra
4. In the DeepInfra App (inside Okta), go to the **Sign On** tab and note the **Client ID** and **Client Secret**
5. For the **Issuer** (your Okta domain): find the link titled *OpenID Provider Metadata*, click it, and copy the `"issuer"` URL from the JSON document
6. **Email [feedback@deepinfra.com](mailto:feedback@deepinfra.com)** with:
* Team ID
* Issuer
* Client ID
* Client Secret
* Admin email (the user who will be team admin)
After setup is complete, users can sign in via:
* Okta dashboard
* DeepInfra's [SSO login page](https://deepinfra.com/login_sso) (enter your Team ID)
## SP-initiated SSO
1. Go to [deepinfra.com/login](https://deepinfra.com/login)
2. Click **Corporate SSO**
3. Enter your **Team ID** and click **SSO Login**
4. Enter your Okta credentials and click **Sign in with Okta**
5. You're redirected to the DeepInfra dashboard
## Team management notes
* Admin can change team member roles (member / admin)
* Admin has access to the billing dashboard
* All team members share the same API tokens and models
* For per-user isolation (separate tokens and models per person), contact us
# Rate Limits
Source: https://docs.deepinfra.com/account/rate-limits
Default concurrent request limits and how to increase them.
## Default limit: 200 concurrent requests per model
Every account has a default limit of **200 concurrent requests per model**. If you query two different models simultaneously, you can handle 400 total concurrent requests (200 per model).
This limit is sufficient for most production applications, including services with hundreds of thousands of daily active users.
## Understanding concurrent vs. requests per minute
The rate limit is on *concurrent* requests, not per-minute volume. Throughput depends on how long each request takes:
| Avg Request Duration | Concurrent Limit | Approx RPM |
| -------------------- | :--------------: | :--------: |
| 1 second | 200 | 12,000 RPM |
| 10 seconds | 200 | 1,200 RPM |
| 60 seconds | 200 | 200 RPM |
As requests complete, new ones can immediately take their place.
## Batch jobs
For large batch jobs (e.g., embedding a knowledge base), use a [token bucket algorithm](https://en.wikipedia.org/wiki/Token_bucket) to stay under 200 concurrent requests. You'll still complete the work in a reasonable time.
## Rate limit errors
You'll receive HTTP **429** with a `Rate limited` message when the limit is exceeded.
Actions to take:
* Retry after a short delay
* Slow down your request rate
* Apply for a limit increase
You may occasionally receive **429** errors when a model becomes very busy, even if you're under the limit. Auto-scaling will kick in shortly. Retry after a brief wait.
## Request a limit increase
You can request a rate limit increase in your [Dashboard → Account](https://deepinfra.com/dash/account). Include context about your use case.
# Signing In
Source: https://docs.deepinfra.com/account/signing-in
Sign in with Google, GitHub, email & password, or corporate SSO — and what to know about linked accounts.
You can sign in to DeepInfra at [deepinfra.com/login](https://deepinfra.com/login) with Google, GitHub, email & password, or Corporate SSO. With Google, GitHub, and SSO we create an account on your first login if you don't have one; for email & password there's a [sign-up page](https://deepinfra.com/signup).
## Signing in with Google
If you don't have a DeepInfra account yet, **Continue with Google** creates one automatically. No password is set on the account — you sign in with Google from then on.
### If you already have an account with your Google email
Signing in with Google automatically links your Google identity to the existing account — there is no confirmation prompt. What happens to the account depends on whether its email was already verified:
* **Verified** — nothing changes: the Google link is simply added, and password sign-in keeps working.
* **Not verified** — the account is treated as unclaimed: your password is removed, all active sessions are signed out, and your name and picture are replaced by your Google profile.
To keep your password and profile, verify your email address **before** your first Google sign-in.
### Changing your email or unlinking Google
Your email address can't be changed while Google is linked — the settings show "Linked from your Google account". There is currently no self-serve way to unlink Google either; contact us at [feedback@deepinfra.com](mailto:feedback@deepinfra.com) to unlink or change your email.
## Signing in with GitHub
Clicking **Continue with GitHub** creates (or signs in to) an account tied to your GitHub identity, using your primary verified GitHub email. GitHub sign-in never links to an existing Google or email account — see the [FAQ](#faq).
### Changing your email
Your account email can only be one of the emails associated with your GitHub account — the email dropdown in the dashboard settings is synced from your GitHub profile. To use a different email, add it to your GitHub account first; a few minutes later it will appear in the dropdown.
## Email and password
Create an account at [deepinfra.com/signup](https://deepinfra.com/signup). You'll receive a verification email — verify promptly: unverified accounts have limited access and can be [claimed by a Google sign-in](#if-you-already-have-an-account-with-your-google-email) with the same email.
* Signing up with an email that's already in use fails — sign in to the existing account instead.
* Forgot your password? Use the reset link on the login page.
* Accounts created via Google have no password, so password sign-in fails with "Invalid email or password" — use **Continue with Google** instead.
## Corporate SSO
Teams using Okta sign in via the **Corporate SSO** button — see [Okta SSO](/account/okta-sso) for setup and sign-in steps.
## Joining via a team invite
Team invites can be accepted with Google or email & password, and only when signing up — the new account automatically joins the inviting team. An existing account can't join a team (see the [FAQ](#faq)).
## Reactivating a deleted account
If you've requested account deletion (in the dashboard account settings) and it hasn't been permanently removed yet, any successful sign-in reactivates the account. If that wasn't intended, contact [feedback@deepinfra.com](mailto:feedback@deepinfra.com).
## FAQ
You're probably in a different account than the one you originally signed up with. Sign in again with the method you used the first time — and make sure you always use the same sign-in method.
This can happen because GitHub sign-in has its own account, separate from the Google / email & password one — even when all of them use the same email address. So if you signed up with GitHub but signed in with Google or email (or the other way around), you ended up in a second, empty account.
Team invites only work when a **new** account is created. An existing account can't join a team, and the same email can't be on both a personal and a team account (one created via an invite). Your options:
* Request deletion of your existing account (in the dashboard account settings), then email [feedback@deepinfra.com](mailto:feedback@deepinfra.com) so we remove it immediately. Once we confirm it's gone, ask the team owner to re-invite you and sign up through the new link — signing in earlier would just [reactivate the account](#reactivating-a-deleted-account).
* Sign up with a different email, e.g. `you+team@example.com` — most mail providers deliver it to the same inbox, but it counts as a separate account.
# Subprocessors
Source: https://docs.deepinfra.com/account/subprocessors
Third-party services DeepInfra uses to provide its services.
DeepInfra uses the following subprocessors to provide its services:
| Name | Nature of Processing |
| --------------------- | -------------------- |
| Stripe | Payment processor |
| Amazon Web Services | Infrastructure |
| Google Cloud Platform | Infrastructure |
*Last updated: September 6, 2024*
# Webhooks
Source: https://docs.deepinfra.com/account/webhooks
Receive inference results asynchronously via HTTP callbacks.
Webhooks are a feature of the [DeepInfra Native API](/apis/deepinfra-native). They are not supported with the OpenAI-compatible API.
Webhooks let you submit an inference request and receive the result via an HTTP callback, instead of waiting for the response synchronously. This is useful for long-running requests or fire-and-forget workloads.
## How it works
Add a `webhook` parameter to your request. The API immediately responds with status `queued`, then calls your webhook URL with the result once inference is complete.
## Text generation example
```javascript JavaScript theme={null}
import { TextGeneration } from "deepinfra";
const client = new TextGeneration(
"https://api.deepinfra.com/v1/inference/deepseek-ai/DeepSeek-V3",
"$DEEPINFRA_TOKEN"
);
const res = await client.generate({
input: "<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\nHello!<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
stop: ["<|eot_id|>"],
webhook: "https://your-app.com/deepinfra-webhook"
});
console.log(res.inference_status.status); // "queued"
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/inference/deepseek-ai/DeepSeek-V3" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"input": "<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\nHello!<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
"stop": ["<|eot_id|>"],
"webhook": "https://your-app.com/deepinfra-webhook"
}'
```
## Embeddings example
```bash theme={null}
curl "https://api.deepinfra.com/v1/inference/Qwen/Qwen3-Embedding-8B" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"inputs": ["I like chocolate"],
"webhook": "https://your-app.com/deepinfra-webhook"
}'
```
## Webhook payload
On success, your endpoint receives:
```json theme={null}
{
"request_id": "R7X9fdlIaF5GlVisBAi5xR3E",
"inference_status": {
"status": "succeeded",
"runtime_ms": 228,
"cost": 0.0001140000022132881
},
"results": { ... }
}
```
On failure:
```json theme={null}
{
"request_id": "RHNShFanUP5ExA8rzgyDWH88",
"inference_status": {
"status": "failed",
"runtime_ms": 0,
"cost": 0.0
}
}
```
## Retry behavior
DeepInfra will make a few retry attempts if your webhook endpoint returns a 4xx or 5xx status code.
# Hermes-Agent
Source: https://docs.deepinfra.com/agents/hermes-agent
Run a fully managed Hermes-Agent instance — Nous Research's autonomous agent, driven over SSH and powered by DeepInfra models.
[Hermes-Agent](https://github.com/NousResearch/hermes-agent) is a self-improving autonomous agent by Nous Research, driven from the terminal. The hosted version gives you a dedicated, always-on instance without running your own server: DeepInfra provisions the VM, installs Hermes, and pre-configures it with a dedicated DeepInfra API key.
Hermes-Agent instances are **SSH only** — the framework runs without a web dashboard, so you connect over SSH after creation.
Want to run Hermes on your own machine instead? See [Hermes Agent integration](/integrations/hermes-agent) for using DeepInfra as a custom provider in a self-hosted setup.
## Plans
Hermes-Agent instances are available on the `budget` (2 vCPU, 2 GB RAM) and `tiny` (2 vCPU, 1 GB RAM) plans. See [plans and billing](/agents/introduction#plans) for details; current per-hour pricing is shown in the **Create Instance** dialog.
## Create an instance
1. Make sure your public SSH key is registered at [Dashboard → SSH Keys](https://deepinfra.com/dash/ssh_keys) — it's your only way in
2. Go to [Dashboard → Hosted Agents](https://deepinfra.com/dash/agents) and click **Create Instance**
3. Enter a **Name**, choose **Hermes-Agent** as the instance type, and pick a plan
4. Click **Create** — setup is fully automated and takes a few minutes
## Connect and run
Once the instance is `running`, click **SSH** in the Connect column to get the exact command, or use the public IP from the details page:
```bash theme={null}
ssh hermes@ -p 2222
```
Then start the agent:
```bash theme={null}
hermes
```
That's it — the instance ships with a working configuration, so Hermes is ready to go on first login.
The instance gets a **new public IP address** on every restart. Grab the current one from the details page.
## Configuration
Hermes keeps its configuration in `~/.hermes/`:
* `~/.hermes/config.yaml` — model, provider, and agent settings. See the [Hermes configuration docs](https://hermes-agent.nousresearch.com/docs/user-guide/configuration).
* `~/.hermes/.env` — secrets. The `DEEPINFRA_API_KEY` line is managed by DeepInfra and refreshed on every instance start, so don't edit it — manual changes to that line are overwritten. The rest of the file is yours: other secrets you add (for example a third-party API key for a tool) stay untouched.
You can point Hermes at any [model from our catalog](https://deepinfra.com/models/text-generation) — for example `deepseek-ai/DeepSeek-V4-Flash` — either by editing `config.yaml` or interactively with `hermes model`. The agent's model calls are billed as regular inference usage against your account.
## FAQ
**Why is there no dashboard link for my instance?**
By design — Hermes-Agent is a terminal-driven framework with no web UI. SSH in and run `hermes`.
**Does my work survive stop/start?**
Yes. The whole filesystem — Hermes state, your files, installed packages — is captured in a snapshot on stop and restored on start. Only the public IP changes.
**Can I install extra tools on the instance?**
Yes, it's your VM — install packages and customize freely over SSH. Automatic [backups](/agents/introduction#backups) cover your changes.
# Introduction to Hosted Agents
Source: https://docs.deepinfra.com/agents/introduction
Fully managed instances running open-source agent frameworks — created in minutes, billed by the hour, pre-wired to DeepInfra inference.
Hosted Agents are fully managed virtual machines that run an open-source agent framework for you. You pick a framework and a plan, and DeepInfra provisions the instance, installs and configures the agent, and wires it to a dedicated DeepInfra API key — no servers to set up, no config files to bootstrap. Your instance is up and running in a few minutes, and its state persists across stops and restarts.
Manage your instances at [Dashboard → Hosted Agents](https://deepinfra.com/dash/agents).
## Available frameworks
| Framework | Access | Description |
| ------------------------------------ | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| [OpenClaw](/agents/openclaw) | Web dashboard + SSH | Your personal AI assistant, reachable on the messaging channels you already use. [Open source](https://github.com/openclaw/openclaw). |
| [Hermes-Agent](/agents/hermes-agent) | SSH only | Self-improving autonomous agent by Nous Research, driven from the terminal. [Open source](https://github.com/NousResearch/hermes-agent). |
## Plans
| Plan | vCPU | RAM | Included monthly transfer | Available for |
| ---------- | ---- | ---- | ------------------------- | ---------------------- |
| `standard` | 2 | 4 GB | 4 TB | OpenClaw |
| `budget` | 2 | 2 GB | 3 TB | OpenClaw, Hermes-Agent |
| `tiny` | 2 | 1 GB | 2 TB | Hermes-Agent |
Current per-hour pricing for each plan is shown in the **Create Instance** dialog and on each instance's details page.
## Billing
Two separate line items make up the cost of a hosted agent:
* **Compute** — a per-hour rate for the instance itself, billed only while the instance is active (creating, starting, running, or stopping). A stopped instance costs nothing.
* **Inference** — whatever the agent spends on model calls. Each instance gets its own DeepInfra API key, and its LLM usage is billed at standard token prices against your account.
Both appear on your [Usage](https://deepinfra.com/dash/usage) page; compute shows up as **Hosted Agent**.
Fair use: instances that exceed an average of their plan's included network transfer per month (inbound + outbound, prorated by uptime) may be temporarily paused. You can start the instance again from the dashboard once usage normalizes.
## Create an instance
Register your public SSH key at [Dashboard → SSH Keys](https://deepinfra.com/dash/ssh_keys). It's installed on every instance you create, and you'll need it to connect over SSH.
Go to [Dashboard → Hosted Agents](https://deepinfra.com/dash/agents) and click **Create Instance**.
Pick a **Name** (up to 64 characters), an **Instance Type** (the agent framework), and a **Plan**, then click **Create**.
Setup is fully automated and takes a few minutes. Once the instance is `running`, connect via the **dashboard** link (OpenClaw) or **SSH** (any framework).
## Instance lifecycle
| State | Description |
| ---------- | --------------------------------------------------------------------------------- |
| `creating` | The instance is being provisioned |
| `starting` | The instance is booting up |
| `stopping` | The instance is shutting down; a snapshot of its state is captured |
| `running` | The instance is active and accessible |
| `stopped` | The instance is shut down — not billed for compute |
| `failed` | The instance hit an unrecoverable error (see the fail reason on its details page) |
Stopping an instance shuts it down so you're not billed for compute while it's idle. Starting it again restores everything as it was.
The instance gets a **new public IP address** on every restart. Check the details page for the current one.
## Manage an instance
From the instance list or its details page you can:
* **Start / Stop** — pause compute billing without losing state.
* **Update** — when a newer framework version is available, upgrade in place. No restart needed; the agent may be unresponsive for a few minutes.
* **Backup** — trigger a snapshot on demand (see below).
* **Delete** — stop and permanently remove the instance. This cannot be undone.
DeepInfra retains administrative SSH access to your instance to automate maintenance tasks (update and restore from backups for example).
## Backups
Backups are created automatically every 24 hours, and one is also captured whenever you stop the instance. Snapshots older than 5 days are cleaned up, with at least 2 always retained. You can create one manually anytime with **Create Backup** on the details page.
To roll back, pick a backup in the **Backups** table and click **Restore**.
Restoring overwrites the instance's current data with the backup. This cannot be undone — take a fresh backup first if you might need the current state.
## HTTP API
Everything above is also available programmatically. Authenticate with your [API key](/account/authentication).
Create an instance:
```bash theme={null}
curl -X POST https://api.deepinfra.com/v1/agents \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "my-agent",
"agent_type_id": "openclaw",
"plan_id": "standard"
}'
```
List your instances:
```bash theme={null}
curl "https://api.deepinfra.com/v1/agents?state=all" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN"
```
Start, stop, or update:
```bash theme={null}
curl -X POST https://api.deepinfra.com/v1/agents/{instance_id}/start \
-H "Authorization: Bearer $DEEPINFRA_TOKEN"
curl -X POST https://api.deepinfra.com/v1/agents/{instance_id}/stop \
-H "Authorization: Bearer $DEEPINFRA_TOKEN"
curl -X POST https://api.deepinfra.com/v1/agents/{instance_id}/update \
-H "Authorization: Bearer $DEEPINFRA_TOKEN"
```
Delete:
```bash theme={null}
curl -X DELETE https://api.deepinfra.com/v1/agents/{instance_id} \
-H "Authorization: Bearer $DEEPINFRA_TOKEN"
```
Other endpoints: `GET /v1/agents/catalog` (frameworks, plans, and current pricing), `GET /v1/agents/{instance_id}/backups`, `POST /v1/agents/{instance_id}/backup`, and `POST /v1/agents/{instance_id}/backups/{backup_id}/restore`. See the [API Reference](https://docs.deepinfra.com/api-reference) for full schemas.
## FAQ
**How many instances can I run?**
Up to 5 active instances per account.
**Where does my instance run?**
Instances run in `us-west-2` by default. Accounts subject to EU data-protection rules are placed in `eu-central-1`. The region is shown on the instance details page.
**What happens to my data when I stop an instance?**
Nothing is lost — state is captured in a snapshot and restored when you start it again. Only the public IP changes.
**Can DeepInfra access my instance?**
DeepInfra retains administrative SSH access for automated maintenance (updates and backup restores for example).
**Why was my instance paused?**
Most likely it exceeded the fair-use network transfer threshold for its plan. Start it again from the dashboard once usage normalizes.
**What happens if my account is suspended?**
If your account is suspended — for example when your balance runs out — your instances are automatically stopped within a minute or two, and creating, starting, or restoring instances is blocked. Nothing is deleted: state is preserved the same way as a regular stop, and once your account is back in good standing you can start your instances again. Keep in mind that a running agent spends on inference on its own, so consider enabling [Automatic Top-Up](https://deepinfra.com/dash/billing) if you plan to leave it unattended.
**Where do I see what my agent is costing me?**
On the [Usage](https://deepinfra.com/dash/usage) page — compute appears as **Hosted Agent**, and the agent's model calls appear as regular inference usage.
# OpenClaw
Source: https://docs.deepinfra.com/agents/openclaw
Run a fully managed OpenClaw instance — your personal AI assistant with a web dashboard, powered by DeepInfra models.
[OpenClaw](https://openclaw.ai) is the wildly popular open-source personal AI assistant ([github.com/openclaw/openclaw](https://github.com/openclaw/openclaw)) that talks to you on the channels you already use — WhatsApp, Telegram, Slack, Discord, and many more — and acts autonomously on your behalf.
The hosted version gives you a dedicated, always-on OpenClaw instance without running your own server: DeepInfra provisions the VM, installs OpenClaw, wires it to a dedicated DeepInfra API key, and keeps it backed up. You get the full OpenClaw web dashboard plus SSH access.
## Plans
OpenClaw instances are available on the `standard` (2 vCPU, 4 GB RAM) and `budget` (2 vCPU, 2 GB RAM) plans. See [plans and billing](/agents/introduction#plans) for details; current per-hour pricing is shown in the **Create Instance** dialog.
## Create an instance
1. Go to [Dashboard → Hosted Agents](https://deepinfra.com/dash/agents) and click **Create Instance**
2. Enter a **Name**, choose **OpenClaw** as the instance type, and pick a plan
3. Click **Create** — setup is fully automated and takes a few minutes
## Open the dashboard
Once the instance is `running`, click **dashboard** in the Connect column (or **Open Dashboard** in the actions menu). The OpenClaw dashboard opens in a new tab, served securely through DeepInfra — no ports to expose or tunnels to set up.
Right after a start, the instance can take up to \~3 minutes to finish booting. If you see a loading page, it refreshes automatically once the dashboard is up.
From the dashboard you can chat with your agent, connect messaging channels, and manage its configuration — everything a self-hosted OpenClaw offers. See the [OpenClaw docs](https://docs.openclaw.ai) for what the assistant itself can do.
## Models
Your instance comes pre-configured with a DeepInfra model, so it works out of the box. You can switch to any model from [our catalog](https://deepinfra.com/models/text-generation) in OpenClaw's settings — for example `deepseek-ai/DeepSeek-V4-Flash`. The agent's model calls are billed as regular inference usage against your account.
## SSH access
Power users can connect directly:
```bash theme={null}
ssh openclaw@ -p 2222
```
The public IP and SSH port are shown on the instance details page. Your account's [SSH keys](https://deepinfra.com/dash/ssh_keys) are installed automatically.
The file `/home/openclaw/config.env` is managed by DeepInfra — manual edits will be overwritten by maintenance. OpenClaw's self-update is disabled on hosted instances; upgrade through the dashboard's **Update** action instead.
## Updating and backups
When a newer OpenClaw version is available, an **Update** action appears — it upgrades in place, with no restart. Backups run automatically every 24 hours; see [Backups](/agents/introduction#backups).
## FAQ
**Why is the dashboard not loading right after I start the instance?**
The VM boots in stages; the dashboard is the last thing to come up (up to \~3 minutes). The loading page retries automatically.
**My agent stopped responding on its channels — what happened?**
Check the instance state at [Dashboard → Hosted Agents](https://deepinfra.com/dash/agents). If it's `stopped` or paused for fair use, start it again. Note that the instance gets a new public IP on restart — dashboard and channel connections are unaffected, but any SSH shortcuts you saved need the new IP.
**Can I customize the instance over SSH?**
Yes — install packages, add skills, edit OpenClaw's own configuration. Just leave `/home/openclaw/config.env` alone and don't re-enable self-update; version upgrades go through the **Update** action.
# Account Email Values
Source: https://docs.deepinfra.com/api-reference/account/account-email-values
https://api.deepinfra.com/openapi.json get /v1/me/emails
# Account Gpu Pool
Source: https://docs.deepinfra.com/api-reference/account/account-gpu-pool
https://api.deepinfra.com/openapi.json get /v1/me/gpu_pool
# Account Rate Limit
Source: https://docs.deepinfra.com/api-reference/account/account-rate-limit
https://api.deepinfra.com/openapi.json get /v1/me/rate_limit
# Account Update Details
Source: https://docs.deepinfra.com/api-reference/account/account-update-details
https://api.deepinfra.com/openapi.json patch /v1/me
# Delete Account
Source: https://docs.deepinfra.com/api-reference/account/delete-account
https://api.deepinfra.com/openapi.json delete /v1/me
# Gpu Pool Gpu Types
Source: https://docs.deepinfra.com/api-reference/account/gpu-pool-gpu-types
https://api.deepinfra.com/openapi.json get /v1/gpu_pool/gpu_types
# Me
Source: https://docs.deepinfra.com/api-reference/account/me
https://api.deepinfra.com/openapi.json get /v1/me
# Request Gpu Pool Change
Source: https://docs.deepinfra.com/api-reference/account/request-gpu-pool-change
https://api.deepinfra.com/openapi.json post /v1/me/gpu_pool/request
# Request Rate Limit Increase
Source: https://docs.deepinfra.com/api-reference/account/request-rate-limit-increase
https://api.deepinfra.com/openapi.json post /v1/me/rate_limit/request
# Team Set Display Name
Source: https://docs.deepinfra.com/api-reference/account/team-set-display-name
https://api.deepinfra.com/openapi.json post /v1/me/team_display_name
# Create Backup
Source: https://docs.deepinfra.com/api-reference/agents/create-backup
https://api.deepinfra.com/openapi.json post /v1/agents/{instance_id}/backup
# Create Dashboard Launch Token
Source: https://docs.deepinfra.com/api-reference/agents/create-dashboard-launch-token
https://api.deepinfra.com/openapi.json post /v1/agents/{instance_id}/launch_token
Mint a single-use launch URL for the dashboard.
Called by the launcher page right when readyz flips ready. The launch URL
is used as a top-level navigation; /launch then sets the oc_auth cookie
and 302s into the proxied dashboard.
The user's bearer token is stashed in Redis under the token's jti and
retrieved (atomic GETDEL) on /launch redeem — this keeps the bearer out of
the URL and out of any signed payload while preserving the existing proxy
auth flow (oc_auth cookie value = bearer token).
Refuses instances whose agent_type has has_dashboard=False (e.g. hermes).
# Create Instance
Source: https://docs.deepinfra.com/api-reference/agents/create-instance
https://api.deepinfra.com/openapi.json post /v1/agents
# Delete Instance
Source: https://docs.deepinfra.com/api-reference/agents/delete-instance
https://api.deepinfra.com/openapi.json delete /v1/agents/{instance_id}
# Get Catalog
Source: https://docs.deepinfra.com/api-reference/agents/get-catalog
https://api.deepinfra.com/openapi.json get /v1/agents/catalog
# Get Instance
Source: https://docs.deepinfra.com/api-reference/agents/get-instance
https://api.deepinfra.com/openapi.json get /v1/agents/{instance_id}
# List Backups
Source: https://docs.deepinfra.com/api-reference/agents/list-backups
https://api.deepinfra.com/openapi.json get /v1/agents/{instance_id}/backups
# List Instances
Source: https://docs.deepinfra.com/api-reference/agents/list-instances
https://api.deepinfra.com/openapi.json get /v1/agents
# Rename Instance
Source: https://docs.deepinfra.com/api-reference/agents/rename-instance
https://api.deepinfra.com/openapi.json patch /v1/agents/{instance_id}
# Restore Backup
Source: https://docs.deepinfra.com/api-reference/agents/restore-backup
https://api.deepinfra.com/openapi.json post /v1/agents/{instance_id}/backups/{backup_id}/restore
# Start Instance
Source: https://docs.deepinfra.com/api-reference/agents/start-instance
https://api.deepinfra.com/openapi.json post /v1/agents/{instance_id}/start
# Stop Instance
Source: https://docs.deepinfra.com/api-reference/agents/stop-instance
https://api.deepinfra.com/openapi.json post /v1/agents/{instance_id}/stop
# Update Instance Version
Source: https://docs.deepinfra.com/api-reference/agents/update-instance-version
https://api.deepinfra.com/openapi.json post /v1/agents/{instance_id}/update
# Openai Audio Speech
Source: https://docs.deepinfra.com/api-reference/audio/openai-audio-speech
https://api.deepinfra.com/openapi.json post /v1/audio/speech
# Openai Audio Transcriptions
Source: https://docs.deepinfra.com/api-reference/audio/openai-audio-transcriptions
https://api.deepinfra.com/openapi.json post /v1/audio/transcriptions
# Openai Audio Translations
Source: https://docs.deepinfra.com/api-reference/audio/openai-audio-translations
https://api.deepinfra.com/openapi.json post /v1/audio/translations
# Create Api Token
Source: https://docs.deepinfra.com/api-reference/authentication/create-api-token
https://api.deepinfra.com/openapi.json post /v1/api-tokens
# Create Scoped Jwt
Source: https://docs.deepinfra.com/api-reference/authentication/create-scoped-jwt
https://api.deepinfra.com/openapi.json post /v1/scoped-jwt
# Create Ssh Key
Source: https://docs.deepinfra.com/api-reference/authentication/create-ssh-key
https://api.deepinfra.com/openapi.json post /v1/ssh_keys
# Delete Api Token
Source: https://docs.deepinfra.com/api-reference/authentication/delete-api-token
https://api.deepinfra.com/openapi.json delete /v1/api-tokens/{api_token}
# Delete Ssh Key
Source: https://docs.deepinfra.com/api-reference/authentication/delete-ssh-key
https://api.deepinfra.com/openapi.json delete /v1/ssh_keys/{ssh_key_id}
# Export Api Token To Vercel
Source: https://docs.deepinfra.com/api-reference/authentication/export-api-token-to-vercel
https://api.deepinfra.com/openapi.json post /v1/api-tokens/{api_token}/vercel_export
# Get Api Token
Source: https://docs.deepinfra.com/api-reference/authentication/get-api-token
https://api.deepinfra.com/openapi.json get /v1/api-tokens/{api_token}
# Get Api Tokens
Source: https://docs.deepinfra.com/api-reference/authentication/get-api-tokens
https://api.deepinfra.com/openapi.json get /v1/api-tokens
# Get Ssh Keys
Source: https://docs.deepinfra.com/api-reference/authentication/get-ssh-keys
https://api.deepinfra.com/openapi.json get /v1/ssh_keys
# Github Cli Login
Source: https://docs.deepinfra.com/api-reference/authentication/github-cli-login
https://api.deepinfra.com/openapi.json get /github/cli/login
deepctl is calling this request waiting for auth token during login.
The token is stored in /github/callback
# Github Login
Source: https://docs.deepinfra.com/api-reference/authentication/github-login
https://api.deepinfra.com/openapi.json get /github/login
Initiate github SSO login flow. Callback is /github/callback
# Google Login
Source: https://docs.deepinfra.com/api-reference/authentication/google-login
https://api.deepinfra.com/openapi.json get /google/login
Initiate Google SSO login flow. Callback is /google/callback
# Inspect Scoped Jwt
Source: https://docs.deepinfra.com/api-reference/authentication/inspect-scoped-jwt
https://api.deepinfra.com/openapi.json get /v1/scoped-jwt
# Okta Login
Source: https://docs.deepinfra.com/api-reference/authentication/okta-login
https://api.deepinfra.com/openapi.json get /okta/login
# Add Funds
Source: https://docs.deepinfra.com/api-reference/billing/add-funds
https://api.deepinfra.com/openapi.json post /payment/funds
# Billing Portal
Source: https://docs.deepinfra.com/api-reference/billing/billing-portal
https://api.deepinfra.com/openapi.json get /payment/billing-portal
# Deepstart Apply
Source: https://docs.deepinfra.com/api-reference/billing/deepstart-apply
https://api.deepinfra.com/openapi.json post /payment/deepstart/application
# Get Checklist
Source: https://docs.deepinfra.com/api-reference/billing/get-checklist
https://api.deepinfra.com/openapi.json get /payment/checklist
# Get Config
Source: https://docs.deepinfra.com/api-reference/billing/get-config
https://api.deepinfra.com/openapi.json get /payment/config
# List Invoices
Source: https://docs.deepinfra.com/api-reference/billing/list-invoices
https://api.deepinfra.com/openapi.json get /payment/invoices
# Set Config
Source: https://docs.deepinfra.com/api-reference/billing/set-config
https://api.deepinfra.com/openapi.json post /payment/config
# Setup Topup
Source: https://docs.deepinfra.com/api-reference/billing/setup-topup
https://api.deepinfra.com/openapi.json post /payment/topup
# Usage
Source: https://docs.deepinfra.com/api-reference/billing/usage
https://api.deepinfra.com/openapi.json get /payment/usage
# Usage Api Token
Source: https://docs.deepinfra.com/api-reference/billing/usage-api-token
https://api.deepinfra.com/openapi.json get /payment/usage/{api_token}
# Usage Rent
Source: https://docs.deepinfra.com/api-reference/billing/usage-rent
https://api.deepinfra.com/openapi.json get /payment/usage/rent
# Usage Tokens
Source: https://docs.deepinfra.com/api-reference/billing/usage-tokens
https://api.deepinfra.com/openapi.json get /payment/usage/tokens
# Anthropic Messages
Source: https://docs.deepinfra.com/api-reference/chat-completions/anthropic-messages
https://api.deepinfra.com/openapi.json post /anthropic/v1/messages
# Anthropic Messages Count Tokens
Source: https://docs.deepinfra.com/api-reference/chat-completions/anthropic-messages-count-tokens
https://api.deepinfra.com/openapi.json post /anthropic/v1/messages/count_tokens
# Openai Chat Completions
Source: https://docs.deepinfra.com/api-reference/chat-completions/openai-chat-completions
https://api.deepinfra.com/openapi.json post /v1/chat/completions
# Deploy Args History
Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-args-history
https://api.deepinfra.com/openapi.json get /deploy/{deploy_id}/config/history
# Deploy Args Restore
Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-args-restore
https://api.deepinfra.com/openapi.json post /deploy/{deploy_id}/config/history/{entry_id}/restore
# Deploy Create
Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-create
https://api.deepinfra.com/openapi.json post /v1/deploy
# Deploy Create Hf
Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-create-hf
https://api.deepinfra.com/openapi.json post /deploy/hf/
# Deploy Create Llm
Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-create-llm
https://api.deepinfra.com/openapi.json post /deploy/llm
# Deploy Delete
Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-delete
https://api.deepinfra.com/openapi.json delete /deploy/{deploy_id}
# Deploy Detailed Stats
Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-detailed-stats
https://api.deepinfra.com/openapi.json get /deploy/{deploy_id}/stats2
# Deploy Gpu Availability
Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-gpu-availability
https://api.deepinfra.com/openapi.json get /deploy/llm/gpu_availability
# Deploy List
Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-list
https://api.deepinfra.com/openapi.json get /deploy/list/
# Deploy List
Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-list-1
https://api.deepinfra.com/openapi.json get /deploy/list
# Deploy Llm Presets
Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-llm-presets
https://api.deepinfra.com/openapi.json get /deploy/llm/presets
DeepInfra presets and mirrored vLLM recipes for ``hf_repo_id``, told apart by
``source``; empty when none. Filter by ``gpu``/``engine``/``source``.
# Deploy Llm Standard Args
Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-llm-standard-args
https://api.deepinfra.com/openapi.json get /deploy/llm/standard_args
# Deploy Llm Suggest Name
Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-llm-suggest-name
https://api.deepinfra.com/openapi.json get /deploy/llm/suggest_name
# Deploy Rebalance
Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-rebalance
https://api.deepinfra.com/openapi.json post /deploy/{deploy_id}/rebalance
Start a GPU pool rebalance: move GPUs from this deployment onto another
deployment you own, one instance at a time and without downtime. Moving all
instances stops this deployment; start it again later to resume it.
# Deploy Rebalance Cancel
Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-rebalance-cancel
https://api.deepinfra.com/openapi.json post /deploy/{deploy_id}/rebalance/cancel
Stop an in-flight GPU pool rebalance; instances already moved stay, and
both deployments keep min/max instances fixed at their current counts.
# Deploy Rebalance Status
Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-rebalance-status
https://api.deepinfra.com/openapi.json get /deploy/{deploy_id}/rebalance
Status of GPU pool rebalances touching this deployment. A just-started
rebalance can take a moment to appear. A finished or cancelled rebalance
leaves both deployments' min/max instances fixed at the final counts; edit
them to resume autoscaling.
# Deploy Start
Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-start
https://api.deepinfra.com/openapi.json post /deploy/{deploy_id}/start
Start a stopped deployment. Re-creates pods via auto-scaling.
# Deploy Stats
Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-stats
https://api.deepinfra.com/openapi.json get /deploy/{deploy_id}/stats
# Deploy Status
Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-status
https://api.deepinfra.com/openapi.json get /deploy/{deploy_id}
# Deploy Stop
Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-stop
https://api.deepinfra.com/openapi.json post /deploy/{deploy_id}/stop
Stop a running deployment. Terminates pods. Can be restarted later.
# Deploy Update
Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-update
https://api.deepinfra.com/openapi.json put /deploy/{deploy_id}
# Deployment Stats
Source: https://docs.deepinfra.com/api-reference/dedicated-models/deployment-stats
https://api.deepinfra.com/openapi.json get /deploy/stats
# Openai Embeddings
Source: https://docs.deepinfra.com/api-reference/embeddings/openai-embeddings
https://api.deepinfra.com/openapi.json post /v1/embeddings
# Cancel Openai Batch
Source: https://docs.deepinfra.com/api-reference/files-&-batches/cancel-openai-batch
https://api.deepinfra.com/openapi.json post /v1/batches/{batch_id}/cancel
# Create Openai Batch
Source: https://docs.deepinfra.com/api-reference/files-&-batches/create-openai-batch
https://api.deepinfra.com/openapi.json post /v1/batches
# Delete File
Source: https://docs.deepinfra.com/api-reference/files-&-batches/delete-file
https://api.deepinfra.com/openapi.json delete /v1/files/{file_id}
# Get File
Source: https://docs.deepinfra.com/api-reference/files-&-batches/get-file
https://api.deepinfra.com/openapi.json get /v1/files/{file_id}
# Get File Content
Source: https://docs.deepinfra.com/api-reference/files-&-batches/get-file-content
https://api.deepinfra.com/openapi.json get /v1/files/{file_id}/content
# List Files
Source: https://docs.deepinfra.com/api-reference/files-&-batches/list-files
https://api.deepinfra.com/openapi.json get /v1/files
# Openai Files
Source: https://docs.deepinfra.com/api-reference/files-&-batches/openai-files
https://api.deepinfra.com/openapi.json post /v1/files
# Retrieve Openai Batch
Source: https://docs.deepinfra.com/api-reference/files-&-batches/retrieve-openai-batch
https://api.deepinfra.com/openapi.json get /v1/batches/{batch_id}
# Retrieve Openai Batches
Source: https://docs.deepinfra.com/api-reference/files-&-batches/retrieve-openai-batches
https://api.deepinfra.com/openapi.json get /v1/batches
# Container Rentals Delete
Source: https://docs.deepinfra.com/api-reference/gpu-rentals/container-rentals-delete
https://api.deepinfra.com/openapi.json delete /v1/containers/{container_id}
# Container Rentals Get
Source: https://docs.deepinfra.com/api-reference/gpu-rentals/container-rentals-get
https://api.deepinfra.com/openapi.json get /v1/containers/{container_id}
# Container Rentals Get Params
Source: https://docs.deepinfra.com/api-reference/gpu-rentals/container-rentals-get-params
https://api.deepinfra.com/openapi.json get /v1/containers/params
# Container Rentals List
Source: https://docs.deepinfra.com/api-reference/gpu-rentals/container-rentals-list
https://api.deepinfra.com/openapi.json get /v1/containers
# Container Rentals Start
Source: https://docs.deepinfra.com/api-reference/gpu-rentals/container-rentals-start
https://api.deepinfra.com/openapi.json post /v1/containers
# Container Rentals Update
Source: https://docs.deepinfra.com/api-reference/gpu-rentals/container-rentals-update
https://api.deepinfra.com/openapi.json patch /v1/containers/{container_id}
# Rent Gpu Availability
Source: https://docs.deepinfra.com/api-reference/gpu-rentals/rent-gpu-availability
https://api.deepinfra.com/openapi.json get /v1/containers/gpu_availability
# Openai Images Edits
Source: https://docs.deepinfra.com/api-reference/image-generation/openai-images-edits
https://api.deepinfra.com/openapi.json post /v1/images/edits
Edit image using OpenAI Images Edits API
# Openai Images Generations
Source: https://docs.deepinfra.com/api-reference/image-generation/openai-images-generations
https://api.deepinfra.com/openapi.json post /v1/images/generations
Generate image using OpenAI Images API
# Openai Images Variations
Source: https://docs.deepinfra.com/api-reference/image-generation/openai-images-variations
https://api.deepinfra.com/openapi.json post /v1/images/variations
Generate a similar image using OpenAI Images Variations API
# Inference Deploy
Source: https://docs.deepinfra.com/api-reference/inference/inference-deploy
https://api.deepinfra.com/openapi.json post /v1/inference/deploy/{deploy_id}
# Inference Model
Source: https://docs.deepinfra.com/api-reference/inference/inference-model
https://api.deepinfra.com/openapi.json post /v1/inference/{model_name}
# Deployment Logs Query
Source: https://docs.deepinfra.com/api-reference/logs-&-metrics/deployment-logs-query
https://api.deepinfra.com/openapi.json get /v1/deployment_logs/query
Query deployment logs.
* Without timestamps (from/to) returns last `limit` messages (in last month).
* With `from` only, returns first `limit` messages after `from` (inclusive).
* With `to` only, returns last `limit` messages before `to` (inclusive).
* With both `from` and `to`, return the first `limit` messages after `from`, but not later than `to`.
* `from` and `to` should be no more than a month apart.
# Get Live Metrics
Source: https://docs.deepinfra.com/api-reference/logs-&-metrics/get-live-metrics
https://api.deepinfra.com/openapi.json get /v1/metrics/live
Get the latest values for the Live metrics section on the web front page.
# Get Request Costs
Source: https://docs.deepinfra.com/api-reference/logs-&-metrics/get-request-costs
https://api.deepinfra.com/openapi.json post /v1/request-costs
# Logs Query
Source: https://docs.deepinfra.com/api-reference/logs-&-metrics/logs-query
https://api.deepinfra.com/openapi.json get /v1/logs/query
Query inference logs.
* Without timestamps (from/to) returns last `limit` messages (in last month).
* With `from` only, returns first `limit` messages after `from` (inclusive).
* With `to` only, returns last `limit` messages before `to` (inclusive).
* With both `from` and `to`, return the first `limit` messages after `from`, but not later than `to`.
* `from` and `to` should be no more than a month apart.
# Create Lora
Source: https://docs.deepinfra.com/api-reference/lora-adapters/create-lora
https://api.deepinfra.com/openapi.json post /v1/lora/create
# Delete Lora
Source: https://docs.deepinfra.com/api-reference/lora-adapters/delete-lora
https://api.deepinfra.com/openapi.json delete /v1/lora/{lora_name}
# Delete Lora Model
Source: https://docs.deepinfra.com/api-reference/lora-adapters/delete-lora-model
https://api.deepinfra.com/openapi.json delete /lora-model/{lora_model_name}
# Get Lora
Source: https://docs.deepinfra.com/api-reference/lora-adapters/get-lora
https://api.deepinfra.com/openapi.json get /v1/lora/{lora_name}
# Get Lora Status
Source: https://docs.deepinfra.com/api-reference/lora-adapters/get-lora-status
https://api.deepinfra.com/openapi.json get /v1/lora/{lora_name}/status
# Get Model Loras
Source: https://docs.deepinfra.com/api-reference/lora-adapters/get-model-loras
https://api.deepinfra.com/openapi.json get /v1/model/{model_name}/loras
# Get User Loras
Source: https://docs.deepinfra.com/api-reference/lora-adapters/get-user-loras
https://api.deepinfra.com/openapi.json get /v1/user/loras
# Update Lora
Source: https://docs.deepinfra.com/api-reference/lora-adapters/update-lora
https://api.deepinfra.com/openapi.json patch /v1/lora/{lora_name}
# Upload Lora Model
Source: https://docs.deepinfra.com/api-reference/lora-adapters/upload-lora-model
https://api.deepinfra.com/openapi.json post /lora-model
# Get Hardware
Source: https://docs.deepinfra.com/api-reference/models/get-hardware
https://api.deepinfra.com/openapi.json get /v2/hardware
# Model Delete
Source: https://docs.deepinfra.com/api-reference/models/model-delete
https://api.deepinfra.com/openapi.json delete /models/{model_name}
# Model Families Names
Source: https://docs.deepinfra.com/api-reference/models/model-families-names
https://api.deepinfra.com/openapi.json get /model-families/names
# Model Family
Source: https://docs.deepinfra.com/api-reference/models/model-family
https://api.deepinfra.com/openapi.json get /model-families/{family_name}
# Model Meta Update
Source: https://docs.deepinfra.com/api-reference/models/model-meta-update
https://api.deepinfra.com/openapi.json post /models/{model_name}/meta
# Model Publicity
Source: https://docs.deepinfra.com/api-reference/models/model-publicity
https://api.deepinfra.com/openapi.json post /models/{model_name}/publicity
# Model Schema
Source: https://docs.deepinfra.com/api-reference/models/model-schema
https://api.deepinfra.com/openapi.json get /models/{model_name}/schema/{variantKey}
# Model Versions
Source: https://docs.deepinfra.com/api-reference/models/model-versions
https://api.deepinfra.com/openapi.json get /models/{model_name}/versions
# Models Deployment List
Source: https://docs.deepinfra.com/api-reference/models/models-deployment-list
https://api.deepinfra.com/openapi.json get /models/deployment/list
# Models Featured
Source: https://docs.deepinfra.com/api-reference/models/models-featured
https://api.deepinfra.com/openapi.json get /models/featured
# Models Info
Source: https://docs.deepinfra.com/api-reference/models/models-info
https://api.deepinfra.com/openapi.json get /models/{model_name}
# Models List
Source: https://docs.deepinfra.com/api-reference/models/models-list
https://api.deepinfra.com/openapi.json get /models/list
# Models Lora List
Source: https://docs.deepinfra.com/api-reference/models/models-lora-list
https://api.deepinfra.com/openapi.json get /models/lora/list
# Openai Models
Source: https://docs.deepinfra.com/api-reference/models/openai-models
https://api.deepinfra.com/openapi.json get /v1/models
# Openrouter Models
Source: https://docs.deepinfra.com/api-reference/models/openrouter-models
https://api.deepinfra.com/openapi.json get /openrouter/models
# Private Models List
Source: https://docs.deepinfra.com/api-reference/models/private-models-list
https://api.deepinfra.com/openapi.json get /models/private/list
# Create Sandbox
Source: https://docs.deepinfra.com/api-reference/sandboxes/create-sandbox
https://api.deepinfra.com/openapi.json post /v1/sandboxes
Create a new sandbox instance with the given plan, image, and settings. The sandbox starts in CREATING state and transitions to RUNNING asynchronously.
# Delete Sandbox
Source: https://docs.deepinfra.com/api-reference/sandboxes/delete-sandbox
https://api.deepinfra.com/openapi.json delete /v1/sandboxes/{sandbox_id}
# Exec Command
Source: https://docs.deepinfra.com/api-reference/sandboxes/exec-command
https://api.deepinfra.com/openapi.json post /v1/sandboxes/{sandbox_id}/exec
Run a command in the sandbox. Streams NDJSON lines (application/x-ndjson): {"stdout": ...}/{"stderr": ...} chunks followed by exactly one terminal {"returncode": N} or {"error": msg} line.
# Get Sandbox
Source: https://docs.deepinfra.com/api-reference/sandboxes/get-sandbox
https://api.deepinfra.com/openapi.json get /v1/sandboxes/{sandbox_id}
# List Sandbox Plans
Source: https://docs.deepinfra.com/api-reference/sandboxes/list-sandbox-plans
https://api.deepinfra.com/openapi.json get /v1/sandboxes/catalog
Returns all available sandbox plans with their resource specs and pricing.
# List Sandboxes
Source: https://docs.deepinfra.com/api-reference/sandboxes/list-sandboxes
https://api.deepinfra.com/openapi.json get /v1/sandboxes
# Read File
Source: https://docs.deepinfra.com/api-reference/sandboxes/read-file
https://api.deepinfra.com/openapi.json get /v1/sandboxes/{sandbox_id}/fs/content
Read a file from an absolute path inside the sandbox; returns raw bytes (application/octet-stream).
# Start Sandbox
Source: https://docs.deepinfra.com/api-reference/sandboxes/start-sandbox
https://api.deepinfra.com/openapi.json post /v1/sandboxes/{sandbox_id}/start
# Stop Sandbox
Source: https://docs.deepinfra.com/api-reference/sandboxes/stop-sandbox
https://api.deepinfra.com/openapi.json post /v1/sandboxes/{sandbox_id}/stop
# Write File
Source: https://docs.deepinfra.com/api-reference/sandboxes/write-file
https://api.deepinfra.com/openapi.json put /v1/sandboxes/{sandbox_id}/fs/content
Write the raw request body (application/octet-stream, max 100 MiB) to an absolute path inside the sandbox.
# Openai Completions
Source: https://docs.deepinfra.com/api-reference/text-completions/openai-completions
https://api.deepinfra.com/openapi.json post /v1/completions
# Create Voice
Source: https://docs.deepinfra.com/api-reference/text-to-speech/create-voice
https://api.deepinfra.com/openapi.json post /v1/voices/add
Create a new voice
# Delete Voice
Source: https://docs.deepinfra.com/api-reference/text-to-speech/delete-voice
https://api.deepinfra.com/openapi.json delete /v1/voices/{voice_id}
# Get Voice
Source: https://docs.deepinfra.com/api-reference/text-to-speech/get-voice
https://api.deepinfra.com/openapi.json get /v1/voices/{voice_id}
Get a voice by its id
# Get Voices
Source: https://docs.deepinfra.com/api-reference/text-to-speech/get-voices
https://api.deepinfra.com/openapi.json get /v1/voices
Get available voices for a given user
# Text To Speech
Source: https://docs.deepinfra.com/api-reference/text-to-speech/text-to-speech
https://api.deepinfra.com/openapi.json post /v1/text-to-speech/{voice_id}
# Text To Speech Stream
Source: https://docs.deepinfra.com/api-reference/text-to-speech/text-to-speech-stream
https://api.deepinfra.com/openapi.json post /v1/text-to-speech/{voice_id}/stream
# Update Voice
Source: https://docs.deepinfra.com/api-reference/text-to-speech/update-voice
https://api.deepinfra.com/openapi.json post /v1/voices/{voice_id}/edit
# Detokenize
Source: https://docs.deepinfra.com/api-reference/tokenizer/detokenize
https://api.deepinfra.com/openapi.json post /v1/detokenize
# Tokenize
Source: https://docs.deepinfra.com/api-reference/tokenizer/tokenize
https://api.deepinfra.com/openapi.json post /v1/tokenize
# Cli Version
Source: https://docs.deepinfra.com/api-reference/utilities/cli-version
https://api.deepinfra.com/openapi.json get /cli/version
# Submit Feedback
Source: https://docs.deepinfra.com/api-reference/utilities/submit-feedback
https://api.deepinfra.com/openapi.json post /v1/feedback
Submit feedback
# Create Video Generation
Source: https://docs.deepinfra.com/api-reference/videos/create-video-generation
https://api.deepinfra.com/openapi.json post /v1/videos
# Create Video Generation
Source: https://docs.deepinfra.com/api-reference/videos/create-video-generation-1
https://api.deepinfra.com/openapi.json post /v1/openai/videos
# Get Video Content
Source: https://docs.deepinfra.com/api-reference/videos/get-video-content
https://api.deepinfra.com/openapi.json get /v1/videos/{video_id}/content
# Get Video Content
Source: https://docs.deepinfra.com/api-reference/videos/get-video-content-1
https://api.deepinfra.com/openapi.json get /v1/openai/videos/{video_id}/content
# Get Video Generation
Source: https://docs.deepinfra.com/api-reference/videos/get-video-generation
https://api.deepinfra.com/openapi.json get /v1/videos/{video_id}
# Get Video Generation
Source: https://docs.deepinfra.com/api-reference/videos/get-video-generation-1
https://api.deepinfra.com/openapi.json get /v1/openai/videos/{video_id}
# Text Completions
Source: https://docs.deepinfra.com/apis/completions
Legacy OpenAI-compatible completions API for raw text generation.
The completions API is the legacy text generation interface — you provide a raw prompt string and the model continues it. For most use cases, the [Chat Completions API](/chat/overview) is simpler and recommended instead.
The endpoint is:
```
POST https://api.deepinfra.com/v1/openai/completions
```
This is an advanced API. You need to know your model's exact prompt format. Different models have different input formats. Check the model's API section on its page for the expected format.
## Example
The example below uses `deepseek-ai/DeepSeek-V3` with its prompt format:
```python Python theme={null}
from openai import OpenAI
openai = OpenAI(
api_key="$DEEPINFRA_TOKEN",
base_url="https://api.deepinfra.com/v1/openai",
)
stream = True # or False
completion = openai.completions.create(
model="deepseek-ai/DeepSeek-V3",
prompt="<｜begin▁of▁sentence｜><｜User｜>Hello!<｜Assistant｜>",
stop=["<｜end▁of▁sentence｜>"],
stream=stream,
)
if stream:
for event in completion:
if event.choices[0].finish_reason:
print(event.choices[0].finish_reason,
event.usage.prompt_tokens,
event.usage.completion_tokens)
else:
print(event.choices[0].text, end="", flush=True)
else:
print(completion.choices[0].text)
print(completion.usage.prompt_tokens, completion.usage.completion_tokens)
```
```javascript JavaScript theme={null}
import OpenAI from "openai";
const openai = new OpenAI({
baseURL: "https://api.deepinfra.com/v1/openai",
apiKey: "$DEEPINFRA_TOKEN",
});
const stream = true; // or false
const completion = await openai.completions.create({
model: "deepseek-ai/DeepSeek-V3",
prompt: "<｜begin▁of▁sentence｜><｜User｜>Hello!<｜Assistant｜>",
stream: stream,
stop: ["<｜end▁of▁sentence｜>"]
});
if (stream) {
for await (const chunk of completion) {
if (chunk.choices[0].finish_reason) {
console.log(chunk.choices[0].finish_reason,
chunk.usage.prompt_tokens,
chunk.usage.completion_tokens);
} else {
process.stdout.write(chunk.choices[0].text);
}
}
} else {
console.log(completion.choices[0].text);
console.log(completion.usage.prompt_tokens, completion.usage.completion_tokens);
}
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"model": "deepseek-ai/DeepSeek-V3",
"prompt": "<｜begin▁of▁sentence｜><｜User｜>Hello!<｜Assistant｜>",
"stop": [
"<｜end▁of▁sentence｜>"
]
}'
```
## Supported parameters
| Parameter | Notes |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `model` | Model name or `MODEL_NAME:VERSION` |
| `prompt` | Raw prompt string in the model's expected format |
| `max_tokens` | Max tokens to generate. Defaults to the model's max context length minus input length |
| `stream` | Stream output via SSE instead of returning the full response at once. Default: `false` |
| `temperature` | Sampling temperature between 0 and 2. Higher values produce more random output; lower values more deterministic. Default: `1.0` |
| `top_p` | Nucleus sampling threshold — only tokens comprising the top `top_p` probability mass are considered. Default: `1.0` |
| `stop` | Up to 4 sequences where the API will stop generating further tokens |
| `n` | Number of completion sequences to return. Default: `1` |
| `echo` | If `true`, the prompt is included at the start of the returned text |
| `logprobs` | Return log probabilities for the generated tokens |
For every model, you can check its prompt format in the API section on its page.
For the complete parameter reference, see the [API reference](/api-reference/text-completions/openai-completions).
# DeepInfra Native API
Source: https://docs.deepinfra.com/apis/deepinfra-native
Advanced API with access to all model types including image generation, speech, object detection, and more.
The DeepInfra Native API gives you access to every model we provide, including model types not covered by the OpenAI-compatible API: image generation, speech recognition, object detection, token classification, fill mask, image classification, zero-shot image classification, and text classification.
For LLMs and embeddings, the [OpenAI-compatible API](/chat/overview) is simpler and recommended. Use the native API when you need model types beyond LLMs/embeddings, or when you need features like [webhooks](/account/webhooks) or [log probabilities](/chat/log-probs).
The base endpoint is:
```
https://api.deepinfra.com/v1/inference/{model_name}
```
## JavaScript client
```bash theme={null}
npm install deepinfra
```
## Text Generation (LLMs)
```javascript theme={null}
import { TextGeneration } from "deepinfra";
const client = new TextGeneration(
"https://api.deepinfra.com/v1/inference/deepseek-ai/DeepSeek-V3",
"$DEEPINFRA_TOKEN"
);
const res = await client.generate({
input: "<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\nHello!<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
stop: ["<|eot_id|>"]
});
console.log(res.results[0].generated_text);
console.log(res.inference_status.tokens_input, res.inference_status.tokens_generated);
```
```bash theme={null}
curl "https://api.deepinfra.com/v1/inference/deepseek-ai/DeepSeek-V3" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"input": "<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\nHello!<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
"stop": ["<|eot_id|>"],
"stream": false
}'
```
## Embeddings
```javascript theme={null}
import { Embeddings } from "deepinfra";
const client = new Embeddings("Qwen/Qwen3-Embedding-8B", "$DEEPINFRA_TOKEN");
const output = await client.generate({
inputs: [
"What is the capital of France?",
"What is the capital of Germany?",
],
});
console.log(output.embeddings[0]);
```
```bash theme={null}
curl -X POST \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-F 'inputs=["I like chocolate"]' \
'https://api.deepinfra.com/v1/inference/Qwen/Qwen3-Embedding-8B'
```
## Image Generation
```javascript theme={null}
import { TextToImage } from "deepinfra";
import { createWriteStream } from "fs";
import { Readable } from "stream";
const model = new TextToImage("stabilityai/stable-diffusion-2-1", "$DEEPINFRA_TOKEN");
const response = await model.generate({
prompt: "a burger with a funny hat on the beach",
});
const result = await fetch(response.images[0]);
if (result.ok && result.body) {
Readable.fromWeb(result.body).pipe(createWriteStream("image.png"));
}
```
```bash theme={null}
curl "https://api.deepinfra.com/v1/inference/stabilityai/stable-diffusion-2-1" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{"prompt": "a burger with a funny hat on the beach"}'
```
## Speech Recognition
```bash theme={null}
curl -X POST \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-F audio=@audio.mp3 \
'https://api.deepinfra.com/v1/inference/openai/whisper-large'
```
## Object Detection
```bash theme={null}
curl -X POST \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-F image=@image.jpg \
'https://api.deepinfra.com/v1/inference/hustvl/yolos-small'
```
## Token Classification
```bash theme={null}
curl -X POST \
-d '{"input": "My name is John Doe and I live in San Francisco."}' \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-H 'Content-Type: application/json' \
'https://api.deepinfra.com/v1/inference/Davlan/bert-base-multilingual-cased-ner-hrl'
```
## Fill Mask
```bash theme={null}
curl -X POST \
-d '{"input": "I need my [MASK] right now!"}' \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-H 'Content-Type: application/json' \
'https://api.deepinfra.com/v1/inference/bert-base-cased'
```
## Image Classification
```bash theme={null}
curl -X POST \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-F image=@image.jpg \
'https://api.deepinfra.com/v1/inference/google/vit-base-patch16-224'
```
## Zero-Shot Image Classification
```bash theme={null}
curl -X POST \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-F image=@image.jpg \
-F 'candidate_labels=["dog", "cat", "car", "horse", "person"]' \
'https://api.deepinfra.com/v1/inference/openai/clip-vit-base-patch32'
```
## Text Classification
```bash theme={null}
curl -X POST \
-d '{"input": "Nvidia announces new AI chips months after latest launch"}' \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-H 'Content-Type: application/json' \
'https://api.deepinfra.com/v1/inference/ProsusAI/finbert'
```
## HTTP / other languages
The native API is plain HTTP — you can use it from any language (Go, C#, Java, PHP, Ruby, C++, etc.) without any SDK dependency.
# Embeddings
Source: https://docs.deepinfra.com/apis/embeddings
Generate embedding vectors from text using the OpenAI-compatible embeddings API.
DeepInfra supports the OpenAI embeddings API for all [embedding models](https://deepinfra.com/models/embeddings).
The endpoint is:
```
POST https://api.deepinfra.com/v1/openai/embeddings
```
## Example
```python Python theme={null}
from openai import OpenAI
openai = OpenAI(
api_key="$DEEPINFRA_TOKEN",
base_url="https://api.deepinfra.com/v1/openai",
)
input_text = "The food was delicious and the waiter..."
# Or a list: ["hello", "world"]
embeddings = openai.embeddings.create(
model="Qwen/Qwen3-Embedding-8B",
input=input_text,
encoding_format="float"
)
print(embeddings.data[0].embedding)
print(embeddings.usage.prompt_tokens)
```
```javascript JavaScript theme={null}
import OpenAI from "openai";
const openai = new OpenAI({
baseURL: "https://api.deepinfra.com/v1/openai",
apiKey: "$DEEPINFRA_TOKEN",
});
const input = "The quick brown fox jumped over the lazy dog";
// Or an array: ["hello", "world"]
const embedding = await openai.embeddings.create({
model: "Qwen/Qwen3-Embedding-8B",
input: input,
encoding_format: "float",
});
console.log(embedding.data[0].embedding);
console.log(embedding.usage.prompt_tokens);
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/embeddings" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"input": "The food was delicious and the waiter...",
"model": "Qwen/Qwen3-Embedding-8B",
"encoding_format": "float"
}'
```
## Batch embeddings
Pass an array as `input` to embed multiple texts in a single request:
```python theme={null}
embeddings = openai.embeddings.create(
model="Qwen/Qwen3-Embedding-8B",
input=["Hello", "World", "How are you?"],
encoding_format="float"
)
for i, item in enumerate(embeddings.data):
print(f"Text {i}: {item.embedding[:5]}...") # First 5 dims
```
## Supported parameters
| Parameter | Notes |
| ----------------- | -------------------------- |
| `model` | Embedding model name |
| `input` | String or array of strings |
| `encoding_format` | `float` only |
## Available models
Browse [all embedding models](https://deepinfra.com/models/embeddings) — includes Qwen3 Embedding, BAAI/bge, sentence-transformers, and more.
# Image Generation
Source: https://docs.deepinfra.com/apis/image-generation
Generate images from text prompts using the OpenAI-compatible images API.
DeepInfra supports the OpenAI-compatible image generation API. The default model is FLUX Schnell.
The endpoint is:
```
POST https://api.deepinfra.com/v1/openai/images/generations
```
Browse [all text-to-image models](https://deepinfra.com/models/text-to-image).
## Example
```python Python theme={null}
import io
import base64
from PIL import Image
from openai import OpenAI
client = OpenAI(
api_key="$DEEPINFRA_TOKEN",
base_url="https://api.deepinfra.com/v1/openai"
)
response = client.images.generate(
prompt="A photo of an astronaut riding a horse on Mars.",
size="1024x1024",
n=1,
)
b64_json = response.data[0].b64_json
image_bytes = base64.b64decode(b64_json)
image = Image.open(io.BytesIO(image_bytes))
image.save("output.png")
```
```javascript JavaScript theme={null}
import * as fs from "fs";
import OpenAI from "openai";
const openai = new OpenAI({
baseURL: "https://api.deepinfra.com/v1/openai",
apiKey: "$DEEPINFRA_TOKEN",
});
const response = await openai.images.generate({
prompt: "A photo of an astronaut riding a horse on Mars.",
size: "1024x1024",
n: 1,
});
const b64Json = response.data[0].b64_json;
const imageBuffer = Buffer.from(b64Json, "base64");
fs.writeFileSync("output.png", imageBuffer);
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/images/generations" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"prompt": "A photo of an astronaut riding a horse on Mars.",
"size": "1024x1024",
"n": 1
}'
```
## Supported parameters
| Parameter | Notes |
| ------------------ | -------------------------------------- |
| `prompt` | Text description of the image |
| `model` | Defaults to FLUX Schnell |
| `size` | Image dimensions (e.g., `"1024x1024"`) |
| `n` | Number of images to generate |
| `response_format` | Only `b64_json` supported |
| `quality`, `style` | Available for compatibility only |
## LoRA image adapters
You can also use custom LoRA adapters for image generation — see [LoRA for Image Generation](/private-models/lora-image).
## Tutorial
For a deeper example including advanced options, see the [Stable Diffusion tutorial](/tutorials/stable-diffusion).
# Reranking
Source: https://docs.deepinfra.com/apis/reranker
Rerank a list of documents by relevance to a query.
Reranker models take a query and a list of candidate documents and return a relevance score for each document. They're typically used as a second-pass filter after an initial vector search to improve retrieval quality in RAG pipelines.
Browse [all reranker models](https://deepinfra.com/models/reranker).
## Endpoint
```
POST https://api.deepinfra.com/v1/inference/{model_name}
```
## Example
```python Python theme={null}
import requests
DEEPINFRA_TOKEN = "$DEEPINFRA_TOKEN"
MODEL = "cross-encoder/ms-marco-MiniLM-L-12-v2"
response = requests.post(
f"https://api.deepinfra.com/v1/inference/{MODEL}",
headers={
"Authorization": f"Bearer {DEEPINFRA_TOKEN}",
"Content-Type": "application/json",
},
json={
"query": "What is the capital of France?",
"documents": [
"Paris is the capital and most populous city of France.",
"Berlin is the capital of Germany.",
"The Eiffel Tower is located in Paris.",
"France is a country in Western Europe.",
],
},
)
result = response.json()
for item in result["scores"]:
print(item)
```
```bash cURL theme={null}
curl -X POST \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"query": "What is the capital of France?",
"documents": [
"Paris is the capital and most populous city of France.",
"Berlin is the capital of Germany.",
"The Eiffel Tower is located in Paris.",
"France is a country in Western Europe."
]
}' \
'https://api.deepinfra.com/v1/inference/cross-encoder/ms-marco-MiniLM-L-12-v2'
```
## Response
```json theme={null}
{
"scores": [0.98, 0.02, 0.45, 0.31]
}
```
Scores are relevance probabilities in the range \[0, 1], in the same order as the input documents. Sort by score descending to get the most relevant documents first.
## Usage in a RAG pipeline
A typical pattern:
1. **Retrieve** — run a vector similarity search to fetch the top-N candidate chunks (e.g. top 50)
2. **Rerank** — pass the query + candidates to a reranker to get relevance scores
3. **Select** — keep only the top-K highest-scoring chunks (e.g. top 5) for the LLM context
This two-stage approach improves precision significantly compared to embedding similarity alone.
```python theme={null}
# 1. Get initial candidates from your vector DB
candidates = vector_db.search(query, top_k=50)
# 2. Rerank
response = requests.post(
"https://api.deepinfra.com/v1/inference/cross-encoder/ms-marco-MiniLM-L-12-v2",
headers={"Authorization": f"Bearer {DEEPINFRA_TOKEN}", "Content-Type": "application/json"},
json={"query": query, "documents": [c["text"] for c in candidates]},
)
scores = response.json()["scores"]
# 3. Select top-K
ranked = sorted(zip(scores, candidates), reverse=True)
top_chunks = [doc for _, doc in ranked[:5]]
```
## Available models
Browse [all reranker models](https://deepinfra.com/models/reranker).
# Speech Recognition
Source: https://docs.deepinfra.com/apis/speech
Transcribe audio to text using Whisper and other speech recognition models.
DeepInfra hosts [Whisper](https://github.com/openai/whisper) and other speech recognition models. Given an audio file, they produce transcribed text with per-sentence timestamps.
Browse [all speech recognition models](https://deepinfra.com/models/automatic-speech-recognition).
## Models
* `openai/whisper-large` — best accuracy
* `openai/whisper-medium`, `openai/whisper-small`, `openai/whisper-base` — faster, lighter
* `openai/whisper-timestamped-medium` — per-word timestamp segmentation
## Example
```bash cURL theme={null}
curl -X POST \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-F audio=@audio.mp3 \
'https://api.deepinfra.com/v1/inference/openai/whisper-large'
```
```javascript JavaScript theme={null}
import { AutomaticSpeechRecognition } from "deepinfra";
import path from "path";
import { fileURLToPath } from "url";
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const DEEPINFRA_API_KEY = "$DEEPINFRA_TOKEN";
const MODEL = "openai/whisper-large";
const client = new AutomaticSpeechRecognition(MODEL, DEEPINFRA_API_KEY);
const input = {
audio: path.join(__dirname, "audio.mp3"),
};
const response = await client.generate(input);
console.log(response.text);
```
## Supported audio formats
* `mp3`
* `wav`
## Response
```json theme={null}
{
"text": "Hello, this is a transcription of the audio file.",
"segments": [
{
"start": 0.0,
"end": 3.5,
"text": "Hello, this is a transcription of the audio file."
}
]
}
```
## Additional parameters
Each model exposes different parameters (language, task, etc.). Check the model's API documentation page for details.
## Tutorial
See the [Whisper tutorial](/tutorials/whisper) for a complete walkthrough.
# Text to Speech
Source: https://docs.deepinfra.com/apis/text-to-speech
Convert text to natural-sounding audio using TTS models.
DeepInfra hosts text-to-speech models that convert text into natural-sounding audio. Browse [all TTS models](https://deepinfra.com/models/text-to-speech).
## Endpoint
```
POST https://api.deepinfra.com/v1/inference/{model_name}
```
## Example
```python Python theme={null}
import requests
DEEPINFRA_TOKEN = "$DEEPINFRA_TOKEN"
MODEL = "hexgrad/Kokoro-82M"
response = requests.post(
f"https://api.deepinfra.com/v1/inference/{MODEL}",
headers={
"Authorization": f"Bearer {DEEPINFRA_TOKEN}",
"Content-Type": "application/json",
},
json={
"text": "Hello! This is a text-to-speech example using DeepInfra.",
},
)
# Save the returned audio
with open("output.wav", "wb") as f:
f.write(response.content)
```
```bash cURL theme={null}
curl -X POST \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-H "Content-Type: application/json" \
-d '{"text": "Hello! This is a text-to-speech example using DeepInfra."}' \
'https://api.deepinfra.com/v1/inference/hexgrad/Kokoro-82M' \
--output output.wav
```
## Additional parameters
Each model may expose additional parameters such as voice selection, speed, and language. Check the model's individual [API documentation page](https://deepinfra.com/models/text-to-speech) for supported options.
## Available models
Browse [all text-to-speech models](https://deepinfra.com/models/text-to-speech).
# Text to Video
Source: https://docs.deepinfra.com/apis/text-to-video
Generate video clips from text prompts.
DeepInfra hosts text-to-video models that generate short video clips from a text description. Browse [all text-to-video models](https://deepinfra.com/models/text-to-video).
## Endpoint
```
POST https://api.deepinfra.com/v1/inference/{model_name}
```
## Example
```python Python theme={null}
import requests
DEEPINFRA_TOKEN = "$DEEPINFRA_TOKEN"
MODEL = "Wan-AI/Wan2.1-T2V-14B"
response = requests.post(
f"https://api.deepinfra.com/v1/inference/{MODEL}",
headers={
"Authorization": f"Bearer {DEEPINFRA_TOKEN}",
"Content-Type": "application/json",
},
json={
"prompt": "A serene mountain lake at sunrise, with mist rising from the water and pine trees reflected on the surface.",
},
)
result = response.json()
# result["video"] contains the URL to the generated video
print(result["video"])
```
```bash cURL theme={null}
curl -X POST \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-H "Content-Type: application/json" \
-d '{"prompt": "A serene mountain lake at sunrise, with mist rising from the water and pine trees reflected on the surface."}' \
'https://api.deepinfra.com/v1/inference/Wan-AI/Wan2.1-T2V-14B'
```
## Tips for good prompts
* Be descriptive about the scene, lighting, and motion
* Specify the camera movement if relevant (e.g. "slow pan", "aerial shot", "close-up")
* Keep prompts focused — overly complex prompts can produce inconsistent results
* Use the [negative prompt](https://deepinfra.com/models/text-to-video) parameter (if supported) to exclude unwanted elements
## Async inference
Video generation is compute-intensive and may take longer than text inference. Consider using [webhooks](/account/webhooks) to receive the result asynchronously rather than polling.
## Available models
Browse [all text-to-video models](https://deepinfra.com/models/text-to-video).
# Batch Endpoints
Source: https://docs.deepinfra.com/batch/batch-endpoints
Reference for the Batch API endpoints — create, retrieve, list, and cancel batch jobs.
An **OpenAI-compatible** Batch API for submitting and managing asynchronous inference jobs. All endpoints are relative to:
```
https://api.deepinfra.com/v1/openai
```
These endpoints operate on the [Batch object](/batch/batch-objects).
## Create a batch
```
POST /batches
```
Creates and starts executing a batch job from an uploaded file of requests.
Creating a batch requires the following parameters:
The time window after which the batch expires. Currently only `"24h"` is supported.
The endpoint to run the batch against. One of the batch-supported endpoints, currently `/v1/chat/completions`, `/v1/completions`, or `/v1/embeddings`. Must match the `url` used on every line of the input file.
The `id` of the uploaded input file (created with `purpose="batch"`).
Up to 16 key–value string pairs, where each key is a string of up to 64 characters and each value is a string of up to 512 characters.
Controls how long the output and error files remain available. An object with two fields:
* **`anchor`** (optional) — must be `"created_at"`. The expiry is measured from when the output file is created. Defaults to `"created_at"`.
* **`seconds`** (optional) — the number of seconds the file stays available after the anchor. An integer between `3600` (1 hour) and `2592000` (30 days). Defaults to `2592000` (30 days).
This endpoint returns a [Batch object](/batch/batch-objects).
```python Python theme={null}
from openai import OpenAI
client = OpenAI(
api_key="$DEEPINFRA_TOKEN",
base_url="https://api.deepinfra.com/v1/openai",
)
batch = client.batches.create(
input_file_id="file_abc123",
endpoint="/v1/chat/completions",
completion_window="24h",
metadata={"description": "nightly eval run"},
output_expires_after={"anchor": "created_at", "seconds": 604800},
)
print(batch.id, batch.status)
```
```javascript JavaScript theme={null}
import OpenAI from "openai";
const client = new OpenAI({
apiKey: "$DEEPINFRA_TOKEN",
baseURL: "https://api.deepinfra.com/v1/openai",
});
const batch = await client.batches.create({
input_file_id: "file_abc123",
endpoint: "/v1/chat/completions",
completion_window: "24h",
metadata: { description: "nightly eval run" },
output_expires_after: { anchor: "created_at", seconds: 604800 },
});
console.log(batch.id, batch.status);
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/batches" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"input_file_id": "file_abc123",
"endpoint": "/v1/chat/completions",
"completion_window": "24h",
"metadata": {"description": "nightly eval run"},
"output_expires_after": {"anchor": "created_at", "seconds": 604800}
}'
```
```json theme={null}
{
"id": "batch_abc123",
"object": "batch",
"endpoint": "/v1/chat/completions",
"errors": null,
"input_file_id": "file_abc123",
"completion_window": "24h",
"status": "validating",
"output_file_id": null,
"error_file_id": null,
"created_at": 1711471533,
"in_progress_at": null,
"expires_at": null,
"finalizing_at": null,
"completed_at": null,
"failed_at": null,
"expired_at": null,
"cancelling_at": null,
"cancelled_at": null,
"request_counts": null,
"metadata": {
"description": "nightly eval run"
},
"model": null,
"usage": null
}
```
## Retrieve a batch
```
GET /batches/{batch_id}
```
Returns information about a specific batch.
Retrieving a batch requires the following parameters:
The `id` of the batch to retrieve.
This endpoint returns a [Batch object](/batch/batch-objects).
```python Python theme={null}
batch = client.batches.retrieve("batch_abc123")
print(batch.id, batch.status)
```
```javascript JavaScript theme={null}
const batch = await client.batches.retrieve("batch_abc123");
console.log(batch.id, batch.status);
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/batches/batch_abc123" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN"
```
```json theme={null}
{
"id": "batch_abc123",
"object": "batch",
"endpoint": "/v1/chat/completions",
"errors": null,
"input_file_id": "file_abc123",
"completion_window": "24h",
"status": "in_progress",
"output_file_id": null,
"error_file_id": null,
"created_at": 1711471533,
"in_progress_at": 1711471538,
"expires_at": null,
"finalizing_at": null,
"completed_at": null,
"failed_at": null,
"expired_at": null,
"cancelling_at": null,
"cancelled_at": null,
"request_counts": {
"total": 100,
"completed": 40,
"failed": 1
},
"metadata": {
"description": "nightly eval run"
},
"model": "deepseek-ai/DeepSeek-V3",
"usage": {
"input_tokens": 4800,
"input_tokens_details": {
"cached_tokens": 0
},
"output_tokens": 3200,
"output_tokens_details": {
"reasoning_tokens": 0
},
"total_tokens": 8000
}
}
```
## List batches
```
GET /batches
```
Listing batches requires the following parameters:
A pagination cursor. The returned list starts from the object right after the batch with this `id`. If omitted, the list starts from the first batch.
An integer between `1` and `100`. The returned list will have at most `limit` elements. Defaults to `20`.
The returned object has the following fields:
| Field | Type | Description |
| ---------- | ------- | ------------------------------------------------- |
| `object` | string | The object type, always `"list"`. |
| `data` | array | A list of [Batch objects](/batch/batch-objects). |
| `first_id` | string | The `id` of the first batch in the list. |
| `last_id` | string | The `id` of the last batch in the list. |
| `has_more` | boolean | `true` if there are more batches after `last_id`. |
```python Python theme={null}
batches = client.batches.list(limit=20)
for batch in batches.data:
print(batch.id, batch.status)
```
```javascript JavaScript theme={null}
const batches = await client.batches.list({ limit: 20 });
for (const batch of batches.data) {
console.log(batch.id, batch.status);
}
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/batches?limit=20" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN"
```
```json theme={null}
{
"object": "list",
"data": [
{
"id": "batch_abc123",
"object": "batch",
"endpoint": "/v1/chat/completions",
"errors": null,
"input_file_id": "file_abc123",
"completion_window": "24h",
"status": "completed",
"output_file_id": "file_out456",
"error_file_id": null,
"created_at": 1711471533,
"in_progress_at": 1711471538,
"expires_at": null,
"finalizing_at": 1711475133,
"completed_at": 1711475134,
"failed_at": null,
"expired_at": null,
"cancelling_at": null,
"cancelled_at": null,
"request_counts": {
"total": 100,
"completed": 100,
"failed": 0
},
"metadata": {
"description": "nightly eval run"
},
"model": "deepseek-ai/DeepSeek-V3",
"usage": {
"input_tokens": 12000,
"input_tokens_details": {
"cached_tokens": 0
},
"output_tokens": 8000,
"output_tokens_details": {
"reasoning_tokens": 0
},
"total_tokens": 20000
}
},
{
"id": "batch_def456",
"object": "batch",
"endpoint": "/v1/embeddings",
"errors": null,
"input_file_id": "file_def456",
"completion_window": "24h",
"status": "in_progress",
"output_file_id": null,
"error_file_id": null,
"created_at": 1711558000,
"in_progress_at": 1711558005,
"expires_at": null,
"finalizing_at": null,
"completed_at": null,
"failed_at": null,
"expired_at": null,
"cancelling_at": null,
"cancelled_at": null,
"request_counts": {
"total": 5000,
"completed": 1200,
"failed": 0
},
"metadata": null,
"model": "Qwen/Qwen3-Embedding-8B",
"usage": {
"input_tokens": 60000,
"input_tokens_details": {
"cached_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"reasoning_tokens": 0
},
"total_tokens": 60000
}
}
],
"first_id": "batch_abc123",
"last_id": "batch_def456",
"has_more": false
}
```
## Cancel a batch
```
POST /batches/{batch_id}/cancel
```
Cancels a given batch. The batch moves to the `cancelling` status until it is finalized, at which point its status becomes `cancelled`. Requests that managed to finish are written to the output file, and the rest are written to the error file as `cancelled`.
Cancelling a batch requires the following parameters:
The `id` of the batch to cancel.
This endpoint returns a [Batch object](/batch/batch-objects).
```python Python theme={null}
batch = client.batches.cancel("batch_abc123")
print(batch.status)
```
```javascript JavaScript theme={null}
const batch = await client.batches.cancel("batch_abc123");
console.log(batch.status);
```
```bash cURL theme={null}
curl -X POST "https://api.deepinfra.com/v1/openai/batches/batch_abc123/cancel" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN"
```
```json theme={null}
{
"id": "batch_abc123",
"object": "batch",
"endpoint": "/v1/chat/completions",
"errors": null,
"input_file_id": "file_abc123",
"completion_window": "24h",
"status": "cancelled",
"output_file_id": "file_out456",
"error_file_id": "file_err789",
"created_at": 1711471533,
"in_progress_at": 1711471538,
"expires_at": null,
"finalizing_at": null,
"completed_at": null,
"failed_at": null,
"expired_at": null,
"cancelling_at": 1711472000,
"cancelled_at": 1711472050,
"request_counts": {
"total": 100,
"completed": 40,
"failed": 0
},
"metadata": {
"description": "nightly eval run"
},
"model": "deepseek-ai/DeepSeek-V3",
"usage": {
"input_tokens": 4800,
"input_tokens_details": {
"cached_tokens": 0
},
"output_tokens": 3200,
"output_tokens_details": {
"reasoning_tokens": 0
},
"total_tokens": 8000
}
}
```
# Batch Objects
Source: https://docs.deepinfra.com/batch/batch-objects
Reference for the objects returned by the Batch API — the Batch and BatchUsage objects.
Reference for the objects returned by the Batch API.
## The Batch object
Most batch endpoints return a `Batch` object describing a batch job.
| Field | Type | Description |
| ------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `id` | string | The batch identifier. |
| `object` | string | The object type, always `"batch"`. |
| `endpoint` | string | The endpoint the batch runs against. Must match the `url` used on every line of the input file. |
| `errors` | object | Optional. Validation errors for the batch. See [errors](#errors). |
| `input_file_id` | string | The `id` of the input file for the batch. |
| `completion_window` | string | The time window after which the batch expires. Currently only `"24h"` is supported. |
| `status` | string | The current status of the batch. See [Batch status](#batch-status). |
| `output_file_id` | string | The `id` of the file containing the outputs of successfully executed requests. |
| `error_file_id` | string | The `id` of the file containing the outputs of requests that failed (failed lines). |
| `created_at` | integer | Unix timestamp (in seconds) for when the batch was created. |
| `in_progress_at` | integer | Unix timestamp for when the batch started processing. `null` until reached. |
| `expires_at` | integer | Unix timestamp for when the batch will expire. `null` until reached. |
| `finalizing_at` | integer | Unix timestamp for when the batch started finalizing. `null` until reached. |
| `completed_at` | integer | Unix timestamp for when the batch completed. `null` until reached. |
| `failed_at` | integer | Unix timestamp for when the batch failed. `null` until reached. |
| `expired_at` | integer | Unix timestamp for when the batch expired. `null` until reached. |
| `cancelling_at` | integer | Unix timestamp for when the batch started cancelling. `null` until reached. |
| `cancelled_at` | integer | Unix timestamp for when the batch was cancelled. `null` until reached. |
| `request_counts` | object | Optional. The counts of requests by status within the batch. See [request\_counts](#request_counts). |
| `metadata` | object | Up to 16 key–value string pairs, where each key is a string of up to 64 characters and each value is a string of up to 512 characters. |
| `model` | string | Optional. The model the inference is run against. |
| `usage` | object | Optional. Token usage statistics for the batch. See [The BatchUsage object](#the-batchusage-object). |
### Batch status
A batch moves through the following statuses:
| Status | Meaning |
| ------------- | ---------------------------------------------------------------------------------------------------------------- |
| `validating` | The input file is being validated before the batch starts. |
| `in_progress` | Validation passed and requests are running. |
| `finalizing` | All requests are done; the output files are being assembled. |
| `completed` | The batch finished; results are ready to download. |
| `failed` | Validation failed. The [`errors`](#errors) field of the Batch object describes why the file failed. |
| `expired` | Not all requests could finish within the 24-hour window; the finished ones are still written to the output file. |
| `cancelling` | The batch is being cancelled. This status may last for some time while in-flight requests finish. |
| `cancelled` | The batch has been cancelled successfully. |
### errors
| Field | Type | Description |
| -------- | ------ | ----------------------------------------------------------------------------- |
| `object` | string | The object type, always `"list"`. |
| `data` | array | Optional. An array of error objects, each with the following optional fields: |
| Field | Type | Description |
| --------- | ------- | --------------------------------------------------------- |
| `code` | string | Optional. A machine-readable error code. |
| `line` | integer | Optional. The line of the input file the error refers to. |
| `message` | string | Optional. A human-readable description of the error. |
| `param` | string | Optional. The parameter the error relates to. |
### request\_counts
| Field | Type | Description |
| ----------- | ------- | -------------------------------------------------------- |
| `total` | integer | The total number of requests in the batch. |
| `completed` | integer | The number of requests that have completed successfully. |
| `failed` | integer | The number of requests that have failed. |
## The BatchUsage object
Token usage statistics aggregated across all requests in the batch.
| Field | Type | Description |
| ----------------------- | ------- | ------------------------------------------------------------------------------------------------- |
| `input_tokens` | integer | The total number of input (prompt) tokens across the batch. |
| `input_tokens_details` | object | A breakdown of the input tokens. Contains `cached_tokens`, and may contain additional fields. |
| `output_tokens` | integer | The total number of output (completion) tokens across the batch. |
| `output_tokens_details` | object | A breakdown of the output tokens. Contains `reasoning_tokens`, and may contain additional fields. |
| `total_tokens` | integer | The total number of tokens used (input + output). |
## Example
Batch object of a completed batch:
```json theme={null}
{
"id": "batch_abc123",
"object": "batch",
"endpoint": "/v1/chat/completions",
"errors": null,
"input_file_id": "file_abc123",
"completion_window": "24h",
"status": "completed",
"output_file_id": "file_out456",
"error_file_id": null,
"created_at": 1711471533,
"in_progress_at": 1711471538,
"expires_at": null,
"finalizing_at": 1711475133,
"completed_at": 1711475134,
"failed_at": null,
"expired_at": null,
"cancelling_at": null,
"cancelled_at": null,
"request_counts": {
"total": 100,
"completed": 100,
"failed": 0
},
"metadata": {
"description": "nightly eval run"
},
"model": "deepseek-ai/DeepSeek-V3",
"usage": {
"input_tokens": 12000,
"input_tokens_details": {
"cached_tokens": 0
},
"output_tokens": 8000,
"output_tokens_details": {
"reasoning_tokens": 0
},
"total_tokens": 20000
}
}
```
# Files API
Source: https://docs.deepinfra.com/batch/file-endpoints
The OpenAI-compatible Files API — upload, list, retrieve, download, and delete files.
An **OpenAI-compatible** Files API for managing batch and fine-tune related files. All endpoints are relative to:
```
https://api.deepinfra.com/v1/openai
```
Fine-tuning is not yet supported.
Currently the only API that uses files is the [Batch API](/batch/introduction), which accepts files of up to 200 MB.
## Objects
### The FileObject
Most file endpoints return a `FileObject` describing an uploaded file.
| Field | Type | Description |
| ------------ | ------- | --------------------------------------------------------------------------------------- |
| `id` | string | The file identifier, referenced in API endpoints. |
| `object` | string | The object type, always `"file"`. |
| `bytes` | integer | The size of the file, in bytes. |
| `created_at` | integer | Unix timestamp (in seconds) for when the file was created. |
| `expires_at` | integer | Unix timestamp (in seconds) for when the file will be deleted. |
| `filename` | string | The name of the file. |
| `purpose` | string | The intended purpose of the file. One of `"batch"`, `"fine-tune"`, or `"batch-output"`. |
```json theme={null}
{
"id": "file_abc123",
"object": "file",
"bytes": 120000,
"created_at": 1677610602,
"expires_at": 1680202602,
"filename": "requests.jsonl",
"purpose": "batch"
}
```
## Create file
```
POST /files
```
Creates a file used in other API endpoints.
Creating a file requires the following parameters:
The file to be uploaded.
The intended purpose of the file. Can be `"batch"` or `"fine-tune"` for upload, but only `"batch"` is currently supported.
Controls how long the file remains available. An object with two fields:
* **`anchor`** (optional) — must be `"created_at"`. The expiry is measured from when the file is created. Defaults to `"created_at"`.
* **`seconds`** (optional) — the number of seconds the file stays available after the anchor. An integer between `3600` (1 hour) and `2592000` (30 days). Defaults to `2592000` (30 days).
This endpoint returns a [FileObject](#the-fileobject).
```python Python theme={null}
from openai import OpenAI
client = OpenAI(
api_key="$DEEPINFRA_TOKEN",
base_url="https://api.deepinfra.com/v1/openai",
)
file = client.files.create(
file=open("requests.jsonl", "rb"),
purpose="batch",
expires_after={"anchor": "created_at", "seconds": 604800},
)
print(file.id)
```
```javascript JavaScript theme={null}
import fs from "fs";
import OpenAI from "openai";
const client = new OpenAI({
apiKey: "$DEEPINFRA_TOKEN",
baseURL: "https://api.deepinfra.com/v1/openai",
});
const file = await client.files.create({
file: fs.createReadStream("requests.jsonl"),
purpose: "batch",
expires_after: { anchor: "created_at", seconds: 604800 },
});
console.log(file.id);
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/files" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-F purpose="batch" \
-F file="@requests.jsonl" \
-F expires_after[anchor]="created_at" \
-F expires_after[seconds]=604800
```
```json theme={null}
{
"id": "file_abc123",
"object": "file",
"bytes": 120000,
"created_at": 1677610602,
"expires_at": 1678215402,
"filename": "requests.jsonl",
"purpose": "batch"
}
```
## List files
```
GET /files
```
Listing files requires the following parameters:
A pagination cursor. The returned list starts from the object right after the file with this `id`. If omitted, the list starts from the first file.
An integer between `1` and `10000`. The returned list will have at most `limit` elements. Defaults to `10000`.
Sort order by `created_at`, either `"asc"` (ascending) or `"desc"` (descending).
Only returns files of the given purpose. If omitted, files are not filtered by purpose.
The returned object has the following fields:
| Field | Type | Description |
| ---------- | ------- | ----------------------------------------------- |
| `object` | string | The object type, always `"list"`. |
| `data` | array | A list of [FileObject](#the-fileobject). |
| `first_id` | string | The `id` of the first file in the list. |
| `last_id` | string | The `id` of the last file in the list. |
| `has_more` | boolean | `true` if there are more files after `last_id`. |
```python Python theme={null}
files = client.files.list(
purpose="batch",
limit=20,
order="desc",
)
for file in files.data:
print(file.id, file.filename)
```
```javascript JavaScript theme={null}
const files = await client.files.list({
purpose: "batch",
limit: 20,
order: "desc",
});
for (const file of files.data) {
console.log(file.id, file.filename);
}
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/files?purpose=batch&limit=20&order=desc" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN"
```
```json theme={null}
{
"object": "list",
"data": [
{
"id": "file_abc123",
"object": "file",
"bytes": 120000,
"created_at": 1677610602,
"expires_at": 1678215402,
"filename": "requests.jsonl",
"purpose": "batch"
},
{
"id": "file_def456",
"object": "file",
"bytes": 84000,
"created_at": 1677520000,
"expires_at": 1678124800,
"filename": "eval.jsonl",
"purpose": "batch"
}
],
"first_id": "file_abc123",
"last_id": "file_def456",
"has_more": false
}
```
## Retrieve a file
```
GET /files/{file_id}
```
Returns information about a specific file.
Retrieving a file requires the following parameters:
The `id` of the file to retrieve.
This endpoint returns a [FileObject](#the-fileobject).
```python Python theme={null}
file = client.files.retrieve("file_abc123")
print(file.id, file.filename)
```
```javascript JavaScript theme={null}
const file = await client.files.retrieve("file_abc123");
console.log(file.id, file.filename);
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/files/file_abc123" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN"
```
```json theme={null}
{
"id": "file_abc123",
"object": "file",
"bytes": 120000,
"created_at": 1677610602,
"expires_at": 1678215402,
"filename": "requests.jsonl",
"purpose": "batch"
}
```
## Retrieve file content
```
GET /files/{file_id}/content
```
Returns the content of a specified file.
Retrieving file content requires the following parameters:
The `id` of the file whose content to retrieve.
This endpoint returns the binary response content.
```python Python theme={null}
content = client.files.content("file_abc123")
print(content.text)
```
```javascript JavaScript theme={null}
const content = await client.files.content("file_abc123");
console.log(await content.text());
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/files/file_abc123/content" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN"
```
## Delete a file
```
DELETE /files/{file_id}
```
Deletes a file and removes it from all storages.
Deleting a file requires the following parameters:
The `id` of the file to delete.
This endpoint returns a `FileDeleted` object with the following fields:
| Field | Type | Description |
| --------- | ------- | --------------------------------- |
| `id` | string | The `id` of the deleted file. |
| `deleted` | boolean | `true` if the file was deleted. |
| `object` | string | The object type, always `"file"`. |
```python Python theme={null}
deleted = client.files.delete("file_abc123")
print(deleted.deleted)
```
```javascript JavaScript theme={null}
const deleted = await client.files.delete("file_abc123");
console.log(deleted.deleted);
```
```bash cURL theme={null}
curl -X DELETE "https://api.deepinfra.com/v1/openai/files/file_abc123" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN"
```
```json theme={null}
{
"id": "file_abc123",
"deleted": true,
"object": "file"
}
```
## File API Limits
| Limit | Value |
| ----------------------------------------------------------- | ------ |
| Maximum size of a single user-created file | 512 MB |
| Maximum file size with `purpose = "batch"` | 200 MB |
| Maximum total file size per user (including response files) | 2 GB |
# Introduction to Batch API
Source: https://docs.deepinfra.com/batch/introduction
Run large, non-urgent inference jobs asynchronously at 20% off — OpenAI-compatible file upload, batch, and results workflow.
The Batch API lets you submit large volumes of requests as a single asynchronous job and get the results back within 24 hours, billed at **20% below real-time pricing**. It's built for workloads that aren't latency-sensitive — embedding a whole corpus, classifying or summarizing a dataset, or running a model over an evaluation set.
It's **OpenAI-compatible**: if you've used OpenAI's Batch API, the workflow is identical — upload a JSONL file of requests, create a batch, poll for completion, and download the results. Point the OpenAI SDK at DeepInfra and your existing batch code works.
The endpoint is:
```
https://api.deepinfra.com/v1/openai
```
The only changes from your existing OpenAI code are the `base_url` and `api_key`, plus using a model from [our catalog](https://deepinfra.com/models).
Any OpenAI-compatible model available for real-time inference on a batch-supported endpoint can also be used in batch.
## Supported endpoints
A batch runs all of its requests against a single endpoint. The supported endpoints are:
* `/v1/chat/completions`
* `/v1/completions`
* `/v1/embeddings`
Other OpenAI batch endpoints — `/v1/images/generations`, `/v1/images/edits`, `/v1/moderations`, `/v1/responses`, `/v1/videos` — are **not yet** supported and are rejected at validation.
## Workflow
## Step 1 — Prepare the input file
The input is a [JSONL](https://jsonlines.org/) file with one request per line. Each line has a `custom_id`, the HTTP `method` (`POST`), the `url` (which must match the batch endpoint), and the request `body` — the exact JSON you'd send to that endpoint in real time.
```jsonl Chat completions theme={null}
{"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "deepseek-ai/DeepSeek-V3", "messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 100}}
{"custom_id": "req-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "deepseek-ai/DeepSeek-V3", "messages": [{"role": "user", "content": "Write a haiku about batching."}], "max_tokens": 100}}
```
```jsonl Embeddings theme={null}
{"custom_id": "doc-1", "method": "POST", "url": "/v1/embeddings", "body": {"model": "Qwen/Qwen3-Embedding-8B", "input": "The food was delicious and the waiter...", "encoding_format": "float"}}
{"custom_id": "doc-2", "method": "POST", "url": "/v1/embeddings", "body": {"model": "Qwen/Qwen3-Embedding-8B", "input": ["first text", "second text"], "encoding_format": "float"}}
```
A few rules:
* **`custom_id` must be unique** across all requests in a file. It's how you match results back to requests, since output order isn't guaranteed.
* **`method` must be `POST`.**
* **`url` must equal the batch's `endpoint`** — you can't mix endpoints in one file.
* **All requests must use the same model.**
* Use a DeepInfra model id (e.g. `deepseek-ai/DeepSeek-V3`), not an OpenAI model name.
* **`body` must match the request format of the corresponding endpoint** — it's the exact JSON you'd send to that endpoint in real time. See [Chat Completions](/chat/overview), [Text Completions](/apis/completions), or [Embeddings](/apis/embeddings).
## Step 2 — Upload the file
Upload the JSONL file with `purpose="batch"`. In return you get a [FileObject](/batch/file-endpoints#the-fileobject) containing the `id` of the uploaded file. For more information on how to upload a file, see [Create file](/batch/file-endpoints#create-file).
```python Python theme={null}
from openai import OpenAI
client = OpenAI(
api_key="$DEEPINFRA_TOKEN",
base_url="https://api.deepinfra.com/v1/openai",
)
batch_input_file = client.files.create(
file=open("requests.jsonl", "rb"),
purpose="batch",
)
print(batch_input_file.id)
```
```javascript JavaScript theme={null}
import fs from "fs";
import OpenAI from "openai";
const client = new OpenAI({
apiKey: "$DEEPINFRA_TOKEN",
baseURL: "https://api.deepinfra.com/v1/openai",
});
const batchInputFile = await client.files.create({
file: fs.createReadStream("requests.jsonl"),
purpose: "batch",
});
console.log(batchInputFile.id);
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/files" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-F purpose="batch" \
-F file="@requests.jsonl"
```
## Step 3 — Create the batch
Create the batch job from the uploaded input file, choosing the `endpoint` to run against. The batch starts executing as soon as it's created. For the exact details of creating a batch request, see [Create a batch](/batch/batch-endpoints#create-a-batch).
```python Python theme={null}
batch = client.batches.create(
input_file_id=batch_input_file.id,
endpoint="/v1/chat/completions",
completion_window="24h",
metadata={"description": "nightly eval run"},
output_expires_after={"anchor": "created_at", "seconds": 604800},
)
print(batch.id, batch.status)
```
```javascript JavaScript theme={null}
const batch = await client.batches.create({
input_file_id: batchInputFile.id,
endpoint: "/v1/chat/completions",
completion_window: "24h",
metadata: { description: "nightly eval run" },
output_expires_after: { anchor: "created_at", seconds: 604800 },
});
console.log(batch.id, batch.status);
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/batches" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"input_file_id": "file_abc123",
"endpoint": "/v1/chat/completions",
"completion_window": "24h",
"metadata": {"description": "nightly eval run"},
"output_expires_after": {"anchor": "created_at", "seconds": 604800}
}'
```
## Step 4 — Check status
A batch runs asynchronously, so after creating it you can check its status to know when the results are ready. Retrieve the batch periodically and watch its `status` until it reaches a terminal state — `completed`, `failed`, `expired`, or `cancelled`. For details on retrieving a batch, see [Retrieve a batch](/batch/batch-endpoints#retrieve-a-batch).
```python Python theme={null}
batch = client.batches.retrieve(batch.id)
print(batch.status)
print(batch.request_counts) # total / completed / failed
```
```javascript JavaScript theme={null}
const updated = await client.batches.retrieve(batch.id);
console.log(updated.status);
console.log(updated.request_counts); // total / completed / failed
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/batches/batch_abc123" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN"
```
A batch moves through several statuses — see [Batch status](/batch/batch-objects#batch-status) for what each one means.
You can track progress by checking the `usage` and `request_counts` fields when checking status.
Once the batch reaches a terminal state, the output and error files will be available, if they contain any information.
## Step 5 — Download the results
Once the batch reaches a terminal state, you can get either the [`errors`](/batch/batch-objects#errors) field, or the output and error files (`output_file_id` and `error_file_id`) from the [Batch object](/batch/batch-objects), depending on the state.
You can download the output and error files using the [Files API](/batch/file-endpoints#retrieve-file-content).
```python Python theme={null}
batch = client.batches.retrieve(batch.id)
# Successful responses
output = client.files.content(batch.output_file_id)
print(output.text)
# Failed / cancelled requests, if any
if batch.error_file_id:
errors = client.files.content(batch.error_file_id)
print(errors.text)
```
```javascript JavaScript theme={null}
const done = await client.batches.retrieve(batch.id);
const output = await client.files.content(done.output_file_id);
console.log(await output.text());
if (done.error_file_id) {
const errors = await client.files.content(done.error_file_id);
console.log(await errors.text());
}
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/files/file_out456/content" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN"
```
Each result line carries the `custom_id` from the input so you can match it back to your request. Successful lines have a `response`; failed lines have an `error`. The `response` has a `body` field with the same format that the real-time API would return. The `error` has `code` and `message` fields that better describe why the line failed.
```jsonl theme={null}
{"id": "batch_req_xyz", "custom_id": "req-1", "response": {"status_code": 200, "body": {"choices": [{"message": {"role": "assistant", "content": "Hello!"}}]}}, "error": null}
{"id": "batch_req_abc", "custom_id": "req-2", "response": null, "error": {"code": "invalid_request", "message": "..."}}
```
## Cancel a batch
You can cancel a batch at any point while its status is non-terminal. Cancelling a batch puts it in the `cancelling` status for some time, after which it moves to `cancelled`.
```python Python theme={null}
client.batches.cancel(batch.id)
```
```javascript JavaScript theme={null}
await client.batches.cancel(batch.id);
```
```bash cURL theme={null}
curl -X POST "https://api.deepinfra.com/v1/openai/batches/batch_abc123/cancel" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN"
```
## Pricing
Batch requests are billed at **20% less** than the corresponding real-time price for the same model and endpoint. The discount is applied automatically — there's nothing extra to configure.
## Rate limits
Batch requests and usage do not affect your real-time rate limits.
There are additional batch-related rate limits:
| Limit | Value |
| --------------------------- | ------ |
| Requests (lines) per file | 50,000 |
| Input file size | 200 MB |
| Embedding inputs per file | 50,000 |
| Concurrent batches per user | 100 |
## Related
* [Chat Completions](/chat/overview)
* [Text Completions](/apis/completions)
* [Embeddings](/apis/embeddings)
# Log Probabilities
Source: https://docs.deepinfra.com/chat/log-probs
Get per-token log probabilities from LLM responses.
You can retrieve the log probability of each generated token. This is useful for uncertainty estimation, token-level filtering, confidence scoring, or building custom sampling logic.
Log probabilities are supported across all request modes:
* **OpenAI-compatible API** — using `logprobs` and `top_logprobs` parameters
* **DeepInfra Native API** — streaming and non-streaming
## OpenAI-compatible API
Set `logprobs: true` in your request. Optionally set `top_logprobs` (1–20) to also get the top alternative tokens at each position.
```python Python theme={null}
from openai import OpenAI
client = OpenAI(
api_key="$DEEPINFRA_TOKEN",
base_url="https://api.deepinfra.com/v1/openai",
)
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3",
messages=[{"role": "user", "content": "Say hello in one word"}],
logprobs=True,
top_logprobs=3,
)
for token in response.choices[0].logprobs.content:
print(f"{token.token!r}: {token.logprob:.4f}")
for alt in token.top_logprobs:
print(f" alt {alt.token!r}: {alt.logprob:.4f}")
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"model": "deepseek-ai/DeepSeek-V3",
"messages": [{"role": "user", "content": "Say hello in one word"}],
"logprobs": true,
"top_logprobs": 3
}'
```
Response structure:
```json theme={null}
{
"choices": [{
"logprobs": {
"content": [
{
"token": "Hello",
"logprob": -0.0023,
"top_logprobs": [
{"token": "Hello", "logprob": -0.0023},
{"token": "Hi", "logprob": -1.42},
{"token": "Hey", "logprob": -3.87}
]
}
]
}
}]
}
```
## DeepInfra Native API (streaming)
The native streaming API returns log probabilities inline with each token as it is generated.
```bash theme={null}
curl -X POST \
-d '{"input": "I have this dream", "stream": true}' \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-H 'Content-Type: application/json' \
'https://api.deepinfra.com/v1/inference/deepseek-ai/DeepSeek-V3'
```
Response (streamed):
```
data: {"token": {"id": 29892, "text": ",", "logprob": -2.65625, "special": false}, "generated_text": null, "details": null}
data: {"token": {"id": 988, "text": " where", "logprob": -0.39575195, "special": false}, "generated_text": null, "details": null}
data: {"token": {"id": 1432, "text": " every", "logprob": -3.15625, "special": false}, "generated_text": null, "details": null}
data: {"token": {"id": 931, "text": " time", "logprob": -0.1385498, "special": false}, "generated_text": null, "details": null}
```
The `logprob` field is the log probability of the generated token (base e). Lower (more negative) values indicate less likely tokens.
# Chat Completions
Source: https://docs.deepinfra.com/chat/overview
OpenAI-compatible chat completions API — just change the base URL and model name.
DeepInfra offers an OpenAI-compatible chat completions API for all [LLM models](https://deepinfra.com/models/text-generation) at the best prices for open-source model inference. For other model types (embeddings, image generation, speech, reranking, and more), see [More APIs](/apis/completions). The endpoint is:
```
https://api.deepinfra.com/v1/openai
```
The only changes you need to make from your existing OpenAI code:
1. Set `base_url` to `https://api.deepinfra.com/v1/openai`
2. Set `api_key` to your DeepInfra token
3. Set `model` to a model from [our catalog](https://deepinfra.com/models)
## Install the SDK
```bash Python theme={null}
pip install openai
```
```bash JavaScript theme={null}
npm install openai
```
## Basic chat completion
```python Python theme={null}
from openai import OpenAI
openai = OpenAI(
api_key="$DEEPINFRA_TOKEN",
base_url="https://api.deepinfra.com/v1/openai",
)
chat_completion = openai.chat.completions.create(
model="deepseek-ai/DeepSeek-V3",
messages=[{"role": "user", "content": "Hello"}],
)
print(chat_completion.choices[0].message.content)
print(chat_completion.usage.prompt_tokens, chat_completion.usage.completion_tokens)
```
```javascript JavaScript theme={null}
import OpenAI from "openai";
const openai = new OpenAI({
apiKey: "$DEEPINFRA_TOKEN",
baseURL: "https://api.deepinfra.com/v1/openai",
});
const completion = await openai.chat.completions.create({
messages: [{ role: "user", content: "Hello" }],
model: "deepseek-ai/DeepSeek-V3",
});
console.log(completion.choices[0].message.content);
console.log(completion.usage.prompt_tokens, completion.usage.completion_tokens);
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"model": "deepseek-ai/DeepSeek-V3",
"messages": [
{
"role": "user",
"content": "Hello!"
}
]
}'
```
## Multi-turn conversations
To create a longer conversation, include the full message history in every request. The model uses this context to provide better answers.
```python Python theme={null}
from openai import OpenAI
openai = OpenAI(
api_key="$DEEPINFRA_TOKEN",
base_url="https://api.deepinfra.com/v1/openai",
)
chat_completion = openai.chat.completions.create(
model="deepseek-ai/DeepSeek-V3",
messages=[
{"role": "system", "content": "Respond like a michelin starred chef."},
{"role": "user", "content": "Can you name at least two different techniques to cook lamb?"},
{"role": "assistant", "content": "Bonjour! Let me tell you, my friend, cooking lamb is an art form..."},
{"role": "user", "content": "Tell me more about the second method."},
],
)
print(chat_completion.choices[0].message.content)
```
```javascript JavaScript theme={null}
import OpenAI from "openai";
const openai = new OpenAI({
baseURL: "https://api.deepinfra.com/v1/openai",
apiKey: "$DEEPINFRA_TOKEN",
});
const completion = await openai.chat.completions.create({
messages: [
{role: "system", content: "Respond like a michelin starred chef."},
{role: "user", content: "Can you name at least two different techniques to cook lamb?"},
{role: "assistant", content: "Bonjour! Let me tell you, my friend, cooking lamb is an art form..."},
{role: "user", content: "Tell me more about the second method."}
],
model: "deepseek-ai/DeepSeek-V3",
});
console.log(completion.choices[0].message.content);
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"model": "deepseek-ai/DeepSeek-V3",
"messages": [
{"role": "system", "content": "Respond like a michelin starred chef."},
{"role": "user", "content": "Can you name at least two different techniques to cook lamb?"},
{"role": "assistant", "content": "Bonjour! Let me tell you..."},
{"role": "user", "content": "Tell me more about the second method."}
]
}'
```
The longer the conversation, the more tokens it uses. The maximum conversation length is determined by the model's context size.
## Supported parameters
| Parameter | Notes |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `model` | Model name, or `MODEL_NAME:VERSION`, or `deploy_id:DEPLOY_ID` |
| `messages` | Roles: `system`, `user`, `assistant` |
| `max_tokens` | Max tokens to generate. See [Max output tokens](#max-output-tokens) |
| `stream` | See [Streaming](/chat/streaming) |
| `temperature` | Sampling temperature between 0 and 2. Higher values produce more random output; lower values more deterministic. Default: `1.0` |
| `top_p` | Nucleus sampling threshold — only tokens comprising the top `top_p` probability mass are considered. Default: `1.0` |
| `stop` | Up to 4 sequences where the API will stop generating further tokens |
| `n` | Number of completion sequences to return. Default: `1` |
| `presence_penalty` | Penalizes tokens that have already appeared in the text, encouraging the model to discuss new topics. Range: -2.0 to 2.0. Default: `0` |
| `frequency_penalty` | Penalizes tokens based on how often they've appeared so far, reducing repetition. Range: -2.0 to 2.0. Default: `0` |
| `response_format` | See [Structured Outputs](/chat/structured-outputs) |
| `tools`, `tool_choice` | See [Tool Calling](/chat/tool-calling) |
| `service_tier` | Select a service tier (`"priority"` or `"flex"`) for tagged models. See [Service Tier](#service-tier) below. |
| `fail_fast` | Reject with HTTP 429 instead of queueing when the model is at capacity. See [Fail fast](#fail-fast) below. Default: `false` |
| `reasoning_effort` | Controls reasoning depth for reasoning models. See [Reasoning Models](/chat/reasoning). |
We may not be 100% compatible with all OpenAI parameters. Let us know on Discord or by email if something you need is missing.
For the complete parameter reference, see the [API reference](/api-reference/chat-completions/openai-chat-completions).
## Service tier
Set the optional `service_tier` parameter to run a request on a non-standard tier. Two tiers are available on tagged models: **priority** (faster, at a premium) and **flex** (cheaper, best-effort). Leave `service_tier` unset for standard real-time scheduling and pricing.
### Priority
Set `service_tier` to `"priority"` to request priority inference on supported models. Priority requests get faster time-to-first-token and higher throughput during peak demand.
Priority inference incurs a 50% surcharge on top of the model's standard per-token price.
```python Python theme={null}
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3",
messages=[{"role": "user", "content": "Hello!"}],
extra_body={"service_tier": "priority"},
)
```
```javascript JavaScript theme={null}
const response = await openai.chat.completions.create({
model: "deepseek-ai/DeepSeek-V3",
messages: [{ role: "user", content: "Hello!" }],
service_tier: "priority",
});
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"model": "deepseek-ai/DeepSeek-V3",
"service_tier": "priority",
"messages": [
{
"role": "user",
"content": "Hello!"
}
]
}'
```
### Flex
Set `service_tier` to `"flex"` to run Chat Completions requests at a lower cost in exchange for slower response times and occasional resource unavailability. It's ideal for non-production or lower-priority tasks such as model evaluations, data enrichment, and asynchronous workloads. When a model is busy, a flex request may wait up to 10 minutes for available capacity before it runs or is rejected with an HTTP 429, so use it for work you can retry.
Flex inference is billed at a 20% discount off the model's standard per-token price.
```python Python theme={null}
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3",
messages=[{"role": "user", "content": "Hello!"}],
extra_body={"service_tier": "flex"},
)
```
```javascript JavaScript theme={null}
const response = await openai.chat.completions.create({
model: "deepseek-ai/DeepSeek-V3",
messages: [{ role: "user", content: "Hello!" }],
service_tier: "flex",
});
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"model": "deepseek-ai/DeepSeek-V3",
"service_tier": "flex",
"messages": [
{
"role": "user",
"content": "Hello!"
}
]
}'
```
The response includes a `service_tier` field confirming which tier was actually used. Not all models support these tiers — check the model page for availability. If a model doesn't support the requested tier, the request is served at the standard tier and billed at the standard price; no error is returned.
## Fail fast
By default, a request sent to a model that is at capacity waits in the queue until capacity frees up. Set the optional `fail_fast` parameter to `true` to get an immediate HTTP 429 instead of waiting.
This is meant for latency-sensitive callers that would rather go somewhere else than sit in a queue — for example, clients that fail over to another provider on rejection. The 429 arrives as soon as the request would have been queued, so you don't spend your latency budget waiting.
```python Python theme={null}
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3",
messages=[{"role": "user", "content": "Hello!"}],
extra_body={"fail_fast": True},
)
```
```javascript JavaScript theme={null}
const response = await openai.chat.completions.create({
model: "deepseek-ai/DeepSeek-V3",
messages: [{ role: "user", content: "Hello!" }],
fail_fast: true,
});
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"model": "deepseek-ai/DeepSeek-V3",
"fail_fast": true,
"messages": [
{
"role": "user",
"content": "Hello!"
}
]
}'
```
Rejection is capacity-aware rather than backlog-triggered: a `fail_fast` request is rejected only when the model is busy enough that serving it would actually be slow. Whenever there is spare capacity the request is admitted and served exactly as if `fail_fast` were unset, so enabling it does not cost you throughput on an idle model.
When a request is rejected, the response is an HTTP 429 carrying the `engine_overloaded` code:
```json theme={null}
{
"error": {
"message": "Model busy, retry later",
"type": "invalid_request_error",
"param": null,
"code": "engine_overloaded"
}
}
```
A rejected request never reaches the model, so no inference happens and nothing is billed.
If you set both `fail_fast: true` and `service_tier: "priority"`, `fail_fast` wins — the explicit request not to wait is honored, and you get a 429 rather than a priority slot in the queue.
## Max output tokens
The maximum number of tokens that can be generated in a single response is model-dependent, with a hard cap of 16384 tokens for most models. Set `max_tokens` to control the limit for a specific request.
### Continuing responses beyond the limit
If you need a longer response, use response continuation: send a follow-up request with the previous response included as an assistant message, and the model will continue from where it left off.
```bash theme={null}
curl "https://api.deepinfra.com/v1/openai/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"model": "deepseek-ai/DeepSeek-V3",
"messages": [
{"role": "user", "content": "Write a very long essay about AI."},
{"role": "assistant", "content": ""}
],
"max_tokens": 4096
}'
```
Note: response continuation cannot extend past the model's total context window. A 400 error is returned when the total context size is exceeded.
## What's next
Stream tokens as they're generated.
Get responses in JSON format.
Give models access to external functions.
Send images alongside text.
Control chain-of-thought reasoning behavior.
# Prompt Cache Retention
Source: https://docs.deepinfra.com/chat/prompt-cache-retention
Retain a prompt prefix for 5 minutes or 1 hour so reuse skips prefill and bills at the cache-read rate.
[Prompt caching](/chat/prompt-caching) reuses the KV cache from a recent request when the beginning of your prompt matches — automatically, and best-effort. **Prompt cache retention** goes further: it lets you explicitly keep a prompt prefix cached for a fixed window — **5 minutes** or **1 hour** — so reuse is guaranteed for that window instead of depending on whether the prefix happens to still be warm.
While a prefix is retained, every request that reuses it **skips prefill** for a faster time to first token and is billed at the discounted **cache-read** rate. Opening the window costs a small **cache-write** premium upfront.
## Quick start
Send `prompt_cache_key` with every request in a session. Add `prompt_cache_options` on the first one to open the window.
```python Python theme={null}
from openai import OpenAI
client = OpenAI(
api_key="$DEEPINFRA_TOKEN",
base_url="https://api.deepinfra.com/v1/openai",
)
resp = client.chat.completions.create(
model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B",
messages=messages, # your large, reused context
extra_body={
"prompt_cache_key": "agent-session-123",
"prompt_cache_options": {"mode": "explicit", "ttl": "1h"},
},
)
print(resp.usage.prompt_tokens_details)
```
```javascript JavaScript theme={null}
import OpenAI from "openai";
const openai = new OpenAI({
apiKey: "$DEEPINFRA_TOKEN",
baseURL: "https://api.deepinfra.com/v1/openai",
});
const resp = await openai.chat.completions.create({
model: "nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B",
messages, // your large, reused context
prompt_cache_key: "agent-session-123",
prompt_cache_options: { mode: "explicit", ttl: "1h" },
});
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"model": "nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B",
"messages": [{"role": "system", "content": "..."}],
"prompt_cache_key": "agent-session-123",
"prompt_cache_options": {"mode": "explicit", "ttl": "1h"}
}'
```
## Request parameters
| Parameter | Type | Description |
| ------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `prompt_cache_key` | `string` | A stable identifier for the context you're caching (for example a session or agent id). Required for retention; reuse is matched on this key plus the prompt content. |
| `prompt_cache_options.mode` | `string` | Currently `"explicit"`. |
| `prompt_cache_options.ttl` | `"5m"` \| `"1h"` | Retention window. Sending it opens the window, or extends an open one. Omit it on reuse requests. |
| `prompt_cache_breakpoint.mode` | `string` | Set to `"explicit"` on a message content part to end the retained prefix there. See [Cache breakpoints](#cache-breakpoints). |
## Window lifecycle
| Request | Effect |
| ------------------------------------ | ------------------------------------------------------------------------------------------------------ |
| First request with a `ttl` | Writes the cache and starts the clock. Still prefills; billed the write premium on the cached portion. |
| Same key, no `ttl` | Reuses the retained prefix at the cache-read rate. Does not move the deadline. |
| Same key, with a `ttl` | Reuses the cache *and* pushes the deadline out. Billed the write premium again. |
| Shorter `ttl` inside a longer window | Never shortens the window. |
| After expiry | Reuse falls back to standard input pricing — no silent charges. |
Retention is scoped to your account **and** `prompt_cache_key`. It works with streaming and non-streaming, chat and text-completions.
You're charged a cache write each time a request writes new blocks **or extends the window** — not just on the first request. For example, send the same prompt with a `ttl` twice and the second call extends the window, so it's charged as another cache write. Send a `ttl` only to open the window or when you deliberately want to extend it; for ordinary reuse, omit `ttl` and pay the cheaper cache-read rate.
## Cache breakpoints
Most prompts are a stable prefix (system instructions, tools, a document) followed by a variable tail. Mark where the reusable prefix ends with a `prompt_cache_breakpoint` on a message content part — retention then applies to everything up to and including that part, and ignores the variable remainder, so the retained cache stays stable across requests even as the question changes.
```json theme={null}
{
"messages": [
{ "role": "system", "content": [
{ "type": "text",
"text": "... large stable system prompt, tools, and reference context ...",
"prompt_cache_breakpoint": { "mode": "explicit" } }
]},
{ "role": "user", "content": "... the variable question — not retained ..." }
],
"prompt_cache_key": "agent-session-123",
"prompt_cache_options": { "mode": "explicit", "ttl": "1h" }
}
```
Put anything that changes between calls (timestamps, user ids, retrieved chunks) **after** the breakpoint, so the prefix before it stays identical and matches in full. Changing a token inside the retained prefix only recomputes from the point of divergence onward — you still get the cache-read rate on the portion that still matches.
## Cache granularity
Caches are written in fixed increments, so the retained portion is always **rounded down** to a whole multiple of the model's cache granularity. Whatever is left over is billed as standard input.
The granularity differs per model. On Nemotron-3-Ultra it is **8,192 tokens**: a 20,000-token prompt retains 16,384 tokens (two increments), and the remaining 3,616 are billed as standard input. A breakpoint shorter than one increment retains nothing.
Check the model page for the granularity of the model you're using — it determines how much of your prompt is actually cacheable.
## Reading the response
Every response reports what happened in `usage.prompt_tokens_details`.
| Field | Type | Description |
| -------------------- | --------- | ------------------------------------------------------------------------------------------------------- |
| `cache_write_tokens` | `integer` | Tokens written to cache, billed at the retention write rate. The remainder is billed as standard input. |
| `cached_tokens` | `integer` | Tokens reused from a retained prefix, billed at the cache-read rate. |
```json Write (first request) theme={null}
"usage": {
"prompt_tokens": 34375,
"total_tokens": 34495,
"completion_tokens": 120,
"prompt_tokens_details": {
"cached_tokens": 0,
"cache_write_tokens": 32768
}
}
```
```json Reuse (later request) theme={null}
"usage": {
"prompt_tokens": 34380,
"total_tokens": 34475,
"completion_tokens": 95,
"prompt_tokens_details": {
"cached_tokens": 32768,
"cache_write_tokens": 0
}
}
```
## Pricing
Relative to the model's standard input price:
| Action | Rate |
| ------------------------------------------ | ------------------------------------------------------------------ |
| Reuse a retained prefix (cache **read**) | model's cache-read rate (e.g. **0.2×** input for Nemotron-3-Ultra) |
| Retain for **5 minutes** (cache **write**) | **1.25×** input |
| Retain for **1 hour** (cache **write**) | **2.0×** input |
| Non-retained input | 1× (standard) |
Only whole cacheable blocks count as cache read/write; any remainder is billed as standard input. The write premium applies only when a request actually **creates or extends** the retention window — reuse inside a window you've already paid for is billed at the read rate.
## Model support
| Model | Retention |
| ------------------------------------------ | --------- |
| `nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B` | 5m, 1h |
| `moonshotai/Kimi-K2.7-Code` | 5m, 1h |
More models to follow. On a model without retention support, `prompt_cache_options` is ignored and the request is billed as standard input.
# Prompt Caching
Source: https://docs.deepinfra.com/chat/prompt-caching
Reduce latency and cost by caching repeated prompt prefixes.
Prompt caching allows DeepInfra to reuse the KV (key-value) cache from previous requests when the beginning of your prompt is identical. This reduces both latency and cost for workloads that repeatedly send the same prefix — such as a long system prompt, a large document, or a fixed set of few-shot examples.
## How it works
When you send a request, DeepInfra checks whether the beginning of your prompt matches a cached prefix from a recent request on the same model. If it does, the cached KV state is reused instead of recomputing it, which:
* **Reduces time-to-first-token** — the model skips processing the cached portion
* **Lowers cost** — cached input tokens are billed at a reduced rate
## Usage
Prompt caching is **automatic** — no extra parameters required. Just structure your prompts so that the reused content appears at the beginning.
```python theme={null}
from openai import OpenAI
client = OpenAI(
api_key="$DEEPINFRA_TOKEN",
base_url="https://api.deepinfra.com/v1/openai",
)
# Long system prompt that stays the same across requests
SYSTEM_PROMPT = """You are a helpful AI assistant with deep expertise in Python.
[... thousands of tokens of instructions or context ...]
"""
# First request — full processing
response1 = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": "How do I use list comprehensions?"},
],
)
# Second request — cached prefix reused, faster and cheaper
response2 = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": "What are Python generators?"},
],
)
```
## Best practices
**Put stable content first.** The cache matches from the beginning of the prompt. Place your system prompt, documents, and few-shot examples before the user's message.
**Keep the prefix identical.** Even a single character difference will invalidate the cache. Avoid dynamic content (timestamps, user IDs, etc.) in the cacheable prefix.
**Longer prefixes save more.** Prompt caching is most effective with long, repeated prefixes — think multi-page documents, long system prompts, or RAG context.
## Common use cases
| Use case | Cached prefix |
| ------------------------------------ | ------------------- |
| Chatbot with a long system prompt | System prompt |
| RAG / document Q\&A | Retrieved documents |
| Few-shot classification | Examples |
| Code assistant with a large codebase | Codebase context |
| Multi-turn conversation | Previous turns |
## Checking cache usage
The response usage object indicates how many tokens were served from cache:
```json theme={null}
{
"usage": {
"prompt_tokens": 5000,
"completion_tokens": 50,
"total_tokens": 5050,
"prompt_tokens_details": {
"cached_tokens": 4800
}
}
}
```
In this example, 4800 of the 5000 input tokens were cached.
## Explicit cache keys
By default, caching is automatic based on prefix matching. The `prompt_cache_key` parameter lets you explicitly tag a request with a cache key, improving cache hit rates when your prompts share the same logical content but differ slightly in formatting or ordering.
We recommend using a **session-scoped key** like `userid-chatsessionid` (e.g. `"user123-chat456"`). Within a single chat session, the conversation history grows incrementally — each new request reuses all previous turns plus one new message. A per-session cache key ensures these near-identical prompts always hit the cache.
```python Python theme={null}
from openai import OpenAI
client = OpenAI(
api_key="$DEEPINFRA_TOKEN",
base_url="https://api.deepinfra.com/v1/openai",
)
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3",
messages=[
{"role": "system", "content": "You are a helpful coding assistant."},
{"role": "user", "content": "How do I use async/await?"},
],
extra_body={"prompt_cache_key": "user123-chat456"},
)
```
```javascript JavaScript theme={null}
import OpenAI from "openai";
const openai = new OpenAI({
apiKey: "$DEEPINFRA_TOKEN",
baseURL: "https://api.deepinfra.com/v1/openai",
});
const response = await openai.chat.completions.create({
model: "deepseek-ai/DeepSeek-V3",
messages: [
{ role: "system", content: "You are a helpful coding assistant." },
{ role: "user", content: "How do I use async/await?" },
],
prompt_cache_key: "user123-chat456",
});
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"model": "deepseek-ai/DeepSeek-V3",
"prompt_cache_key": "user123-chat456",
"messages": [
{
"role": "system",
"content": "You are a helpful coding assistant."
},
{
"role": "user",
"content": "How do I use async/await?"
}
]
}'
```
Requests with the same `prompt_cache_key` and model will share a KV cache, even if their prompt prefixes aren't byte-for-byte identical.
| Parameter | Type | Description |
| ------------------ | -------- | ---------------------------------------------------------------------------------------- |
| `prompt_cache_key` | `string` | An explicit key for cache lookup. Requests with the same key and model share a KV cache. |
Need a prefix to stay cached for a guaranteed window? [Prompt cache retention](/chat/prompt-cache-retention) lets you retain a prompt for 5 minutes or 1 hour with `prompt_cache_options`, billed at a cache-write premium upfront and reused at the cache-read rate.
## Notes
* Prompt caching is available on supported models — check the model page for details
* Cache entries expire after a period of inactivity
* Caches are per-model and per-account
* When using `prompt_cache_key`, the key is scoped per-model and per-account
# Reasoning Models
Source: https://docs.deepinfra.com/chat/reasoning
Configure chain-of-thought reasoning with reasoning_effort and the reasoning parameter.
Some models on DeepInfra support extended chain-of-thought reasoning — the model "thinks through" a problem step by step before producing a final answer. By default, reasoning models produce a reasoning trace alongside the response. You can control this behavior with the `reasoning_effort` parameter.
## Supported models
Reasoning is available on models that support chain-of-thought, including:
* `deepseek-ai/DeepSeek-R1`
Check the [model catalog](https://deepinfra.com/models) for the latest list.
## Controlling reasoning effort
Use `reasoning_effort` to control how much reasoning the model performs. Higher effort means deeper thinking but more output tokens and higher latency.
```python Python theme={null}
from openai import OpenAI
client = OpenAI(
api_key="$DEEPINFRA_TOKEN",
base_url="https://api.deepinfra.com/v1/openai",
)
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-R1",
messages=[{"role": "user", "content": "Prove that the square root of 2 is irrational."}],
extra_body={"reasoning_effort": "high"},
)
print(response.choices[0].message.content)
```
```javascript JavaScript theme={null}
import OpenAI from "openai";
const openai = new OpenAI({
apiKey: "$DEEPINFRA_TOKEN",
baseURL: "https://api.deepinfra.com/v1/openai",
});
const response = await openai.chat.completions.create({
model: "deepseek-ai/DeepSeek-R1",
messages: [{ role: "user", content: "Prove that the square root of 2 is irrational." }],
reasoning_effort: "high",
});
console.log(response.choices[0].message.content);
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"model": "deepseek-ai/DeepSeek-R1",
"reasoning_effort": "high",
"messages": [
{
"role": "user",
"content": "Prove that the square root of 2 is irrational."
}
]
}'
```
## Disabling reasoning
Set `reasoning_effort` to `"none"` to disable chain-of-thought entirely. The model will respond directly without a reasoning trace — faster and cheaper.
```python Python theme={null}
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-R1",
messages=[{"role": "user", "content": "What is the capital of France?"}],
extra_body={"reasoning_effort": "none"},
)
```
```javascript JavaScript theme={null}
const response = await openai.chat.completions.create({
model: "deepseek-ai/DeepSeek-R1",
messages: [{ role: "user", content: "What is the capital of France?" }],
reasoning_effort: "none",
});
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"model": "deepseek-ai/DeepSeek-R1",
"reasoning_effort": "none",
"messages": [
{
"role": "user",
"content": "What is the capital of France?"
}
]
}'
```
## The `reasoning` parameter
For more granular control, use the `reasoning` object instead of `reasoning_effort`:
```python Python theme={null}
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-R1",
messages=[{"role": "user", "content": "Solve this step by step: 15! / 13!"}],
extra_body={
"reasoning": {
"effort": "medium",
"enabled": True,
}
},
)
```
```javascript JavaScript theme={null}
const response = await openai.chat.completions.create({
model: "deepseek-ai/DeepSeek-R1",
messages: [{ role: "user", content: "Solve this step by step: 15! / 13!" }],
reasoning: {
effort: "medium",
enabled: true,
},
});
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"model": "deepseek-ai/DeepSeek-R1",
"reasoning": {
"effort": "medium",
"enabled": true
},
"messages": [
{
"role": "user",
"content": "Solve this step by step: 15! / 13!"
}
]
}'
```
Setting `"enabled": false` is equivalent to `reasoning_effort: "none"`.
## When to use reasoning
| Use case | Recommended setting |
| --------------------------------------- | --------------------------------------- |
| Math, logic, and code problems | `"high"` (default for reasoning models) |
| Multi-step analysis | `"medium"` or `"high"` |
| Simple Q\&A, translation, summarization | `"none"` |
| Cost-sensitive workloads | `"none"` or `"low"` |
## Supported parameters
| Parameter | Type | Description |
| ------------------- | --------- | ------------------------------------------------------------------ |
| `reasoning_effort` | `string` | Controls reasoning depth: `"none"`, `"low"`, `"medium"`, `"high"`. |
| `reasoning` | `object` | Fine-grained reasoning config. |
| `reasoning.effort` | `string` | Same values as `reasoning_effort`. |
| `reasoning.enabled` | `boolean` | Explicitly enable or disable reasoning. |
## Notes
* Reasoning tokens count toward output token billing
* Disabling reasoning on a reasoning model makes it behave like a standard chat model
* `reasoning_effort: "none"` is equivalent to `reasoning: { enabled: false }`
* Not all models support reasoning — using these parameters on a non-reasoning model has no effect
Full chat completions API reference.
Stream reasoning responses token by token.
Cache long prompts for faster reasoning.
# Streaming
Source: https://docs.deepinfra.com/chat/streaming
Stream chat completion responses token by token using server-sent events.
DeepInfra supports streaming responses via server-sent events (SSE), the same protocol as OpenAI. Set `stream: true` in your request to enable it.
## Examples
```python Python theme={null}
from openai import OpenAI
openai = OpenAI(
api_key="$DEEPINFRA_TOKEN",
base_url="https://api.deepinfra.com/v1/openai",
)
stream = openai.chat.completions.create(
model="deepseek-ai/DeepSeek-V3",
messages=[{"role": "user", "content": "Hello"}],
stream=True,
)
for event in stream:
if event.choices[0].finish_reason:
print(event.choices[0].finish_reason,
event.usage['prompt_tokens'],
event.usage['completion_tokens'])
else:
print(event.choices[0].delta.content, end="", flush=True)
```
```javascript JavaScript theme={null}
import OpenAI from "openai";
const openai = new OpenAI({
apiKey: "$DEEPINFRA_TOKEN",
baseURL: "https://api.deepinfra.com/v1/openai",
});
const completion = await openai.chat.completions.create({
messages: [{ role: "user", content: "Hello" }],
model: "deepseek-ai/DeepSeek-V3",
stream: true,
});
for await (const chunk of completion) {
if (chunk.choices[0].finish_reason) {
console.log(chunk.choices[0].finish_reason,
chunk.usage.prompt_tokens,
chunk.usage.completion_tokens);
} else {
process.stdout.write(chunk.choices[0].delta.content);
}
}
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"model": "deepseek-ai/DeepSeek-V3",
"stream": true,
"messages": [
{
"role": "user",
"content": "Hello!"
}
]
}'
```
## SSE format
Each streamed chunk is a `data:` line containing a JSON object:
```
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"delta":{"content":"Hello"},"finish_reason":null}]}
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":10,"completion_tokens":5}}
data: [DONE]
```
The final chunk before `[DONE]` contains usage information.
## Notes
* Streaming works for all supported models
* Usage stats are available in the last chunk (when `finish_reason` is set)
* The `completion_tokens` and `prompt_tokens` counts are the same as non-streaming
# Structured Outputs
Source: https://docs.deepinfra.com/chat/structured-outputs
Get model responses in JSON format using response_format.
In addition to text, the DeepInfra API can return responses in JSON format. This is supported in both our inference API and our OpenAI-compatible API, across [many of our models](https://deepinfra.com/models?q=json).
There are two modes:
| Mode | How to use | When to use |
| ------------- | ----------------------------------------------- | ---------------------------------- |
| `json_object` | `{"type": "json_object"}` | Any valid JSON object, schema-free |
| `json_schema` | `{"type": "json_schema", "json_schema": {...}}` | Enforces a strict output schema |
## json\_object mode
The simplest way to get JSON output. The model returns a valid JSON object but you don't control the exact shape.
```python Python theme={null}
import openai
import json
client = openai.OpenAI(
base_url="https://api.deepinfra.com/v1/openai",
api_key="$DEEPINFRA_TOKEN",
)
messages = [
{
"role": "user",
"content": "Provide a JSON list of 3 famous scientific breakthroughs in the past century, all of the countries which contributed, and in what year."
}
]
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3",
messages=messages,
response_format={"type": "json_object"},
)
print(response.choices[0].message.content)
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"model": "deepseek-ai/DeepSeek-V3",
"messages": [
{
"role": "user",
"content": "Provide a JSON list of 3 famous scientific breakthroughs."
}
],
"response_format": {"type": "json_object"}
}'
```
## json\_schema mode
Enforces a strict output schema using [JSON Schema](https://json-schema.org/). The model is constrained to produce only values that match your schema — useful when downstream code depends on a fixed structure.
```python Python theme={null}
import openai
import json
client = openai.OpenAI(
base_url="https://api.deepinfra.com/v1/openai",
api_key="$DEEPINFRA_TOKEN",
)
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3",
messages=[
{
"role": "user",
"content": "Extract the name, country, and year from: 'Alexander Fleming discovered Penicillin in the UK in 1928.'"
}
],
response_format={
"type": "json_schema",
"json_schema": {
"name": "breakthrough",
"strict": True,
"schema": {
"type": "object",
"properties": {
"name": {"type": "string"},
"country": {"type": "string"},
"year": {"type": "integer"}
},
"required": ["name", "country", "year"],
"additionalProperties": False
}
}
}
)
print(json.loads(response.choices[0].message.content))
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"model": "deepseek-ai/DeepSeek-V3",
"messages": [
{
"role": "user",
"content": "Extract the name, country, and year from: '\''Alexander Fleming discovered Penicillin in the UK in 1928.'\''"
}
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "breakthrough",
"strict": true,
"schema": {
"type": "object",
"properties": {
"name": {"type": "string"},
"country": {"type": "string"},
"year": {"type": "integer"}
},
"required": ["name", "country", "year"],
"additionalProperties": false
}
}
}
}'
```
Output:
```json theme={null}
{"name": "Penicillin", "country": "UK", "year": 1928}
```
## Tips
**Always prompt the model to produce JSON.** While not strictly required for `json_object`, mentioning the expected format in your prompt improves consistency.
**Prefer `json_schema` for production.** When your code depends on specific field names or types, `json_schema` with `"strict": true` eliminates shape surprises.
**Watch for truncation.** If the model stops due to `max_tokens` or `length`, the JSON may be incomplete. Always validate before parsing.
## Caveats
JSON mode can affect model alignment. When forced to produce structured output, some models are more likely to hallucinate values rather than say "I don't know." This is especially visible for prompts about real-time data (weather, stock prices, etc.).
Example: asking "What's the weather in San Francisco?" with JSON mode enabled may cause the model to fabricate a weather forecast rather than explaining it doesn't have real-time data.
**Best practices:**
* Use JSON mode for structured data extraction tasks, not for general question answering
* Keep prompts specific about the expected schema
* Validate model output before using it in production systems
* Use lower temperatures (\< 0.7) for more consistent structure
# Tool Calling
Source: https://docs.deepinfra.com/chat/tool-calling
Let models call external functions — the foundation of AI agents.
Tool calling (also known as function calling) is the most important capability for building AI agents. It lets models decide when to invoke external tools — web search, code execution, database queries, API calls — and seamlessly weave the results into a final response. Without reliable tool calling, agentic systems break down.
**Tool call accuracy is a top priority for us.** We invest significant engineering effort in ensuring that function call parsing, argument extraction, and round-trip reliability are correct across all supported models. In third-party benchmarks like the [K2-Vendor-Verifier evaluation](https://github.com/MoonshotAI/K2-Vendor-Verifier?tab=readme-ov-file#k2-0905-evaluation-results), DeepInfra achieves a top accuracy score for `moonshotai/Kimi-K2-Instruct` — among the highest of any provider tested.
We provide an OpenAI-compatible tool calling API. For more background, see the [DeepInfra blog](https://deepinfra.com/blog/function-calling-feature).
## Setup
```python Python theme={null}
import openai
import json
client = openai.OpenAI(
base_url="https://api.deepinfra.com/v1/openai",
api_key="$DEEPINFRA_TOKEN",
)
```
```javascript JavaScript theme={null}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.deepinfra.com/v1/openai",
apiKey: "$DEEPINFRA_TOKEN",
});
```
## Define your function
```python Python theme={null}
def get_current_weather(location):
"""Get the current weather in a given location"""
if "tokyo" in location.lower():
return json.dumps({"location": "Tokyo", "temperature": "75"})
elif "san francisco" in location.lower():
return json.dumps({"location": "San Francisco", "temperature": "60"})
elif "paris" in location.lower():
return json.dumps({"location": "Paris", "temperature": "70"})
else:
return json.dumps({"location": location, "temperature": "unknown"})
```
```javascript JavaScript theme={null}
async function get_current_weather(location) {
if (location.toLowerCase().includes("tokyo")) {
return JSON.stringify({"location": "Tokyo", "temperature": "75"});
} else if (location.toLowerCase().includes("san francisco")) {
return JSON.stringify({"location": "San Francisco", "temperature": "60"});
} else if (location.toLowerCase().includes("paris")) {
return JSON.stringify({"location": "Paris", "temperature": "70"});
} else {
return JSON.stringify({"location": location, "temperature": "unknown"});
}
}
```
## Step 1: Send tools to the model
```python Python theme={null}
tools = [{
"type": "function",
"function": {
"name": "get_current_weather",
"description": "Get the current weather in a given location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA"
}
},
"required": ["location"]
},
}
}]
messages = [{"role": "user", "content": "What is the weather in San Francisco?"}]
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3",
messages=messages,
tools=tools,
tool_choice="auto",
)
tool_calls = response.choices[0].message.tool_calls
for tool_call in tool_calls:
print(tool_call.model_dump())
```
```javascript JavaScript theme={null}
const tools = [{
"type": "function",
"function": {
"name": "get_current_weather",
"description": "Get the current weather in a given location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA"
}
},
"required": ["location"]
},
}
}];
const messages = [{"role": "user", "content": "What is the weather in San Francisco?"}];
const response = await client.chat.completions.create({
model: "deepseek-ai/DeepSeek-V3",
messages: messages,
tools: tools,
tool_choice: "auto",
});
const tool_calls = response.choices[0].message.tool_calls;
for (const tool_call of tool_calls) {
console.log(tool_call);
}
```
Output:
```
{'id': 'call_X0xYqdnoUonPJpQ6HEadxLHE', 'function': {'arguments': '{"location": "San Francisco"}', 'name': 'get_current_weather'}, 'type': 'function'}
```
## Step 2: Execute the function and send results back
```python Python theme={null}
# Extend conversation with assistant's reply
messages.append(response.choices[0].message)
for tool_call in tool_calls:
function_name = tool_call.function.name
if function_name == "get_current_weather":
function_args = json.loads(tool_call.function.arguments)
function_response = get_current_weather(
location=function_args.get("location")
)
messages.append({
"tool_call_id": tool_call.id,
"role": "tool",
"content": function_response,
})
# Get a new response from the model with function results
second_response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3",
messages=messages,
tools=tools,
tool_choice="auto",
)
print(second_response.choices[0].message.content)
```
```javascript JavaScript theme={null}
// Extend conversation with assistant's reply
messages.push(response.choices[0].message);
for (const tool_call of tool_calls) {
const function_name = tool_call.function.name;
if (function_name == "get_current_weather") {
const function_args = JSON.parse(tool_call.function.arguments);
const function_response = await get_current_weather(function_args.location);
messages.push({
"tool_call_id": tool_call.id,
"role": "tool",
"content": function_response,
});
}
}
const second_response = await client.chat.completions.create({
model: "deepseek-ai/DeepSeek-V3",
messages: messages,
tools: tools,
tool_choice: "auto",
});
console.log(second_response.choices[0].message.content);
```
Output:
```
The current temperature in San Francisco, CA is 60 degrees.
```
## Tips
* Write clear, detailed function descriptions — model quality depends heavily on them
* Use lower temperatures (\< 1.0) to avoid erratic parameter values
* Avoid system messages when using tool calling
* Model quality degrades with more functions — keep the list focused
* Keep `top_p` and `top_k` at their defaults
## Supported features
| Feature | Supported |
| --------------------- | -------------------- |
| Single tool calls | ✅ |
| Parallel tool calls | ✅ (quality may vary) |
| `tool_choice: "auto"` | ✅ |
| `tool_choice: "none"` | ✅ |
| Streaming mode | ✅ |
| Nested calls | ❌ |
## Notes
* Function definitions count toward your input token usage
* Inference usage is counted as normal when using tool calling
# Vision & OCR
Source: https://docs.deepinfra.com/chat/vision
Send images to multimodal models for visual understanding and text extraction.
DeepInfra hosts multimodal models that accept both images and text as input and produce text output. These models use the standard OpenAI vision API format and cover two major use cases:
* **Visual understanding** — describe images, answer questions about visual content, compare images, analyze charts
* **OCR (Optical Character Recognition)** — extract text from scanned documents, receipts, invoices, screenshots, handwritten notes, and PDFs
## Available vision models
* [Qwen/Qwen2.5-VL-32B-Instruct](https://deepinfra.com/Qwen/Qwen2.5-VL-32B-Instruct)
* [Qwen/Qwen2.5-VL-7B-Instruct](https://deepinfra.com/Qwen/Qwen2.5-VL-7B-Instruct)
* [meta-llama/Llama-3.2-11B-Vision-Instruct](https://deepinfra.com/meta-llama/Llama-3.2-11B-Vision-Instruct)
## Available OCR models
We host a growing set of OCR-specialized models for high-accuracy text extraction. Browse the [full OCR model catalog](https://deepinfra.com/models/ocr/).
OCR models currently use the same vision API format below. A dedicated OCR endpoint optimized for document processing is coming soon.
## Quick start
Images are passed in two ways:
1. **URL** — pass a link to a publicly accessible image
2. **Base64** — encode the image and include it directly in the request
### Image URL
```bash theme={null}
curl "https://api.deepinfra.com/v1/openai/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"model": "Qwen/Qwen2.5-VL-32B-Instruct",
"messages": [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://shared.deepinfra.com/models/llava-hf/llava-1.5-7b-hf/cover_image.ed4fba7a25b147e7fe6675e9f760585e11274e8ee72596e6412447260493cd4f-s600.webp"
}
},
{
"type": "text",
"text": "What'\''s in this image?"
}
]
}
]
}'
```
### Base64 encoded image
```python theme={null}
from openai import OpenAI
import base64
import requests
openai = OpenAI(
api_key="$DEEPINFRA_TOKEN",
base_url="https://api.deepinfra.com/v1/openai",
)
image_url = "https://shared.deepinfra.com/models/llava-hf/llava-1.5-7b-hf/cover_image.ed4fba7a25b147e7fe6675e9f760585e11274e8ee72596e6412447260493cd4f-s600.webp"
base64_image = base64.b64encode(requests.get(image_url).content).decode("utf-8")
chat_completion = openai.chat.completions.create(
model="Qwen/Qwen2.5-VL-32B-Instruct",
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": f"data:image/jpeg;base64,{base64_image}"
}
},
{
"type": "text",
"text": "What's in this image?"
}
]
}
]
)
print(chat_completion.choices[0].message.content)
```
## OCR example
Extract all text from a document image:
```python theme={null}
from openai import OpenAI
import base64
openai = OpenAI(
api_key="$DEEPINFRA_TOKEN",
base_url="https://api.deepinfra.com/v1/openai",
)
with open("invoice.png", "rb") as f:
base64_image = base64.b64encode(f.read()).decode("utf-8")
response = openai.chat.completions.create(
model="Qwen/Qwen2.5-VL-32B-Instruct",
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {"url": f"data:image/png;base64,{base64_image}"}
},
{
"type": "text",
"text": "Extract all text from this document. Preserve the structure and layout as much as possible."
}
]
}
]
)
print(response.choices[0].message.content)
```
Common OCR prompts:
* `"Extract all text from this image."` — basic extraction
* `"Extract all text and return it as structured JSON with field names and values."` — structured extraction (e.g. invoices, forms)
* `"Transcribe the handwritten text in this image."` — handwriting recognition
* `"List all line items, quantities, and prices from this receipt."` — targeted extraction
## Multiple images
You can pass multiple images in a single request by including multiple `image_url` content items:
```bash theme={null}
curl "https://api.deepinfra.com/v1/openai/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"model": "Qwen/Qwen2.5-VL-32B-Instruct",
"messages": [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {"url": "https://example.com/page1.jpg"}
},
{
"type": "image_url",
"image_url": {"url": "https://example.com/page2.jpg"}
},
{
"type": "text",
"text": "Extract all text from both pages."
}
]
}
]
}'
```
## Pricing and token counting
Images are tokenized and billed as input tokens. The number of tokens consumed by an image is reported in the response under `"usage": {"prompt_tokens": ...}`.
Different models work with different image resolutions. You can still pass images of any resolution — the model will rescale them automatically. Check the model's documentation page for supported resolutions.
## Limitations
* Supported image formats: **jpg**, **png**, **webp**
* Maximum image size: **20MB**
* The `detail` parameter (image fidelity) is not currently supported
# GPU Instances
Source: https://docs.deepinfra.com/gpu-instances/overview
Rent dedicated B200 GPU instances with SSH access for training, fine-tuning, and custom workloads.
DeepInfra GPU Instances give you dedicated access to NVIDIA B200 hardware — the most powerful GPUs available. You get a container with SSH access and full control over your environment, billed by the hour.
## Available GPUs
| GPU | Memory | Best for |
| -------------- | ----------- | -------------------------------------------- |
| **B200-180GB** | 180GB HBM3e | Large-scale inference, fine-tuning, training |
## Key features
* **Dedicated access** — no sharing with other users
* **SSH access** — connect directly to your container
* **Full environment control** — bring your own Docker image or use ours
* **Pay-per-use** — billed by the hour, only while running
* **Quick setup** — running in minutes
## Use cases
* LLM training and fine-tuning
* Large-scale batch inference
* Research and experimentation
* Development environments with GPU access
## Web UI
### Start a new container
1. Go to [Dashboard → GPU Instances](https://deepinfra.com/dash/instances)
2. Click **New Container**
3. **Select GPU configuration** — choose from available B200/B300 configs. Each shows:
* GPU type, quantity, and memory (e.g., `1xB200-180GB`, `8xB200-180GB`)
* Hourly pricing
* Availability status
4. **Enter container details**:
* **Container Name** — a descriptive name
* **SSH Key** — paste your public SSH key (format: `ssh-rsa AAAAB3NzaC1yc2E...`)
5. Accept the NVIDIA license agreements and cryptocurrency mining policy
6. Click **I agree to the above**
### Connect to a running container
1. Wait for container status to show `running`
2. Click on the SSH login field to copy the command
3. Run `ssh ubuntu@` in your terminal
### Stop a container
1. Click on the container in the instances list
2. Click **Terminate**
3. Type `confirm` and click **Terminate**
All container data is permanently lost when terminated. Save your work before stopping.
## HTTP API
### Create a container
```bash theme={null}
curl -X POST https://api.deepinfra.com/v1/containers \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "my-cluster",
"gpu_config": "8xB200-180GB",
"container_image": "di-cont-ubuntu-torch:latest",
"cloud_init_user_data": "#cloud-config\nusers:\n- name: ubuntu\n shell: /bin/bash\n sudo: '\''ALL=(ALL) NOPASSWD:ALL'\''\n ssh_authorized_keys:\n - ssh-rsa AAAAB3NzaC1yc2E..."
}'
```
### Get container details
```bash theme={null}
curl -X GET https://api.deepinfra.com/v1/containers/{container_id} \
-H "Authorization: Bearer $DEEPINFRA_TOKEN"
```
Once `running`, connect via SSH:
```bash theme={null}
ssh ubuntu@
```
### List containers
```bash theme={null}
curl -X GET https://api.deepinfra.com/v1/containers \
-H "Authorization: Bearer $DEEPINFRA_TOKEN"
```
### Terminate a container
```bash theme={null}
curl -X DELETE https://api.deepinfra.com/v1/containers/{container_id} \
-H "Authorization: Bearer $DEEPINFRA_TOKEN"
```
## Container lifecycle
| State | Description |
| --------------- | ------------------------------------------------- |
| `creating` | Container is being initialized |
| `starting` | Container is booting up |
| `running` | Container is active and accessible |
| `shutting_down` | Container is being terminated |
| `failed` | Container failed to start or encountered an error |
| `deleted` | Container has been permanently removed |
# What is DeepInfra
Source: https://docs.deepinfra.com/index
AI inference cloud — OpenAI-compatible API, 100s of open-source models, private GPU deployments, and GPU rental.
DeepInfra is an AI inference cloud that makes it simple to run the latest machine learning models at scale — LLMs, vision, embeddings, image generation, video generation, speech, and more.
## What you can do
OpenAI-compatible API for 100+ LLMs. Swap your base URL, keep your code.
Multimodal models for visual understanding and document text extraction.
State-of-the-art embedding and reranker models for search and RAG.
FLUX, Stable Diffusion, text-to-video, and more.
Speech recognition (Whisper) and text-to-speech models.
Run your own fine-tuned LLM on A100 / H100 / H200 / B200 / B300 with autoscaling.
## Why DeepInfra
**Drop-in OpenAI replacement.** Point your existing OpenAI SDK to `https://api.deepinfra.com/v1/openai` and your code works without changes. No migration required.
**Best price for open-source models.** DeepInfra consistently offers the lowest prices for open-source model inference. You only pay per token — no idle GPU time, no minimums, no seat fees. DeepInfra is also the provider with the most models on [OpenRouter](https://openrouter.ai/provider/deepinfra).
**Always-fresh model catalog.** DeepInfra is typically among the first providers to deploy a newly released model.
**Private deployments for compliance and customization.** Need to run your own fine-tuned weights, or require data isolation? Deploy a dedicated instance on A100/H100/H200/B200/B300 with autoscaling and a private endpoint — competitive GPU pricing, deployable in just a few clicks.
**GPU Clusters for training and full control.** Rent a B200 or B300 cluster with SSH access and run whatever you want.
## Get started in 60 seconds
Make your first API call — no installation required.
## Quick example
```python theme={null}
from openai import OpenAI
client = OpenAI(
api_key="$DEEPINFRA_TOKEN",
base_url="https://api.deepinfra.com/v1/openai",
)
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3",
messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)
```
Get your API key from the [Dashboard](https://deepinfra.com/dash/api_keys).
# AI SDK (Vercel)
Source: https://docs.deepinfra.com/integrations/ai-sdk
Use DeepInfra models with the Vercel AI SDK for TypeScript/JavaScript.
The [AI SDK](https://sdk.vercel.ai/) by Vercel is the AI toolkit for TypeScript and JavaScript from the creators of Next.js. DeepInfra has a first-party AI SDK provider.
See the [full AI SDK docs for DeepInfra](https://sdk.vercel.ai/providers/ai-sdk-providers/deepinfra).
## Installation
```bash theme={null}
npm install ai @ai-sdk/deepinfra
```
Get your API key from the [Dashboard](https://deepinfra.com/dash/api_keys).
## Text generation
```javascript theme={null}
import { createDeepInfra } from "@ai-sdk/deepinfra";
import { generateText } from "ai";
const deepinfra = createDeepInfra({ apiKey: "$DEEPINFRA_TOKEN" });
const { text, usage, finishReason } = await generateText({
model: deepinfra("deepseek-ai/DeepSeek-V3"),
prompt: "Write a vegetarian lasagna recipe for 4 people.",
});
console.log(text);
```
With a system message:
```javascript theme={null}
const { text } = await generateText({
model: deepinfra("deepseek-ai/DeepSeek-V3"),
prompt: "Write a vegetarian lasagna recipe for 4 people.",
system: "You are a professional writer. You write simple, clear, and concise content.",
});
```
## Streaming
```javascript theme={null}
import { createDeepInfra } from "@ai-sdk/deepinfra";
import { streamText } from "ai";
const deepinfra = createDeepInfra({ apiKey: "$DEEPINFRA_TOKEN" });
const result = streamText({
model: deepinfra("deepseek-ai/DeepSeek-V3"),
prompt: "Invent a new holiday and describe its traditions.",
system: "You are a professional writer. You write simple, clear, and concise content.",
});
for await (const textPart of result.textStream) {
process.stdout.write(textPart);
}
```
## Conversations
```javascript theme={null}
const { text } = await generateText({
model: deepinfra("deepseek-ai/DeepSeek-V3"),
messages: [
{ role: "system", content: "Respond like a michelin starred chef." },
{ role: "user", content: "Can you name at least two different techniques to cook lamb?" },
{ role: "assistant", content: "Bonjour! Let me tell you..." },
{ role: "user", content: "Tell me more about the second method." },
],
});
```
Streaming conversations work the same way — use `streamText` with `messages`.
## Structured data
Generate typed, structured output using Zod schemas:
```javascript theme={null}
import { createDeepInfra } from "@ai-sdk/deepinfra";
import { generateObject } from "ai";
import { z } from "zod";
const deepinfra = createDeepInfra({ apiKey: "$DEEPINFRA_TOKEN" });
const { object } = await generateObject({
model: deepinfra("deepseek-ai/DeepSeek-V3"),
schema: z.object({
recipe: z.object({
name: z.string(),
ingredients: z.array(z.object({ name: z.string(), amount: z.string() })),
steps: z.array(z.string()),
}),
}),
prompt: "Generate a lasagna recipe.",
});
console.log(object.recipe.name);
```
Enum output:
```javascript theme={null}
const { object } = await generateObject({
model: deepinfra("deepseek-ai/DeepSeek-V3"),
output: "enum",
enum: ["action", "comedy", "drama", "horror", "sci-fi"],
prompt: 'Classify: "A group of astronauts travel through a wormhole..."',
});
```
## Tool / function calling
```javascript theme={null}
import { createDeepInfra } from "@ai-sdk/deepinfra";
import { generateText, tool } from "ai";
import { z } from "zod";
const deepinfra = createDeepInfra({ apiKey: "$DEEPINFRA_TOKEN" });
const result = await generateText({
model: deepinfra("deepseek-ai/DeepSeek-V3"),
tools: {
weather: tool({
description: "Get the weather in a location",
parameters: z.object({
location: z.string().describe("The location to get the weather for"),
}),
execute: async ({ location }) => ({
location,
temperature: 72 + Math.floor(Math.random() * 21) - 10,
}),
}),
},
prompt: "What is the weather in San Francisco?",
maxSteps: 2,
});
console.log(result.text);
```
# Anthropic SDK & Claude Code
Source: https://docs.deepinfra.com/integrations/anthropic
Use DeepInfra models with the Anthropic Messages API, Claude Code, and the Anthropic SDK.
DeepInfra exposes an Anthropic-compatible Messages API. This means tools that target the Anthropic API — Claude Code, the Anthropic Python and TypeScript SDKs, and any framework with an Anthropic adapter — can point at DeepInfra and use open-source models.
## Endpoint
```
https://api.deepinfra.com/anthropic
```
Two endpoints are available:
| Endpoint | Description |
| ------------------------------------------ | ---------------------------------- |
| `POST /anthropic/v1/messages` | Create a message (chat completion) |
| `POST /anthropic/v1/messages/count_tokens` | Count tokens for a message request |
## Authentication
Both standard Anthropic authentication methods are supported:
| Header | Example |
| --------------- | ------------------------- |
| `Authorization` | `Bearer $DEEPINFRA_TOKEN` |
| `x-api-key` | `$DEEPINFRA_TOKEN` |
You can also pass `anthropic-version` and `anthropic-beta` headers as needed.
## Using the Anthropic SDK
```bash Python theme={null}
pip install anthropic
```
```bash JavaScript theme={null}
npm install @anthropic-ai/sdk
```
```python Python theme={null}
import anthropic
client = anthropic.Anthropic(
base_url="https://api.deepinfra.com/anthropic",
api_key="$DEEPINFRA_TOKEN",
)
message = client.messages.create(
model="deepseek-ai/DeepSeek-V3",
max_tokens=1024,
messages=[
{"role": "user", "content": "Hello!"}
],
)
print(message.content[0].text)
```
```javascript JavaScript theme={null}
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic({
baseURL: "https://api.deepinfra.com/anthropic",
apiKey: "$DEEPINFRA_TOKEN",
});
const message = await client.messages.create({
model: "deepseek-ai/DeepSeek-V3",
max_tokens: 1024,
messages: [
{ role: "user", content: "Hello!" },
],
});
console.log(message.content[0].text);
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/anthropic/v1/messages" \
-H "Content-Type: application/json" \
-H "x-api-key: $DEEPINFRA_TOKEN" \
-H "anthropic-version: 2023-06-01" \
-d '{
"model": "deepseek-ai/DeepSeek-V3",
"max_tokens": 1024,
"messages": [
{
"role": "user",
"content": "Hello!"
}
]
}'
```
## Using with Claude Code
Claude Code can use DeepInfra as its backend. To keep your normal Claude Code setup untouched, add a dedicated shell function to your `~/.bashrc` or `~/.zshrc`:
```bash theme={null}
deepinfra() {
export ANTHROPIC_BASE_URL=https://api.deepinfra.com/anthropic
export ANTHROPIC_AUTH_TOKEN=$DEEPINFRA_TOKEN
export ANTHROPIC_MODEL=deepseek-ai/DeepSeek-V3.1-Terminus
export ANTHROPIC_DEFAULT_HAIKU_MODEL=Qwen/Qwen3-30B-A3B
export CLAUDE_CODE_MAX_OUTPUT_TOKENS=16384
claude "$@"
}
```
Then run `deepinfra` instead of `claude` to launch Claude Code via DeepInfra. Your regular `claude` command stays unchanged.
### Model override environment variables
Claude Code uses model aliases (`opus`, `sonnet`, `haiku`) internally. You can remap each alias to a DeepInfra model using these environment variables:
| Environment variable | Description | Example |
| -------------------------------- | ---------------------------------------------------------------------------------------- | ------------------------------------ |
| `ANTHROPIC_MODEL` | The primary model Claude Code uses for all tasks | `deepseek-ai/DeepSeek-V3.1-Terminus` |
| `ANTHROPIC_DEFAULT_OPUS_MODEL` | Model used for the `opus` alias (complex reasoning) | `deepseek-ai/DeepSeek-R1` |
| `ANTHROPIC_DEFAULT_SONNET_MODEL` | Model used for the `sonnet` alias (daily coding) | `deepseek-ai/DeepSeek-V3.1-Terminus` |
| `ANTHROPIC_DEFAULT_HAIKU_MODEL` | Model used for the `haiku` alias and background tasks (tab completions, commit messages) | `Qwen/Qwen3-30B-A3B` |
| `CLAUDE_CODE_SUBAGENT_MODEL` | Model used for subagents (parallel background tasks) | `Qwen/Qwen3-30B-A3B` |
A more complete example with all overrides:
```bash theme={null}
deepinfra() {
export ANTHROPIC_BASE_URL=https://api.deepinfra.com/anthropic
export ANTHROPIC_AUTH_TOKEN=$DEEPINFRA_TOKEN
export ANTHROPIC_MODEL=deepseek-ai/DeepSeek-V3.1-Terminus
export ANTHROPIC_DEFAULT_OPUS_MODEL=deepseek-ai/DeepSeek-R1
export ANTHROPIC_DEFAULT_SONNET_MODEL=deepseek-ai/DeepSeek-V3.1-Terminus
export ANTHROPIC_DEFAULT_HAIKU_MODEL=Qwen/Qwen3-30B-A3B
export CLAUDE_CODE_SUBAGENT_MODEL=Qwen/Qwen3-30B-A3B
export CLAUDE_CODE_MAX_OUTPUT_TOKENS=16384
claude "$@"
}
```
`ANTHROPIC_DEFAULT_HAIKU_MODEL` is used for lightweight background tasks like tab completions and commit messages. Pick a fast, cheap model here to keep costs low. The older `ANTHROPIC_SMALL_FAST_MODEL` variable is deprecated — use `ANTHROPIC_DEFAULT_HAIKU_MODEL` instead.
## Streaming
Streaming works the same as the Anthropic API — use `stream=True` (Python) or `stream: true` (JS/cURL):
```python theme={null}
with client.messages.stream(
model="deepseek-ai/DeepSeek-V3",
max_tokens=1024,
messages=[{"role": "user", "content": "Write a short poem about open source."}],
) as stream:
for text in stream.text_stream:
print(text, end="", flush=True)
```
## Token counting
Count the tokens in a message request before sending it:
```bash theme={null}
curl "https://api.deepinfra.com/anthropic/v1/messages/count_tokens" \
-H "Content-Type: application/json" \
-H "x-api-key: $DEEPINFRA_TOKEN" \
-d '{
"model": "deepseek-ai/DeepSeek-V3",
"messages": [
{
"role": "user",
"content": "Hello, how are you?"
}
]
}'
```
## Notes
* You are running open-source models via the Anthropic protocol, not Anthropic's Claude models.
* Model names use DeepInfra identifiers (e.g. `deepseek-ai/DeepSeek-V3`), not Anthropic model names.
* Not all Anthropic-specific features may be supported. Standard message creation, streaming, and token counting work as expected.
Use the OpenAI-compatible API instead.
API keys and scoped JWTs.
# AutoGen
Source: https://docs.deepinfra.com/integrations/autogen
Build multi-agent LLM applications with AutoGen using DeepInfra endpoints.
[AutoGen](https://github.com/microsoft/autogen) is a framework for building LLM applications with multiple agents that converse to solve tasks. It works with DeepInfra via the OpenAI-compatible API.
## Installation
```bash theme={null}
pip install pyautogen
```
## Configuration
Point AutoGen at the DeepInfra endpoint using `base_url`:
```python theme={null}
import autogen
config_list = [
{
"model": "deepseek-ai/DeepSeek-V3",
"base_url": "https://api.deepinfra.com/v1/openai",
"api_key": ""
}
]
llm_config = {"config_list": config_list, "seed": 42}
assistant = autogen.AssistantAgent("assistant", llm_config=llm_config)
user_proxy = autogen.UserProxyAgent("user_proxy", code_execution_config={"work_dir": "coding"})
user_proxy.initiate_chat(assistant, message="What time is it right now?")
```
You can use any [OpenAI-compatible LLM](https://deepinfra.com/models/text-generation) from DeepInfra.
## How it works
In the example above, two agents converse to solve the task:
1. The **assistant** agent generates a Python code snippet to get the current time
2. The **user\_proxy** agent automatically detects and executes the code block
3. The result is sent back to the assistant, which summarizes it
Example output:
````text theme={null}
user_proxy (to assistant):
What time is it now?
--------------------------------------------------------------------------------
assistant (to user_proxy):
To get the current time, you can use the `datetime` module in Python...
```python
import datetime
current_time = datetime.datetime.now()
print(current_time.strftime("%I:%M %p"))
```
--------------------------------------------------------------------------------
user_proxy (to assistant):
exitcode: 0 (execution succeeded)
Code output:
02:20 PM
--------------------------------------------------------------------------------
assistant (to user_proxy):
The current time is 02:20 PM.
````
# Hermes Agent
Source: https://docs.deepinfra.com/integrations/hermes-agent
Use DeepInfra models with Hermes Agent by adding a custom OpenAI-compatible provider.
Don't want to host it yourself? DeepInfra can run Hermes for you — see [Hosted Agents: Hermes-Agent](/agents/hermes-agent).
[Hermes Agent](https://github.com/NousResearch/hermes-agent) is a self-hosted, self-improving autonomous agent by Nous Research. DeepInfra isn't a built-in provider, but Hermes works with any OpenAI-compatible endpoint (`/v1/chat/completions`) — so you add DeepInfra as a `custom` provider and use any [LLM from our catalog](https://deepinfra.com/models/text-generation).
## Configure `~/.hermes/config.yaml`
```yaml theme={null}
model:
default: deepseek-ai/DeepSeek-V4-Flash
provider: custom
base_url: https://api.deepinfra.com/v1/openai
api_key: ${DEEPINFRA_TOKEN}
context_length: 1048576
```
Get your API key from the [Dashboard](https://deepinfra.com/dash/api_keys).
* `provider: custom` is required for any non-built-in OpenAI-compatible endpoint, and `base_url` overrides `provider`.
* The DeepInfra model id passes through verbatim in `default` — no reformatting needed.
* `api_key` falls back to the `OPENAI_API_KEY` environment variable if omitted, and secrets can live in `~/.hermes/.env` instead of the YAML.
* Set `context_length` explicitly. Above it's the model's full 1M-token window; Hermes needs roughly 64k minimum for agent functionality. Check a model's window via [`/v1/openai/models?filter=with_meta`](https://api.deepinfra.com/v1/openai/models?filter=with_meta\&sort_by=openclaw).
## Interactive alternative
Instead of editing the YAML by hand, run the model wizard and pick the custom endpoint:
```bash theme={null}
hermes model
# Choose: "Custom endpoint"
# Base URL: https://api.deepinfra.com/v1/openai
# API key:
# Model: deepseek-ai/DeepSeek-V4-Flash
```
Hermes persists the choice and reuses it on every run.
## Run it
```bash theme={null}
hermes
```
## Learn more
Hermes' provider overview and custom-endpoint option.
The full `config.yaml` reference.
DeepInfra's OpenAI-compatible API.
API keys and scoped JWTs.
# LangChain
Source: https://docs.deepinfra.com/integrations/langchain
Use DeepInfra models with LangChain for LLM-powered applications.
[LangChain](https://python.langchain.com/) is a framework for building applications powered by language models. DeepInfra integrates with LangChain via official adapters for LLMs, chat models, and embeddings.
## Available adapters
* [Chat adapter](https://python.langchain.com/docs/integrations/chat/deepinfra) — for chat-based LLMs
* [LLM adapter](https://python.langchain.com/docs/integrations/llms/deepinfra) — for text generation LLMs
* [Embeddings adapter](https://python.langchain.com/docs/integrations/text_embedding/deepinfra) — for embedding models
## Installation
```bash theme={null}
pip install langchain langchain-community
```
Set your API token:
```python theme={null}
import os
os.environ["DEEPINFRA_API_TOKEN"] = ""
```
## LLM examples
```python theme={null}
import os
from langchain_community.llms import DeepInfra
from langchain.prompts import PromptTemplate
from langchain.chains import LLMChain
os.environ["DEEPINFRA_API_TOKEN"] = ""
llm = DeepInfra(model_id="deepseek-ai/DeepSeek-V3")
llm.model_kwargs = {
"temperature": 0.7,
"repetition_penalty": 1.2,
"max_new_tokens": 250,
"top_p": 0.9,
}
# Basic inference
print(llm.invoke("Who let the dogs out?"))
# Streaming inference
for chunk in llm.stream("Who let the dogs out?"):
print(chunk)
# Chain with prompt template
template = """Question: {question}
Answer: Let's think step by step."""
prompt = PromptTemplate(template=template, input_variables=["question"])
llm_chain = prompt | llm
print(llm_chain.invoke("Can penguins reach the North pole?"))
```
## Chat examples
```python theme={null}
import os
from langchain_community.chat_models import ChatDeepInfra
from langchain_core.messages import HumanMessage
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
os.environ["DEEPINFRA_API_TOKEN"] = ""
messages = [
HumanMessage(content="Translate this sentence from English to French. I love programming.")
]
# Synchronous
chat = ChatDeepInfra(model="deepseek-ai/DeepSeek-V3")
print(chat.invoke(messages))
# Async
async def async_example():
chat = ChatDeepInfra(model="deepseek-ai/DeepSeek-V3")
await chat.agenerate([messages])
# Streaming
chat_stream = ChatDeepInfra(
streaming=True,
verbose=True,
callbacks=[StreamingStdOutCallbackHandler()],
)
print(chat_stream.invoke(messages))
```
## Embeddings
```python theme={null}
import os
from langchain_community.embeddings import DeepInfraEmbeddings
os.environ["DEEPINFRA_API_TOKEN"] = ""
embeddings = DeepInfraEmbeddings(
model_id="Qwen/Qwen3-Embedding-8B",
query_instruction="",
embed_instruction="",
)
docs = ["Dog is not a cat", "Beta is the second letter of Greek alphabet"]
document_result = embeddings.embed_documents(docs)
print(document_result)
```
# LlamaIndex
Source: https://docs.deepinfra.com/integrations/llama-index
Use DeepInfra LLMs and embeddings with LlamaIndex.
[LlamaIndex](https://www.llamaindex.ai) is a popular data framework for LLM applications — RAG pipelines, agents, and more. DeepInfra integrates with LlamaIndex for both LLMs and embeddings.
## LLMs
### Installation
```bash theme={null}
pip install llama-index-llms-deepinfra
```
### Initialization
```python theme={null}
from llama_index.llms.deepinfra import DeepInfraLLM
import asyncio
llm = DeepInfraLLM(
model="deepseek-ai/DeepSeek-V3",
api_key="$DEEPINFRA_TOKEN",
temperature=0.5,
max_tokens=50,
additional_kwargs={"top_p": 0.9},
)
```
### Synchronous
```python theme={null}
# Complete
response = llm.complete("Hello World!")
print(response.text)
# Stream complete
for completion in llm.stream_complete("Once upon a time"):
print(completion.delta, end="")
# Chat
from llama_index.core.base.llms.types import ChatMessage
messages = [ChatMessage(role="user", content="Tell me a joke.")]
chat_response = llm.chat(messages)
print(chat_response.message.content)
# Stream chat
messages = [
ChatMessage(role="system", content="You are a helpful assistant."),
ChatMessage(role="user", content="Tell me a story."),
]
for chat_response in llm.stream_chat(messages):
print(chat_response.delta, end="")
```
### Asynchronous
```python theme={null}
# Async complete
async def async_complete():
response = await llm.acomplete("Hello Async World!")
print(response.text)
asyncio.run(async_complete())
# Async stream complete
async def async_stream_complete():
response = await llm.astream_complete("Once upon an async time")
async for completion in response:
print(completion.delta, end="")
asyncio.run(async_stream_complete())
# Async chat
async def async_chat():
messages = [ChatMessage(role="user", content="Tell me an async joke.")]
chat_response = await llm.achat(messages)
print(chat_response.message.content)
asyncio.run(async_chat())
# Async stream chat
async def async_stream_chat():
messages = [
ChatMessage(role="system", content="You are a helpful assistant."),
ChatMessage(role="user", content="Tell me an async story."),
]
response = await llm.astream_chat(messages)
async for chat_response in response:
print(chat_response.delta, end="")
asyncio.run(async_stream_chat())
```
## Embeddings
### Installation
```bash theme={null}
pip install llama-index llama-index-embeddings-deepinfra
```
### Initialization
```python theme={null}
from dotenv import load_dotenv, find_dotenv
from llama_index.embeddings.deepinfra import DeepInfraEmbeddingModel
_ = load_dotenv(find_dotenv())
model = DeepInfraEmbeddingModel(
model_id="Qwen/Qwen3-Embedding-8B",
api_token="$DEEPINFRA_TOKEN",
normalize=True,
text_prefix="text: ",
query_prefix="query: ",
)
```
### Synchronous requests
```python theme={null}
# Single text
response = model.get_text_embedding("hello world")
print(response)
# Batch
texts = ["hello world", "goodbye world"]
response_batch = model.get_text_embedding_batch(texts)
print(response_batch)
# Query
query_response = model.get_query_embedding("hello world")
print(query_response)
```
### Asynchronous requests
```python theme={null}
import asyncio
async def main():
async_response = await model.aget_text_embedding("hello world")
print(async_response)
asyncio.run(main())
```
# Pi
Source: https://docs.deepinfra.com/integrations/pi
Use DeepInfra models with the Pi coding agent via a custom OpenAI-compatible provider.
[Pi](https://github.com/earendil-works/pi) is a coding agent. It doesn't ship DeepInfra as a built-in provider, but it works with any OpenAI-compatible endpoint — so you can add DeepInfra as a custom provider and run any [LLM from our catalog](https://deepinfra.com/models/text-generation).
## Configure the provider
Add a `deepinfra` provider to your Pi configuration. See Pi's [custom provider docs](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/custom-provider.md) for where the config file lives.
```json theme={null}
{
"providers": {
"deepinfra": {
"name": "DeepInfra",
"baseUrl": "https://api.deepinfra.com/v1/openai",
"apiKey": "$DEEPINFRA_TOKEN",
"api": "openai-completions",
"models": [
{
"id": "deepseek-ai/DeepSeek-V4-Flash",
"name": "DeepSeek V4 Flash",
"reasoning": true,
"input": ["text"],
"cost": { "input": 0.09, "output": 0.18, "cacheRead": 0.018 },
"compat": {
"supportsDeveloperRole": false,
"thinkingFormat": "deepseek"
},
"contextWindow": 1048576,
"maxTokens": 65536
}
]
}
}
}
```
Get your API key from the [Dashboard](https://deepinfra.com/dash/api_keys).
## Run it
Set your API key in the environment and launch Pi against the model:
```bash theme={null}
export DEEPINFRA_TOKEN=
pi --model deepinfra/deepseek-ai/DeepSeek-V4-Flash
```
## Notes on the values
* **`id`** (`deepseek-ai/DeepSeek-V4-Flash`) must match exactly — it's case-sensitive and is passed straight through as the model name.
* The **`compat`** block applies to DeepSeek models only — omit it for other models:
* **`supportsDeveloperRole: false`** is required for DeepSeek models. Because they're marked `"reasoning": true`, Pi defaults to sending the system prompt with the OpenAI-only `developer` role, which is rejected with a `422` error. This setting makes Pi use the standard `system` role instead.
* **`thinkingFormat: "deepseek"`** matches how DeepSeek models stream reasoning tokens (as `reasoning_content`), so the model's thinking renders properly in Pi.
* **`contextWindow`** is the model's full 1M-token window. **`maxTokens`** (`65536`) is just a per-request output cap you can raise or lower to taste.
* The **`cost`** block only drives Pi's local spend display; the numbers above are DeepInfra's current per-million-token rates for this model, but they don't affect requests either way.
To get the current context window and pricing for this or any other model, fetch [`/v1/openai/models?filter=with_meta`](https://api.deepinfra.com/v1/openai/models?filter=with_meta\&sort_by=openclaw), or see the [model's page](https://deepinfra.com/deepseek-ai/DeepSeek-V4-Flash).
## Troubleshooting
**`422` role validation error** — an error like:
```
Error: 422: {"message":"Input should be ", ..., "param":"messages.0...role"}
```
means Pi sent the system prompt with the `developer` role, which isn't accepted. This happens by default with DeepSeek models because they're configured with `"reasoning": true`. Fix it by adding `"compat": { "supportsDeveloperRole": false }` to the model entry, as shown in the config above.
**`401` / `402` errors** — if requests fail even though the config looks right, confirm your API key and balance with a direct API call. This bypasses Pi entirely and tells you whether the issue is your DeepInfra account or the Pi config:
```bash theme={null}
curl -s https://api.deepinfra.com/v1/openai/chat/completions \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"model": "deepseek-ai/DeepSeek-V4-Flash",
"messages": [{"role": "user", "content": "Say hello in one word."}]
}'
```
If this returns a normal completion, your key and balance are good. A `401` means the key is wrong; a `402`/quota error means the account needs funding.
## Learn more
Pi's guide to defining a custom provider.
How Pi handles providers and models.
DeepInfra's OpenAI-compatible API.
API keys and scoped JWTs.
# Models
Source: https://docs.deepinfra.com/models
Browse 100+ open-source models available on DeepInfra.
DeepInfra hosts a large number of the most popular machine learning models. You can find [the full list here](https://deepinfra.com/models), conveniently split into categories based on their functionality.
We are constantly adding more. DeepInfra is usually amongst the first to add a new model once it is available, and offers the best prices for open-source model inference.
## Model categories
* **[Text generation / LLMs](https://deepinfra.com/models/text-generation)** — Llama, DeepSeek, Mistral, Qwen, Gemma, and more
* **[Embeddings](https://deepinfra.com/models/embeddings)** — Qwen3 Embedding, BAAI/bge, sentence-transformers, and more
* **[Rerankers](https://deepinfra.com/models/reranker)** — Cross-encoder rerankers for RAG pipelines
* **[Vision / multimodal](https://deepinfra.com/models/multimodal)** — Qwen2.5-VL, Llama Vision, and more
* **[OCR](https://deepinfra.com/models/ocr)** — Specialized models for document text extraction
* **[Text to image](https://deepinfra.com/models/text-to-image)** — FLUX, Stable Diffusion, and more
* **[Text to video](https://deepinfra.com/models/text-to-video)** — Generate video clips from text prompts
* **[Text to speech](https://deepinfra.com/models/text-to-speech)** — Convert text to natural-sounding audio
* **[Speech recognition](https://deepinfra.com/models/automatic-speech-recognition)** — Whisper and other ASR models
## Model pages
Each model has a dedicated page where you can:
* Try it out interactively
* See its API documentation
* Grab ready-to-use code examples
## Private models
We also support deploying [custom models](/private-models/overview) on DeepInfra infrastructure. Run your own fine-tuned or trained-from-scratch LLM on dedicated A100/H100/H200/B200/B300 GPUs.
## Specifying model versions
Some models have more than one version available. You can infer against a particular version using `{"model": "MODEL_NAME:VERSION", ...}` format.
You can also infer against a `deploy_id` using `{"model": "deploy_id:DEPLOY_ID", ...}`. This is especially useful for [Custom LLMs](/private-models/custom-llms) — you can start inferring before the deployment finishes and before you have the model name + version pair.
## Model deprecation
Due to the fast-paced AI world, newer and better models are released every day. Occasionally we have to deprecate older models to maintain quality and affordability.
When a model is deprecated:
* **You'll receive at least 1 week's advance notice** before the deprecation date
* **Your applications won't break** — after deprecation, inference requests are automatically forwarded to a recommended replacement model
* **You'll get an email** notifying recent users of the model, including the deprecation date
You can browse the current list of available models at [deepinfra.com/models](https://deepinfra.com/models).
## Suggest a model
If you think there is a model that we should run, let us know at [info@deepinfra.com](mailto:info@deepinfra.com). We read every email.
# Custom LLMs
Source: https://docs.deepinfra.com/private-models/custom-llms
Deploy your own LLM on dedicated A100/H100/H200/B200/B300 GPUs with autoscaling and an OpenAI-compatible endpoint.
Run a dedicated instance of your public or private LLM on DeepInfra infrastructure. Your model gets its own GPU allocation, autoscaling, and an OpenAI-compatible API endpoint.
## Overview
**Benefits:**
* Predictable response times (no sharing with other users)
* Autoscaling support
* Run your own fine-tuned or trained-from-scratch model
* Full OpenAI API compatibility
**Trade-offs:**
* Billed per GPU-hour, not per token — you need sufficient load to justify the cost
Public models like Mixtral are shared across many users, giving very competitive per-token pricing. A private deployment gives you full GPU access, so you pay for GPU uptime regardless of traffic.
## Deployment configuration
A deployment has fixed parameters:
| Parameter | Description |
| ---------------- | ----------------------------------------------------------------------------- |
| `model_name` | Name used for inference calls |
| `gpu` | `A100-80GB`, `H100-80GB`, `H200-141GB`, `B200-180GB`, `B300-288GB` (and more) |
| `num_gpus` | Number of GPUs (model weights must fit with room for KV cache) |
| `max_batch_size` | Max parallel requests; additional requests are queued |
| `weights` | Hugging Face repo (public or private) |
And dynamic settings (can be changed while running):
| Setting | Description |
| --------------- | ------------------------------------------ |
| `min_instances` | Minimum running copies (0 = scale to zero) |
| `max_instances` | Maximum copies during high load |
## Create a deployment
### Web UI
Go to [Dashboard → New Deployment → Custom LLM](https://deepinfra.com/dash/deployments?new=custom-llm).
### HTTP API
```bash theme={null}
curl -X POST https://api.deepinfra.com/deploy/llm \
-d '{
"model_name": "test-model",
"gpu": "A100-80GB",
"num_gpus": 2,
"max_batch_size": 64,
"hf": {
"repo": "deepseek-ai/DeepSeek-V3"
},
"settings": {
"min_instances": 0,
"max_instances": 1
}
}' \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer $DEEPINFRA_TOKEN"
```
The model's full name will be `YOUR_GITHUB_USERNAME/model-name`.
## Monitor your deployment
Track status via the [Dashboard → Deployments](https://deepinfra.com/dash/deployments) or via HTTP:
```bash theme={null}
curl https://api.deepinfra.com/deploy/list \
-H "Authorization: Bearer $DEEPINFRA_TOKEN"
```
## Use your deployment
Once running, inference via:
* **Web demo**: `https://deepinfra.com/FULLNAME`
* **OpenAI ChatCompletions API**
* **OpenAI Completions API**
* **DeepInfra inference API**
```bash theme={null}
curl "https://api.deepinfra.com/v1/openai/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"model": "YOUR_USERNAME/test-model",
"messages": [{"role": "user", "content": "Hello!"}]
}'
```
You can also use `deploy_id` before the model is running:
```json theme={null}
{"model": "deploy_id:YOUR_DEPLOY_ID", ...}
```
## Update scaling settings
```bash theme={null}
curl -X PUT https://api.deepinfra.com/deploy/DEPLOY_ID \
-d '{"settings": {"min_instances": 2, "max_instances": 2}}' \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer $DEPLOY_API_KEY"
```
## Delete a deployment
* Use the trash icon in [Dashboard → Deployments](https://deepinfra.com/dash/deployments)
* Or: `DELETE https://api.deepinfra.com/deploy/DEPLOY_ID`
## Limitations
* **4 GPU limit per user** (e.g., 4×1GPU or 1×4GPU). Contact us for more.
* GPU availability is not guaranteed during scale-up — you're only billed for what runs
* Billing happens weekly in a separate invoice
* Quantization is not currently supported (in progress)
* `deploy_id` may not be immediately available while the model is deploying
Forgetting to shut down a deployment is a common mistake. For example, leaving 2 GPUs running over a weekend (64 hours) at $2/GPU-hour costs $256. Set spending limits in [billing settings](https://deepinfra.com/dash/billing).
# LoRA Adapters
Source: https://docs.deepinfra.com/private-models/lora
Deploy LoRA fine-tuned language models on DeepInfra.
Deploy LoRA adapter models on top of base models hosted at DeepInfra. Your adapter is loaded on a supported base model and served with the standard OpenAI-compatible API.
## Prerequisites
1. A LoRA adapter model hosted on Hugging Face
2. A base model that supports LoRA at DeepInfra (see supported base models in the upload form)
3. A Hugging Face token if your LoRA adapter is private
4. A DeepInfra account and API key
## Deploy a LoRA model
1. Go to [Dashboard](https://deepinfra.com/dash)
2. Click **New Deployment**
3. Click the **LoRA Model** tab
4. Fill in the form:
* **LoRA model name** — name used to reference this deployment
* **Hugging Face Model Name** — path to your LoRA adapter on Hugging Face
* **Hugging Face Token** — optional, required for private repos
## Example
Using the public adapter `askardeepinfra/llama-3.1-8B-rank-32-example-lora` (base: `meta-llama/Meta-Llama-3.1-8B-Instruct`):
1. Go to Dashboard → New Deployment → LoRA Model
2. Fill in:
* **LoRA model name**: `asdf/lora-example`
* **Hugging Face Model Name**: `askardeepinfra/llama-3.1-8B-rank-32-example-lora`
3. Click **Upload**
The deployment appears in [Dashboard → Deployments](https://deepinfra.com/dash/deployments). Initial state is `Initializing` → `Deploying` → `Running`.
Once running, your model page is at `https://deepinfra.com/asdf/lora-example`.
## Inference
```bash theme={null}
curl "https://api.deepinfra.com/v1/openai/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_API_KEY" \
-d '{
"model": "asdf/lora-example",
"messages": [
{
"role": "user",
"content": "Hello!"
}
]
}'
```
The LoRA model name is used directly in the `model` field — the same as any other model.
# LoRA for Image Generation
Source: https://docs.deepinfra.com/private-models/lora-image
Deploy LoRA adapters for text-to-image generation using models from Civitai.
Deploy LoRA adapters for image generation on top of supported base image models. Source your LoRA from [Civitai](https://civitai.com).
## Prerequisites
1. A public LoRA model from Civitai
2. A DeepInfra account
## Deploy an image LoRA
1. Go to [Dashboard](https://deepinfra.com/dash)
2. Click **New Deployment**
3. Click the **LoRA text to image** tab
4. Fill in the form:
* **LoRA model name** — name used to reference this deployment
* **Base Model** — select the base model for this LoRA
* **Civitai URL** — URL of the LoRA model from civitai.com
The deployment appears in [Dashboard → Deployments](https://deepinfra.com/dash/deployments) with state `Initializing`. Deployment time varies from 5 seconds to 1 minute depending on LoRA size. Once `Running`, the model is ready to use.
## Inference
### Direct API endpoint
```bash theme={null}
curl "https://api.deepinfra.com/yourname/yourmodel" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"prompt": "A cat in anime style",
"lora_scale": 0.7
}'
```
### OpenAI-compatible endpoint
```bash theme={null}
curl "https://api.deepinfra.com/v1/openai/images/generations" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"model": "yourname/yourmodel",
"prompt": "A cat in anime style"
}'
```
## Notes
* Only public Civitai models are supported
* Base models support the same parameters as their original versions
* `lora_scale` controls the strength of the LoRA adaptation (0.0–1.0)
# Deploy Private Models
Source: https://docs.deepinfra.com/private-models/overview
Run your own LLMs and image models on dedicated GPU infrastructure with autoscaling.
DeepInfra allows you to deploy your own models on dedicated infrastructure — your weights, your endpoint, your isolation.
## Why run private models?
* **Compliance** — data stays on dedicated infrastructure, not shared with other users
* **Custom weights** — deploy fine-tuned or trained-from-scratch models
* **Predictable latency** — no sharing with other users means consistent response times
* **Autoscaling** — scale from 0 to many instances automatically based on load
* **Competitive GPU pricing** — some of the lowest per-GPU-hour rates available, with no lock-in
* **Simple deployment** — up and running in just a couple of clicks from the dashboard
## What you can deploy
Deploy any Hugging Face LLM on A100/H100/H200/B200/B300 GPUs with the OpenAI-compatible API.
Deploy LoRA fine-tuned language models on top of supported base models.
Deploy LoRA adapters for image generation from Civitai.
## GPU options
Private model deployments run on:
* **A100-80GB** — proven workhorse for LLM inference, great value
* **H100-80GB** — fast and widely supported
* **H200-141GB** — large HBM3e memory, ideal for big models
* **B200-180GB** — NVIDIA Blackwell, significantly faster for inference workloads
* **B300-288GB** — latest NVIDIA Blackwell Ultra, highest performance available
## Pricing model
Unlike shared inference (pay per token), private deployments are billed per GPU-hour. You pay for the time your GPUs are running, regardless of traffic.
Leaving a custom deployment running by mistake can rack up costs quickly. For example, forgetting to shut down a 2-GPU deployment over a weekend (64 hours) costs \~\$256 USD. Always set spending limits in [payment settings](https://deepinfra.com/dash/billing).
## Getting started
1. Go to [Dashboard → Deployments](https://deepinfra.com/dash/deployments)
2. Click **New Deployment**
3. Choose your deployment type (Custom LLM, LoRA, or LoRA Image)
4. Fill in the configuration and deploy
See the specific guides for each deployment type:
* [Custom LLMs](/private-models/custom-llms)
* [LoRA Adapters](/private-models/lora)
* [LoRA Image Models](/private-models/lora-image)
# Quickstart
Source: https://docs.deepinfra.com/quickstart
Make your first API call in 60 seconds — no installation required.
You don't need to install anything to do your first inference. You only need [your access token](https://deepinfra.com/dash/api_keys). DeepInfra gives you access to 100+ open-source models at the best prices available.
## Step 1: Get your API key
Go to the [Dashboard](https://deepinfra.com/dash/api_keys) and create an API key (new to DeepInfra? See [Signing In](/account/signing-in)). If you're logged in, examples throughout the docs will have your token pre-filled.
## Step 2: Make your first API call
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/openai/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"model": "deepseek-ai/DeepSeek-V3",
"messages": [
{
"role": "user",
"content": "Hello!"
}
]
}'
```
```python Python theme={null}
from openai import OpenAI
client = OpenAI(
api_key="$DEEPINFRA_TOKEN",
base_url="https://api.deepinfra.com/v1/openai",
)
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3",
messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)
```
```javascript JavaScript theme={null}
import OpenAI from "openai";
const openai = new OpenAI({
apiKey: "$DEEPINFRA_TOKEN",
baseURL: "https://api.deepinfra.com/v1/openai",
});
const response = await openai.chat.completions.create({
model: "deepseek-ai/DeepSeek-V3",
messages: [{ role: "user", content: "Hello!" }],
});
console.log(response.choices[0].message.content);
```
The response looks like this:
```json theme={null}
{
"id": "chatcmpl-guMTxWgpFf",
"object": "chat.completion",
"created": 1694623155,
"model": "deepseek-ai/DeepSeek-V3",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "Hello! It's nice to meet you. Is there something I can help you with?"
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 15,
"completion_tokens": 16,
"total_tokens": 31,
"estimated_cost": 0.0000268
}
}
```
## That's it
You're using the [OpenAI Chat Completions API](/chat/overview) — the same interface you already know. The only changes are:
* **Base URL**: `https://api.deepinfra.com/v1/openai`
* **API key**: your DeepInfra token
* **Model**: any model from [our catalog](https://deepinfra.com/models)
The official OpenAI Python and Node.js libraries work out of the box.
## Install the SDK (optional)
```bash Python theme={null}
pip install openai
```
```bash JavaScript theme={null}
npm install openai
```
## Next steps
Learn about the full chat completions API.
Stream responses token by token.
Give models access to external functions.
Browse 100+ available models.
# Request Compression
Source: https://docs.deepinfra.com/request-compression
Compress large request bodies with gzip to cut upload bandwidth and latency.
DeepInfra accepts **gzip-compressed request bodies**. For large payloads — long chat
histories, big embeddings batches, sizable inputs — compressing the request shrinks the
bytes you upload (JSON and text typically compress 4–8×), which lowers your egress and can
speed up requests over slower or metered connections.
To use it, gzip the request body and set the `Content-Encoding: gzip` header. It works on
**every** endpoint (chat completions, embeddings, and the rest), including requests whose
response streams back — the request body is a single upload regardless of how the response
is returned.
```bash cURL theme={null}
# Pipe the JSON body through gzip and send it with Content-Encoding: gzip
echo '{
"model": "deepseek-ai/DeepSeek-V3",
"messages": [{"role": "user", "content": "Hello!"}]
}' | gzip | curl "https://api.deepinfra.com/v1/openai/chat/completions" \
-H "Content-Type: application/json" \
-H "Content-Encoding: gzip" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
--data-binary @-
```
```python Python theme={null}
import gzip
import json
import requests
payload = json.dumps({
"model": "deepseek-ai/DeepSeek-V3",
"messages": [{"role": "user", "content": "Hello!"}],
}).encode("utf-8")
response = requests.post(
"https://api.deepinfra.com/v1/openai/chat/completions",
data=gzip.compress(payload),
headers={
"Content-Type": "application/json",
"Content-Encoding": "gzip",
"Authorization": "Bearer $DEEPINFRA_TOKEN",
},
)
print(response.json())
```
```javascript JavaScript theme={null}
import { gzipSync } from "node:zlib";
const payload = JSON.stringify({
model: "deepseek-ai/DeepSeek-V3",
messages: [{ role: "user", content: "Hello!" }],
});
const response = await fetch("https://api.deepinfra.com/v1/openai/chat/completions", {
method: "POST",
headers: {
"Content-Type": "application/json",
"Content-Encoding": "gzip",
Authorization: "Bearer $DEEPINFRA_TOKEN",
},
body: gzipSync(payload),
});
console.log(await response.json());
```
## When it helps
Request compression pays off in proportion to how large your request body is:
* **Worth it** for large bodies — long conversations, large embeddings batches, big inputs —
where trimming the upload meaningfully cuts bandwidth and transfer time.
* **Not worth it** for small requests (a short prompt). gzip adds a small header and a little
CPU on both ends, so for tiny bodies there's nothing to gain.
**gzip only.** `Content-Encoding: gzip` is the only supported request compression. Other
codings (`deflate`, `br`, `zstd`) are not decompressed — a body compressed with those will
fail to parse. Use `gzip`.
# Stable Diffusion
Source: https://docs.deepinfra.com/tutorials/stable-diffusion
Generate images with Stable Diffusion and SDXL on DeepInfra.
DeepInfra supports a wide variety of text-to-image models, including Stable Diffusion 1.4, 1.5, 2.1, SDXL, and many derivative models.
Browse [all text-to-image models](https://deepinfra.com/models/text-to-image).
## Quick start
This example uses `stability-ai/sdxl`:
```javascript JavaScript theme={null}
import { Sdxl } from "deepinfra";
import { createWriteStream } from "fs";
import { Readable } from "stream";
const DEEPINFRA_API_KEY = "$DEEPINFRA_TOKEN";
const model = new Sdxl(DEEPINFRA_API_KEY);
const response = await model.generate({
input: {
prompt: "a burger with a funny hat on the beach",
},
});
const result = await fetch(response.output[0]);
if (result.ok && result.body) {
Readable.fromWeb(result.body).pipe(createWriteStream("image.png"));
}
```
```bash cURL theme={null}
curl "https://api.deepinfra.com/v1/inference/stability-ai/sdxl" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-d '{
"input": {
"prompt": "a burger with a funny hat on the beach"
}
}'
```
## Advanced options
Each model has its own set of parameters including negative prompts, number of inference steps, guidance scale, seed, and more. Check the model's page for all available options:
* [stability-ai/sdxl](https://deepinfra.com/stability-ai/sdxl)
* [stabilityai/stable-diffusion-2-1](https://deepinfra.com/stabilityai/stable-diffusion-2-1)
## Using the OpenAI-compatible API
You can also use the OpenAI images API for image generation — see [Image Generation](/apis/image-generation).
## Custom LoRA adapters
To use custom style LoRA adapters for image generation, see [LoRA for Image Generation](/private-models/lora-image).
# Whisper Speech Recognition
Source: https://docs.deepinfra.com/tutorials/whisper
Transcribe audio to text with OpenAI Whisper on DeepInfra.
[Whisper](https://github.com/openai/whisper) is OpenAI's speech recognition model. Given an audio file, it produces transcribed text with per-sentence timestamps. DeepInfra hosts multiple Whisper variants.
Browse [all speech recognition models](https://deepinfra.com/models?type=automatic-speech-recognition).
## Models
| Model | Notes |
| ----------------------------------- | ------------------- |
| `openai/whisper-large` | Best accuracy |
| `openai/whisper-medium` | Balanced |
| `openai/whisper-small` | Fast |
| `openai/whisper-base` | Smallest |
| `openai/whisper-timestamped-medium` | Per-word timestamps |
By default, Whisper produces per-sentence timestamp segmentation. `whisper-timestamped` gives per-word timestamps.
## Example
```javascript JavaScript theme={null}
import { AutomaticSpeechRecognition } from "deepinfra";
import path from "path";
import { fileURLToPath } from "url";
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const client = new AutomaticSpeechRecognition(
"openai/whisper-large",
"$DEEPINFRA_TOKEN"
);
const response = await client.generate({
audio: path.join(__dirname, "audio.mp3"),
});
console.log(response.text);
```
```bash cURL theme={null}
curl -X POST \
-H "Authorization: Bearer $DEEPINFRA_TOKEN" \
-F audio=@audio.mp3 \
'https://api.deepinfra.com/v1/inference/openai/whisper-large'
```
## Supported formats
* `mp3`
* `wav`
## Additional parameters
Each Whisper variant supports parameters like `language`, `task` (transcribe vs. translate), and more. Check the model's documentation page for the full list:
* [openai/whisper-large](https://deepinfra.com/openai/whisper-large)