# Authentication Source: https://docs.deepinfra.com/account/authentication API tokens and scoped JWT for secure, scope-limited inference access. This page covers authenticating **API requests**. For signing in to the dashboard itself, see [Signing In](/account/signing-in). DeepInfra supports two authentication methods: 1. **API keys** — full-access tokens for your own use 2. **Scoped JWTs** — short-lived, scope-limited tokens you can issue to third parties ## API keys Get your API keys from the [Dashboard](https://deepinfra.com/dash/api_keys). Use them in the `Authorization` header: ```bash theme={null} Authorization: Bearer $DEEPINFRA_API_KEY ``` ## Scoped JWT Scoped JWT tokens let you grant limited inference access to third parties without sharing your API key. You can restrict the token by: * **Models allowed** — specific model(s) only * **Expiration** — time-limited (up to 1 year) * **Spending limit** — maximum USD spend Usage is counted against the API key that was used to sign the JWT. ### Create a scoped JWT ```bash theme={null} curl -X POST "https://api.deepinfra.com/v1/scoped-jwt" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "api_key_name": "auto", "models": ["deepseek-ai/DeepSeek-V4-Flash-0731"], "expires_delta": 3600, "spending_limit": 1.0 }' ``` Response: ```json theme={null} {"token": "jwt:eyJhbGciOiJIUzI1NiIsImtpZCxxxxxxxxxxxxxxxxxx"} ``` This creates a token limited to `deepseek-ai/DeepSeek-V4-Flash-0731`, expiring in 1 hour, with a \$1.00 spending limit. **Optional fields** (omit to remove restriction): * `models` — allow any model * `expires_delta` — no expiration (defaults to 1 year) * `spending_limit` — no spending limit * Use `expires_at` (unix timestamp) instead of `expires_delta` if preferred ### Inspect a JWT ```bash theme={null} curl "https://api.deepinfra.com/v1/scoped-jwt?jwtoken=XXXX" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" ``` ```json theme={null} { "expires_at": 1738843515, "models": ["deepseek-ai/DeepSeek-V4-Flash-0731"], "spending_limit": 1 } ``` ### Use a scoped JWT Use it exactly like a regular API key in the `Authorization` header: ```bash theme={null} curl "https://api.deepinfra.com/v1/openai/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $SCOPED_JWT" \ -d '{ "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "messages": [{"role": "user", "content": "Hello!"}] }' ``` Requests using disallowed models, expired tokens, or over-budget tokens will be rejected. # Data Privacy Source: https://docs.deepinfra.com/account/data-privacy How DeepInfra handles your data during inference — what's stored, what's not. This page explains how DeepInfra handles data when you use our inference APIs. ## Summary * **Input data is not stored to disk** during inference — it exists only in memory while the request is being processed * **Output data is not stored** — it is sent to you and then deleted from memory * **We do not train on your data** (except Google/Anthropic model exceptions below) * **We do not share your data with third parties** (except Google/Anthropic model exceptions below) ## Data privacy When using DeepInfra inference APIs, we do not store the data you submit to our APIs on disk. We only hold it in memory during the inference process. Once inference is complete, the data is deleted from memory. The same applies to outputs — once sent back to you, they are deleted. **Exception:** Outputs of image generation models are stored for a short period of time to allow easy access (e.g., for the model demo page). ### Google models If you opt to use a Google model, Google will store the output as outlined in their [Privacy Notice](https://cloud.google.com/terms/cloud-privacy-notice). ### Anthropic models If you opt to use an Anthropic model, Anthropic will store the output as outlined in their [Trust Center](https://trust.anthropic.com/). ## Bulk inference APIs When using our bulk inference APIs (submitting multiple requests in a single call), we may need to store data for a longer period, potentially on disk in encrypted form. Once inference is complete and results are returned to you, the data is deleted from disk and memory after a short retention period. ## No training We do not use data you submit to our APIs for training models, except when using Google or Anthropic models, where the receiving company's training policy applies. ## No sharing We do not share data you submit to our APIs with any third party, except when using Google or Anthropic models, where we are required to transfer data to those endpoints to fulfill the request. ## Logs We do not log the content of your requests. We log only metadata useful for debugging: request ID, cost, sampling parameters. When using the Google model, Google logs prompts and responses for a limited period solely to detect violations of their [Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy). ## Personal information Personal information and data provided when using certain models through our API may be shared with the relevant model API endpoints, as specified at the time of use. # Okta SSO Source: https://docs.deepinfra.com/account/okta-sso Configure Okta as a Single Sign-On provider for your DeepInfra team. DeepInfra supports Okta-based Single Sign-On (SSO) via OpenID Connect for teams that need centralized identity management. For individual sign-in options (Google, GitHub, email), see [Signing In](/account/signing-in). ## Supported features * SSO initiated via Okta (IdP-initiated) * SSO initiated via DeepInfra (SP-initiated) * Automatic account creation on first sign-in ## Configuration steps 1. **Install the DeepInfra app** in your Okta instance 2. **Fill in the configuration**: * **Team ID** — your Okta subdomain is a good starting point. For multi-tenancy (multiple teams in one Okta instance), use `subdomain-group`. Lowercase only, starting with subdomain, dashes for separators. * **Use Stage** — leave blank 3. **Assign users or groups** who should have access to DeepInfra 4. In the DeepInfra App (inside Okta), go to the **Sign On** tab and note the **Client ID** and **Client Secret** 5. For the **Issuer** (your Okta domain): find the link titled *OpenID Provider Metadata*, click it, and copy the `"issuer"` URL from the JSON document 6. **Email [feedback@deepinfra.com](mailto:feedback@deepinfra.com)** with: * Team ID * Issuer * Client ID * Client Secret * Admin email (the user who will be team admin) After setup is complete, users can sign in via: * Okta dashboard * DeepInfra's [SSO login page](https://deepinfra.com/login_sso) (enter your Team ID) ## SP-initiated SSO 1. Go to [deepinfra.com/login](https://deepinfra.com/login) 2. Click **Corporate SSO** 3. Enter your **Team ID** and click **SSO Login** 4. Enter your Okta credentials and click **Sign in with Okta** 5. You're redirected to the DeepInfra dashboard ## Team management notes * Admin can change team member roles (member / admin) * Admin has access to the billing dashboard * All team members share the same API tokens and models * For per-user isolation (separate tokens and models per person), contact us # Rate Limits Source: https://docs.deepinfra.com/account/rate-limits Default concurrent request limits and how to increase them. ## Default limit: 200 concurrent requests per model Every account has a default limit of **200 concurrent requests per model**. If you query two different models simultaneously, you can handle 400 total concurrent requests (200 per model). This limit is sufficient for most production applications, including services with hundreds of thousands of daily active users. ## Understanding concurrent vs. requests per minute The rate limit is on *concurrent* requests, not per-minute volume. Throughput depends on how long each request takes: | Avg Request Duration | Concurrent Limit | Approx RPM | | - | :-: | :-: | | 1 second | 200 | 12,000 RPM | | 10 seconds | 200 | 1,200 RPM | | 60 seconds | 200 | 200 RPM | As requests complete, new ones can immediately take their place. ## Batch jobs For large batch jobs (e.g., embedding a knowledge base), use a [token bucket algorithm](https://en.wikipedia.org/wiki/Token_bucket) to stay under 200 concurrent requests. You'll still complete the work in a reasonable time. ## Rate limit errors You'll receive HTTP **429** with a `Rate limited` message when the limit is exceeded. Actions to take: * Retry after a short delay * Slow down your request rate * Apply for a limit increase > [!NOTE] > > You may occasionally receive **429** errors when a model becomes very busy, even if you're under the limit. Auto-scaling will kick in shortly. Retry after a brief wait. ## Request a limit increase You can request a rate limit increase in your [Dashboard → Account](https://deepinfra.com/dash/account). Include context about your use case. # Signing In Source: https://docs.deepinfra.com/account/signing-in Sign in with Google, GitHub, email & password, or corporate SSO — and what to know about linked accounts. You can sign in to DeepInfra at [deepinfra.com/login](https://deepinfra.com/login) with Google, GitHub, email & password, or Corporate SSO. With Google, GitHub, and SSO we create an account on your first login if you don't have one; for email & password there's a [sign-up page](https://deepinfra.com/signup). ## Signing in with Google If you don't have a DeepInfra account yet, **Continue with Google** creates one automatically. No password is set on the account — you sign in with Google from then on. ### If you already have an account with your Google email Signing in with Google automatically links your Google identity to the existing account — there is no confirmation prompt. What happens to the account depends on whether its email was already verified: * **Verified** — nothing changes: the Google link is simply added, and password sign-in keeps working. * **Not verified** — the account is treated as unclaimed: your password is removed, all active sessions are signed out, and your name and picture are replaced by your Google profile. > [!WARNING] > > To keep your password and profile, verify your email address **before** your first Google sign-in. ### Changing your email or unlinking Google Your email address can't be changed while Google is linked — the settings show "Linked from your Google account". There is currently no self-serve way to unlink Google either; contact us at [feedback@deepinfra.com](mailto:feedback@deepinfra.com) to unlink or change your email. ## Signing in with GitHub Clicking **Continue with GitHub** creates (or signs in to) an account tied to your GitHub identity, using your primary verified GitHub email. GitHub sign-in never links to an existing Google or email account — see the [FAQ](#faq). ### Changing your email Your account email can only be one of the emails associated with your GitHub account — the email dropdown in the dashboard settings is synced from your GitHub profile. To use a different email, add it to your GitHub account first; a few minutes later it will appear in the dropdown. ## Email and password Create an account at [deepinfra.com/signup](https://deepinfra.com/signup). You'll receive a verification email — verify promptly: unverified accounts have limited access and can be [claimed by a Google sign-in](#if-you-already-have-an-account-with-your-google-email) with the same email. * Signing up with an email that's already in use fails — sign in to the existing account instead. * Forgot your password? Use the reset link on the login page. * Accounts created via Google have no password, so password sign-in fails with "Invalid email or password" — use **Continue with Google** instead. ## Corporate SSO Teams using Okta sign in via the **Corporate SSO** button — see [Okta SSO](/account/okta-sso) for setup and sign-in steps. ## Joining via a team invite Team invites can be accepted with Google or email & password, and only when signing up — the new account automatically joins the inviting team. An existing account can't join a team (see the [FAQ](#faq)). ## Reactivating a deleted account If you've requested account deletion (in the dashboard account settings) and it hasn't been permanently removed yet, any successful sign-in reactivates the account. If that wasn't intended, contact [feedback@deepinfra.com](mailto:feedback@deepinfra.com). ## FAQ > **I signed in, but my API keys, usage, or balance are missing** > > You're probably in a different account than the one you originally signed up with. Sign in again with the method you used the first time — and make sure you always use the same sign-in method. > > This can happen because GitHub sign-in has its own account, separate from the Google / email & password one — even when all of them use the same email address. So if you signed up with GitHub but signed in with Google or email (or the other way around), you ended up in a second, empty account. > **I opened a team invite link and signed in, but I wasn't added to the team** > > Team invites only work when a **new** account is created. An existing account can't join a team, and the same email can't be on both a personal and a team account (one created via an invite). Your options: > > * Request deletion of your existing account (in the dashboard account settings), then email [feedback@deepinfra.com](mailto:feedback@deepinfra.com) so we remove it immediately. Once we confirm it's gone, ask the team owner to re-invite you and sign up through the new link — signing in earlier would just [reactivate the account](#reactivating-a-deleted-account). > * Sign up with a different email, e.g. `you+team@example.com` — most mail providers deliver it to the same inbox, but it counts as a separate account. # Subprocessors Source: https://docs.deepinfra.com/account/subprocessors Third-party services DeepInfra uses to provide its services. DeepInfra uses the following subprocessors to provide its services: | Name | Nature of Processing | | - | - | | Stripe | Payment processor | | Amazon Web Services | Infrastructure | | Google Cloud Platform | Infrastructure | *Last updated: September 6, 2024* # Webhooks Source: https://docs.deepinfra.com/account/webhooks Receive inference results asynchronously via HTTP callbacks. Webhooks are a feature of the [DeepInfra Native API](/apis/deepinfra-native). They are not supported with the OpenAI-compatible API. Webhooks let you submit an inference request and receive the result via an HTTP callback, instead of waiting for the response synchronously. This is useful for long-running requests or fire-and-forget workloads. ## How it works Add a `webhook` parameter to your request. The API immediately responds with status `queued`, then calls your webhook URL with the result once inference is complete. ## Text generation example ```javascript JavaScript theme={null} import { TextGeneration } from "deepinfra"; const client = new TextGeneration( "https://api.deepinfra.com/v1/inference/deepseek-ai/DeepSeek-V4-Flash-0731", process.env.DEEPINFRA_API_KEY ); const res = await client.generate({ input: "<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\nHello!<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n", stop: ["<|eot_id|>"], webhook: "https://your-app.com/deepinfra-webhook" }); console.log(res.inference_status.status); // "queued" ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/inference/deepseek-ai/DeepSeek-V4-Flash-0731" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "input": "<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\nHello!<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n", "stop": ["<|eot_id|>"], "webhook": "https://your-app.com/deepinfra-webhook" }' ``` ## Embeddings example ```bash theme={null} curl "https://api.deepinfra.com/v1/inference/Qwen/Qwen3-Embedding-8B" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "inputs": ["I like chocolate"], "webhook": "https://your-app.com/deepinfra-webhook" }' ``` ## Webhook payload On success, your endpoint receives: ```json theme={null} { "request_id": "R7X9fdlIaF5GlVisBAi5xR3E", "inference_status": { "status": "succeeded", "runtime_ms": 228, "cost": 0.0001140000022132881 }, "results": { ... } } ``` On failure: ```json theme={null} { "request_id": "RHNShFanUP5ExA8rzgyDWH88", "inference_status": { "status": "failed", "runtime_ms": 0, "cost": 0.0 } } ``` ## Retry behavior DeepInfra will make a few retry attempts if your webhook endpoint returns a 4xx or 5xx status code. # Hermes-Agent Source: https://docs.deepinfra.com/agents/hermes-agent Run a fully managed Hermes-Agent instance — Nous Research's autonomous agent, driven over SSH and powered by DeepInfra models. [Hermes-Agent](https://github.com/NousResearch/hermes-agent) is a self-improving autonomous agent by Nous Research, driven from the terminal. The hosted version gives you a dedicated, always-on instance without running your own server: DeepInfra provisions the VM, installs Hermes, and pre-configures it with a dedicated DeepInfra API key. Hermes-Agent instances are **SSH only** — the framework runs without a web dashboard, so you connect over SSH after creation. > [!NOTE] > > Want to run Hermes on your own machine instead? See [Hermes Agent integration](/integrations/hermes-agent) for using DeepInfra as a custom provider in a self-hosted setup. ## Plans Hermes-Agent instances are available on the `budget` (2 vCPU, 2 GB RAM) and `tiny` (2 vCPU, 1 GB RAM) plans. See [plans and billing](/agents/introduction#plans) for details; current per-hour pricing is shown in the **Create Instance** dialog. ## Create an instance 1. Make sure your public SSH key is registered at [Dashboard → SSH Keys](https://deepinfra.com/dash/ssh_keys) — it's your only way in 2. Go to [Dashboard → Hosted Agents](https://deepinfra.com/dash/agents) and click **Create Instance** 3. Enter a **Name**, choose **Hermes-Agent** as the instance type, and pick a plan 4. Click **Create** — setup is fully automated and takes a few minutes ## Connect and run Once the instance is `running`, click **SSH** in the Connect column to get the exact command, or use the public IP from the details page: ```bash theme={null} ssh hermes@ -p 2222 ``` Then start the agent: ```bash theme={null} hermes ``` That's it — the instance ships with a working configuration, so Hermes is ready to go on first login. > [!WARNING] > > The instance gets a **new public IP address** on every restart. Grab the current one from the details page. ## Configuration Hermes keeps its configuration in `~/.hermes/`: * `~/.hermes/config.yaml` — model, provider, and agent settings. See the [Hermes configuration docs](https://hermes-agent.nousresearch.com/docs/user-guide/configuration). * `~/.hermes/.env` — secrets. The `DEEPINFRA_API_KEY` line is managed by DeepInfra and refreshed on every instance start, so don't edit it — manual changes to that line are overwritten. The rest of the file is yours: other secrets you add (for example a third-party API key for a tool) stay untouched. You can point Hermes at any [model from our catalog](https://deepinfra.com/models/text-generation) — for example `deepseek-ai/DeepSeek-V4-Flash` — either by editing `config.yaml` or interactively with `hermes model`. The agent's model calls are billed as regular inference usage against your account. ## Updating You update Hermes yourself, whenever you want — DeepInfra doesn't update it for you. New versions are listed on the [Hermes releases page](https://github.com/NousResearch/hermes-agent/releases). Click **Update** on the instance to get the exact commands, with copy buttons. Connect over SSH, then run: ```bash theme={null} git -C /opt/hermes/src remote set-branches origin main && hermes update ``` The `git` part makes the install track upstream's `main` branch, which is what `hermes update` pulls. The update takes a few minutes and keeps your settings and chat history. Since it brings in the latest upstream code, consider [creating a backup](/agents/introduction#backups) first so you can roll back. > [!NOTE] > > The version shown in the dashboard is the one DeepInfra installed, and it doesn't change when you update Hermes yourself. Run `hermes --version` on the instance to see what you're running. ## FAQ **Why is there no dashboard link for my instance?** By design — Hermes-Agent is a terminal-driven framework with no web UI. SSH in and run `hermes`. **Does my work survive stop/start?** Yes. The whole filesystem — Hermes state, your files, installed packages — is captured in a snapshot on stop and restored on start. Only the public IP changes. **Can I install extra tools on the instance?** Yes, it's your VM — install packages and customize freely over SSH. Automatic [backups](/agents/introduction#backups) cover your changes. # Introduction to Hosted Agents Source: https://docs.deepinfra.com/agents/introduction Fully managed instances running open-source agent frameworks — created in minutes, billed by the hour, pre-wired to DeepInfra inference. Hosted Agents are fully managed virtual machines that run an open-source agent framework for you. You pick a framework and a plan, and DeepInfra provisions the instance, installs and configures the agent, and wires it to a dedicated DeepInfra API key — no servers to set up, no config files to bootstrap. Your instance is up and running in a few minutes, and its state persists across stops and restarts. Manage your instances at [Dashboard → Hosted Agents](https://deepinfra.com/dash/agents). ## Available frameworks | Framework | Access | Description | | - | - | - | | [OpenClaw](/agents/openclaw) | Web dashboard + SSH | Your personal AI assistant, reachable on the messaging channels you already use. [Open source](https://github.com/openclaw/openclaw). | | [Hermes-Agent](/agents/hermes-agent) | SSH only | Self-improving autonomous agent by Nous Research, driven from the terminal. [Open source](https://github.com/NousResearch/hermes-agent). | ## Plans | Plan | vCPU | RAM | Included monthly transfer | Available for | | - | - | - | - | - | | `standard` | 2 | 4 GB | 4 TB | OpenClaw | | `budget` | 2 | 2 GB | 3 TB | OpenClaw, Hermes-Agent | | `tiny` | 2 | 1 GB | 2 TB | Hermes-Agent | Current per-hour pricing for each plan is shown in the **Create Instance** dialog and on each instance's details page. ## Billing Two separate line items make up the cost of a hosted agent: * **Compute** — a per-hour rate for the instance itself, billed only while the instance is active (creating, starting, running, or stopping). A stopped instance costs nothing. * **Inference** — whatever the agent spends on model calls. Each instance gets its own DeepInfra API key, and its LLM usage is billed at standard token prices against your account. Both appear on your [Usage](https://deepinfra.com/dash/usage) page; compute shows up as **Hosted Agent**. > [!NOTE] > > Fair use: instances that exceed an average of their plan's included network transfer per month (inbound + outbound, prorated by uptime) may be temporarily paused. You can start the instance again from the dashboard once usage normalizes. ## Create an instance **1. Add an SSH key** Register your public SSH key at [Dashboard → SSH Keys](https://deepinfra.com/dash/ssh_keys). It's installed on every instance you create, and you'll need it to connect over SSH. **2. Open Hosted Agents** Go to [Dashboard → Hosted Agents](https://deepinfra.com/dash/agents) and click **Create Instance**. **3. Configure and create** Pick a **Name** (up to 64 characters), an **Instance Type** (the agent framework), and a **Plan**, then click **Create**. **4. Wait for it to come up** Setup is fully automated and takes a few minutes. Once the instance is `running`, connect via the **dashboard** link (OpenClaw) or **SSH** (any framework). ## Instance lifecycle | State | Description | | - | - | | `creating` | The instance is being provisioned | | `starting` | The instance is booting up | | `stopping` | The instance is shutting down; a snapshot of its state is captured | | `running` | The instance is active and accessible | | `stopped` | The instance is shut down — not billed for compute | | `failed` | The instance hit an unrecoverable error (see the fail reason on its details page) | Stopping an instance shuts it down so you're not billed for compute while it's idle. Starting it again restores everything as it was. > [!WARNING] > > The instance gets a **new public IP address** on every restart. Check the details page for the current one. ## Manage an instance From the instance list or its details page you can: * **Start / Stop** — pause compute billing without losing state. * **Update** — for OpenClaw, when a newer version is available, upgrade in place. No restart needed; the agent may be unresponsive for a few minutes. For Hermes-Agent, **Update** shows the commands to update it yourself over SSH, whenever you want (see [Updating Hermes-Agent](/agents/hermes-agent#updating)). * **Backup** — trigger a snapshot on demand (see below). * **Delete** — stop and permanently remove the instance. This cannot be undone. > [!NOTE] > > DeepInfra retains administrative SSH access to your instance to automate maintenance tasks (update and restore from backups for example). ## Backups Backups are created automatically every 24 hours, and one is also captured whenever you stop the instance. Snapshots older than 5 days are cleaned up, with at least 2 always retained. You can create one manually anytime with **Create Backup** on the details page. To roll back, pick a backup in the **Backups** table and click **Restore**. > [!WARNING] > > Restoring overwrites the instance's current data with the backup. This cannot be undone — take a fresh backup first if you might need the current state. ## HTTP API Everything above is also available programmatically. Authenticate with your [API key](/account/authentication). Create an instance: ```bash theme={null} curl -X POST https://api.deepinfra.com/v1/agents \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "my-agent", "agent_type_id": "openclaw", "plan_id": "standard" }' ``` List your instances: ```bash theme={null} curl "https://api.deepinfra.com/v1/agents?state=all" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" ``` Start, stop, or update: ```bash theme={null} curl -X POST https://api.deepinfra.com/v1/agents/{instance_id}/start \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" curl -X POST https://api.deepinfra.com/v1/agents/{instance_id}/stop \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" curl -X POST https://api.deepinfra.com/v1/agents/{instance_id}/update \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" ``` `/update` applies to OpenClaw. Hermes-Agent instances are [updated over SSH](/agents/hermes-agent#updating) instead, so for them `/update` returns `409` with the command to run. Delete: ```bash theme={null} curl -X DELETE https://api.deepinfra.com/v1/agents/{instance_id} \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" ``` Other endpoints: `GET /v1/agents/catalog` (frameworks, plans, and current pricing), `GET /v1/agents/{instance_id}/backups`, `POST /v1/agents/{instance_id}/backup`, and `POST /v1/agents/{instance_id}/backups/{backup_id}/restore`. See the [API Reference](https://docs.deepinfra.com/api-reference) for full schemas. ## FAQ **How many instances can I run?** Up to 5 active instances per account. **Where does my instance run?** Instances run in `us-west-2` by default. Accounts subject to EU data-protection rules are placed in `eu-central-1`. The region is shown on the instance details page. **What happens to my data when I stop an instance?** Nothing is lost — state is captured in a snapshot and restored when you start it again. Only the public IP changes. **Can DeepInfra access my instance?** DeepInfra retains administrative SSH access for automated maintenance (updates and backup restores for example). **Why was my instance paused?** Most likely it exceeded the fair-use network transfer threshold for its plan. Start it again from the dashboard once usage normalizes. **What happens if my account is suspended?** If your account is suspended — for example when your balance runs out — your instances are automatically stopped within a minute or two, and creating, starting, or restoring instances is blocked. Nothing is deleted: state is preserved the same way as a regular stop, and once your account is back in good standing you can start your instances again. Keep in mind that a running agent spends on inference on its own, so consider enabling [Automatic Top-Up](https://deepinfra.com/dash/billing) if you plan to leave it unattended. **Where do I see what my agent is costing me?** On the [Usage](https://deepinfra.com/dash/usage) page — compute appears as **Hosted Agent**, and the agent's model calls appear as regular inference usage. # OpenClaw Source: https://docs.deepinfra.com/agents/openclaw Run a fully managed OpenClaw instance — your personal AI assistant with a web dashboard, powered by DeepInfra models. [OpenClaw](https://openclaw.ai) is the wildly popular open-source personal AI assistant ([github.com/openclaw/openclaw](https://github.com/openclaw/openclaw)) that talks to you on the channels you already use — WhatsApp, Telegram, Slack, Discord, and many more — and acts autonomously on your behalf. The hosted version gives you a dedicated, always-on OpenClaw instance without running your own server: DeepInfra provisions the VM, installs OpenClaw, wires it to a dedicated DeepInfra API key, and keeps it backed up. You get the full OpenClaw web dashboard plus SSH access. ## Plans OpenClaw instances are available on the `standard` (2 vCPU, 4 GB RAM) and `budget` (2 vCPU, 2 GB RAM) plans. See [plans and billing](/agents/introduction#plans) for details; current per-hour pricing is shown in the **Create Instance** dialog. ## Create an instance 1. Go to [Dashboard → Hosted Agents](https://deepinfra.com/dash/agents) and click **Create Instance** 2. Enter a **Name**, choose **OpenClaw** as the instance type, and pick a plan 3. Click **Create** — setup is fully automated and takes a few minutes ## Open the dashboard Once the instance is `running`, click **dashboard** in the Connect column (or **Open Dashboard** in the actions menu). The OpenClaw dashboard opens in a new tab, served securely through DeepInfra — no ports to expose or tunnels to set up. > [!NOTE] > > Right after a start, the instance can take up to \~3 minutes to finish booting. If you see a loading page, it refreshes automatically once the dashboard is up. From the dashboard you can chat with your agent, connect messaging channels, and manage its configuration — everything a self-hosted OpenClaw offers. See the [OpenClaw docs](https://docs.openclaw.ai) for what the assistant itself can do. ## Models Your instance comes pre-configured with a DeepInfra model, so it works out of the box. You can switch to any model from [our catalog](https://deepinfra.com/models/text-generation) in OpenClaw's settings — for example `deepseek-ai/DeepSeek-V4-Flash`. The agent's model calls are billed as regular inference usage against your account. ## SSH access Power users can connect directly: ```bash theme={null} ssh openclaw@ -p 2222 ``` The public IP and SSH port are shown on the instance details page. Your account's [SSH keys](https://deepinfra.com/dash/ssh_keys) are installed automatically. > [!WARNING] > > The file `/home/openclaw/config.env` is managed by DeepInfra — manual edits will be overwritten by maintenance. OpenClaw's self-update is disabled on hosted instances; upgrade through the dashboard's **Update** action instead. ## Updating and backups When a newer OpenClaw version is available, an **Update** action appears — it upgrades in place, with no restart. Backups run automatically every 24 hours; see [Backups](/agents/introduction#backups). ## FAQ **Why is the dashboard not loading right after I start the instance?** The VM boots in stages; the dashboard is the last thing to come up (up to \~3 minutes). The loading page retries automatically. **My agent stopped responding on its channels — what happened?** Check the instance state at [Dashboard → Hosted Agents](https://deepinfra.com/dash/agents). If it's `stopped` or paused for fair use, start it again. Note that the instance gets a new public IP on restart — dashboard and channel connections are unaffected, but any SSH shortcuts you saved need the new IP. **Can I customize the instance over SSH?** Yes — install packages, add skills, edit OpenClaw's own configuration. Just leave `/home/openclaw/config.env` alone and don't re-enable self-update; version upgrades go through the **Update** action. # Account Email Values Source: https://docs.deepinfra.com/api-reference/account/account-email-values https://api.deepinfra.com/openapi.json get /v1/me/emails # Account Gpu Pool Source: https://docs.deepinfra.com/api-reference/account/account-gpu-pool https://api.deepinfra.com/openapi.json get /v1/me/gpu_pool # Account Rate Limit Source: https://docs.deepinfra.com/api-reference/account/account-rate-limit https://api.deepinfra.com/openapi.json get /v1/me/rate_limit # Account Update Details Source: https://docs.deepinfra.com/api-reference/account/account-update-details https://api.deepinfra.com/openapi.json patch /v1/me # Cancel Gpu Pool Request Source: https://docs.deepinfra.com/api-reference/account/cancel-gpu-pool-request https://api.deepinfra.com/openapi.json delete /v1/me/gpu_pool/request/{request_id} Withdraw the caller's open request. # Delete Account Source: https://docs.deepinfra.com/api-reference/account/delete-account https://api.deepinfra.com/openapi.json delete /v1/me # Gpu Pool Gpu Types Source: https://docs.deepinfra.com/api-reference/account/gpu-pool-gpu-types https://api.deepinfra.com/openapi.json get /v1/gpu_pool/gpu_types # Me Source: https://docs.deepinfra.com/api-reference/account/me https://api.deepinfra.com/openapi.json get /v1/me # Request Gpu Pool Change Source: https://docs.deepinfra.com/api-reference/account/request-gpu-pool-change https://api.deepinfra.com/openapi.json post /v1/me/gpu_pool/request File or amend the caller's single open GPU limit request. # Request Rate Limit Increase Source: https://docs.deepinfra.com/api-reference/account/request-rate-limit-increase https://api.deepinfra.com/openapi.json post /v1/me/rate_limit/request # Team Set Display Name Source: https://docs.deepinfra.com/api-reference/account/team-set-display-name https://api.deepinfra.com/openapi.json post /v1/me/team_display_name # Create Backup Source: https://docs.deepinfra.com/api-reference/agents/create-backup https://api.deepinfra.com/openapi.json post /v1/agents/{instance_id}/backup # Create Dashboard Launch Token Source: https://docs.deepinfra.com/api-reference/agents/create-dashboard-launch-token https://api.deepinfra.com/openapi.json post /v1/agents/{instance_id}/launch_token Mint a single-use launch URL for the dashboard. Called by the launcher page right when readyz flips ready. The launch URL is used as a top-level navigation; /launch then sets the oc_auth cookie and 302s into the proxied dashboard. The user's bearer token is stashed in Redis under the token's jti and retrieved (atomic GETDEL) on /launch redeem — this keeps the bearer out of the URL and out of any signed payload while preserving the existing proxy auth flow (oc_auth cookie value = bearer token). Refuses instances whose agent_type has has_dashboard=False (e.g. hermes). # Create Instance Source: https://docs.deepinfra.com/api-reference/agents/create-instance https://api.deepinfra.com/openapi.json post /v1/agents # Delete Instance Source: https://docs.deepinfra.com/api-reference/agents/delete-instance https://api.deepinfra.com/openapi.json delete /v1/agents/{instance_id} # Get Catalog Source: https://docs.deepinfra.com/api-reference/agents/get-catalog https://api.deepinfra.com/openapi.json get /v1/agents/catalog # Get Instance Source: https://docs.deepinfra.com/api-reference/agents/get-instance https://api.deepinfra.com/openapi.json get /v1/agents/{instance_id} # List Backups Source: https://docs.deepinfra.com/api-reference/agents/list-backups https://api.deepinfra.com/openapi.json get /v1/agents/{instance_id}/backups # List Instances Source: https://docs.deepinfra.com/api-reference/agents/list-instances https://api.deepinfra.com/openapi.json get /v1/agents # Rename Instance Source: https://docs.deepinfra.com/api-reference/agents/rename-instance https://api.deepinfra.com/openapi.json patch /v1/agents/{instance_id} # Restore Backup Source: https://docs.deepinfra.com/api-reference/agents/restore-backup https://api.deepinfra.com/openapi.json post /v1/agents/{instance_id}/backups/{backup_id}/restore # Start Instance Source: https://docs.deepinfra.com/api-reference/agents/start-instance https://api.deepinfra.com/openapi.json post /v1/agents/{instance_id}/start # Stop Instance Source: https://docs.deepinfra.com/api-reference/agents/stop-instance https://api.deepinfra.com/openapi.json post /v1/agents/{instance_id}/stop # Update Instance Version Source: https://docs.deepinfra.com/api-reference/agents/update-instance-version https://api.deepinfra.com/openapi.json post /v1/agents/{instance_id}/update # Openai Audio Speech Source: https://docs.deepinfra.com/api-reference/audio/openai-audio-speech https://api.deepinfra.com/openapi.json post /v1/audio/speech # Openai Audio Transcriptions Source: https://docs.deepinfra.com/api-reference/audio/openai-audio-transcriptions https://api.deepinfra.com/openapi.json post /v1/audio/transcriptions # Openai Audio Translations Source: https://docs.deepinfra.com/api-reference/audio/openai-audio-translations https://api.deepinfra.com/openapi.json post /v1/audio/translations # Create Api Token Source: https://docs.deepinfra.com/api-reference/authentication/create-api-token https://api.deepinfra.com/openapi.json post /v1/api-tokens Create an API token. The reply carries the secret; no later call returns it. # Create Scoped Jwt Source: https://docs.deepinfra.com/api-reference/authentication/create-scoped-jwt https://api.deepinfra.com/openapi.json post /v1/scoped-jwt # Create Ssh Key Source: https://docs.deepinfra.com/api-reference/authentication/create-ssh-key https://api.deepinfra.com/openapi.json post /v1/ssh_keys # Delete Api Token By Id Source: https://docs.deepinfra.com/api-reference/authentication/delete-api-token-by-id https://api.deepinfra.com/openapi.json delete /v1/api-tokens/id/{token_id} Delete an API token, by token_id. token_id comes from Get Api Tokens (`GET /v1/api-tokens`). # Delete Ssh Key Source: https://docs.deepinfra.com/api-reference/authentication/delete-ssh-key https://api.deepinfra.com/openapi.json delete /v1/ssh_keys/{ssh_key_id} # Export Api Token To Vercel By Body Source: https://docs.deepinfra.com/api-reference/authentication/export-api-token-to-vercel-by-body https://api.deepinfra.com/openapi.json post /v1/api-tokens/vercel_export # Get Api Token By Id Source: https://docs.deepinfra.com/api-reference/authentication/get-api-token-by-id https://api.deepinfra.com/openapi.json get /v1/api-tokens/id/{token_id} One API token's metadata, by token_id. token_id comes from Get Api Tokens (`GET /v1/api-tokens`). # Get Api Tokens Source: https://docs.deepinfra.com/api-reference/authentication/get-api-tokens https://api.deepinfra.com/openapi.json get /v1/api-tokens # Get Ssh Keys Source: https://docs.deepinfra.com/api-reference/authentication/get-ssh-keys https://api.deepinfra.com/openapi.json get /v1/ssh_keys # Github Cli Login Source: https://docs.deepinfra.com/api-reference/authentication/github-cli-login https://api.deepinfra.com/openapi.json get /github/cli/login deepctl is calling this request waiting for auth token during login. The token is stored in /github/callback # Github Login Source: https://docs.deepinfra.com/api-reference/authentication/github-login https://api.deepinfra.com/openapi.json get /github/login Initiate github SSO login flow. Callback is /github/callback # Google Login Source: https://docs.deepinfra.com/api-reference/authentication/google-login https://api.deepinfra.com/openapi.json get /google/login Initiate Google SSO login flow. Callback is /google/callback # Inspect Scoped Jwt Source: https://docs.deepinfra.com/api-reference/authentication/inspect-scoped-jwt https://api.deepinfra.com/openapi.json get /v1/scoped-jwt # Okta Login Source: https://docs.deepinfra.com/api-reference/authentication/okta-login https://api.deepinfra.com/openapi.json get /okta/login # Add Card Checkout Source: https://docs.deepinfra.com/api-reference/billing/add-card-checkout https://api.deepinfra.com/openapi.json post /payment/payment-methods/setup Checkout page that saves a card, billing address included, so Add Funds can show it later. # Add Funds Source: https://docs.deepinfra.com/api-reference/billing/add-funds https://api.deepinfra.com/openapi.json post /payment/funds # Billing Portal Source: https://docs.deepinfra.com/api-reference/billing/billing-portal https://api.deepinfra.com/openapi.json get /payment/billing-portal # Deepstart Apply Source: https://docs.deepinfra.com/api-reference/billing/deepstart-apply https://api.deepinfra.com/openapi.json post /payment/deepstart/application # Get Checklist Source: https://docs.deepinfra.com/api-reference/billing/get-checklist https://api.deepinfra.com/openapi.json get /payment/checklist # Get Config Source: https://docs.deepinfra.com/api-reference/billing/get-config https://api.deepinfra.com/openapi.json get /payment/config # Get Key Limits Source: https://docs.deepinfra.com/api-reference/billing/get-key-limits https://api.deepinfra.com/openapi.json get /payment/key-limits # List Invoices Source: https://docs.deepinfra.com/api-reference/billing/list-invoices https://api.deepinfra.com/openapi.json get /payment/invoices # Set Config Source: https://docs.deepinfra.com/api-reference/billing/set-config https://api.deepinfra.com/openapi.json post /payment/config # Set Key Limit Source: https://docs.deepinfra.com/api-reference/billing/set-key-limit https://api.deepinfra.com/openapi.json post /payment/key-limits # Setup Topup Source: https://docs.deepinfra.com/api-reference/billing/setup-topup https://api.deepinfra.com/openapi.json post /payment/topup # Usage Source: https://docs.deepinfra.com/api-reference/billing/usage https://api.deepinfra.com/openapi.json get /payment/usage # Usage Api Token Source: https://docs.deepinfra.com/api-reference/billing/usage-api-token https://api.deepinfra.com/openapi.json get /payment/usage/{api_token} Deprecated: use Usage Api Token By Id (`GET /payment/usage/id/{token_id}`). # Usage Api Token By Id Source: https://docs.deepinfra.com/api-reference/billing/usage-api-token-by-id https://api.deepinfra.com/openapi.json get /payment/usage/id/{token_id} Usage for a single API token, by token_id. token_id comes from Get Api Tokens (`GET /v1/api-tokens`). # Usage Rent Source: https://docs.deepinfra.com/api-reference/billing/usage-rent https://api.deepinfra.com/openapi.json get /payment/usage/rent # Usage Tokens Source: https://docs.deepinfra.com/api-reference/billing/usage-tokens https://api.deepinfra.com/openapi.json get /payment/usage/tokens # Anthropic Messages Source: https://docs.deepinfra.com/api-reference/chat-completions/anthropic-messages https://api.deepinfra.com/openapi.json post /anthropic/v1/messages # Anthropic Messages Count Tokens Source: https://docs.deepinfra.com/api-reference/chat-completions/anthropic-messages-count-tokens https://api.deepinfra.com/openapi.json post /anthropic/v1/messages/count_tokens # Openai Chat Completions Source: https://docs.deepinfra.com/api-reference/chat-completions/openai-chat-completions https://api.deepinfra.com/openapi.json post /v1/chat/completions # List Models Source: https://docs.deepinfra.com/api-reference/decisions/list-models https://api.deepinfra.com/openapi.json get /typesafe/v1/models # Systemone Source: https://docs.deepinfra.com/api-reference/decisions/systemone https://api.deepinfra.com/openapi.json post /v1/decisions # Deploy Args History Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-args-history https://api.deepinfra.com/openapi.json get /deploy/{deploy_id}/config/history # Deploy Args Restore Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-args-restore https://api.deepinfra.com/openapi.json post /deploy/{deploy_id}/config/history/{entry_id}/restore # Deploy Create Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-create https://api.deepinfra.com/openapi.json post /v1/deploy # Deploy Create Hf Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-create-hf https://api.deepinfra.com/openapi.json post /deploy/hf/ # Deploy Create Llm Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-create-llm https://api.deepinfra.com/openapi.json post /deploy/llm # Deploy Delete Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-delete https://api.deepinfra.com/openapi.json delete /deploy/{deploy_id} # Deploy Detailed Stats Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-detailed-stats https://api.deepinfra.com/openapi.json get /deploy/{deploy_id}/stats2 # Deploy Gpu Availability Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-gpu-availability https://api.deepinfra.com/openapi.json get /deploy/llm/gpu_availability # Deploy List Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-list https://api.deepinfra.com/openapi.json get /deploy/list/ # Deploy List Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-list-1 https://api.deepinfra.com/openapi.json get /deploy/list # Deploy Llm Presets Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-llm-presets https://api.deepinfra.com/openapi.json get /deploy/llm/presets DeepInfra presets and mirrored vLLM recipes for ``hf_repo_id``, told apart by ``source``; empty when none. Filter by ``gpu``/``engine``/``source``. # Deploy Llm Standard Args Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-llm-standard-args https://api.deepinfra.com/openapi.json get /deploy/llm/standard_args # Deploy Llm Suggest Name Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-llm-suggest-name https://api.deepinfra.com/openapi.json get /deploy/llm/suggest_name # Deploy Rebalance Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-rebalance https://api.deepinfra.com/openapi.json post /deploy/{deploy_id}/rebalance Start a GPU pool rebalance: move GPUs from this deployment onto another deployment you own, a target instance at a time unless concurrency raises it. A source instance is never left half-moved, so one larger target instance stops several of this deployment's at once, and one larger source instance is replaced by several target instances at once. Moving all instances stops this deployment; start it again later to resume it. # Deploy Rebalance Cancel Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-rebalance-cancel https://api.deepinfra.com/openapi.json post /deploy/{deploy_id}/rebalance/cancel Stop an in-flight GPU pool rebalance; instances already moved stay, and both deployments keep min/max instances fixed at their current counts. # Deploy Rebalance Status Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-rebalance-status https://api.deepinfra.com/openapi.json get /deploy/{deploy_id}/rebalance Status of GPU pool rebalances touching this deployment. A just-started rebalance can take a moment to appear. A finished or cancelled rebalance leaves both deployments' min/max instances fixed at the final counts; edit them to resume autoscaling. # Deploy Start Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-start https://api.deepinfra.com/openapi.json post /deploy/{deploy_id}/start Start a stopped deployment. Re-creates pods via auto-scaling. # Deploy Stats Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-stats https://api.deepinfra.com/openapi.json get /deploy/{deploy_id}/stats # Deploy Status Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-status https://api.deepinfra.com/openapi.json get /deploy/{deploy_id} # Deploy Stop Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-stop https://api.deepinfra.com/openapi.json post /deploy/{deploy_id}/stop Stop a running deployment. Terminates pods. Can be restarted later. # Deploy Update Source: https://docs.deepinfra.com/api-reference/dedicated-models/deploy-update https://api.deepinfra.com/openapi.json put /deploy/{deploy_id} # Deployment Stats Source: https://docs.deepinfra.com/api-reference/dedicated-models/deployment-stats https://api.deepinfra.com/openapi.json get /deploy/stats # Openai Embeddings Source: https://docs.deepinfra.com/api-reference/embeddings/openai-embeddings https://api.deepinfra.com/openapi.json post /v1/embeddings # Cancel Openai Batch Source: https://docs.deepinfra.com/api-reference/files-&-batches/cancel-openai-batch https://api.deepinfra.com/openapi.json post /v1/batches/{batch_id}/cancel # Create Openai Batch Source: https://docs.deepinfra.com/api-reference/files-&-batches/create-openai-batch https://api.deepinfra.com/openapi.json post /v1/batches # Delete File Source: https://docs.deepinfra.com/api-reference/files-&-batches/delete-file https://api.deepinfra.com/openapi.json delete /v1/files/{file_id} # Get File Source: https://docs.deepinfra.com/api-reference/files-&-batches/get-file https://api.deepinfra.com/openapi.json get /v1/files/{file_id} # Get File Content Source: https://docs.deepinfra.com/api-reference/files-&-batches/get-file-content https://api.deepinfra.com/openapi.json get /v1/files/{file_id}/content # List Files Source: https://docs.deepinfra.com/api-reference/files-&-batches/list-files https://api.deepinfra.com/openapi.json get /v1/files # Openai Files Source: https://docs.deepinfra.com/api-reference/files-&-batches/openai-files https://api.deepinfra.com/openapi.json post /v1/files # Retrieve Openai Batch Source: https://docs.deepinfra.com/api-reference/files-&-batches/retrieve-openai-batch https://api.deepinfra.com/openapi.json get /v1/batches/{batch_id} # Retrieve Openai Batches Source: https://docs.deepinfra.com/api-reference/files-&-batches/retrieve-openai-batches https://api.deepinfra.com/openapi.json get /v1/batches # Container Rentals Delete Source: https://docs.deepinfra.com/api-reference/gpu-rentals/container-rentals-delete https://api.deepinfra.com/openapi.json delete /v1/containers/{container_id} # Container Rentals Get Source: https://docs.deepinfra.com/api-reference/gpu-rentals/container-rentals-get https://api.deepinfra.com/openapi.json get /v1/containers/{container_id} # Container Rentals Get Params Source: https://docs.deepinfra.com/api-reference/gpu-rentals/container-rentals-get-params https://api.deepinfra.com/openapi.json get /v1/containers/params # Container Rentals List Source: https://docs.deepinfra.com/api-reference/gpu-rentals/container-rentals-list https://api.deepinfra.com/openapi.json get /v1/containers # Container Rentals Start Source: https://docs.deepinfra.com/api-reference/gpu-rentals/container-rentals-start https://api.deepinfra.com/openapi.json post /v1/containers # Container Rentals Update Source: https://docs.deepinfra.com/api-reference/gpu-rentals/container-rentals-update https://api.deepinfra.com/openapi.json patch /v1/containers/{container_id} # Rent Gpu Availability Source: https://docs.deepinfra.com/api-reference/gpu-rentals/rent-gpu-availability https://api.deepinfra.com/openapi.json get /v1/containers/gpu_availability # Get Generated Image Source: https://docs.deepinfra.com/api-reference/image-generation/get-generated-image https://api.deepinfra.com/openapi.json get /v1/images/{image_id} Serve a `response_format=url` image; unauthenticated, 404 once expired. # Openai Images Edits Source: https://docs.deepinfra.com/api-reference/image-generation/openai-images-edits https://api.deepinfra.com/openapi.json post /v1/images/edits Edit image using OpenAI Images Edits API # Openai Images Generations Source: https://docs.deepinfra.com/api-reference/image-generation/openai-images-generations https://api.deepinfra.com/openapi.json post /v1/images/generations Generate image using OpenAI Images API # Openai Images Variations Source: https://docs.deepinfra.com/api-reference/image-generation/openai-images-variations https://api.deepinfra.com/openapi.json post /v1/images/variations Generate a similar image using OpenAI Images Variations API # Inference Deploy Source: https://docs.deepinfra.com/api-reference/inference/inference-deploy https://api.deepinfra.com/openapi.json post /v1/inference/deploy/{deploy_id} # Inference Model Source: https://docs.deepinfra.com/api-reference/inference/inference-model https://api.deepinfra.com/openapi.json post /v1/inference/{model_name} # Deployment Logs Query Source: https://docs.deepinfra.com/api-reference/logs-&-metrics/deployment-logs-query https://api.deepinfra.com/openapi.json get /v1/deployment_logs/query Query deployment logs. * Without timestamps (from/to) returns last `limit` messages (in last month). * With `from` only, returns first `limit` messages after `from` (inclusive). * With `to` only, returns last `limit` messages before `to` (inclusive). * With both `from` and `to`, return the first `limit` messages after `from`, but not later than `to`. * `from` and `to` should be no more than a month apart. # Get Live Metrics Source: https://docs.deepinfra.com/api-reference/logs-&-metrics/get-live-metrics https://api.deepinfra.com/openapi.json get /v1/metrics/live Get the latest values for the Live metrics section on the web front page. # Get Request Costs Source: https://docs.deepinfra.com/api-reference/logs-&-metrics/get-request-costs https://api.deepinfra.com/openapi.json post /v1/request-costs # Logs Query Source: https://docs.deepinfra.com/api-reference/logs-&-metrics/logs-query https://api.deepinfra.com/openapi.json get /v1/logs/query Query inference logs. * Without timestamps (from/to) returns last `limit` messages (in last month). * With `from` only, returns first `limit` messages after `from` (inclusive). * With `to` only, returns last `limit` messages before `to` (inclusive). * With both `from` and `to`, return the first `limit` messages after `from`, but not later than `to`. * `from` and `to` should be no more than a month apart. # Create Lora Source: https://docs.deepinfra.com/api-reference/lora-adapters/create-lora https://api.deepinfra.com/openapi.json post /v1/lora/create # Delete Lora Source: https://docs.deepinfra.com/api-reference/lora-adapters/delete-lora https://api.deepinfra.com/openapi.json delete /v1/lora/{lora_name} # Delete Lora Model Source: https://docs.deepinfra.com/api-reference/lora-adapters/delete-lora-model https://api.deepinfra.com/openapi.json delete /lora-model/{lora_model_name} # Get Lora Source: https://docs.deepinfra.com/api-reference/lora-adapters/get-lora https://api.deepinfra.com/openapi.json get /v1/lora/{lora_name} # Get Lora Status Source: https://docs.deepinfra.com/api-reference/lora-adapters/get-lora-status https://api.deepinfra.com/openapi.json get /v1/lora/{lora_name}/status # Get Model Loras Source: https://docs.deepinfra.com/api-reference/lora-adapters/get-model-loras https://api.deepinfra.com/openapi.json get /v1/model/{model_name}/loras # Get User Loras Source: https://docs.deepinfra.com/api-reference/lora-adapters/get-user-loras https://api.deepinfra.com/openapi.json get /v1/user/loras # Update Lora Source: https://docs.deepinfra.com/api-reference/lora-adapters/update-lora https://api.deepinfra.com/openapi.json patch /v1/lora/{lora_name} # Upload Lora Model Source: https://docs.deepinfra.com/api-reference/lora-adapters/upload-lora-model https://api.deepinfra.com/openapi.json post /lora-model # Get Hardware Source: https://docs.deepinfra.com/api-reference/models/get-hardware https://api.deepinfra.com/openapi.json get /v2/hardware # Model Delete Source: https://docs.deepinfra.com/api-reference/models/model-delete https://api.deepinfra.com/openapi.json delete /models/{model_name} # Model Families Names Source: https://docs.deepinfra.com/api-reference/models/model-families-names https://api.deepinfra.com/openapi.json get /model-families/names # Model Family Source: https://docs.deepinfra.com/api-reference/models/model-family https://api.deepinfra.com/openapi.json get /model-families/{family_name} # Model Meta Update Source: https://docs.deepinfra.com/api-reference/models/model-meta-update https://api.deepinfra.com/openapi.json post /models/{model_name}/meta # Model Publicity Source: https://docs.deepinfra.com/api-reference/models/model-publicity https://api.deepinfra.com/openapi.json post /models/{model_name}/publicity # Model Schema Source: https://docs.deepinfra.com/api-reference/models/model-schema https://api.deepinfra.com/openapi.json get /models/{model_name}/schema/{variantKey} # Models Deployment List Source: https://docs.deepinfra.com/api-reference/models/models-deployment-list https://api.deepinfra.com/openapi.json get /models/deployment/list # Models Featured Source: https://docs.deepinfra.com/api-reference/models/models-featured https://api.deepinfra.com/openapi.json get /models/featured # Models Info Source: https://docs.deepinfra.com/api-reference/models/models-info https://api.deepinfra.com/openapi.json get /models/{model_name} # Models List Source: https://docs.deepinfra.com/api-reference/models/models-list https://api.deepinfra.com/openapi.json get /models/list # Models Lora List Source: https://docs.deepinfra.com/api-reference/models/models-lora-list https://api.deepinfra.com/openapi.json get /models/lora/list # Openai Models Source: https://docs.deepinfra.com/api-reference/models/openai-models https://api.deepinfra.com/openapi.json get /v1/models # Openrouter Models Source: https://docs.deepinfra.com/api-reference/models/openrouter-models https://api.deepinfra.com/openapi.json get /openrouter/models # Private Models List Source: https://docs.deepinfra.com/api-reference/models/private-models-list https://api.deepinfra.com/openapi.json get /models/private/list # Openai Responses Source: https://docs.deepinfra.com/api-reference/responses/openai-responses https://api.deepinfra.com/openapi.json post /v1/responses # Openai Responses Not Found Source: https://docs.deepinfra.com/api-reference/responses/openai-responses-not-found https://api.deepinfra.com/openapi.json get /v1/responses/{response_id} # Openai Responses Not Found Source: https://docs.deepinfra.com/api-reference/responses/openai-responses-not-found-1 https://api.deepinfra.com/openapi.json delete /v1/responses/{response_id} # Create Sandbox Source: https://docs.deepinfra.com/api-reference/sandboxes/create-sandbox https://api.deepinfra.com/openapi.json post /v1/sandboxes Create a new sandbox instance with the given plan and settings. The sandbox starts in CREATING state and transitions to RUNNING asynchronously. # Delete Sandbox Source: https://docs.deepinfra.com/api-reference/sandboxes/delete-sandbox https://api.deepinfra.com/openapi.json delete /v1/sandboxes/{sandbox_id} # Exec Command Source: https://docs.deepinfra.com/api-reference/sandboxes/exec-command https://api.deepinfra.com/openapi.json post /v1/sandboxes/{sandbox_id}/exec Run a command in the sandbox. Streams NDJSON lines (application/x-ndjson): {"stdout": ...}/{"stderr": ...} chunks followed by exactly one terminal {"returncode": N} or {"error": msg} line. # Get Sandbox Source: https://docs.deepinfra.com/api-reference/sandboxes/get-sandbox https://api.deepinfra.com/openapi.json get /v1/sandboxes/{sandbox_id} # List Sandbox Plans Source: https://docs.deepinfra.com/api-reference/sandboxes/list-sandbox-plans https://api.deepinfra.com/openapi.json get /v1/sandboxes/catalog Returns all available sandbox plans with their resource specs and pricing. # List Sandboxes Source: https://docs.deepinfra.com/api-reference/sandboxes/list-sandboxes https://api.deepinfra.com/openapi.json get /v1/sandboxes # Read File Source: https://docs.deepinfra.com/api-reference/sandboxes/read-file https://api.deepinfra.com/openapi.json get /v1/sandboxes/{sandbox_id}/fs/content Read a file from an absolute path inside the sandbox; returns raw bytes (application/octet-stream). # Start Sandbox Source: https://docs.deepinfra.com/api-reference/sandboxes/start-sandbox https://api.deepinfra.com/openapi.json post /v1/sandboxes/{sandbox_id}/start # Stop Sandbox Source: https://docs.deepinfra.com/api-reference/sandboxes/stop-sandbox https://api.deepinfra.com/openapi.json post /v1/sandboxes/{sandbox_id}/stop # Write File Source: https://docs.deepinfra.com/api-reference/sandboxes/write-file https://api.deepinfra.com/openapi.json put /v1/sandboxes/{sandbox_id}/fs/content Write the raw request body (application/octet-stream, max 100 MiB) to an absolute path inside the sandbox. # Openai Completions Source: https://docs.deepinfra.com/api-reference/text-completions/openai-completions https://api.deepinfra.com/openapi.json post /v1/completions # Create Voice Source: https://docs.deepinfra.com/api-reference/text-to-speech/create-voice https://api.deepinfra.com/openapi.json post /v1/voices/add Create a new voice # Delete Voice Source: https://docs.deepinfra.com/api-reference/text-to-speech/delete-voice https://api.deepinfra.com/openapi.json delete /v1/voices/{voice_id} # Get Voice Source: https://docs.deepinfra.com/api-reference/text-to-speech/get-voice https://api.deepinfra.com/openapi.json get /v1/voices/{voice_id} Get a voice by its id # Get Voices Source: https://docs.deepinfra.com/api-reference/text-to-speech/get-voices https://api.deepinfra.com/openapi.json get /v1/voices Get available voices for a given user # Text To Speech Source: https://docs.deepinfra.com/api-reference/text-to-speech/text-to-speech https://api.deepinfra.com/openapi.json post /v1/text-to-speech/{voice_id} # Text To Speech Stream Source: https://docs.deepinfra.com/api-reference/text-to-speech/text-to-speech-stream https://api.deepinfra.com/openapi.json post /v1/text-to-speech/{voice_id}/stream # Update Voice Source: https://docs.deepinfra.com/api-reference/text-to-speech/update-voice https://api.deepinfra.com/openapi.json post /v1/voices/{voice_id}/edit # Detokenize Source: https://docs.deepinfra.com/api-reference/tokenizer/detokenize https://api.deepinfra.com/openapi.json post /v1/detokenize # Tokenize Source: https://docs.deepinfra.com/api-reference/tokenizer/tokenize https://api.deepinfra.com/openapi.json post /v1/tokenize # Cli Version Source: https://docs.deepinfra.com/api-reference/utilities/cli-version https://api.deepinfra.com/openapi.json get /cli/version # Submit Feedback Source: https://docs.deepinfra.com/api-reference/utilities/submit-feedback https://api.deepinfra.com/openapi.json post /v1/feedback Submit feedback # Create Video Generation Source: https://docs.deepinfra.com/api-reference/videos/create-video-generation https://api.deepinfra.com/openapi.json post /v1/videos # Create Video Generation Source: https://docs.deepinfra.com/api-reference/videos/create-video-generation-1 https://api.deepinfra.com/openapi.json post /v1/openai/videos # Get Video Content Source: https://docs.deepinfra.com/api-reference/videos/get-video-content https://api.deepinfra.com/openapi.json get /v1/videos/{video_id}/content # Get Video Content Source: https://docs.deepinfra.com/api-reference/videos/get-video-content-1 https://api.deepinfra.com/openapi.json get /v1/openai/videos/{video_id}/content # Get Video Generation Source: https://docs.deepinfra.com/api-reference/videos/get-video-generation https://api.deepinfra.com/openapi.json get /v1/videos/{video_id} # Get Video Generation Source: https://docs.deepinfra.com/api-reference/videos/get-video-generation-1 https://api.deepinfra.com/openapi.json get /v1/openai/videos/{video_id} # Text Completions Source: https://docs.deepinfra.com/apis/completions Legacy OpenAI-compatible completions API for raw text generation. The completions API is the legacy text generation interface — you provide a raw prompt string and the model continues it. For most use cases, the [Chat Completions API](/chat/overview) is simpler and recommended instead. The endpoint is: ``` POST https://api.deepinfra.com/v1/openai/completions ``` > [!WARNING] > > This is an advanced API. You need to know your model's exact prompt format. Different models have different input formats. Check the model's API section on its page for the expected format. ## Example The example below uses `deepseek-ai/DeepSeek-V4-Flash-0731` with its prompt format: ```python Python theme={null} import os from openai import OpenAI openai = OpenAI( api_key=os.environ["DEEPINFRA_API_KEY"], base_url="https://api.deepinfra.com/v1/openai", ) stream = True # or False completion = openai.completions.create( model="deepseek-ai/DeepSeek-V4-Flash-0731", prompt="<｜begin▁of▁sentence｜><｜User｜>Hello!<｜Assistant｜>", stop=["<｜end▁of▁sentence｜>"], stream=stream, ) if stream: for event in completion: if event.choices[0].finish_reason: print(event.choices[0].finish_reason, event.usage.prompt_tokens, event.usage.completion_tokens) else: print(event.choices[0].text, end="", flush=True) else: print(completion.choices[0].text) print(completion.usage.prompt_tokens, completion.usage.completion_tokens) ``` ```javascript JavaScript theme={null} import OpenAI from "openai"; const openai = new OpenAI({ baseURL: "https://api.deepinfra.com/v1/openai", apiKey: process.env.DEEPINFRA_API_KEY, }); const stream = true; // or false const completion = await openai.completions.create({ model: "deepseek-ai/DeepSeek-V4-Flash-0731", prompt: "<｜begin▁of▁sentence｜><｜User｜>Hello!<｜Assistant｜>", stream: stream, stop: ["<｜end▁of▁sentence｜>"] }); if (stream) { for await (const chunk of completion) { if (chunk.choices[0].finish_reason) { console.log(chunk.choices[0].finish_reason, chunk.usage.prompt_tokens, chunk.usage.completion_tokens); } else { process.stdout.write(chunk.choices[0].text); } } } else { console.log(completion.choices[0].text); console.log(completion.usage.prompt_tokens, completion.usage.completion_tokens); } ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "prompt": "<｜begin▁of▁sentence｜><｜User｜>Hello!<｜Assistant｜>", "stop": [ "<｜end▁of▁sentence｜>" ] }' ``` ## Supported parameters | Parameter | Notes | | - | - | | `model` | Model name or `MODEL_NAME:VERSION` | | `prompt` | Raw prompt string in the model's expected format | | `max_tokens` | Max tokens to generate. Defaults to the model's max context length minus input length | | `stream` | Stream output via SSE instead of returning the full response at once. Default: `false` | | `temperature` | Sampling temperature between 0 and 2. Higher values produce more random output; lower values more deterministic. Default: `1.0` | | `top_p` | Nucleus sampling threshold — only tokens comprising the top `top_p` probability mass are considered. Default: `1.0` | | `stop` | Up to 4 sequences where the API will stop generating further tokens | | `n` | Number of completion sequences to return. Default: `1` | | `echo` | If `true`, the prompt is included at the start of the returned text | | `logprobs` | Return log probabilities for the generated tokens | For every model, you can check its prompt format in the API section on its page. For the complete parameter reference, see the [API reference](/api-reference/text-completions/openai-completions). # DeepInfra Native API Source: https://docs.deepinfra.com/apis/deepinfra-native Advanced API with access to all model types including image generation, speech, and zero-shot image classification. The DeepInfra Native API gives you access to every model we provide, including model types not covered by the OpenAI-compatible API: image generation, speech recognition, and zero-shot image classification. For LLMs and embeddings, the [OpenAI-compatible API](/chat/overview) is simpler and recommended. Use the native API when you need model types beyond LLMs/embeddings, or when you need features like [webhooks](/account/webhooks) or [log probabilities](/chat/log-probs). The base endpoint is: ``` https://api.deepinfra.com/v1/inference/{model_name} ``` ## JavaScript client ```bash theme={null} npm install deepinfra ``` ## Text Generation (LLMs) This endpoint takes a **raw prompt** — it does not apply the model's chat template. Pass pre-formatted text, or use the [OpenAI-compatible chat API](/chat/overview), which applies the template for you. Each model's page lists its chat template. ```javascript theme={null} import { TextGeneration } from "deepinfra"; const client = new TextGeneration( "https://api.deepinfra.com/v1/inference/deepseek-ai/DeepSeek-V4-Flash-0731", process.env.DEEPINFRA_API_KEY ); const res = await client.generate({ input: "The capital of France is", max_new_tokens: 20 }); console.log(res.results[0].generated_text); console.log(res.num_input_tokens, res.num_tokens); ``` ```bash theme={null} curl "https://api.deepinfra.com/v1/inference/deepseek-ai/DeepSeek-V4-Flash-0731" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "input": "The capital of France is", "max_new_tokens": 20, "stream": false }' ``` ## Embeddings ```javascript theme={null} import { Embeddings } from "deepinfra"; const client = new Embeddings("Qwen/Qwen3-Embedding-8B", process.env.DEEPINFRA_API_KEY); const output = await client.generate({ inputs: [ "What is the capital of France?", "What is the capital of Germany?", ], }); console.log(output.embeddings[0]); ``` ```bash theme={null} curl -X POST \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -F 'inputs=["I like chocolate"]' \ 'https://api.deepinfra.com/v1/inference/Qwen/Qwen3-Embedding-8B' ``` ## Image Generation ```javascript theme={null} import { TextToImage } from "deepinfra"; import { createWriteStream } from "fs"; import { Readable } from "stream"; const model = new TextToImage("stabilityai/sdxl-turbo", process.env.DEEPINFRA_API_KEY); const response = await model.generate({ prompt: "a burger with a funny hat on the beach", }); const result = await fetch(response.images[0]); if (result.ok && result.body) { Readable.fromWeb(result.body).pipe(createWriteStream("image.jpg")); } ``` ```bash theme={null} curl "https://api.deepinfra.com/v1/inference/stabilityai/sdxl-turbo" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{"prompt": "a burger with a funny hat on the beach"}' ``` ## Speech Recognition ```bash theme={null} curl -X POST \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -F audio=@audio.mp3 \ 'https://api.deepinfra.com/v1/inference/openai/whisper-large-v3' ``` ## Zero-Shot Image Classification ```bash theme={null} curl -X POST \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -F image=@image.jpg \ -F 'candidate_labels=["dog", "cat", "car", "horse", "person"]' \ 'https://api.deepinfra.com/v1/inference/openai/clip-vit-base-patch32' ``` ## HTTP / other languages The native API is plain HTTP — you can use it from any language (Go, C#, Java, PHP, Ruby, C++, etc.) without any SDK dependency. # Embeddings Source: https://docs.deepinfra.com/apis/embeddings Generate embedding vectors from text and images using the OpenAI-compatible embeddings API. DeepInfra supports the OpenAI embeddings API for all [embedding models](https://deepinfra.com/models/embeddings). The endpoint is: ``` POST https://api.deepinfra.com/v1/openai/embeddings ``` ## Example ```python Python theme={null} import os from openai import OpenAI openai = OpenAI( api_key=os.environ["DEEPINFRA_API_KEY"], base_url="https://api.deepinfra.com/v1/openai", ) input_text = "The food was delicious and the waiter..." # Or a list: ["hello", "world"] embeddings = openai.embeddings.create( model="Qwen/Qwen3-Embedding-8B", input=input_text, encoding_format="float" ) print(embeddings.data[0].embedding) print(embeddings.usage.prompt_tokens) ``` ```javascript JavaScript theme={null} import OpenAI from "openai"; const openai = new OpenAI({ baseURL: "https://api.deepinfra.com/v1/openai", apiKey: process.env.DEEPINFRA_API_KEY, }); const input = "The quick brown fox jumped over the lazy dog"; // Or an array: ["hello", "world"] const embedding = await openai.embeddings.create({ model: "Qwen/Qwen3-Embedding-8B", input: input, encoding_format: "float", }); console.log(embedding.data[0].embedding); console.log(embedding.usage.prompt_tokens); ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/embeddings" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "input": "The food was delicious and the waiter...", "model": "Qwen/Qwen3-Embedding-8B", "encoding_format": "float" }' ``` ## Batch embeddings Pass an array as `input` to embed multiple texts in a single request: ```python theme={null} embeddings = openai.embeddings.create( model="Qwen/Qwen3-Embedding-8B", input=["Hello", "World", "How are you?"], encoding_format="float" ) for i, item in enumerate(embeddings.data): print(f"Text {i}: {item.embedding[:5]}...") # First 5 dims ``` ## Image inputs Multimodal embedding models such as `nvidia/Nemotron-3.5-Embed-1B-BF16` and `nvidia/Nemotron-3.5-Embed-8B-BF16` embed images, for example screenshots of document pages, into the same vector space as text. To embed an image, pass an `input` item as a list of content parts instead of a string. The parts of one item are embedded together, so a page image and its extracted text produce a single embedding. These are asymmetric retrieval models: set `input_type` to `document` for the pages you index and to `query` for search queries. ```python theme={null} import base64 with open("report-page-1.png", "rb") as f: page_url = "data:image/png;base64," + base64.b64encode(f.read()).decode() # An https:// URL to the image works too. documents = openai.embeddings.create( model="nvidia/Nemotron-3.5-Embed-1B-BF16", input=[ "Revenue grew 12% year over year.", [{"type": "image_url", "image_url": {"url": page_url}}], [ {"type": "text", "text": "Q3 report, page 1"}, {"type": "image_url", "image_url": {"url": page_url}}, ], ], encoding_format="float", extra_body={"input_type": "document"}, ) query = openai.embeddings.create( model="nvidia/Nemotron-3.5-Embed-1B-BF16", input="How fast did revenue grow last quarter?", encoding_format="float", extra_body={"input_type": "query"}, ) print(documents.usage) ``` The Nemotron-3.5-Embed models take at most one image per item. ### Image tokens The response reports the tokens of the items that contain an image as `image_tokens`: ```json theme={null} "usage": { "prompt_tokens": 2428, "total_tokens": 2428, "prompt_tokens_details": { "image_tokens": 2415 } } ``` `prompt_tokens` is the total. `image_tokens` is the part billed at the model's image-token price, and the rest is billed at its regular input price. Text-only items (the first item above) bill at the input price. Text inside an item that also contains an image (the third item) counts as image tokens. Prices are listed on each model's page. For the Nemotron-3.5-Embed models, an image is scaled so that its longest edge is at most 1120 px (smaller images are not scaled up) and split into a grid of 28 px tiles. It costs `rows × columns + rows` tokens: | Image | Tokens | | - | - | | A4 or Letter page | about 1,200–1,280 | | 1920×1080 screenshot | 943 | | 512×512 | 380 | | 1120×1120 or larger square (maximum) | 1,640 | ## Supported parameters | Parameter | Notes | | - | - | | `model` | Embedding model name | | `input` | String, or an array of strings and content-part lists (see [Image inputs](#image-inputs)) | | `input_type` | `query` or `document` (alias `passage`), for asymmetric retrieval models | | `encoding_format` | `float` only | ## Available models Browse [all embedding models](https://deepinfra.com/models/embeddings) — includes Qwen3 Embedding, BAAI/bge, sentence-transformers, and more. # Image Generation Source: https://docs.deepinfra.com/apis/image-generation Generate images from text prompts using the OpenAI-compatible images API. DeepInfra supports the OpenAI-compatible image generation API. The default model is FLUX Schnell. The endpoint is: ``` POST https://api.deepinfra.com/v1/openai/images/generations ``` Browse [all text-to-image models](https://deepinfra.com/models/text-to-image). ## Example ```python Python theme={null} import os import io import base64 from PIL import Image from openai import OpenAI client = OpenAI( api_key=os.environ["DEEPINFRA_API_KEY"], base_url="https://api.deepinfra.com/v1/openai" ) response = client.images.generate( prompt="A photo of an astronaut riding a horse on Mars.", size="1024x1024", n=1, ) b64_json = response.data[0].b64_json image_bytes = base64.b64decode(b64_json) image = Image.open(io.BytesIO(image_bytes)) image.save("output.png") ``` ```javascript JavaScript theme={null} import * as fs from "fs"; import OpenAI from "openai"; const openai = new OpenAI({ baseURL: "https://api.deepinfra.com/v1/openai", apiKey: process.env.DEEPINFRA_API_KEY, }); const response = await openai.images.generate({ prompt: "A photo of an astronaut riding a horse on Mars.", size: "1024x1024", n: 1, }); const b64Json = response.data[0].b64_json; const imageBuffer = Buffer.from(b64Json, "base64"); fs.writeFileSync("output.png", imageBuffer); ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/images/generations" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "prompt": "A photo of an astronaut riding a horse on Mars.", "size": "1024x1024", "n": 1 }' ``` ## Supported parameters | Parameter | Notes | | - | - | | `prompt` | Text description of the image | | `model` | Defaults to FLUX Schnell | | `size` | Image dimensions (e.g., `"1024x1024"`) | | `n` | Number of images to generate | | `response_format` | Only `b64_json` supported | | `quality`, `style` | Available for compatibility only | ## LoRA image adapters You can also use custom LoRA adapters for image generation — see [LoRA for Image Generation](/private-models/lora-image). ## Tutorial For a deeper example including advanced options, see the [Stable Diffusion tutorial](/tutorials/stable-diffusion). # Reranking Source: https://docs.deepinfra.com/apis/reranker Rerank a list of documents by relevance to a query. Reranker models take a query and a list of candidate documents and return a relevance score for each document. They're typically used as a second-pass filter after an initial vector search to improve retrieval quality in RAG pipelines. Browse [all reranker models](https://deepinfra.com/models/reranker). ## Endpoint ``` POST https://api.deepinfra.com/v1/inference/{model_name} ``` ## Example ```python Python theme={null} import os import requests DEEPINFRA_API_KEY = os.environ["DEEPINFRA_API_KEY"] MODEL = "Qwen/Qwen3-Reranker-8B" response = requests.post( f"https://api.deepinfra.com/v1/inference/{MODEL}", headers={ "Authorization": f"Bearer {DEEPINFRA_API_KEY}", "Content-Type": "application/json", }, json={ "queries": ["What is the capital of France?"], "documents": [ "Paris is the capital and most populous city of France.", "Berlin is the capital of Germany.", "The Eiffel Tower is located in Paris.", "France is a country in Western Europe.", ], }, ) result = response.json() for item in result["scores"]: print(item) ``` ```bash cURL theme={null} curl -X POST \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "queries": ["What is the capital of France?"], "documents": [ "Paris is the capital and most populous city of France.", "Berlin is the capital of Germany.", "The Eiffel Tower is located in Paris.", "France is a country in Western Europe." ] }' \ 'https://api.deepinfra.com/v1/inference/Qwen/Qwen3-Reranker-8B' ``` ## Response ```json theme={null} { "scores": [0.9485635161399841, 0.00010493390436749905, 0.020901929587125778, 0.00019765298929996789], "input_tokens": 348 } ``` Scores are in the same order as the input documents, and a higher score means a more relevant document. Sort by score descending to get the most relevant documents first. What a score means depends on the model: | Model | Score | | - | - | | `Qwen/Qwen3-Reranker-0.6B`, `Qwen/Qwen3-Reranker-4B`, `Qwen/Qwen3-Reranker-8B` | Probability in \[0, 1] | | `nvidia/llama-nemotron-rerank-vl-1b-v2` | Probability in \[0, 1] | | `nvidia/Nemotron-3.5-Rerank-8B-BF16` | Raw logit, unbounded and often negative | Compare scores only within one model. A cutoff tuned on one reranker (for example, keep documents scoring above 0.5) does not carry over to another. `queries` and `documents` must be the same length, except that a single query is broadcast across many documents (the common case shown above). The Qwen3 rerankers take an optional `instruction` field that steers the reranking task; it defaults to `"Given a web search query, retrieve relevant passages that answer the query"`. Other models ignore it. ## Usage in a RAG pipeline A typical pattern: 1. **Retrieve** — run a vector similarity search to fetch the top-N candidate chunks (e.g. top 50) 2. **Rerank** — pass the query + candidates to a reranker to get relevance scores 3. **Select** — keep only the top-K highest-scoring chunks (e.g. top 5) for the LLM context This two-stage approach improves precision significantly compared to embedding similarity alone. ```python theme={null} # 1. Get initial candidates from your vector DB candidates = vector_db.search(query, top_k=50) # 2. Rerank response = requests.post( "https://api.deepinfra.com/v1/inference/Qwen/Qwen3-Reranker-8B", headers={"Authorization": f"Bearer {DEEPINFRA_API_KEY}", "Content-Type": "application/json"}, json={"queries": [query], "documents": [c["text"] for c in candidates]}, ) scores = response.json()["scores"] # 3. Select top-K ranked = sorted(zip(scores, candidates), reverse=True) top_chunks = [doc for _, doc in ranked[:5]] ``` ## Available models Browse [all reranker models](https://deepinfra.com/models/reranker). # Speech Recognition Source: https://docs.deepinfra.com/apis/speech Transcribe audio to text using Whisper and other speech recognition models. DeepInfra hosts [Whisper](https://github.com/openai/whisper) and other speech recognition models. Given an audio file, they produce transcribed text with per-sentence timestamps. Browse [all speech recognition models](https://deepinfra.com/models/automatic-speech-recognition). ## Models Browse [all ASR models](https://deepinfra.com/models/automatic-speech-recognition). ## Example ```bash cURL theme={null} curl -X POST \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -F audio=@audio.mp3 \ 'https://api.deepinfra.com/v1/inference/openai/whisper-large-v3' ``` ```javascript JavaScript theme={null} import { AutomaticSpeechRecognition } from "deepinfra"; import path from "path"; import { fileURLToPath } from "url"; const __filename = fileURLToPath(import.meta.url); const __dirname = path.dirname(__filename); const DEEPINFRA_API_KEY = process.env.DEEPINFRA_API_KEY; const MODEL = "openai/whisper-large-v3"; const client = new AutomaticSpeechRecognition(MODEL, DEEPINFRA_API_KEY); const input = { audio: path.join(__dirname, "audio.mp3"), }; const response = await client.generate(input); console.log(response.text); ``` ## Supported audio formats * `mp3` * `wav` ## Response ```json theme={null} { "text": "Hello, this is a transcription of the audio file.", "segments": [ { "start": 0.0, "end": 3.5, "text": "Hello, this is a transcription of the audio file." } ] } ``` ## Additional parameters Each model exposes different parameters (language, task, etc.). Check the model's API documentation page for details. ## Tutorial See the [Whisper tutorial](/tutorials/whisper) for a complete walkthrough. # Text to Speech Source: https://docs.deepinfra.com/apis/text-to-speech Convert text to natural-sounding audio using TTS models. DeepInfra hosts text-to-speech models that convert text into natural-sounding audio. Browse [all TTS models](https://deepinfra.com/models/text-to-speech). ## Endpoint ``` POST https://api.deepinfra.com/v1/inference/{model_name} ``` ## Example ```python Python theme={null} import os import requests DEEPINFRA_API_KEY = os.environ["DEEPINFRA_API_KEY"] MODEL = "hexgrad/Kokoro-82M" response = requests.post( f"https://api.deepinfra.com/v1/inference/{MODEL}", headers={ "Authorization": f"Bearer {DEEPINFRA_API_KEY}", "Content-Type": "application/json", }, json={ "text": "Hello! This is a text-to-speech example using DeepInfra.", }, ) # Save the returned audio with open("output.wav", "wb") as f: f.write(response.content) ``` ```bash cURL theme={null} curl -X POST \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -H "Content-Type: application/json" \ -d '{"text": "Hello! This is a text-to-speech example using DeepInfra."}' \ 'https://api.deepinfra.com/v1/inference/hexgrad/Kokoro-82M' \ --output output.wav ``` ## Additional parameters Each model may expose additional parameters such as voice selection, speed, and language. Check the model's individual [API documentation page](https://deepinfra.com/models/text-to-speech) for supported options. ## Available models Browse [all text-to-speech models](https://deepinfra.com/models/text-to-speech). # Text to Video Source: https://docs.deepinfra.com/apis/text-to-video Generate video clips from text prompts. DeepInfra hosts text-to-video models that generate short video clips from a text description. Browse [all text-to-video models](https://deepinfra.com/models/text-to-video). ## Endpoint ``` POST https://api.deepinfra.com/v1/inference/{model_name} ``` ## Example ```python Python theme={null} import os import requests DEEPINFRA_API_KEY = os.environ["DEEPINFRA_API_KEY"] MODEL = "Wan-AI/Wan2.2-T2V-A14B" response = requests.post( f"https://api.deepinfra.com/v1/inference/{MODEL}", headers={ "Authorization": f"Bearer {DEEPINFRA_API_KEY}", "Content-Type": "application/json", }, json={ "prompt": "A serene mountain lake at sunrise, with mist rising from the water and pine trees reflected on the surface.", }, ) result = response.json() # video_url is a `data:video/mp4;base64,...` URI, not an HTTP link. # Decode it to save the clip: import base64 header, _, payload = result["video_url"].partition(",") with open("video.mp4", "wb") as f: f.write(base64.b64decode(payload)) ``` ```bash cURL theme={null} curl -X POST \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -H "Content-Type: application/json" \ -d '{"prompt": "A serene mountain lake at sunrise, with mist rising from the water and pine trees reflected on the surface."}' \ 'https://api.deepinfra.com/v1/inference/Wan-AI/Wan2.2-T2V-A14B' ``` ## Response ```json theme={null} { "request_id": "RwKufnhcUDyHEYbVN2W1GWgr", "inference_status": { "runtime_ms": 267363, "cost": 0.375, "output_length": 5 }, "video_url": "data:video/mp4;base64,AAAAIGZ0eXBpc29tAAACAGlzb21pc28y...", "seed": null } ``` `video_url` carries the whole clip inline as a `data:` URI — it is not an HTTP link, so decode the base64 payload to write the file. `output_length` is the clip duration in seconds, which is what these models are billed on. ## Tips for good prompts * Be descriptive about the scene, lighting, and motion * Specify the camera movement if relevant (e.g. "slow pan", "aerial shot", "close-up") * Keep prompts focused — overly complex prompts can produce inconsistent results * Use the [negative prompt](https://deepinfra.com/models/text-to-video) parameter (if supported) to exclude unwanted elements ## Async inference Video generation is compute-intensive and may take longer than text inference. Consider using [webhooks](/account/webhooks) to receive the result asynchronously rather than polling. ## Available models Browse [all text-to-video models](https://deepinfra.com/models/text-to-video). # Batch Endpoints Source: https://docs.deepinfra.com/batch/batch-endpoints Reference for the Batch API endpoints — create, retrieve, list, and cancel batch jobs. An **OpenAI-compatible** Batch API for submitting and managing asynchronous inference jobs. All endpoints are relative to: ``` https://api.deepinfra.com/v1/openai ``` These endpoints operate on the [Batch object](/batch/batch-objects). ## Create a batch ``` POST /batches ``` Creates and starts executing a batch job from an uploaded file of requests. Creating a batch requires the following parameters: > **completion\_window** > > The time window after which the batch expires. Currently only `"24h"` is supported. > **endpoint** > > The endpoint to run the batch against. One of the batch-supported endpoints, currently `/v1/chat/completions`, `/v1/completions`, or `/v1/embeddings`. Must match the `url` used on every line of the input file. > **input\_file\_id** > > The `id` of the uploaded input file (created with `purpose="batch"`). > **metadata (optional)** > > Up to 16 key–value string pairs, where each key is a string of up to 64 characters and each value is a string of up to 512 characters. > **output\_expires\_after (optional)** > > Controls how long the output and error files remain available. An object with two fields: > > * **`anchor`** (optional) — must be `"created_at"`. The expiry is measured from when the output file is created. Defaults to `"created_at"`. > * **`seconds`** (optional) — the number of seconds the file stays available after the anchor. An integer between `3600` (1 hour) and `2592000` (30 days). Defaults to `2592000` (30 days). This endpoint returns a [Batch object](/batch/batch-objects). ```python Python theme={null} import os from openai import OpenAI client = OpenAI( api_key=os.environ["DEEPINFRA_API_KEY"], base_url="https://api.deepinfra.com/v1/openai", ) batch = client.batches.create( input_file_id="file_abc123", endpoint="/v1/chat/completions", completion_window="24h", metadata={"description": "nightly eval run"}, output_expires_after={"anchor": "created_at", "seconds": 604800}, ) print(batch.id, batch.status) ``` ```javascript JavaScript theme={null} import OpenAI from "openai"; const client = new OpenAI({ apiKey: process.env.DEEPINFRA_API_KEY, baseURL: "https://api.deepinfra.com/v1/openai", }); const batch = await client.batches.create({ input_file_id: "file_abc123", endpoint: "/v1/chat/completions", completion_window: "24h", metadata: { description: "nightly eval run" }, output_expires_after: { anchor: "created_at", seconds: 604800 }, }); console.log(batch.id, batch.status); ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/batches" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "input_file_id": "file_abc123", "endpoint": "/v1/chat/completions", "completion_window": "24h", "metadata": {"description": "nightly eval run"}, "output_expires_after": {"anchor": "created_at", "seconds": 604800} }' ``` ```json theme={null} { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": null, "input_file_id": "file_abc123", "completion_window": "24h", "status": "validating", "output_file_id": null, "error_file_id": null, "created_at": 1711471533, "in_progress_at": null, "expires_at": null, "finalizing_at": null, "completed_at": null, "failed_at": null, "expired_at": null, "cancelling_at": null, "cancelled_at": null, "request_counts": null, "metadata": { "description": "nightly eval run" }, "model": null, "usage": null } ``` ## Retrieve a batch ``` GET /batches/{batch_id} ``` Returns information about a specific batch. Retrieving a batch requires the following parameters: > **batch\_id** > > The `id` of the batch to retrieve. This endpoint returns a [Batch object](/batch/batch-objects). ```python Python theme={null} batch = client.batches.retrieve("batch_abc123") print(batch.id, batch.status) ``` ```javascript JavaScript theme={null} const batch = await client.batches.retrieve("batch_abc123"); console.log(batch.id, batch.status); ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/batches/batch_abc123" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" ``` ```json theme={null} { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": null, "input_file_id": "file_abc123", "completion_window": "24h", "status": "in_progress", "output_file_id": null, "error_file_id": null, "created_at": 1711471533, "in_progress_at": 1711471538, "expires_at": null, "finalizing_at": null, "completed_at": null, "failed_at": null, "expired_at": null, "cancelling_at": null, "cancelled_at": null, "request_counts": { "total": 100, "completed": 40, "failed": 1 }, "metadata": { "description": "nightly eval run" }, "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "usage": { "input_tokens": 4800, "input_tokens_details": { "cached_tokens": 0 }, "output_tokens": 3200, "output_tokens_details": { "reasoning_tokens": 0 }, "total_tokens": 8000 } } ``` ## List batches ``` GET /batches ``` Listing batches requires the following parameters: > **after (optional)** > > A pagination cursor. The returned list starts from the object right after the batch with this `id`. If omitted, the list starts from the first batch. > **limit (optional)** > > An integer between `1` and `100`. The returned list will have at most `limit` elements. Defaults to `20`. The returned object has the following fields: | Field | Type | Description | | - | - | - | | `object` | string | The object type, always `"list"`. | | `data` | array | A list of [Batch objects](/batch/batch-objects). | | `first_id` | string | The `id` of the first batch in the list. | | `last_id` | string | The `id` of the last batch in the list. | | `has_more` | boolean | `true` if there are more batches after `last_id`. | ```python Python theme={null} batches = client.batches.list(limit=20) for batch in batches.data: print(batch.id, batch.status) ``` ```javascript JavaScript theme={null} const batches = await client.batches.list({ limit: 20 }); for (const batch of batches.data) { console.log(batch.id, batch.status); } ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/batches?limit=20" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" ``` ```json theme={null} { "object": "list", "data": [ { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": null, "input_file_id": "file_abc123", "completion_window": "24h", "status": "completed", "output_file_id": "file_out456", "error_file_id": null, "created_at": 1711471533, "in_progress_at": 1711471538, "expires_at": null, "finalizing_at": 1711475133, "completed_at": 1711475134, "failed_at": null, "expired_at": null, "cancelling_at": null, "cancelled_at": null, "request_counts": { "total": 100, "completed": 100, "failed": 0 }, "metadata": { "description": "nightly eval run" }, "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "usage": { "input_tokens": 12000, "input_tokens_details": { "cached_tokens": 0 }, "output_tokens": 8000, "output_tokens_details": { "reasoning_tokens": 0 }, "total_tokens": 20000 } }, { "id": "batch_def456", "object": "batch", "endpoint": "/v1/embeddings", "errors": null, "input_file_id": "file_def456", "completion_window": "24h", "status": "in_progress", "output_file_id": null, "error_file_id": null, "created_at": 1711558000, "in_progress_at": 1711558005, "expires_at": null, "finalizing_at": null, "completed_at": null, "failed_at": null, "expired_at": null, "cancelling_at": null, "cancelled_at": null, "request_counts": { "total": 5000, "completed": 1200, "failed": 0 }, "metadata": null, "model": "Qwen/Qwen3-Embedding-8B", "usage": { "input_tokens": 60000, "input_tokens_details": { "cached_tokens": 0 }, "output_tokens": 0, "output_tokens_details": { "reasoning_tokens": 0 }, "total_tokens": 60000 } } ], "first_id": "batch_abc123", "last_id": "batch_def456", "has_more": false } ``` ## Cancel a batch ``` POST /batches/{batch_id}/cancel ``` Cancels a given batch. The batch moves to the `cancelling` status until it is finalized, at which point its status becomes `cancelled`. Requests that managed to finish are written to the output file, and the rest are written to the error file as `cancelled`. Cancelling a batch requires the following parameters: > **batch\_id** > > The `id` of the batch to cancel. This endpoint returns a [Batch object](/batch/batch-objects). ```python Python theme={null} batch = client.batches.cancel("batch_abc123") print(batch.status) ``` ```javascript JavaScript theme={null} const batch = await client.batches.cancel("batch_abc123"); console.log(batch.status); ``` ```bash cURL theme={null} curl -X POST "https://api.deepinfra.com/v1/openai/batches/batch_abc123/cancel" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" ``` ```json theme={null} { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": null, "input_file_id": "file_abc123", "completion_window": "24h", "status": "cancelled", "output_file_id": "file_out456", "error_file_id": "file_err789", "created_at": 1711471533, "in_progress_at": 1711471538, "expires_at": null, "finalizing_at": null, "completed_at": null, "failed_at": null, "expired_at": null, "cancelling_at": 1711472000, "cancelled_at": 1711472050, "request_counts": { "total": 100, "completed": 40, "failed": 0 }, "metadata": { "description": "nightly eval run" }, "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "usage": { "input_tokens": 4800, "input_tokens_details": { "cached_tokens": 0 }, "output_tokens": 3200, "output_tokens_details": { "reasoning_tokens": 0 }, "total_tokens": 8000 } } ``` # Batch Objects Source: https://docs.deepinfra.com/batch/batch-objects Reference for the objects returned by the Batch API — the Batch and BatchUsage objects. Reference for the objects returned by the Batch API. ## The Batch object Most batch endpoints return a `Batch` object describing a batch job. | Field | Type | Description | | - | - | - | | `id` | string | The batch identifier. | | `object` | string | The object type, always `"batch"`. | | `endpoint` | string | The endpoint the batch runs against. Must match the `url` used on every line of the input file. | | `errors` | object | Optional. Validation errors for the batch. See [errors](#errors). | | `input_file_id` | string | The `id` of the input file for the batch. | | `completion_window` | string | The time window after which the batch expires. Currently only `"24h"` is supported. | | `status` | string | The current status of the batch. See [Batch status](#batch-status). | | `output_file_id` | string | The `id` of the file containing the outputs of successfully executed requests. | | `error_file_id` | string | The `id` of the file containing the outputs of requests that failed (failed lines). | | `created_at` | integer | Unix timestamp (in seconds) for when the batch was created. | | `in_progress_at` | integer | Unix timestamp for when the batch started processing. `null` until reached. | | `expires_at` | integer | Unix timestamp for when the batch will expire. `null` until reached. | | `finalizing_at` | integer | Unix timestamp for when the batch started finalizing. `null` until reached. | | `completed_at` | integer | Unix timestamp for when the batch completed. `null` until reached. | | `failed_at` | integer | Unix timestamp for when the batch failed. `null` until reached. | | `expired_at` | integer | Unix timestamp for when the batch expired. `null` until reached. | | `cancelling_at` | integer | Unix timestamp for when the batch started cancelling. `null` until reached. | | `cancelled_at` | integer | Unix timestamp for when the batch was cancelled. `null` until reached. | | `request_counts` | object | Optional. The counts of requests by status within the batch. See [request\_counts](#request_counts). | | `metadata` | object | Up to 16 key–value string pairs, where each key is a string of up to 64 characters and each value is a string of up to 512 characters. | | `model` | string | Optional. The model the inference is run against. | | `usage` | object | Optional. Token usage statistics for the batch. See [The BatchUsage object](#the-batchusage-object). | ### Batch status A batch moves through the following statuses: | Status | Meaning | | - | - | | `validating` | The input file is being validated before the batch starts. | | `in_progress` | Validation passed and requests are running. | | `finalizing` | All requests are done; the output files are being assembled. | | `completed` | The batch finished; results are ready to download. | | `failed` | Validation failed. The [`errors`](#errors) field of the Batch object describes why the file failed. | | `expired` | Not all requests could finish within the 24-hour window; the finished ones are still written to the output file. | | `cancelling` | The batch is being cancelled. This status may last for some time while in-flight requests finish. | | `cancelled` | The batch has been cancelled successfully. | ### errors | Field | Type | Description | | - | - | - | | `object` | string | The object type, always `"list"`. | | `data` | array | Optional. An array of error objects, each with the following optional fields: | | Field | Type | Description | | - | - | - | | `code` | string | Optional. A machine-readable error code. | | `line` | integer | Optional. The line of the input file the error refers to. | | `message` | string | Optional. A human-readable description of the error. | | `param` | string | Optional. The parameter the error relates to. | ### request\_counts | Field | Type | Description | | - | - | - | | `total` | integer | The total number of requests in the batch. | | `completed` | integer | The number of requests that have completed successfully. | | `failed` | integer | The number of requests that have failed. | ## The BatchUsage object Token usage statistics aggregated across all requests in the batch. | Field | Type | Description | | - | - | - | | `input_tokens` | integer | The total number of input (prompt) tokens across the batch. | | `input_tokens_details` | object | A breakdown of the input tokens. Contains `cached_tokens`, and may contain additional fields. | | `output_tokens` | integer | The total number of output (completion) tokens across the batch. | | `output_tokens_details` | object | A breakdown of the output tokens. Contains `reasoning_tokens`, and may contain additional fields. | | `total_tokens` | integer | The total number of tokens used (input + output). | ## Example Batch object of a completed batch: ```json theme={null} { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "errors": null, "input_file_id": "file_abc123", "completion_window": "24h", "status": "completed", "output_file_id": "file_out456", "error_file_id": null, "created_at": 1711471533, "in_progress_at": 1711471538, "expires_at": null, "finalizing_at": 1711475133, "completed_at": 1711475134, "failed_at": null, "expired_at": null, "cancelling_at": null, "cancelled_at": null, "request_counts": { "total": 100, "completed": 100, "failed": 0 }, "metadata": { "description": "nightly eval run" }, "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "usage": { "input_tokens": 12000, "input_tokens_details": { "cached_tokens": 0 }, "output_tokens": 8000, "output_tokens_details": { "reasoning_tokens": 0 }, "total_tokens": 20000 } } ``` # Files API Source: https://docs.deepinfra.com/batch/file-endpoints The OpenAI-compatible Files API — upload, list, retrieve, download, and delete files. An **OpenAI-compatible** Files API for managing batch and fine-tune related files. All endpoints are relative to: ``` https://api.deepinfra.com/v1/openai ``` > [!NOTE] > > Fine-tuning is not yet supported. Currently the only API that uses files is the [Batch API](/batch/introduction), which accepts files of up to 200 MB. ## Objects ### The FileObject Most file endpoints return a `FileObject` describing an uploaded file. | Field | Type | Description | | - | - | - | | `id` | string | The file identifier, referenced in API endpoints. | | `object` | string | The object type, always `"file"`. | | `bytes` | integer | The size of the file, in bytes. | | `created_at` | integer | Unix timestamp (in seconds) for when the file was created. | | `expires_at` | integer | Unix timestamp (in seconds) for when the file will be deleted. | | `filename` | string | The name of the file. | | `purpose` | string | The intended purpose of the file. One of `"batch"`, `"fine-tune"`, or `"batch-output"`. | ```json theme={null} { "id": "file_abc123", "object": "file", "bytes": 120000, "created_at": 1677610602, "expires_at": 1680202602, "filename": "requests.jsonl", "purpose": "batch" } ``` ## Create file ``` POST /files ``` Creates a file used in other API endpoints. Creating a file requires the following parameters: > **file** > > The file to be uploaded. > **purpose** > > The intended purpose of the file. Can be `"batch"` or `"fine-tune"` for upload, but only `"batch"` is currently supported. > **expires\_after (optional)** > > Controls how long the file remains available. An object with two fields: > > * **`anchor`** (optional) — must be `"created_at"`. The expiry is measured from when the file is created. Defaults to `"created_at"`. > * **`seconds`** (optional) — the number of seconds the file stays available after the anchor. An integer between `3600` (1 hour) and `2592000` (30 days). Defaults to `2592000` (30 days). This endpoint returns a [FileObject](#the-fileobject). ```python Python theme={null} import os from openai import OpenAI client = OpenAI( api_key=os.environ["DEEPINFRA_API_KEY"], base_url="https://api.deepinfra.com/v1/openai", ) file = client.files.create( file=open("requests.jsonl", "rb"), purpose="batch", expires_after={"anchor": "created_at", "seconds": 604800}, ) print(file.id) ``` ```javascript JavaScript theme={null} import fs from "fs"; import OpenAI from "openai"; const client = new OpenAI({ apiKey: process.env.DEEPINFRA_API_KEY, baseURL: "https://api.deepinfra.com/v1/openai", }); const file = await client.files.create({ file: fs.createReadStream("requests.jsonl"), purpose: "batch", expires_after: { anchor: "created_at", seconds: 604800 }, }); console.log(file.id); ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/files" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -F purpose="batch" \ -F file="@requests.jsonl" \ -F expires_after[anchor]="created_at" \ -F expires_after[seconds]=604800 ``` ```json theme={null} { "id": "file_abc123", "object": "file", "bytes": 120000, "created_at": 1677610602, "expires_at": 1678215402, "filename": "requests.jsonl", "purpose": "batch" } ``` ## List files ``` GET /files ``` Listing files requires the following parameters: > **after (optional)** > > A pagination cursor. The returned list starts from the object right after the file with this `id`. If omitted, the list starts from the first file. > **limit (optional)** > > An integer between `1` and `10000`. The returned list will have at most `limit` elements. Defaults to `10000`. > **order (optional)** > > Sort order by `created_at`, either `"asc"` (ascending) or `"desc"` (descending). > **purpose (optional)** > > Only returns files of the given purpose. If omitted, files are not filtered by purpose. The returned object has the following fields: | Field | Type | Description | | - | - | - | | `object` | string | The object type, always `"list"`. | | `data` | array | A list of [FileObject](#the-fileobject). | | `first_id` | string | The `id` of the first file in the list. | | `last_id` | string | The `id` of the last file in the list. | | `has_more` | boolean | `true` if there are more files after `last_id`. | ```python Python theme={null} files = client.files.list( purpose="batch", limit=20, order="desc", ) for file in files.data: print(file.id, file.filename) ``` ```javascript JavaScript theme={null} const files = await client.files.list({ purpose: "batch", limit: 20, order: "desc", }); for (const file of files.data) { console.log(file.id, file.filename); } ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/files?purpose=batch&limit=20&order=desc" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" ``` ```json theme={null} { "object": "list", "data": [ { "id": "file_abc123", "object": "file", "bytes": 120000, "created_at": 1677610602, "expires_at": 1678215402, "filename": "requests.jsonl", "purpose": "batch" }, { "id": "file_def456", "object": "file", "bytes": 84000, "created_at": 1677520000, "expires_at": 1678124800, "filename": "eval.jsonl", "purpose": "batch" } ], "first_id": "file_abc123", "last_id": "file_def456", "has_more": false } ``` ## Retrieve a file ``` GET /files/{file_id} ``` Returns information about a specific file. Retrieving a file requires the following parameters: > **file\_id** > > The `id` of the file to retrieve. This endpoint returns a [FileObject](#the-fileobject). ```python Python theme={null} file = client.files.retrieve("file_abc123") print(file.id, file.filename) ``` ```javascript JavaScript theme={null} const file = await client.files.retrieve("file_abc123"); console.log(file.id, file.filename); ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/files/file_abc123" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" ``` ```json theme={null} { "id": "file_abc123", "object": "file", "bytes": 120000, "created_at": 1677610602, "expires_at": 1678215402, "filename": "requests.jsonl", "purpose": "batch" } ``` ## Retrieve file content ``` GET /files/{file_id}/content ``` Returns the content of a specified file. Retrieving file content requires the following parameters: > **file\_id** > > The `id` of the file whose content to retrieve. This endpoint returns the binary response content. ```python Python theme={null} content = client.files.content("file_abc123") print(content.text) ``` ```javascript JavaScript theme={null} const content = await client.files.content("file_abc123"); console.log(await content.text()); ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/files/file_abc123/content" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" ``` ## Delete a file ``` DELETE /files/{file_id} ``` Deletes a file and removes it from all storages. Deleting a file requires the following parameters: > **file\_id** > > The `id` of the file to delete. This endpoint returns a `FileDeleted` object with the following fields: | Field | Type | Description | | - | - | - | | `id` | string | The `id` of the deleted file. | | `deleted` | boolean | `true` if the file was deleted. | | `object` | string | The object type, always `"file"`. | ```python Python theme={null} deleted = client.files.delete("file_abc123") print(deleted.deleted) ``` ```javascript JavaScript theme={null} const deleted = await client.files.delete("file_abc123"); console.log(deleted.deleted); ``` ```bash cURL theme={null} curl -X DELETE "https://api.deepinfra.com/v1/openai/files/file_abc123" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" ``` ```json theme={null} { "id": "file_abc123", "deleted": true, "object": "file" } ``` ## File API Limits | Limit | Value | | - | - | | Maximum size of a single user-created file | 512 MB | | Maximum file size with `purpose = "batch"` | 200 MB | | Maximum total file size per user (including response files) | 2 GB | # Introduction to Batch API Source: https://docs.deepinfra.com/batch/introduction Run large, non-urgent inference jobs asynchronously at 20% off — OpenAI-compatible file upload, batch, and results workflow. The Batch API lets you submit large volumes of requests as a single asynchronous job and get the results back within 24 hours, billed at **20% below real-time pricing**. It's built for workloads that aren't latency-sensitive — embedding a whole corpus, classifying or summarizing a dataset, or running a model over an evaluation set. It's **OpenAI-compatible**: if you've used OpenAI's Batch API, the workflow is identical — upload a JSONL file of requests, create a batch, poll for completion, and download the results. Point the OpenAI SDK at DeepInfra and your existing batch code works. The endpoint is: ``` https://api.deepinfra.com/v1/openai ``` The only changes from your existing OpenAI code are the `base_url` and `api_key`, plus using a model from [our catalog](https://deepinfra.com/models). > [!NOTE] > > Any OpenAI-compatible model available for real-time inference on a batch-supported endpoint can also be used in batch. ## Supported endpoints A batch runs all of its requests against a single endpoint. The supported endpoints are: * `/v1/chat/completions` * `/v1/completions` * `/v1/embeddings` > [!WARNING] > > Other OpenAI batch endpoints — `/v1/images/generations`, `/v1/images/edits`, `/v1/moderations`, `/v1/responses`, `/v1/videos` — are **not yet** supported and are rejected at validation. ## Workflow **1. Prepare a JSONL input file** **2. Upload the file** **3. Create the batch** **4. Poll for completion** **5. Download the results** ## Step 1 — Prepare the input file The input is a [JSONL](https://jsonlines.org/) file with one request per line. Each line has a `custom_id`, the HTTP `method` (`POST`), the `url` (which must match the batch endpoint), and the request `body` — the exact JSON you'd send to that endpoint in real time. ```jsonl Chat completions theme={null} {"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "deepseek-ai/DeepSeek-V4-Flash-0731", "messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 100}} {"custom_id": "req-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "deepseek-ai/DeepSeek-V4-Flash-0731", "messages": [{"role": "user", "content": "Write a haiku about batching."}], "max_tokens": 100}} ``` ```jsonl Embeddings theme={null} {"custom_id": "doc-1", "method": "POST", "url": "/v1/embeddings", "body": {"model": "Qwen/Qwen3-Embedding-8B", "input": "The food was delicious and the waiter...", "encoding_format": "float"}} {"custom_id": "doc-2", "method": "POST", "url": "/v1/embeddings", "body": {"model": "Qwen/Qwen3-Embedding-8B", "input": ["first text", "second text"], "encoding_format": "float"}} ``` A few rules: * **`custom_id` must be unique** across all requests in a file. It's how you match results back to requests, since output order isn't guaranteed. * **`method` must be `POST`.** * **`url` must equal the batch's `endpoint`** — you can't mix endpoints in one file. * **All requests must use the same model.** * Use a DeepInfra model id (e.g. `deepseek-ai/DeepSeek-V4-Flash-0731`), not an OpenAI model name. * **`body` must match the request format of the corresponding endpoint** — it's the exact JSON you'd send to that endpoint in real time. See [Chat Completions](/chat/overview), [Text Completions](/apis/completions), or [Embeddings](/apis/embeddings). ## Step 2 — Upload the file Upload the JSONL file with `purpose="batch"`. In return you get a [FileObject](/batch/file-endpoints#the-fileobject) containing the `id` of the uploaded file. For more information on how to upload a file, see [Create file](/batch/file-endpoints#create-file). ```python Python theme={null} import os from openai import OpenAI client = OpenAI( api_key=os.environ["DEEPINFRA_API_KEY"], base_url="https://api.deepinfra.com/v1/openai", ) batch_input_file = client.files.create( file=open("requests.jsonl", "rb"), purpose="batch", ) print(batch_input_file.id) ``` ```javascript JavaScript theme={null} import fs from "fs"; import OpenAI from "openai"; const client = new OpenAI({ apiKey: process.env.DEEPINFRA_API_KEY, baseURL: "https://api.deepinfra.com/v1/openai", }); const batchInputFile = await client.files.create({ file: fs.createReadStream("requests.jsonl"), purpose: "batch", }); console.log(batchInputFile.id); ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/files" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -F purpose="batch" \ -F file="@requests.jsonl" ``` ## Step 3 — Create the batch Create the batch job from the uploaded input file, choosing the `endpoint` to run against. The batch starts executing as soon as it's created. For the exact details of creating a batch request, see [Create a batch](/batch/batch-endpoints#create-a-batch). ```python Python theme={null} batch = client.batches.create( input_file_id=batch_input_file.id, endpoint="/v1/chat/completions", completion_window="24h", metadata={"description": "nightly eval run"}, output_expires_after={"anchor": "created_at", "seconds": 604800}, ) print(batch.id, batch.status) ``` ```javascript JavaScript theme={null} const batch = await client.batches.create({ input_file_id: batchInputFile.id, endpoint: "/v1/chat/completions", completion_window: "24h", metadata: { description: "nightly eval run" }, output_expires_after: { anchor: "created_at", seconds: 604800 }, }); console.log(batch.id, batch.status); ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/batches" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "input_file_id": "file_abc123", "endpoint": "/v1/chat/completions", "completion_window": "24h", "metadata": {"description": "nightly eval run"}, "output_expires_after": {"anchor": "created_at", "seconds": 604800} }' ``` ## Step 4 — Check status A batch runs asynchronously, so after creating it you can check its status to know when the results are ready. Retrieve the batch periodically and watch its `status` until it reaches a terminal state — `completed`, `failed`, `expired`, or `cancelled`. For details on retrieving a batch, see [Retrieve a batch](/batch/batch-endpoints#retrieve-a-batch). ```python Python theme={null} batch = client.batches.retrieve(batch.id) print(batch.status) print(batch.request_counts) # total / completed / failed ``` ```javascript JavaScript theme={null} const updated = await client.batches.retrieve(batch.id); console.log(updated.status); console.log(updated.request_counts); // total / completed / failed ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/batches/batch_abc123" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" ``` A batch moves through several statuses — see [Batch status](/batch/batch-objects#batch-status) for what each one means. You can track progress by checking the `usage` and `request_counts` fields when checking status. Once the batch reaches a terminal state, the output and error files will be available, if they contain any information. ## Step 5 — Download the results Once the batch reaches a terminal state, you can get either the [`errors`](/batch/batch-objects#errors) field, or the output and error files (`output_file_id` and `error_file_id`) from the [Batch object](/batch/batch-objects), depending on the state. You can download the output and error files using the [Files API](/batch/file-endpoints#retrieve-file-content). ```python Python theme={null} batch = client.batches.retrieve(batch.id) # Successful responses output = client.files.content(batch.output_file_id) print(output.text) # Failed / cancelled requests, if any if batch.error_file_id: errors = client.files.content(batch.error_file_id) print(errors.text) ``` ```javascript JavaScript theme={null} const done = await client.batches.retrieve(batch.id); const output = await client.files.content(done.output_file_id); console.log(await output.text()); if (done.error_file_id) { const errors = await client.files.content(done.error_file_id); console.log(await errors.text()); } ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/files/file_out456/content" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" ``` Each result line carries the `custom_id` from the input so you can match it back to your request. Successful lines have a `response`; failed lines have an `error`. The `response` has a `body` field with the same format that the real-time API would return. The `error` has `code` and `message` fields that better describe why the line failed. ```jsonl theme={null} {"id": "batch_req_xyz", "custom_id": "req-1", "response": {"status_code": 200, "body": {"choices": [{"message": {"role": "assistant", "content": "Hello!"}}]}}, "error": null} {"id": "batch_req_abc", "custom_id": "req-2", "response": null, "error": {"code": "invalid_request", "message": "..."}} ``` ## Cancel a batch You can cancel a batch at any point while its status is non-terminal. Cancelling a batch puts it in the `cancelling` status for some time, after which it moves to `cancelled`. ```python Python theme={null} client.batches.cancel(batch.id) ``` ```javascript JavaScript theme={null} await client.batches.cancel(batch.id); ``` ```bash cURL theme={null} curl -X POST "https://api.deepinfra.com/v1/openai/batches/batch_abc123/cancel" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" ``` ## Pricing Batch requests are billed at **20% less** than the corresponding real-time price for the same model and endpoint. The discount is applied automatically — there's nothing extra to configure. ## Rate limits Batch requests and usage do not affect your real-time rate limits. There are additional batch-related rate limits: | Limit | Value | | - | - | | Requests (lines) per file | 50,000 | | Input file size | 200 MB | | Embedding inputs per file | 50,000 | | Concurrent batches per user | 100 | ## Related * [Chat Completions](/chat/overview) * [Text Completions](/apis/completions) * [Embeddings](/apis/embeddings) # Log Probabilities Source: https://docs.deepinfra.com/chat/log-probs Get per-token log probabilities from LLM responses. You can retrieve the log probability of each generated token. This is useful for uncertainty estimation, token-level filtering, confidence scoring, or building custom sampling logic. Log probabilities are supported across all request modes: * **OpenAI-compatible API** — using `logprobs` and `top_logprobs` parameters * **DeepInfra Native API** — streaming and non-streaming ## OpenAI-compatible API Set `logprobs: true` in your request. Optionally set `top_logprobs` (1–20) to also get the top alternative tokens at each position. ```python Python theme={null} import os from openai import OpenAI client = OpenAI( api_key=os.environ["DEEPINFRA_API_KEY"], base_url="https://api.deepinfra.com/v1/openai", ) response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash-0731", messages=[{"role": "user", "content": "Say hello in one word"}], logprobs=True, top_logprobs=3, ) for token in response.choices[0].logprobs.content: print(f"{token.token!r}: {token.logprob:.4f}") for alt in token.top_logprobs: print(f" alt {alt.token!r}: {alt.logprob:.4f}") ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "messages": [{"role": "user", "content": "Say hello in one word"}], "logprobs": true, "top_logprobs": 3 }' ``` Response structure: ```json theme={null} { "choices": [{ "logprobs": { "content": [ { "token": "Hello", "logprob": -0.0023, "top_logprobs": [ {"token": "Hello", "logprob": -0.0023}, {"token": "Hi", "logprob": -1.42}, {"token": "Hey", "logprob": -3.87} ] } ] } }] } ``` ## DeepInfra Native API (streaming) The native streaming API returns log probabilities inline with each token as it is generated. ```bash theme={null} curl -X POST \ -d '{"input": "I have this dream", "stream": true}' \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -H 'Content-Type: application/json' \ 'https://api.deepinfra.com/v1/inference/deepseek-ai/DeepSeek-V4-Flash-0731' ``` Response (streamed): ``` data: {"token": {"id": 29892, "text": ",", "logprob": -2.65625, "special": false}, "generated_text": null, "details": null} data: {"token": {"id": 988, "text": " where", "logprob": -0.39575195, "special": false}, "generated_text": null, "details": null} data: {"token": {"id": 1432, "text": " every", "logprob": -3.15625, "special": false}, "generated_text": null, "details": null} data: {"token": {"id": 931, "text": " time", "logprob": -0.1385498, "special": false}, "generated_text": null, "details": null} ``` The `logprob` field is the log probability of the generated token (base e). Lower (more negative) values indicate less likely tokens. # Chat Completions Source: https://docs.deepinfra.com/chat/overview OpenAI-compatible chat completions API — just change the base URL and model name. DeepInfra offers an OpenAI-compatible chat completions API for all [LLM models](https://deepinfra.com/models/text-generation) at the best prices for open-source model inference. For other model types (embeddings, image generation, speech, reranking, and more), see [More APIs](/apis/completions). The endpoint is: ``` https://api.deepinfra.com/v1/openai ``` The only changes you need to make from your existing OpenAI code: 1. Set `base_url` to `https://api.deepinfra.com/v1/openai` 2. Set `api_key` to your DeepInfra token 3. Set `model` to a model from [our catalog](https://deepinfra.com/models) ## Install the SDK ```bash Python theme={null} pip install openai ``` ```bash JavaScript theme={null} npm install openai ``` ## Basic chat completion ```python Python theme={null} import os from openai import OpenAI openai = OpenAI( api_key=os.environ["DEEPINFRA_API_KEY"], base_url="https://api.deepinfra.com/v1/openai", ) chat_completion = openai.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash-0731", messages=[{"role": "user", "content": "Hello"}], ) print(chat_completion.choices[0].message.content) print(chat_completion.usage.prompt_tokens, chat_completion.usage.completion_tokens) ``` ```javascript JavaScript theme={null} import OpenAI from "openai"; const openai = new OpenAI({ apiKey: process.env.DEEPINFRA_API_KEY, baseURL: "https://api.deepinfra.com/v1/openai", }); const completion = await openai.chat.completions.create({ messages: [{ role: "user", content: "Hello" }], model: "deepseek-ai/DeepSeek-V4-Flash-0731", }); console.log(completion.choices[0].message.content); console.log(completion.usage.prompt_tokens, completion.usage.completion_tokens); ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "messages": [ { "role": "user", "content": "Hello!" } ] }' ``` ## Multi-turn conversations To create a longer conversation, include the full message history in every request. The model uses this context to provide better answers. ```python Python theme={null} import os from openai import OpenAI openai = OpenAI( api_key=os.environ["DEEPINFRA_API_KEY"], base_url="https://api.deepinfra.com/v1/openai", ) chat_completion = openai.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash-0731", messages=[ {"role": "system", "content": "Respond like a michelin starred chef."}, {"role": "user", "content": "Can you name at least two different techniques to cook lamb?"}, {"role": "assistant", "content": "Bonjour! Let me tell you, my friend, cooking lamb is an art form..."}, {"role": "user", "content": "Tell me more about the second method."}, ], ) print(chat_completion.choices[0].message.content) ``` ```javascript JavaScript theme={null} import OpenAI from "openai"; const openai = new OpenAI({ baseURL: "https://api.deepinfra.com/v1/openai", apiKey: process.env.DEEPINFRA_API_KEY, }); const completion = await openai.chat.completions.create({ messages: [ {role: "system", content: "Respond like a michelin starred chef."}, {role: "user", content: "Can you name at least two different techniques to cook lamb?"}, {role: "assistant", content: "Bonjour! Let me tell you, my friend, cooking lamb is an art form..."}, {role: "user", content: "Tell me more about the second method."} ], model: "deepseek-ai/DeepSeek-V4-Flash-0731", }); console.log(completion.choices[0].message.content); ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "messages": [ {"role": "system", "content": "Respond like a michelin starred chef."}, {"role": "user", "content": "Can you name at least two different techniques to cook lamb?"}, {"role": "assistant", "content": "Bonjour! Let me tell you..."}, {"role": "user", "content": "Tell me more about the second method."} ] }' ``` The longer the conversation, the more tokens it uses. The maximum conversation length is determined by the model's context size. ## Supported parameters | Parameter | Notes | | - | - | | `model` | Model name, or `MODEL_NAME:VERSION`, or `deploy_id:DEPLOY_ID` | | `messages` | Roles: `system`, `user`, `assistant` | | `max_tokens` | Max tokens to generate. See [Max output tokens](#max-output-tokens) | | `stream` | See [Streaming](/chat/streaming) | | `temperature` | Sampling temperature between 0 and 2. Higher values produce more random output; lower values more deterministic. Default: `1.0` | | `top_p` | Nucleus sampling threshold — only tokens comprising the top `top_p` probability mass are considered. Default: `1.0` | | `stop` | Up to 4 sequences where the API will stop generating further tokens | | `n` | Number of completion sequences to return. Default: `1` | | `presence_penalty` | Penalizes tokens that have already appeared in the text, encouraging the model to discuss new topics. Range: -2.0 to 2.0. Default: `0` | | `frequency_penalty` | Penalizes tokens based on how often they've appeared so far, reducing repetition. Range: -2.0 to 2.0. Default: `0` | | `response_format` | See [Structured Outputs](/chat/structured-outputs) | | `tools`, `tool_choice` | See [Tool Calling](/chat/tool-calling) | | `service_tier` | Select a service tier (`"priority"` or `"flex"`) for tagged models. See [Service Tier](#service-tier) below. | | `fail_fast` | Reject with HTTP 429 instead of queueing when the model is at capacity. See [Fail fast](#fail-fast) below. Default: `false` | | `reasoning_effort` | Controls reasoning depth for reasoning models. See [Reasoning Models](/chat/reasoning). | > [!NOTE] > > We may not be 100% compatible with all OpenAI parameters. Let us know on Discord or by email if something you need is missing. For the complete parameter reference, see the [API reference](/api-reference/chat-completions/openai-chat-completions). ## Service tier Set the optional `service_tier` parameter to run a request on a non-standard tier. Two tiers are available on tagged models: **priority** (faster, at a premium) and **flex** (cheaper, best-effort). Leave `service_tier` unset for standard real-time scheduling and pricing. ### Priority Set `service_tier` to `"priority"` to request priority inference on supported models. Priority requests get faster time-to-first-token and higher throughput during peak demand. > [!WARNING] > > Priority inference incurs a 50% surcharge on top of the model's standard per-token price. ```python Python theme={null} response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash-0731", messages=[{"role": "user", "content": "Hello!"}], extra_body={"service_tier": "priority"}, ) ``` ```javascript JavaScript theme={null} const response = await openai.chat.completions.create({ model: "deepseek-ai/DeepSeek-V4-Flash-0731", messages: [{ role: "user", content: "Hello!" }], service_tier: "priority", }); ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "service_tier": "priority", "messages": [ { "role": "user", "content": "Hello!" } ] }' ``` ### Flex Set `service_tier` to `"flex"` to run Chat Completions requests at a lower cost in exchange for slower response times and occasional resource unavailability. It's ideal for non-production or lower-priority tasks such as model evaluations, data enrichment, and asynchronous workloads. When a model is busy, a flex request may wait up to 10 minutes for available capacity before it runs or is rejected with an HTTP 429, so use it for work you can retry. > [!NOTE] > > Flex inference is billed at a 20% discount off the model's standard per-token price. ```python Python theme={null} response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash-0731", messages=[{"role": "user", "content": "Hello!"}], extra_body={"service_tier": "flex"}, ) ``` ```javascript JavaScript theme={null} const response = await openai.chat.completions.create({ model: "deepseek-ai/DeepSeek-V4-Flash-0731", messages: [{ role: "user", content: "Hello!" }], service_tier: "flex", }); ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "service_tier": "flex", "messages": [ { "role": "user", "content": "Hello!" } ] }' ``` The response includes a `service_tier` field confirming which tier was actually used. Not all models support these tiers — check the model page for availability. If a model doesn't support the requested tier, the request is served at the standard tier and billed at the standard price; no error is returned. ## Fail fast By default, a request sent to a model that is at capacity waits in the queue until capacity frees up. Set the optional `fail_fast` parameter to `true` to get an immediate HTTP 429 instead of waiting. This is meant for latency-sensitive callers that would rather go somewhere else than sit in a queue — for example, clients that fail over to another provider on rejection. The 429 arrives as soon as the request would have been queued, so you don't spend your latency budget waiting. ```python Python theme={null} response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash-0731", messages=[{"role": "user", "content": "Hello!"}], extra_body={"fail_fast": True}, ) ``` ```javascript JavaScript theme={null} const response = await openai.chat.completions.create({ model: "deepseek-ai/DeepSeek-V4-Flash-0731", messages: [{ role: "user", content: "Hello!" }], fail_fast: true, }); ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "fail_fast": true, "messages": [ { "role": "user", "content": "Hello!" } ] }' ``` Rejection is capacity-aware rather than backlog-triggered: a `fail_fast` request is rejected only when the model is busy enough that serving it would actually be slow. Whenever there is spare capacity the request is admitted and served exactly as if `fail_fast` were unset, so enabling it does not cost you throughput on an idle model. When a request is rejected, the response is an HTTP 429 carrying the `engine_overloaded` code: ```json theme={null} { "error": { "message": "Model busy, retry later", "type": "invalid_request_error", "param": null, "code": "engine_overloaded" } } ``` > [!NOTE] > > A rejected request never reaches the model, so no inference happens and nothing is billed. If you set both `fail_fast: true` and `service_tier: "priority"`, `fail_fast` wins — the explicit request not to wait is honored, and you get a 429 rather than a priority slot in the queue. ## Max output tokens The maximum number of tokens that can be generated in a single response is model-dependent, with a hard cap of 16384 tokens for most models. Set `max_tokens` to control the limit for a specific request. ### Continuing responses beyond the limit If you need a longer response, use response continuation: send a follow-up request with the previous response included as an assistant message, and the model will continue from where it left off. ```bash theme={null} curl "https://api.deepinfra.com/v1/openai/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "messages": [ {"role": "user", "content": "Write a very long essay about AI."}, {"role": "assistant", "content": ""} ], "max_tokens": 4096 }' ``` Note: response continuation cannot extend past the model's total context window. A 400 error is returned when the total context size is exceeded. ## What's next > **[Streaming](/chat/streaming)** > > Stream tokens as they're generated. > **[Structured Outputs](/chat/structured-outputs)** > > Get responses in JSON format. > **[Tool Calling](/chat/tool-calling)** > > Give models access to external functions. > **[Vision](/chat/vision)** > > Send images alongside text. > **[Reasoning Models](/chat/reasoning)** > > Control chain-of-thought reasoning behavior. # Prompt Cache Retention Source: https://docs.deepinfra.com/chat/prompt-cache-retention Retain a prompt prefix for 5 minutes or 1 hour so reuse skips prefill and bills at the cache-read rate. [Prompt caching](/chat/prompt-caching) reuses the KV cache from a recent request when the beginning of your prompt matches — automatically, and best-effort. **Prompt cache retention** goes further: it lets you explicitly keep a prompt prefix cached for a fixed window — **5 minutes** or **1 hour** — so reuse is guaranteed for that window instead of depending on whether the prefix happens to still be warm. While a prefix is retained, every request that reuses it **skips prefill** for a faster time to first token and is billed at the discounted **cache-read** rate. Opening the window costs a small **cache-write** premium upfront. ## Quick start Send `prompt_cache_key` with every request in a session. Add `prompt_cache_options` on the first one to open the window. ```python Python theme={null} import os from openai import OpenAI client = OpenAI( api_key=os.environ["DEEPINFRA_API_KEY"], base_url="https://api.deepinfra.com/v1/openai", ) resp = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B", messages=messages, # your large, reused context extra_body={ "prompt_cache_key": "agent-session-123", "prompt_cache_options": {"mode": "explicit", "ttl": "1h"}, }, ) print(resp.usage.prompt_tokens_details) ``` ```javascript JavaScript theme={null} import OpenAI from "openai"; const openai = new OpenAI({ apiKey: process.env.DEEPINFRA_API_KEY, baseURL: "https://api.deepinfra.com/v1/openai", }); const resp = await openai.chat.completions.create({ model: "nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B", messages, // your large, reused context prompt_cache_key: "agent-session-123", prompt_cache_options: { mode: "explicit", ttl: "1h" }, }); ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "model": "nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B", "messages": [{"role": "system", "content": "..."}], "prompt_cache_key": "agent-session-123", "prompt_cache_options": {"mode": "explicit", "ttl": "1h"} }' ``` ## Request parameters | Parameter | Type | Description | | - | - | - | | `prompt_cache_key` | `string` | A stable identifier for the context you're caching (for example a session or agent id). Required for retention; reuse is matched on this key plus the prompt content. | | `prompt_cache_options.mode` | `string` | Currently `"explicit"`. | | `prompt_cache_options.ttl` | `"5m"` \| `"1h"` | Retention window. Sending it opens the window, or extends an open one. Omit it on reuse requests. | | `prompt_cache_breakpoint.mode` | `string` | Set to `"explicit"` on a message content part to end the retained prefix there. See [Cache breakpoints](#cache-breakpoints). | ## Window lifecycle | Request | Effect | | - | - | | First request with a `ttl` | Writes the cache and starts the clock. Still prefills; billed the write premium on the cached portion. | | Same key, no `ttl` | Reuses the retained prefix at the cache-read rate. Does not move the deadline. | | Same key, with a `ttl` | Reuses the cache *and* pushes the deadline out. Billed the write premium again. | | Shorter `ttl` inside a longer window | Never shortens the window. | | After expiry | Reuse falls back to standard input pricing — no silent charges. | Retention is scoped to your account **and** `prompt_cache_key`. It works with streaming and non-streaming, chat and text-completions. > [!WARNING] > > You're charged a cache write each time a request writes new blocks **or extends the window** — not just on the first request. For example, send the same prompt with a `ttl` twice and the second call extends the window, so it's charged as another cache write. Send a `ttl` only to open the window or when you deliberately want to extend it; for ordinary reuse, omit `ttl` and pay the cheaper cache-read rate. ## Cache breakpoints Most prompts are a stable prefix (system instructions, tools, a document) followed by a variable tail. Mark where the reusable prefix ends with a `prompt_cache_breakpoint` on a message content part — retention then applies to everything up to and including that part, and ignores the variable remainder, so the retained cache stays stable across requests even as the question changes. ```json theme={null} { "messages": [ { "role": "system", "content": [ { "type": "text", "text": "... large stable system prompt, tools, and reference context ...", "prompt_cache_breakpoint": { "mode": "explicit" } } ]}, { "role": "user", "content": "... the variable question — not retained ..." } ], "prompt_cache_key": "agent-session-123", "prompt_cache_options": { "mode": "explicit", "ttl": "1h" } } ``` > [!NOTE] > > Put anything that changes between calls (timestamps, user ids, retrieved chunks) **after** the breakpoint, so the prefix before it stays identical and matches in full. Changing a token inside the retained prefix only recomputes from the point of divergence onward — you still get the cache-read rate on the portion that still matches. ## Cache granularity Caches are written in fixed increments, so the retained portion is always **rounded down** to a whole multiple of the model's cache granularity. Whatever is left over is billed as standard input. The granularity differs per model. On Nemotron-3-Ultra it is **8,192 tokens**: a 20,000-token prompt retains 16,384 tokens (two increments), and the remaining 3,616 are billed as standard input. A breakpoint shorter than one increment retains nothing. > [!NOTE] > > Check the model page for the granularity of the model you're using — it determines how much of your prompt is actually cacheable. ## Reading the response Every response reports what happened in `usage.prompt_tokens_details`. | Field | Type | Description | | - | - | - | | `cache_write_tokens` | `integer` | Tokens written to cache, billed at the retention write rate. The remainder is billed as standard input. | | `cached_tokens` | `integer` | Tokens reused from a retained prefix, billed at the cache-read rate. | ```json Write (first request) theme={null} "usage": { "prompt_tokens": 34375, "total_tokens": 34495, "completion_tokens": 120, "prompt_tokens_details": { "cached_tokens": 0, "cache_write_tokens": 32768 } } ``` ```json Reuse (later request) theme={null} "usage": { "prompt_tokens": 34380, "total_tokens": 34475, "completion_tokens": 95, "prompt_tokens_details": { "cached_tokens": 32768, "cache_write_tokens": 0 } } ``` ## Pricing Relative to the model's standard input price: | Action | Rate | | - | - | | Reuse a retained prefix (cache **read**) | model's cache-read rate (e.g. **0.2×** input for Nemotron-3-Ultra) | | Retain for **5 minutes** (cache **write**) | **1.25×** input | | Retain for **1 hour** (cache **write**) | **2.0×** input | | Non-retained input | 1× (standard) | Only whole cacheable blocks count as cache read/write; any remainder is billed as standard input. The write premium applies only when a request actually **creates or extends** the retention window — reuse inside a window you've already paid for is billed at the read rate. ## Model support | Model | Retention | | - | - | | `nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B` | 5m, 1h | | `moonshotai/Kimi-K2.7-Code` | 5m, 1h | More models to follow. On a model without retention support, `prompt_cache_options` is ignored and the request is billed as standard input. # Prompt Caching Source: https://docs.deepinfra.com/chat/prompt-caching Reduce latency and cost by caching repeated prompt prefixes. Prompt caching allows DeepInfra to reuse the KV (key-value) cache from previous requests when the beginning of your prompt is identical. This reduces both latency and cost for workloads that repeatedly send the same prefix — such as a long system prompt, a large document, or a fixed set of few-shot examples. ## How it works When you send a request, DeepInfra checks whether the beginning of your prompt matches a cached prefix from a recent request on the same model. If it does, the cached KV state is reused instead of recomputing it, which: * **Reduces time-to-first-token** — the model skips processing the cached portion * **Lowers cost** — cached input tokens are billed at a reduced rate ## Usage Prompt caching is **automatic** — no extra parameters required. Just structure your prompts so that the reused content appears at the beginning. ```python theme={null} import os from openai import OpenAI client = OpenAI( api_key=os.environ["DEEPINFRA_API_KEY"], base_url="https://api.deepinfra.com/v1/openai", ) # Long system prompt that stays the same across requests SYSTEM_PROMPT = """You are a helpful AI assistant with deep expertise in Python. [... thousands of tokens of instructions or context ...] """ # First request — full processing response1 = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash-0731", messages=[ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": "How do I use list comprehensions?"}, ], ) # Second request — cached prefix reused, faster and cheaper response2 = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash-0731", messages=[ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": "What are Python generators?"}, ], ) ``` ## Best practices **Put stable content first.** The cache matches from the beginning of the prompt. Place your system prompt, documents, and few-shot examples before the user's message. **Keep the prefix identical.** Even a single character difference will invalidate the cache. Avoid dynamic content (timestamps, user IDs, etc.) in the cacheable prefix. **Longer prefixes save more.** Prompt caching is most effective with long, repeated prefixes — think multi-page documents, long system prompts, or RAG context. ## Common use cases | Use case | Cached prefix | | - | - | | Chatbot with a long system prompt | System prompt | | RAG / document Q\&A | Retrieved documents | | Few-shot classification | Examples | | Code assistant with a large codebase | Codebase context | | Multi-turn conversation | Previous turns | ## Checking cache usage The response usage object indicates how many tokens were served from cache: ```json theme={null} { "usage": { "prompt_tokens": 5000, "completion_tokens": 50, "total_tokens": 5050, "prompt_tokens_details": { "cached_tokens": 4800 } } } ``` In this example, 4800 of the 5000 input tokens were cached. ## Explicit cache keys By default, caching is automatic based on prefix matching. The `prompt_cache_key` parameter lets you explicitly tag a request with a cache key, improving cache hit rates when your prompts share the same logical content but differ slightly in formatting or ordering. We recommend using a **session-scoped key** like `userid-chatsessionid` (e.g. `"user123-chat456"`). Within a single chat session, the conversation history grows incrementally — each new request reuses all previous turns plus one new message. A per-session cache key ensures these near-identical prompts always hit the cache. ```python Python theme={null} import os from openai import OpenAI client = OpenAI( api_key=os.environ["DEEPINFRA_API_KEY"], base_url="https://api.deepinfra.com/v1/openai", ) response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash-0731", messages=[ {"role": "system", "content": "You are a helpful coding assistant."}, {"role": "user", "content": "How do I use async/await?"}, ], extra_body={"prompt_cache_key": "user123-chat456"}, ) ``` ```javascript JavaScript theme={null} import OpenAI from "openai"; const openai = new OpenAI({ apiKey: process.env.DEEPINFRA_API_KEY, baseURL: "https://api.deepinfra.com/v1/openai", }); const response = await openai.chat.completions.create({ model: "deepseek-ai/DeepSeek-V4-Flash-0731", messages: [ { role: "system", content: "You are a helpful coding assistant." }, { role: "user", content: "How do I use async/await?" }, ], prompt_cache_key: "user123-chat456", }); ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "prompt_cache_key": "user123-chat456", "messages": [ { "role": "system", "content": "You are a helpful coding assistant." }, { "role": "user", "content": "How do I use async/await?" } ] }' ``` Requests with the same `prompt_cache_key` and model will share a KV cache, even if their prompt prefixes aren't byte-for-byte identical. | Parameter | Type | Description | | - | - | - | | `prompt_cache_key` | `string` | An explicit key for cache lookup. Requests with the same key and model share a KV cache. | > [!NOTE] > > Need a prefix to stay cached for a guaranteed window? [Prompt cache retention](/chat/prompt-cache-retention) lets you retain a prompt for 5 minutes or 1 hour with `prompt_cache_options`, billed at a cache-write premium upfront and reused at the cache-read rate. ## Notes * Prompt caching is available on supported models — check the model page for details * Cache entries expire after a period of inactivity * Caches are per-model and per-account * When using `prompt_cache_key`, the key is scoped per-model and per-account # Reasoning Models Source: https://docs.deepinfra.com/chat/reasoning Configure chain-of-thought reasoning with reasoning_effort and the reasoning parameter. Some models on DeepInfra support extended chain-of-thought reasoning — the model "thinks through" a problem step by step before producing a final answer. By default, reasoning models produce a reasoning trace alongside the response. You can control this behavior with the `reasoning_effort` parameter. ## Supported models Reasoning is available on models that support chain-of-thought, including: * `deepseek-ai/DeepSeek-V4-Flash-0731` * `deepseek-ai/DeepSeek-V4-Pro-0813` * `zai-org/GLM-5.2` * `moonshotai/Kimi-K3` * `inclusionAI/Ling-3.0-flash` Check the [model catalog](https://deepinfra.com/models) for the latest list. ## Controlling reasoning effort Use `reasoning_effort` to control how much reasoning the model performs. Higher effort means deeper thinking but more output tokens and higher latency. ```python Python theme={null} import os from openai import OpenAI client = OpenAI( api_key=os.environ["DEEPINFRA_API_KEY"], base_url="https://api.deepinfra.com/v1/openai", ) response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash-0731", messages=[{"role": "user", "content": "Prove that the square root of 2 is irrational."}], extra_body={"reasoning_effort": "high"}, ) print(response.choices[0].message.content) ``` ```javascript JavaScript theme={null} import OpenAI from "openai"; const openai = new OpenAI({ apiKey: process.env.DEEPINFRA_API_KEY, baseURL: "https://api.deepinfra.com/v1/openai", }); const response = await openai.chat.completions.create({ model: "deepseek-ai/DeepSeek-V4-Flash-0731", messages: [{ role: "user", content: "Prove that the square root of 2 is irrational." }], reasoning_effort: "high", }); console.log(response.choices[0].message.content); ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "reasoning_effort": "high", "messages": [ { "role": "user", "content": "Prove that the square root of 2 is irrational." } ] }' ``` ## Disabling reasoning Set `reasoning_effort` to `"none"` to disable chain-of-thought entirely. The model will respond directly without a reasoning trace — faster and cheaper. ```python Python theme={null} response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash-0731", messages=[{"role": "user", "content": "What is the capital of France?"}], extra_body={"reasoning_effort": "none"}, ) ``` ```javascript JavaScript theme={null} const response = await openai.chat.completions.create({ model: "deepseek-ai/DeepSeek-V4-Flash-0731", messages: [{ role: "user", content: "What is the capital of France?" }], reasoning_effort: "none", }); ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "reasoning_effort": "none", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' ``` ## The `reasoning` parameter For more granular control, use the `reasoning` object instead of `reasoning_effort`: ```python Python theme={null} response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash-0731", messages=[{"role": "user", "content": "Solve this step by step: 15! / 13!"}], extra_body={ "reasoning": { "effort": "medium", "enabled": True, } }, ) ``` ```javascript JavaScript theme={null} const response = await openai.chat.completions.create({ model: "deepseek-ai/DeepSeek-V4-Flash-0731", messages: [{ role: "user", content: "Solve this step by step: 15! / 13!" }], reasoning: { effort: "medium", enabled: true, }, }); ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "reasoning": { "effort": "medium", "enabled": true }, "messages": [ { "role": "user", "content": "Solve this step by step: 15! / 13!" } ] }' ``` Setting `"enabled": false` is equivalent to `reasoning_effort: "none"`. ## When to use reasoning | Use case | Recommended setting | | - | - | | Math, logic, and code problems | `"high"` (default for reasoning models) | | Multi-step analysis | `"medium"` or `"high"` | | Simple Q\&A, translation, summarization | `"none"` | | Cost-sensitive workloads | `"none"` or `"low"` | ## Supported parameters | Parameter | Type | Description | | - | - | - | | `reasoning_effort` | `string` | Controls reasoning depth: `"none"`, `"low"`, `"medium"`, `"high"`. | | `reasoning` | `object` | Fine-grained reasoning config. | | `reasoning.effort` | `string` | Same values as `reasoning_effort`. | | `reasoning.enabled` | `boolean` | Explicitly enable or disable reasoning. | ## Notes * Reasoning tokens count toward output token billing * Disabling reasoning on a reasoning model makes it behave like a standard chat model * `reasoning_effort: "none"` is equivalent to `reasoning: { enabled: false }` * Not all models support reasoning — using these parameters on a non-reasoning model has no effect > **[Chat Completions](/chat/overview)** > > Full chat completions API reference. > **[Streaming](/chat/streaming)** > > Stream reasoning responses token by token. > **[Prompt Caching](/chat/prompt-caching)** > > Cache long prompts for faster reasoning. # Streaming Source: https://docs.deepinfra.com/chat/streaming Stream chat completion responses token by token using server-sent events. DeepInfra supports streaming responses via server-sent events (SSE), the same protocol as OpenAI. Set `stream: true` in your request to enable it. ## Examples ```python Python theme={null} import os from openai import OpenAI openai = OpenAI( api_key=os.environ["DEEPINFRA_API_KEY"], base_url="https://api.deepinfra.com/v1/openai", ) stream = openai.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash-0731", messages=[{"role": "user", "content": "Hello"}], stream=True, ) for event in stream: if event.choices[0].finish_reason: print(event.choices[0].finish_reason, event.usage['prompt_tokens'], event.usage['completion_tokens']) else: print(event.choices[0].delta.content, end="", flush=True) ``` ```javascript JavaScript theme={null} import OpenAI from "openai"; const openai = new OpenAI({ apiKey: process.env.DEEPINFRA_API_KEY, baseURL: "https://api.deepinfra.com/v1/openai", }); const completion = await openai.chat.completions.create({ messages: [{ role: "user", content: "Hello" }], model: "deepseek-ai/DeepSeek-V4-Flash-0731", stream: true, }); for await (const chunk of completion) { if (chunk.choices[0].finish_reason) { console.log(chunk.choices[0].finish_reason, chunk.usage.prompt_tokens, chunk.usage.completion_tokens); } else { process.stdout.write(chunk.choices[0].delta.content); } } ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "stream": true, "messages": [ { "role": "user", "content": "Hello!" } ] }' ``` ## SSE format Each streamed chunk is a `data:` line containing a JSON object: ``` data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"delta":{"content":"Hello"},"finish_reason":null}]} data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":10,"completion_tokens":5}} data: [DONE] ``` The final chunk before `[DONE]` contains usage information. ## Notes * Streaming works for all supported models * Usage stats are available in the last chunk (when `finish_reason` is set) * The `completion_tokens` and `prompt_tokens` counts are the same as non-streaming # Structured Outputs Source: https://docs.deepinfra.com/chat/structured-outputs Get model responses in JSON format using response_format. In addition to text, the DeepInfra API can return responses in JSON format. This is supported in both our inference API and our OpenAI-compatible API, across [many of our models](https://deepinfra.com/models?q=json). There are two modes: | Mode | How to use | When to use | | - | - | - | | `json_object` | `{"type": "json_object"}` | Any valid JSON object, schema-free | | `json_schema` | `{"type": "json_schema", "json_schema": {...}}` | Enforces a strict output schema | ## json\_object mode The simplest way to get JSON output. The model returns a valid JSON object but you don't control the exact shape. ```python Python theme={null} import os import openai import json client = openai.OpenAI( base_url="https://api.deepinfra.com/v1/openai", api_key=os.environ["DEEPINFRA_API_KEY"], ) messages = [ { "role": "user", "content": "Provide a JSON list of 3 famous scientific breakthroughs in the past century, all of the countries which contributed, and in what year." } ] response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash-0731", messages=messages, response_format={"type": "json_object"}, ) print(response.choices[0].message.content) ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "messages": [ { "role": "user", "content": "Provide a JSON list of 3 famous scientific breakthroughs." } ], "response_format": {"type": "json_object"} }' ``` ## json\_schema mode Enforces a strict output schema using [JSON Schema](https://json-schema.org/). The model is constrained to produce only values that match your schema — useful when downstream code depends on a fixed structure. ```python Python theme={null} import os import openai import json client = openai.OpenAI( base_url="https://api.deepinfra.com/v1/openai", api_key=os.environ["DEEPINFRA_API_KEY"], ) response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash-0731", messages=[ { "role": "user", "content": "Extract the name, country, and year from: 'Alexander Fleming discovered Penicillin in the UK in 1928.'" } ], response_format={ "type": "json_schema", "json_schema": { "name": "breakthrough", "strict": True, "schema": { "type": "object", "properties": { "name": {"type": "string"}, "country": {"type": "string"}, "year": {"type": "integer"} }, "required": ["name", "country", "year"], "additionalProperties": False } } } ) print(json.loads(response.choices[0].message.content)) ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "messages": [ { "role": "user", "content": "Extract the name, country, and year from: '\''Alexander Fleming discovered Penicillin in the UK in 1928.'\''" } ], "response_format": { "type": "json_schema", "json_schema": { "name": "breakthrough", "strict": true, "schema": { "type": "object", "properties": { "name": {"type": "string"}, "country": {"type": "string"}, "year": {"type": "integer"} }, "required": ["name", "country", "year"], "additionalProperties": false } } } }' ``` Output: ```json theme={null} {"name": "Penicillin", "country": "UK", "year": 1928} ``` ## Tips **Always prompt the model to produce JSON.** While not strictly required for `json_object`, mentioning the expected format in your prompt improves consistency. **Prefer `json_schema` for production.** When your code depends on specific field names or types, `json_schema` with `"strict": true` eliminates shape surprises. **Watch for truncation.** If the model stops due to `max_tokens` or `length`, the JSON may be incomplete. Always validate before parsing. ## Caveats > [!WARNING] > > JSON mode can affect model alignment. When forced to produce structured output, some models are more likely to hallucinate values rather than say "I don't know." This is especially visible for prompts about real-time data (weather, stock prices, etc.). > > Example: asking "What's the weather in San Francisco?" with JSON mode enabled may cause the model to fabricate a weather forecast rather than explaining it doesn't have real-time data. **Best practices:** * Use JSON mode for structured data extraction tasks, not for general question answering * Keep prompts specific about the expected schema * Validate model output before using it in production systems * Use lower temperatures (\< 0.7) for more consistent structure # Tool Calling Source: https://docs.deepinfra.com/chat/tool-calling Let models call external functions — the foundation of AI agents. Tool calling (also known as function calling) is the most important capability for building AI agents. It lets models decide when to invoke external tools — web search, code execution, database queries, API calls — and seamlessly weave the results into a final response. Without reliable tool calling, agentic systems break down. **Tool call accuracy is a top priority for us.** We invest significant engineering effort in ensuring that function call parsing, argument extraction, and round-trip reliability are correct across all supported models. In Moonshot's [K2-Vendor-Verifier evaluation](https://github.com/MoonshotAI/K2-Vendor-Verifier?tab=readme-ov-file#k2-0905-evaluation-results) of Kimi K2, DeepInfra scored among the highest accuracy of any provider tested. The current Kimi model in our catalog is [`moonshotai/Kimi-K3`](https://deepinfra.com/moonshotai/Kimi-K3). We provide an OpenAI-compatible tool calling API. For more background, see the [DeepInfra blog](https://deepinfra.com/blog/function-calling-feature). ## Setup ```python Python theme={null} import os import openai import json client = openai.OpenAI( base_url="https://api.deepinfra.com/v1/openai", api_key=os.environ["DEEPINFRA_API_KEY"], ) ``` ```javascript JavaScript theme={null} import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://api.deepinfra.com/v1/openai", apiKey: process.env.DEEPINFRA_API_KEY, }); ``` ## Define your function ```python Python theme={null} def get_current_weather(location): """Get the current weather in a given location""" if "tokyo" in location.lower(): return json.dumps({"location": "Tokyo", "temperature": "75"}) elif "san francisco" in location.lower(): return json.dumps({"location": "San Francisco", "temperature": "60"}) elif "paris" in location.lower(): return json.dumps({"location": "Paris", "temperature": "70"}) else: return json.dumps({"location": location, "temperature": "unknown"}) ``` ```javascript JavaScript theme={null} async function get_current_weather(location) { if (location.toLowerCase().includes("tokyo")) { return JSON.stringify({"location": "Tokyo", "temperature": "75"}); } else if (location.toLowerCase().includes("san francisco")) { return JSON.stringify({"location": "San Francisco", "temperature": "60"}); } else if (location.toLowerCase().includes("paris")) { return JSON.stringify({"location": "Paris", "temperature": "70"}); } else { return JSON.stringify({"location": location, "temperature": "unknown"}); } } ``` ## Step 1: Send tools to the model ```python Python theme={null} tools = [{ "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA" } }, "required": ["location"] }, } }] messages = [{"role": "user", "content": "What is the weather in San Francisco?"}] response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash-0731", messages=messages, tools=tools, tool_choice="auto", ) tool_calls = response.choices[0].message.tool_calls for tool_call in tool_calls: print(tool_call.model_dump()) ``` ```javascript JavaScript theme={null} const tools = [{ "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA" } }, "required": ["location"] }, } }]; const messages = [{"role": "user", "content": "What is the weather in San Francisco?"}]; const response = await client.chat.completions.create({ model: "deepseek-ai/DeepSeek-V4-Flash-0731", messages: messages, tools: tools, tool_choice: "auto", }); const tool_calls = response.choices[0].message.tool_calls; for (const tool_call of tool_calls) { console.log(tool_call); } ``` Output: ``` {'id': 'call_X0xYqdnoUonPJpQ6HEadxLHE', 'function': {'arguments': '{"location": "San Francisco"}', 'name': 'get_current_weather'}, 'type': 'function'} ``` ## Step 2: Execute the function and send results back ```python Python theme={null} # Extend conversation with assistant's reply messages.append(response.choices[0].message) for tool_call in tool_calls: function_name = tool_call.function.name if function_name == "get_current_weather": function_args = json.loads(tool_call.function.arguments) function_response = get_current_weather( location=function_args.get("location") ) messages.append({ "tool_call_id": tool_call.id, "role": "tool", "content": function_response, }) # Get a new response from the model with function results second_response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash-0731", messages=messages, tools=tools, tool_choice="auto", ) print(second_response.choices[0].message.content) ``` ```javascript JavaScript theme={null} // Extend conversation with assistant's reply messages.push(response.choices[0].message); for (const tool_call of tool_calls) { const function_name = tool_call.function.name; if (function_name == "get_current_weather") { const function_args = JSON.parse(tool_call.function.arguments); const function_response = await get_current_weather(function_args.location); messages.push({ "tool_call_id": tool_call.id, "role": "tool", "content": function_response, }); } } const second_response = await client.chat.completions.create({ model: "deepseek-ai/DeepSeek-V4-Flash-0731", messages: messages, tools: tools, tool_choice: "auto", }); console.log(second_response.choices[0].message.content); ``` Output: ``` The current temperature in San Francisco, CA is 60 degrees. ``` ## Tips * Write clear, detailed function descriptions — model quality depends heavily on them * Use lower temperatures (\< 1.0) to avoid erratic parameter values * Avoid system messages when using tool calling * Model quality degrades with more functions — keep the list focused * Keep `top_p` and `top_k` at their defaults ## Supported features | Feature | Supported | | - | - | | Single tool calls | ✅ | | Parallel tool calls | ✅ (quality may vary) | | `tool_choice: "auto"` | ✅ | | `tool_choice: "none"` | ✅ | | Streaming mode | ✅ | | Nested calls | ❌ | ## Notes * Function definitions count toward your input token usage * Inference usage is counted as normal when using tool calling # Vision & OCR Source: https://docs.deepinfra.com/chat/vision Send images to multimodal models for visual understanding and text extraction. DeepInfra hosts multimodal models that accept both images and text as input and produce text output. These models use the standard OpenAI vision API format and cover two major use cases: * **Visual understanding** — describe images, answer questions about visual content, compare images, analyze charts * **OCR (Optical Character Recognition)** — extract text from scanned documents, receipts, invoices, screenshots, handwritten notes, and PDFs ## Available vision models Browse [all vision models](https://deepinfra.com/models/ocr). ## Available OCR models We host a growing set of OCR-specialized models for high-accuracy text extraction. Browse the [full OCR model catalog](https://deepinfra.com/models/ocr/). OCR models currently use the same vision API format below. A dedicated OCR endpoint optimized for document processing is coming soon. ## Quick start Images are passed in two ways: 1. **URL** — pass a link to a publicly accessible image 2. **Base64** — encode the image and include it directly in the request ### Image URL ```bash theme={null} curl "https://api.deepinfra.com/v1/openai/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "model": "Qwen/Qwen3-VL-235B-A22B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://shared.deepinfra.com/models/Qwen/Qwen3-VL-235B-A22B-Instruct/cover_image.188ae740a50d6f2f48bed81f834ffbe2fe5a0f38bd96dbd312ef804c18aad0c4.webp" } }, { "type": "text", "text": "What'\''s in this image?" } ] } ] }' ``` ### Base64 encoded image ```python theme={null} import os from openai import OpenAI import base64 import requests openai = OpenAI( api_key=os.environ["DEEPINFRA_API_KEY"], base_url="https://api.deepinfra.com/v1/openai", ) image_url = "https://shared.deepinfra.com/models/Qwen/Qwen3-VL-235B-A22B-Instruct/cover_image.188ae740a50d6f2f48bed81f834ffbe2fe5a0f38bd96dbd312ef804c18aad0c4.webp" base64_image = base64.b64encode(requests.get(image_url).content).decode("utf-8") chat_completion = openai.chat.completions.create( model="Qwen/Qwen3-VL-235B-A22B-Instruct", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": f"data:image/jpeg;base64,{base64_image}" } }, { "type": "text", "text": "What's in this image?" } ] } ] ) print(chat_completion.choices[0].message.content) ``` ## OCR example Extract all text from a document image: ```python theme={null} import os from openai import OpenAI import base64 openai = OpenAI( api_key=os.environ["DEEPINFRA_API_KEY"], base_url="https://api.deepinfra.com/v1/openai", ) with open("invoice.png", "rb") as f: base64_image = base64.b64encode(f.read()).decode("utf-8") response = openai.chat.completions.create( model="Qwen/Qwen3-VL-235B-A22B-Instruct", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": {"url": f"data:image/png;base64,{base64_image}"} }, { "type": "text", "text": "Extract all text from this document. Preserve the structure and layout as much as possible." } ] } ] ) print(response.choices[0].message.content) ``` Common OCR prompts: * `"Extract all text from this image."` — basic extraction * `"Extract all text and return it as structured JSON with field names and values."` — structured extraction (e.g. invoices, forms) * `"Transcribe the handwritten text in this image."` — handwriting recognition * `"List all line items, quantities, and prices from this receipt."` — targeted extraction ## Multiple images You can pass multiple images in a single request by including multiple `image_url` content items: ```bash theme={null} curl "https://api.deepinfra.com/v1/openai/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "model": "Qwen/Qwen3-VL-235B-A22B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "image_url", "image_url": {"url": "https://example.com/page1.jpg"} }, { "type": "image_url", "image_url": {"url": "https://example.com/page2.jpg"} }, { "type": "text", "text": "Extract all text from both pages." } ] } ] }' ``` ## Pricing and token counting Images are tokenized and billed as input tokens. The number of tokens consumed by an image is reported in the response under `"usage": {"prompt_tokens": ...}`. Different models work with different image resolutions. You can still pass images of any resolution — the model will rescale them automatically. Check the model's documentation page for supported resolutions. ## Limitations * Supported image formats: **jpg**, **png**, **webp** * Maximum image size: **20MB** * The `detail` parameter (image fidelity) is not currently supported # GPU Instances Source: https://docs.deepinfra.com/gpu-instances/overview Rent dedicated B200 GPU instances with SSH access for training, fine-tuning, and custom workloads. DeepInfra GPU Instances give you dedicated access to NVIDIA B200 hardware — the most powerful GPUs available. You get a container with SSH access and full control over your environment, billed by the hour. ## Available GPUs | GPU | Memory | Best for | | - | - | - | | **B200-180GB** | 180GB HBM3e | Large-scale inference, fine-tuning, training | ## Key features * **Dedicated access** — no sharing with other users * **SSH access** — connect directly to your container * **Full environment control** — bring your own Docker image or use ours * **Pay-per-use** — billed by the hour, only while running * **Quick setup** — running in minutes ## Use cases * LLM training and fine-tuning * Large-scale batch inference * Research and experimentation * Development environments with GPU access ## Web UI ### Start a new container 1. Go to [Dashboard → GPU Instances](https://deepinfra.com/dash/instances) 2. Click **New Container** 3. **Select GPU configuration** — choose from available B200/B300 configs. Each shows: * GPU type, quantity, and memory (e.g., `1xB200-180GB`, `8xB200-180GB`) * Hourly pricing * Availability status 4. **Enter container details**: * **Container Name** — a descriptive name * **SSH Key** — paste your public SSH key (format: `ssh-rsa AAAAB3NzaC1yc2E...`) 5. Accept the NVIDIA license agreements and cryptocurrency mining policy 6. Click **I agree to the above** ### Connect to a running container 1. Wait for container status to show `running` 2. Click on the SSH login field to copy the command 3. Run `ssh ubuntu@` in your terminal ### Stop a container 1. Click on the container in the instances list 2. Click **Terminate** 3. Type `confirm` and click **Terminate** > [!WARNING] > > All container data is permanently lost when terminated. Save your work before stopping. ## HTTP API ### Create a container ```bash theme={null} curl -X POST https://api.deepinfra.com/v1/containers \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "my-cluster", "gpu_config": "8xB200-180GB", "container_image": "di-cont-ubuntu-torch:latest", "cloud_init_user_data": "#cloud-config\nusers:\n- name: ubuntu\n shell: /bin/bash\n sudo: '\''ALL=(ALL) NOPASSWD:ALL'\''\n ssh_authorized_keys:\n - ssh-rsa AAAAB3NzaC1yc2E..." }' ``` ### Get container details ```bash theme={null} curl -X GET https://api.deepinfra.com/v1/containers/{container_id} \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" ``` Once `running`, connect via SSH: ```bash theme={null} ssh ubuntu@ ``` ### List containers ```bash theme={null} curl -X GET https://api.deepinfra.com/v1/containers \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" ``` ### Terminate a container ```bash theme={null} curl -X DELETE https://api.deepinfra.com/v1/containers/{container_id} \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" ``` ## Container lifecycle | State | Description | | - | - | | `creating` | Container is being initialized | | `starting` | Container is booting up | | `running` | Container is active and accessible | | `shutting_down` | Container is being terminated | | `failed` | Container failed to start or encountered an error | | `deleted` | Container has been permanently removed | # What is DeepInfra Source: https://docs.deepinfra.com/index AI inference cloud — OpenAI-compatible API, 100s of open-source models, private GPU deployments, and GPU rental. DeepInfra is an AI inference cloud that makes it simple to run the latest machine learning models at scale — LLMs, vision, embeddings, image generation, video generation, speech, and more. ## What you can do > **[LLMs & Chat](/chat/overview)** > > OpenAI-compatible API for 100+ LLMs. Swap your base URL, keep your code. > **[Vision & OCR](/chat/vision)** > > Multimodal models for visual understanding and document text extraction. > **[Embeddings & Reranking](/apis/embeddings)** > > State-of-the-art embedding and reranker models for search and RAG. > **[Image & Video Generation](/apis/image-generation)** > > FLUX, Stable Diffusion, text-to-video, and more. > **[Speech](/apis/speech)** > > Speech recognition (Whisper) and text-to-speech models. > **[Deploy Private Models](/private-models/overview)** > > Run your own fine-tuned LLM on A100 / H100 / H200 / B200 / B300 with autoscaling. ## Why DeepInfra **Drop-in OpenAI replacement.** Point your existing OpenAI SDK to `https://api.deepinfra.com/v1/openai` and your code works without changes. No migration required. **Best price for open-source models.** DeepInfra consistently offers the lowest prices for open-source model inference. You only pay per token — no idle GPU time, no minimums, no seat fees. DeepInfra is also the provider with the most models on [OpenRouter](https://openrouter.ai/provider/deepinfra). **Always-fresh model catalog.** DeepInfra is typically among the first providers to deploy a newly released model. **Private deployments for compliance and customization.** Need to run your own fine-tuned weights, or require data isolation? Deploy a dedicated instance on A100/H100/H200/B200/B300 with autoscaling and a private endpoint — competitive GPU pricing, deployable in just a few clicks. **GPU Clusters for training and full control.** Rent a B200 or B300 cluster with SSH access and run whatever you want. ## Get started in 60 seconds > **[Quickstart](/quickstart)** > > Make your first API call — no installation required. ## Quick example ```python theme={null} import os from openai import OpenAI client = OpenAI( api_key=os.environ["DEEPINFRA_API_KEY"], base_url="https://api.deepinfra.com/v1/openai", ) response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash-0731", messages=[{"role": "user", "content": "Hello!"}], ) print(response.choices[0].message.content) ``` Get your API key from the [Dashboard](https://deepinfra.com/dash/api_keys). # AI SDK (Vercel) Source: https://docs.deepinfra.com/integrations/ai-sdk Use DeepInfra models with the Vercel AI SDK for TypeScript/JavaScript. The [AI SDK](https://sdk.vercel.ai/) by Vercel is the AI toolkit for TypeScript and JavaScript from the creators of Next.js. DeepInfra has a first-party AI SDK provider. See the [full AI SDK docs for DeepInfra](https://sdk.vercel.ai/providers/ai-sdk-providers/deepinfra). ## Installation ```bash theme={null} npm install ai @ai-sdk/deepinfra ``` Get your API key from the [Dashboard](https://deepinfra.com/dash/api_keys). ## Text generation ```javascript theme={null} import { createDeepInfra } from "@ai-sdk/deepinfra"; import { generateText } from "ai"; const deepinfra = createDeepInfra({ apiKey: process.env.DEEPINFRA_API_KEY }); const { text, usage, finishReason } = await generateText({ model: deepinfra("deepseek-ai/DeepSeek-V4-Flash-0731"), prompt: "Write a vegetarian lasagna recipe for 4 people.", }); console.log(text); ``` With a system message: ```javascript theme={null} const { text } = await generateText({ model: deepinfra("deepseek-ai/DeepSeek-V4-Flash-0731"), prompt: "Write a vegetarian lasagna recipe for 4 people.", system: "You are a professional writer. You write simple, clear, and concise content.", }); ``` ## Streaming ```javascript theme={null} import { createDeepInfra } from "@ai-sdk/deepinfra"; import { streamText } from "ai"; const deepinfra = createDeepInfra({ apiKey: process.env.DEEPINFRA_API_KEY }); const result = streamText({ model: deepinfra("deepseek-ai/DeepSeek-V4-Flash-0731"), prompt: "Invent a new holiday and describe its traditions.", system: "You are a professional writer. You write simple, clear, and concise content.", }); for await (const textPart of result.textStream) { process.stdout.write(textPart); } ``` ## Conversations ```javascript theme={null} const { text } = await generateText({ model: deepinfra("deepseek-ai/DeepSeek-V4-Flash-0731"), messages: [ { role: "system", content: "Respond like a michelin starred chef." }, { role: "user", content: "Can you name at least two different techniques to cook lamb?" }, { role: "assistant", content: "Bonjour! Let me tell you..." }, { role: "user", content: "Tell me more about the second method." }, ], }); ``` Streaming conversations work the same way — use `streamText` with `messages`. ## Structured data Generate typed, structured output using Zod schemas: ```javascript theme={null} import { createDeepInfra } from "@ai-sdk/deepinfra"; import { generateObject } from "ai"; import { z } from "zod"; const deepinfra = createDeepInfra({ apiKey: process.env.DEEPINFRA_API_KEY }); const { object } = await generateObject({ model: deepinfra("deepseek-ai/DeepSeek-V4-Flash-0731"), schema: z.object({ recipe: z.object({ name: z.string(), ingredients: z.array(z.object({ name: z.string(), amount: z.string() })), steps: z.array(z.string()), }), }), prompt: "Generate a lasagna recipe.", }); console.log(object.recipe.name); ``` Enum output: ```javascript theme={null} const { object } = await generateObject({ model: deepinfra("deepseek-ai/DeepSeek-V4-Flash-0731"), output: "enum", enum: ["action", "comedy", "drama", "horror", "sci-fi"], prompt: 'Classify: "A group of astronauts travel through a wormhole..."', }); ``` ## Tool / function calling ```javascript theme={null} import { createDeepInfra } from "@ai-sdk/deepinfra"; import { generateText, tool } from "ai"; import { z } from "zod"; const deepinfra = createDeepInfra({ apiKey: process.env.DEEPINFRA_API_KEY }); const result = await generateText({ model: deepinfra("deepseek-ai/DeepSeek-V4-Flash-0731"), tools: { weather: tool({ description: "Get the weather in a location", parameters: z.object({ location: z.string().describe("The location to get the weather for"), }), execute: async ({ location }) => ({ location, temperature: 72 + Math.floor(Math.random() * 21) - 10, }), }), }, prompt: "What is the weather in San Francisco?", maxSteps: 2, }); console.log(result.text); ``` # Anthropic SDK & Claude Code Source: https://docs.deepinfra.com/integrations/anthropic Use DeepInfra models with the Anthropic Messages API, Claude Code, and the Anthropic SDK. DeepInfra exposes an Anthropic-compatible Messages API. This means tools that target the Anthropic API — Claude Code, the Anthropic Python and TypeScript SDKs, and any framework with an Anthropic adapter — can point at DeepInfra and use open-source models. ## Endpoint ``` https://api.deepinfra.com/anthropic ``` Two endpoints are available: | Endpoint | Description | | - | - | | `POST /anthropic/v1/messages` | Create a message (chat completion) | | `POST /anthropic/v1/messages/count_tokens` | Count tokens for a message request | ## Authentication Both standard Anthropic authentication methods are supported: | Header | Example | | - | - | | `Authorization` | `Bearer $DEEPINFRA_API_KEY` | | `x-api-key` | `$DEEPINFRA_API_KEY` | You can also pass `anthropic-version` and `anthropic-beta` headers as needed. ## Using the Anthropic SDK ### Installation ```bash Python theme={null} pip install anthropic ``` ```bash JavaScript theme={null} npm install @anthropic-ai/sdk ``` ### Create a message Point the client at the DeepInfra endpoint and pass a DeepInfra model name: ```python Python theme={null} import os import anthropic client = anthropic.Anthropic( base_url="https://api.deepinfra.com/anthropic", api_key=os.environ["DEEPINFRA_API_KEY"], ) message = client.messages.create( model="zai-org/GLM-5.2", max_tokens=1024, messages=[ {"role": "user", "content": "Hello!"} ], ) print(message.content[0].text) ``` ```javascript JavaScript theme={null} import Anthropic from "@anthropic-ai/sdk"; const client = new Anthropic({ baseURL: "https://api.deepinfra.com/anthropic", apiKey: process.env.DEEPINFRA_API_KEY, }); const message = await client.messages.create({ model: "zai-org/GLM-5.2", max_tokens: 1024, messages: [ { role: "user", content: "Hello!" }, ], }); console.log(message.content[0].text); ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/anthropic/v1/messages" \ -H "Content-Type: application/json" \ -H "x-api-key: $DEEPINFRA_API_KEY" \ -H "anthropic-version: 2023-06-01" \ -d '{ "model": "zai-org/GLM-5.2", "max_tokens": 1024, "messages": [ { "role": "user", "content": "Hello!" } ] }' ``` ## Using with Claude Code Claude Code can use DeepInfra as its backend. Claude Code exposes four model slots — `opus`, `sonnet`, `fable`, and `haiku` — plus one extra custom entry in the `/model` picker. You map each slot to a DeepInfra model. ### Recommended models Claude Code's slots are a capability/cost ladder — `haiku` \< `sonnet` \< `opus` \< `fable`. `fable` sits *above* `opus`: it's the slowest and most expensive slot, meant for the hardest long-horizon work. Map DeepInfra models onto it the same way, most capable first: | Slot | Model | Context | Why | | - | - | - | - | | `fable` | `moonshotai/Kimi-K3` | 1M | Strongest open-weights agentic coder (Terminal-Bench 2.1 88.3, SWE Marathon 42.0, vendor-stated). Also the priciest of the set by a wide margin, and the slowest to finish — exactly the tradeoff the `fable` slot exists for. | | `opus` | `deepseek-ai/DeepSeek-V4-Pro-0813` | 1M | 1.6T-param MoE built for advanced reasoning and long-running agent tasks. The official release, with much stronger agentic behaviour than the preview. Your escalation model when `sonnet` stalls. | | `sonnet` | `deepseek-ai/DeepSeek-V4-Flash-0731` | 1M | Daily driver. Near-flagship agentic quality (Terminal-Bench 2.1 82.7, vendor-stated) at a small fraction of the cost of the slots above it, with the full 1M context — the right economics for the model you run all day. | | `haiku` | `inclusionAI/Ling-3.0-flash` | 128K | Background tasks. \~400 tok/s and the cheapest of the set — tuned for token efficiency under tight latency and serving-cost budgets. | | custom | `zai-org/GLM-5.2` | 1M | Top *third-party-measured* open-weights model on Terminal-Bench 2.0 (81.0%), the closest benchmark to real agentic terminal work. Independently verified rather than vendor-stated — a good second opinion when a task matters and you want a different model family on it. | All five support prompt caching, which makes cache reads substantially cheaper than fresh input — worth enabling for long agent sessions. See each model's page on [deepinfra.com/models](https://deepinfra.com/models) for current per-token pricing. > [!NOTE] > > Use the dated slugs — `DeepSeek-V4-Pro-0813` and `DeepSeek-V4-Flash-0731`. The undated `deepseek-ai/DeepSeek-V4-Pro` and `deepseek-ai/DeepSeek-V4-Flash` are the earlier preview checkpoints, which the dated releases supersede. ### Option A: `.claude/settings.json` (recommended) Instead of exporting environment variables, put the configuration in a Claude Code settings file. Use `~/.claude/settings.json` to apply it to all projects, or `/.claude/settings.local.json` for a single project (that file is gitignored by default, so your key stays out of version control). Get an API key at [deepinfra.com/dash/api\_keys](https://deepinfra.com/dash/api_keys). ```json .claude/settings.json theme={null} { "env": { "ANTHROPIC_BASE_URL": "https://api.deepinfra.com/anthropic", "ANTHROPIC_AUTH_TOKEN": "", "ANTHROPIC_DEFAULT_FABLE_MODEL": "moonshotai/Kimi-K3", "ANTHROPIC_DEFAULT_FABLE_MODEL_NAME": "Kimi K3", "ANTHROPIC_DEFAULT_OPUS_MODEL": "deepseek-ai/DeepSeek-V4-Pro-0813", "ANTHROPIC_DEFAULT_OPUS_MODEL_NAME": "DeepSeek V4 Pro 0813", "ANTHROPIC_DEFAULT_SONNET_MODEL": "deepseek-ai/DeepSeek-V4-Flash-0731", "ANTHROPIC_DEFAULT_SONNET_MODEL_NAME": "DeepSeek V4 Flash 0731", "ANTHROPIC_DEFAULT_HAIKU_MODEL": "inclusionAI/Ling-3.0-flash", "ANTHROPIC_DEFAULT_HAIKU_MODEL_NAME": "Ling 3.0 Flash", "ANTHROPIC_CUSTOM_MODEL_OPTION": "zai-org/GLM-5.2", "ANTHROPIC_CUSTOM_MODEL_OPTION_NAME": "GLM-5.2" } } ``` Reload Claude Code (or restart your VS Code window) and all five models show up in the `/model` picker. To go back to Anthropic's own models, delete the `env` block and reload. Notes on this file: * `ANTHROPIC_BASE_URL` has **no** `/v1` suffix — Claude Code appends `/v1/messages` itself. * `ANTHROPIC_AUTH_TOKEN` is sent as `Authorization: Bearer …`. `ANTHROPIC_API_KEY` (sent as `x-api-key`) works too. * Each `*_MODEL_NAME` sets the label shown in the picker; it is cosmetic. * `ANTHROPIC_CUSTOM_MODEL_OPTION` adds one extra picker entry. Any other DeepInfra model still works via `/model ` as free text. * **VS Code extension:** if the picker doesn't pick up the config after a window reload, also add the same variables under `claudeCode.environmentVariables` in your VS Code settings. ### Option B: shell function with environment variables If you'd rather keep your default Claude Code setup untouched and switch per-invocation, add a dedicated shell function to your `~/.bashrc` or `~/.zshrc`: ```bash theme={null} deepinfra() { export ANTHROPIC_BASE_URL=https://api.deepinfra.com/anthropic export ANTHROPIC_AUTH_TOKEN=$DEEPINFRA_API_KEY export ANTHROPIC_MODEL=deepseek-ai/DeepSeek-V4-Flash-0731 export ANTHROPIC_DEFAULT_FABLE_MODEL=moonshotai/Kimi-K3 export ANTHROPIC_DEFAULT_OPUS_MODEL=deepseek-ai/DeepSeek-V4-Pro-0813 export ANTHROPIC_DEFAULT_SONNET_MODEL=deepseek-ai/DeepSeek-V4-Flash-0731 export ANTHROPIC_DEFAULT_HAIKU_MODEL=inclusionAI/Ling-3.0-flash export CLAUDE_CODE_SUBAGENT_MODEL=inclusionAI/Ling-3.0-flash export CLAUDE_CODE_MAX_OUTPUT_TOKENS=16384 claude "$@" } ``` Then run `deepinfra` instead of `claude` to launch Claude Code via DeepInfra. Your regular `claude` command stays unchanged. ### Model override environment variables The same variable names work in both options — as JSON keys under `env`, or as shell exports: | Environment variable | Description | Example | | - | - | - | | `ANTHROPIC_MODEL` | The primary model Claude Code uses for all tasks | `deepseek-ai/DeepSeek-V4-Flash-0731` | | `ANTHROPIC_DEFAULT_FABLE_MODEL` | Model used for the `fable` alias (deepest reasoning, slowest, priciest) | `moonshotai/Kimi-K3` | | `ANTHROPIC_DEFAULT_OPUS_MODEL` | Model used for the `opus` alias (hard tasks, escalation) | `deepseek-ai/DeepSeek-V4-Pro-0813` | | `ANTHROPIC_DEFAULT_SONNET_MODEL` | Model used for the `sonnet` alias (daily coding) | `deepseek-ai/DeepSeek-V4-Flash-0731` | | `ANTHROPIC_DEFAULT_HAIKU_MODEL` | Model used for the `haiku` alias and background tasks (tab completions, commit messages) | `inclusionAI/Ling-3.0-flash` | | `CLAUDE_CODE_SUBAGENT_MODEL` | Model used for subagents (parallel background tasks) | `inclusionAI/Ling-3.0-flash` | | `ANTHROPIC_CUSTOM_MODEL_OPTION` | Adds one extra entry to the `/model` picker | `zai-org/GLM-5.2` | Each `*_MODEL` variable above has an optional `*_MODEL_NAME` companion (e.g. `ANTHROPIC_DEFAULT_OPUS_MODEL_NAME`) that sets the display label in the picker. > [!NOTE] > > `ANTHROPIC_DEFAULT_HAIKU_MODEL` is used for lightweight background tasks like tab completions and commit messages. Pick a fast, cheap model here to keep costs low. The older `ANTHROPIC_SMALL_FAST_MODEL` variable is deprecated — use `ANTHROPIC_DEFAULT_HAIKU_MODEL` instead. ## Streaming Streaming works the same as the Anthropic API — use `stream=True` (Python) or `stream: true` (JS/cURL): ```python theme={null} with client.messages.stream( model="zai-org/GLM-5.2", max_tokens=1024, messages=[{"role": "user", "content": "Write a short poem about open source."}], ) as stream: for text in stream.text_stream: print(text, end="", flush=True) ``` ## Token counting Count the tokens in a message request before sending it: ```bash theme={null} curl "https://api.deepinfra.com/anthropic/v1/messages/count_tokens" \ -H "Content-Type: application/json" \ -H "x-api-key: $DEEPINFRA_API_KEY" \ -d '{ "model": "zai-org/GLM-5.2", "messages": [ { "role": "user", "content": "Hello, how are you?" } ] }' ``` ## Notes * You are running open-source models via the Anthropic protocol, not Anthropic's Claude models. * Model names use DeepInfra identifiers (e.g. `zai-org/GLM-5.2`), not Anthropic model names. * Not all Anthropic-specific features may be supported. Standard message creation, streaming, and token counting work as expected. > **[Chat Completions](/chat/overview)** > > Use the OpenAI-compatible API instead. > **[Authentication](/account/authentication)** > > API keys and scoped JWTs. # AutoGen Source: https://docs.deepinfra.com/integrations/autogen Build multi-agent LLM applications with AutoGen using DeepInfra endpoints. [AutoGen](https://github.com/microsoft/autogen) is a framework for building LLM applications with multiple agents that converse to solve tasks. It works with DeepInfra via the OpenAI-compatible API. ## Installation ```bash theme={null} pip install pyautogen ``` ## Configuration Point AutoGen at the DeepInfra endpoint using `base_url`: ```python theme={null} import autogen config_list = [ { "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "base_url": "https://api.deepinfra.com/v1/openai", "api_key": "" } ] llm_config = {"config_list": config_list, "seed": 42} assistant = autogen.AssistantAgent("assistant", llm_config=llm_config) user_proxy = autogen.UserProxyAgent("user_proxy", code_execution_config={"work_dir": "coding"}) user_proxy.initiate_chat(assistant, message="What time is it right now?") ``` You can use any [OpenAI-compatible LLM](https://deepinfra.com/models/text-generation) from DeepInfra. ## How it works In the example above, two agents converse to solve the task: 1. The **assistant** agent generates a Python code snippet to get the current time 2. The **user\_proxy** agent automatically detects and executes the code block 3. The result is sent back to the assistant, which summarizes it Example output: ````text theme={null} user_proxy (to assistant): What time is it now? -------------------------------------------------------------------------------- assistant (to user_proxy): To get the current time, you can use the `datetime` module in Python... ```python import datetime current_time = datetime.datetime.now() print(current_time.strftime("%I:%M %p")) ``` -------------------------------------------------------------------------------- user_proxy (to assistant): exitcode: 0 (execution succeeded) Code output: 02:20 PM -------------------------------------------------------------------------------- assistant (to user_proxy): The current time is 02:20 PM. ```` # Clef (decision models) Source: https://docs.deepinfra.com/integrations/clef Send a state and a set of typed questions to a decision model and get a probability for every allowed option, instead of generated text. [Clef](https://deepinfra.com/Cloudflare/clef) is a decision model. You give it a **state** and a set of **typed questions**, and it scores every allowed option of every question in a single pass. It returns probabilities, not prose. There is no text to parse and no output that can drift from the schema you sent. The distribution is the answer: you get every option with its probability, not just the winner, so you can threshold on confidence instead of guessing whether a generated label is trustworthy. ## The endpoint Decision models are not served on `/v1/openai/chat/completions`. Send them to `/v1/decisions`: ``` POST https://api.deepinfra.com/v1/decisions ``` Two other paths accept the same body and return the same reply: | Path | Use it when | | - | - | | `/v1/decisions` | You are calling DeepInfra directly. | | `/v1/systemone` | You are porting code that already targets a System One server. | | `/typesafe/v1/systemone` | You use a [TypeSafe](https://docs.typesafe.ai/api) or Jev SDK — set its base URL to `https://api.deepinfra.com/typesafe`. | ## Quickstart ```bash theme={null} curl https://api.deepinfra.com/v1/decisions \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "Cloudflare/clef", "state": "Help! My payouts have been failing for 3 days.", "questions": { "is_urgent": { "type": "noul", "instructions": "Does this convey urgency?" }, "department": { "type": "choice", "instructions": "Which team should handle this?", "criteria": { "billing": "Payments", "technical": "Bugs", "sales": "Pricing" } }, "frustration": { "type": "score", "instructions": "How frustrated is the customer?", "criteria": ["Calm", "Frustrated", "Very angry"] } } }' ``` The reply mirrors the question keys you chose: ```json theme={null} { "model": "Cloudflare/clef", "answers": { "is_urgent": { "type": "noul", "noul": 0.93 }, "department": { "type": "choice", "choice": "billing", "probabilities": { "billing": 0.88, "technical": 0.1, "sales": 0.02 }, "confidence": 0.88 }, "frustration": { "type": "score", "score": 1.67, "legend": { "0": "Calm", "1": "Frustrated", "2": "Very angry" }, "probabilities": { "0": 0.04, "1": 0.25, "2": 0.71 }, "confidence": 0.45 } }, "usage": { "input_tokens": 96, "output_tokens": 3 } } ``` ## Question types Every question is one of three shapes. The shape decides the reply. | Type | Asks | Reply | | - | - | - | | `noul` | A yes/no question | `noul` — the probability of yes | | `choice` | Pick one label from `criteria` | `choice` — the winner, plus `probabilities` and `confidence` | | `score` | Rate on an ordered scale | `score` — the expected level, plus `legend`, `probabilities` and `confidence` | Give `choice` at least two options and `score` at least two levels. The score is the probability-weighted average of the level indices, so `1.67` on a three-level scale means the model leans strongly toward the last level. ## State `state` does not have to be a string. Pass any JSON object and Clef reads it as structured input: ```json theme={null} { "model": "Cloudflare/clef", "state": { "invoice": { "vendor": "Acme", "total": 1250.0, "currency": "USD", "status": "overdue" } }, "questions": { "large": { "type": "noul", "instructions": "Is the total above 1000 USD?" } } } ``` Passing the record as JSON rather than as prose helps: the model sees the same field boundaries your code does, and questions about a specific field ("is `total` above the approval threshold") become unambiguous. ## Images When the answer depends on something Clef has to look at rather than read, attach images. Up to 4 per request, each a `data:` URI or an object with `content_type` and `base64`. Remote URLs are not fetched; download the image and embed it. ```json theme={null} { "model": "Cloudflare/clef", "state": "Which invoice is overdue?", "images": [ "data:image/png;base64,iVBORw0KGgo...", { "content_type": "image/jpeg", "base64": "/9j/4AAQSkZJRg..." } ], "questions": { "overdue": { "type": "noul", "instructions": "Is any invoice past due?" } } } ``` Clef reads the images before the state, so a question can refer to both. The whole request body must stay under 13 MiB. This is how document review, screenshot triage and scanned-form classification work without a separate OCR step. ## Python The request is plain JSON over HTTP, so anything that posts JSON works: ```python theme={null} import requests def decide(state, questions, model="Cloudflare/clef"): r = requests.post( "https://api.deepinfra.com/v1/decisions", headers={"Authorization": "Bearer YOUR_DEEPINFRA_API_KEY"}, json={"model": model, "state": state, "questions": questions}, timeout=60, ) r.raise_for_status() return r.json()["answers"] answers = decide( state={"ticket": "Payouts failing for 3 days", "account_age_days": 812}, questions={ "urgent": {"type": "noul", "instructions": "Does this convey urgency?"}, "team": { "type": "choice", "instructions": "Which team should handle this?", "criteria": {"billing": "Payments", "technical": "Bugs"}, }, }, ) if answers["urgent"]["noul"] > 0.8 and answers["team"]["confidence"] > 0.7: route(answers["team"]["choice"]) else: escalate_to_human() ``` ## Acting on confidence The distribution is the point, so use it. The usual production pattern is to act automatically above a confidence threshold and route everything else to a human or a stronger model: ```python theme={null} def route_or_escalate(answer, auto_threshold=0.85, margin=0.15): top = answer["probabilities"][answer["choice"]] runner_up = max((p for k, p in answer["probabilities"].items() if k != answer["choice"]), default=0.0) if top >= auto_threshold and top - runner_up >= margin: return ("auto", answer["choice"]) return ("review", answer) ``` The margin matters as much as the threshold. A 0.5/0.45 split never looks confident under a healthy margin, but it would pass a bare 0.4 threshold while being effectively a coin flip. ## One request, many questions Every question is scored against the same state in a single call. Prefer one request with many questions over many requests with one question each: it is one round trip, and the answers come back together. ## Pricing Clef bills on input tokens at **\$0.24 per 1M tokens**. Output tokens are free. `usage.input_tokens` in the reply is what you are billed for. ## What Clef is not for Clef does not generate text. If you need prose — a summary, a reply draft, a rationale — use a chat model. A common and effective pairing is to let Clef make the decision and let a chat model explain it, rather than asking a chat model to do both and hoping the decision is well-formed. Clef also does not explain itself. It gives you the distribution, not a reason. Building an audit trail on top of `probabilities` is straightforward; building one on top of a generated explanation invites confabulation. ## Related > **[Clef on the model page](https://deepinfra.com/Cloudflare/clef)** > > Try a request in the browser without writing code. > **[Clef-Flash](https://huggingface.co/Cloudflare/clef-flash)** > > The smaller, faster variant of the same model. # CrewAI Source: https://docs.deepinfra.com/integrations/crewai Run CrewAI agents on DeepInfra models through CrewAI's built-in DeepInfra provider, embed knowledge with DeepInfra embedding models, and execute agent code in DeepInfra Sandboxes. [CrewAI](https://docs.crewai.com/) is a Python framework for building crews of role-playing agents that collaborate on tasks. DeepInfra is one of its built-in providers: prefix a model id with `deepinfra/` and CrewAI sends the requests through its native OpenAI client, reading your key from `DEEPINFRA_API_KEY`, with no base URL to configure and no LiteLLM dependency. Tool calling, streaming, JSON-schema structured outputs and token usage reporting all work over this path, with any [chat model from our catalog](https://deepinfra.com/models/text-generation). ## Installation ```bash theme={null} pip install "crewai>=1.15.27" ``` The DeepInfra provider shipped in `crewai` 1.15.27 and needs no extras. Get your API key from the [Dashboard](https://deepinfra.com/dash/api_keys) and export it as `DEEPINFRA_API_KEY`. It is the same key you use for inference, embeddings and Sandboxes. ```bash theme={null} export DEEPINFRA_API_KEY="" ``` ## Configuration Pass the model id with the `deepinfra/` prefix. DeepInfra ids already have the form `org/model`, so a full reference has three segments; everything after the prefix reaches the API unchanged, so use the exact id from the catalog. ```python theme={null} from crewai import LLM llm = LLM(model="deepinfra/deepseek-ai/DeepSeek-V4-Flash-0731", temperature=0.2) ``` Pass the `llm` object to every `Agent` that should run on DeepInfra; different agents in one crew can use different models. Because the whole configuration fits in the model string, it also works wherever CrewAI takes a model name instead of an `LLM` object, including `agents.yaml`: ```yaml theme={null} researcher: role: Researcher goal: Find and summarise the facts the task asks for backstory: A careful analyst who cites what the tools return. llm: deepinfra/deepseek-ai/DeepSeek-V4-Flash-0731 ``` To route requests through a proxy or gateway, set `DEEPINFRA_BASE_URL`; passing `base_url` and `api_key` to `LLM` directly works as well. ## Build a crew The example below gives an agent a tool and runs a single task. The agent calls the tool through OpenAI-style function calling, receives the result, and writes the answer. ```python theme={null} from crewai import Agent, Crew, LLM, Task from crewai.tools import tool llm = LLM(model="deepinfra/deepseek-ai/DeepSeek-V4-Flash-0731", temperature=0.2) @tool("Exchange rate") def exchange_rate(pair: str) -> str: """Return the current exchange rate for a currency pair such as 'EUR/USD'.""" rates = {"EUR/USD": 1.0842, "GBP/USD": 1.2650, "USD/JPY": 156.30} return f"{pair}: {rates.get(pair.upper(), 'unknown pair')}" analyst = Agent( role="FX analyst", goal="Answer currency questions with numbers from the exchange rate tool", backstory="You never guess a rate; you always call the tool first.", llm=llm, tools=[exchange_rate], ) task = Task( description="How many US dollars is 250 euros? Use the tool for the rate and show the arithmetic.", expected_output="One sentence with the rate used and the converted amount.", agent=analyst, ) result = Crew(agents=[analyst], tasks=[task]).kickoff() print(result.raw) print(result.token_usage) ``` Example output: ```text theme={null} 250 euros × 1.0842 = 271.05 US dollars, using the EUR/USD rate of 1.0842. total_tokens=906 prompt_tokens=830 cached_prompt_tokens=256 completion_tokens=76 reasoning_tokens=0 cache_creation_tokens=0 successful_requests=2 ``` Token counts come from DeepInfra's `usage` object on each response, so `result.token_usage` matches what you are billed for. `cached_prompt_tokens` shows DeepInfra's prompt caching kicking in on the second request of the run. ## Knowledge and memory with DeepInfra embeddings CrewAI's knowledge sources and memory store text as embeddings. Its `openai` embedder accepts an `api_base`, so point it at DeepInfra and pick an [embedding model](https://deepinfra.com/models/embeddings). The same `embedder` dictionary works on `Crew(embedder=...)`, on `Agent(embedder=...)` and in memory configuration. ```python theme={null} import os from crewai import Agent, Crew, LLM, Task from crewai.knowledge.source.string_knowledge_source import StringKnowledgeSource llm = LLM(model="deepinfra/deepseek-ai/DeepSeek-V4-Flash-0731") facts = StringKnowledgeSource( content=( "Orbital Coffee opened in 2019 in Sofia. Its house blend is called Perihelion. " "The roastery runs on a 15 kg Loring roaster and ships wholesale on Tuesdays." ) ) embedder = { "provider": "openai", "config": { "api_key": os.environ["DEEPINFRA_API_KEY"], "api_base": "https://api.deepinfra.com/v1/openai", "model_name": "BAAI/bge-m3", }, } agent = Agent( role="Support agent", goal="Answer questions about Orbital Coffee from the knowledge base", backstory="You answer only from what the knowledge base says.", llm=llm, ) task = Task( description="What is Orbital Coffee's house blend called, and on which day does wholesale ship?", expected_output="A one-sentence answer.", agent=agent, ) crew = Crew(agents=[agent], tasks=[task], knowledge_sources=[facts], embedder=embedder) print(crew.kickoff().raw) ``` Example output: ```text theme={null} Orbital Coffee's house blend is called Perihelion, and wholesale ships on Tuesdays. ``` | Embedding model | Dimensions | Max input | Notes | | - | - | - | - | | `BAAI/bge-m3` | 1024 | 8,192 tokens | Multilingual, a solid default for knowledge bases | | `Qwen/Qwen3-Embedding-4B` | 2560 | 32,768 tokens | Higher quality, larger vectors | | `Qwen/Qwen3-Embedding-8B` | 4096 | 32,768 tokens | Highest quality in the catalog | | `intfloat/multilingual-e5-large` | 1024 | 512 tokens | Small and fast for short passages | Keep the same embedding model for the life of a knowledge base or memory store; vectors from different models are not comparable. ## Run agent code in a Sandbox Agents that write code need somewhere safe to run it. [DeepInfra Sandboxes](/sandboxes/overview) are isolated microVMs created with one API call, and the `deepinfra` Python SDK wraps them. The tool below runs Python in a fresh sandbox per call and returns the output to the agent. ```bash theme={null} pip install deepinfra ``` ```python theme={null} from crewai.tools import BaseTool from deepinfra import Sandbox from pydantic import BaseModel, Field class RunPythonInput(BaseModel): code: str = Field(..., description="Python source to execute.") class DeepInfraSandboxPython(BaseTool): name: str = "Run Python in a DeepInfra sandbox" description: str = ( "Executes Python code in an isolated DeepInfra Sandbox and returns " "stdout, stderr and the exit code." ) args_schema: type[BaseModel] = RunPythonInput plan: str = "nano" def _run(self, code: str) -> str: with Sandbox.create(plan=self.plan, timeout="10m", tags={"framework": "crewai"}) as sb: result = sb.run_python(code) return f"exit={result.returncode}\nstdout:\n{result.stdout}\nstderr:\n{result.stderr}" ``` Add `DeepInfraSandboxPython()` to an agent's `tools` list like any other tool. A nano sandbox is running within about half a second of the create call, and the context manager terminates it when the call returns, so you pay only for the seconds the code runs. Files the agent needs to keep between calls belong in `/workspace`; to carry state across calls, create one `Sandbox` in the tool's constructor and reuse it instead of the context manager. Note the account limit of 25 active sandboxes when many agents run in parallel. ## Choosing a model The agent loop depends on reliable tool calling. The models below are verified with CrewAI's function-calling path on DeepInfra; prefix any of them with `deepinfra/` when you pass it to `LLM`. Context lengths are from the [model catalog](https://api.deepinfra.com/v1/openai/models?filter=with_meta\&sort_by=crewai), which is public and also carries current pricing. | Model | Context | Good for | | - | - | - | | `deepseek-ai/DeepSeek-V4-Flash-0731` | 1M | Default in these examples: very low cost, prompt caching, long documents | | `Qwen/Qwen3.5-27B` | 262K | Fast and inexpensive, with vision and reasoning | | `zai-org/GLM-5.3` | 1M | Strong reasoning on complex multi-step tasks | | `meta-llama/Llama-4-Scout-17B-16E-Instruct` | 327K | Llama ecosystem, vision, wide language coverage | | `moonshotai/Kimi-K3` | 1M | Complex coding and long-horizon agent work, at a premium price | | `google/gemma-4-31B-it-turbo` | 262K | Cheap, capable generalist | > [!NOTE] > > To compare other models, list the catalog with context windows and pricing via [`/v1/openai/models?filter=with_meta&sort_by=crewai`](https://api.deepinfra.com/v1/openai/models?filter=with_meta\&sort_by=crewai). Each entry's `metadata` block has `context_length`, `max_tokens`, and per-million-token `pricing`; the `id` is what you pass as `model`, after the `deepinfra/` prefix. > > ```bash theme={null} > curl -s "https://api.deepinfra.com/v1/openai/models?filter=with_meta&sort_by=crewai" \ > | jq '.data[] | select(.metadata.tags | index("chat")) | {id, context_length: .metadata.context_length, pricing: .metadata.pricing}' > ``` ## Tips * **Context window.** CrewAI looks model ids up in a table of OpenAI models to size its context management, and assumes 8,192 tokens for ids it does not know, DeepInfra models included. With the default `respect_context_window=True`, an agent on a DeepInfra model starts summarising its conversation at roughly 7,000 tokens even when the model supports far more. Set `respect_context_window=False` on agents that need the full window; CrewAI then sends the full history and surfaces the provider error if a prompt ever exceeds the model's real limit. * **Model id format.** The part after `deepinfra/` must be `org/model`. A reference without the organisation, such as `deepinfra/DeepSeek-V4-Flash-0731`, raises a `ValueError` when the `LLM` is constructed rather than failing on the first request. * **Older CrewAI versions.** Before `crewai` 1.15.27 the `deepinfra/` prefix routed through LiteLLM and needed `pip install "crewai[litellm]"`. If you cannot upgrade, create the LLM with `custom_openai=True`, `base_url="https://api.deepinfra.com/v1/openai"` and `api_key=os.environ["DEEPINFRA_API_KEY"]` instead; that form uses CrewAI's native OpenAI client and still works on current versions. On current versions, `is_litellm=True` keeps the LiteLLM path for those who need it. * **Structured outputs.** Pass a Pydantic model as `response_format` on the `LLM` or as `output_pydantic` on a `Task`. CrewAI sends it as a strict JSON schema, which DeepInfra honours. * **Streaming.** Set `stream=True` on the `LLM` to receive tokens as they are generated. Usage is still reported on the final chunk. ## Learn more > **[CrewAI LLM docs](https://docs.crewai.com/en/concepts/llms)** > > Every `LLM` parameter, the DeepInfra provider section, and the YAML configuration format. > **[Chat Completions](/chat/overview)** > > DeepInfra's OpenAI-compatible endpoint, including [Tool Calling](/chat/tool-calling) and [Structured Outputs](/chat/structured-outputs). > **[Embeddings](/apis/embeddings)** > > Embedding models for knowledge bases and memory. > **[Sandboxes](/sandboxes/overview)** > > Isolated microVMs for running agent-written code. > [!NOTE] > > Tested with `crewai` 1.15.27 and `deepinfra` 0.3.0. # Hermes Agent Source: https://docs.deepinfra.com/integrations/hermes-agent Use DeepInfra models with Hermes Agent through its built-in DeepInfra provider. > [!NOTE] > > Don't want to host it yourself? DeepInfra can run Hermes for you — see [Hosted Agents: Hermes-Agent](/agents/hermes-agent). [Hermes Agent](https://github.com/NousResearch/hermes-agent) is a self-hosted, self-improving autonomous agent by Nous Research. DeepInfra is a built-in Hermes provider: set `provider: deepinfra`, export your key as `DEEPINFRA_API_KEY`, and pick any [LLM from our catalog](https://deepinfra.com/models/text-generation). Hermes reads the model list, context lengths and pricing live from the DeepInfra catalog, so there is no base URL or context length to configure. ## Configure `~/.hermes/config.yaml` ```yaml theme={null} model: default: deepseek-ai/DeepSeek-V4-Flash provider: deepinfra ``` Get your API key from the [Dashboard](https://deepinfra.com/dash/api_keys) and export it, or put the same line in `~/.hermes/.env`: ```bash theme={null} export DEEPINFRA_API_KEY="" ``` > [!NOTE] > > * The DeepInfra model id passes through verbatim in `default` — no reformatting needed. > * Reasoning is controlled through DeepInfra's `reasoning_effort` field, so `agent.reasoning_effort`, `/reasoning ` and `--reasoning` work in both directions: an effort turns thinking on for models that default off (DeepSeek-V4.x), and `/reasoning none` turns it off for models that default on. > * Hermes needs roughly 64k of context for agent functionality. Check a model's window and pricing via [`/v1/openai/models?filter=with_meta&sort_by=hermes`](https://api.deepinfra.com/v1/openai/models?filter=with_meta\&sort_by=hermes). > * On a Hermes release older than the built-in provider, use `provider: custom` with `base_url: https://api.deepinfra.com/v1/openai`, `api_key: ${DEEPINFRA_API_KEY}` and an explicit `context_length`. ## Run it ```bash theme={null} hermes ``` To try a model without editing the YAML, pass the provider and model on the command line: ```bash theme={null} hermes chat --provider deepinfra -m deepseek-ai/DeepSeek-V4-Flash -q "Say hello in one word." ``` ## Learn more > **[AI Providers](https://hermes-agent.nousresearch.com/docs/integrations/providers#deepinfra)** > > Hermes' provider overview, including the DeepInfra section. > **[Configuration](https://hermes-agent.nousresearch.com/docs/user-guide/configuration)** > > The full `config.yaml` reference. > **[Chat Completions](/chat/overview)** > > DeepInfra's OpenAI-compatible API. > **[NVIDIA OpenShell](/integrations/openshell)** > > Run Hermes in a sandbox whose only allowed destination is DeepInfra. > [!NOTE] > > Tested with Hermes Agent v0.21.5 (2026.9.24). # LangChain Source: https://docs.deepinfra.com/integrations/langchain Use DeepInfra models with LangChain for LLM-powered applications. [LangChain](https://python.langchain.com/) is a framework for building applications powered by language models. DeepInfra integrates with LangChain via official adapters for LLMs, chat models, and embeddings. ## Available adapters * [Chat adapter](https://python.langchain.com/docs/integrations/chat/deepinfra) — for chat-based LLMs * [LLM adapter](https://python.langchain.com/docs/integrations/llms/deepinfra) — for text generation LLMs * [Embeddings adapter](https://python.langchain.com/docs/integrations/text_embedding/deepinfra) — for embedding models ## Installation ```bash theme={null} pip install langchain langchain-community ``` Set your API token: ```python theme={null} import os os.environ["DEEPINFRA_API_TOKEN"] = "" ``` ## LLM examples ```python theme={null} import os from langchain_community.llms import DeepInfra from langchain.prompts import PromptTemplate from langchain.chains import LLMChain os.environ["DEEPINFRA_API_TOKEN"] = "" llm = DeepInfra(model_id="deepseek-ai/DeepSeek-V4-Flash-0731") llm.model_kwargs = { "temperature": 0.7, "repetition_penalty": 1.2, "max_new_tokens": 250, "top_p": 0.9, } # Basic inference print(llm.invoke("Who let the dogs out?")) # Streaming inference for chunk in llm.stream("Who let the dogs out?"): print(chunk) # Chain with prompt template template = """Question: {question} Answer: Let's think step by step.""" prompt = PromptTemplate(template=template, input_variables=["question"]) llm_chain = prompt | llm print(llm_chain.invoke("Can penguins reach the North pole?")) ``` ## Chat examples ```python theme={null} import os from langchain_community.chat_models import ChatDeepInfra from langchain_core.messages import HumanMessage from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler os.environ["DEEPINFRA_API_TOKEN"] = "" messages = [ HumanMessage(content="Translate this sentence from English to French. I love programming.") ] # Synchronous chat = ChatDeepInfra(model="deepseek-ai/DeepSeek-V4-Flash-0731") print(chat.invoke(messages)) # Async async def async_example(): chat = ChatDeepInfra(model="deepseek-ai/DeepSeek-V4-Flash-0731") await chat.agenerate([messages]) # Streaming chat_stream = ChatDeepInfra( streaming=True, verbose=True, callbacks=[StreamingStdOutCallbackHandler()], ) print(chat_stream.invoke(messages)) ``` ## Embeddings ```python theme={null} import os from langchain_community.embeddings import DeepInfraEmbeddings os.environ["DEEPINFRA_API_TOKEN"] = "" embeddings = DeepInfraEmbeddings( model_id="Qwen/Qwen3-Embedding-8B", query_instruction="", embed_instruction="", ) docs = ["Dog is not a cat", "Beta is the second letter of Greek alphabet"] document_result = embeddings.embed_documents(docs) print(document_result) ``` # LlamaIndex Source: https://docs.deepinfra.com/integrations/llama-index Use DeepInfra LLMs and embeddings with LlamaIndex. [LlamaIndex](https://www.llamaindex.ai) is a popular data framework for LLM applications — RAG pipelines, agents, and more. DeepInfra integrates with LlamaIndex for both LLMs and embeddings. ## LLMs ### Installation ```bash theme={null} pip install llama-index-llms-deepinfra ``` ### Initialization ```python theme={null} import os from llama_index.llms.deepinfra import DeepInfraLLM import asyncio llm = DeepInfraLLM( model="deepseek-ai/DeepSeek-V4-Flash-0731", api_key=os.environ["DEEPINFRA_API_KEY"], temperature=0.5, max_tokens=50, additional_kwargs={"top_p": 0.9}, ) ``` ### Synchronous ```python theme={null} # Complete response = llm.complete("Hello World!") print(response.text) # Stream complete for completion in llm.stream_complete("Once upon a time"): print(completion.delta, end="") # Chat from llama_index.core.base.llms.types import ChatMessage messages = [ChatMessage(role="user", content="Tell me a joke.")] chat_response = llm.chat(messages) print(chat_response.message.content) # Stream chat messages = [ ChatMessage(role="system", content="You are a helpful assistant."), ChatMessage(role="user", content="Tell me a story."), ] for chat_response in llm.stream_chat(messages): print(chat_response.delta, end="") ``` ### Asynchronous ```python theme={null} # Async complete async def async_complete(): response = await llm.acomplete("Hello Async World!") print(response.text) asyncio.run(async_complete()) # Async stream complete async def async_stream_complete(): response = await llm.astream_complete("Once upon an async time") async for completion in response: print(completion.delta, end="") asyncio.run(async_stream_complete()) # Async chat async def async_chat(): messages = [ChatMessage(role="user", content="Tell me an async joke.")] chat_response = await llm.achat(messages) print(chat_response.message.content) asyncio.run(async_chat()) # Async stream chat async def async_stream_chat(): messages = [ ChatMessage(role="system", content="You are a helpful assistant."), ChatMessage(role="user", content="Tell me an async story."), ] response = await llm.astream_chat(messages) async for chat_response in response: print(chat_response.delta, end="") asyncio.run(async_stream_chat()) ``` ## Embeddings ### Installation ```bash theme={null} pip install llama-index llama-index-embeddings-deepinfra ``` ### Initialization ```python theme={null} import os from dotenv import load_dotenv, find_dotenv from llama_index.embeddings.deepinfra import DeepInfraEmbeddingModel _ = load_dotenv(find_dotenv()) model = DeepInfraEmbeddingModel( model_id="Qwen/Qwen3-Embedding-8B", api_token=os.environ["DEEPINFRA_API_KEY"], normalize=True, text_prefix="text: ", query_prefix="query: ", ) ``` ### Synchronous requests ```python theme={null} # Single text response = model.get_text_embedding("hello world") print(response) # Batch texts = ["hello world", "goodbye world"] response_batch = model.get_text_embedding_batch(texts) print(response_batch) # Query query_response = model.get_query_embedding("hello world") print(query_response) ``` ### Asynchronous requests ```python theme={null} import asyncio async def main(): async_response = await model.aget_text_embedding("hello world") print(async_response) asyncio.run(main()) ``` # Microsoft Agent Framework Source: https://docs.deepinfra.com/integrations/microsoft-agent-framework Run Microsoft Agent Framework agents on DeepInfra models in Python and .NET through the OpenAI-compatible API. [Microsoft Agent Framework](https://learn.microsoft.com/agent-framework/) is Microsoft's open-source SDK for building AI agents and multi-agent workflows in Python and .NET. Its OpenAI clients work with any OpenAI-compatible endpoint, so you can point them at DeepInfra and use any [LLM from our catalog](https://deepinfra.com/models/text-generation). There is no DeepInfra-specific package; you use the standard OpenAI Chat Completions client with a custom base URL. ## Installation ```bash Python theme={null} pip install agent-framework-openai ``` ```bash .NET theme={null} dotnet add package Microsoft.Agents.AI.OpenAI ``` Get your API key from the [Dashboard](https://deepinfra.com/dash/api_keys) and export it as `DEEPINFRA_API_KEY`. ## Configuration Create the Chat Completions client with DeepInfra's base URL and your API key, then build an agent from it. Everything else works as described in the [Agent Framework docs](https://learn.microsoft.com/agent-framework/user-guide/overview). ```python Python theme={null} import asyncio import os from agent_framework import Agent from agent_framework.openai import OpenAIChatCompletionClient async def main() -> None: client = OpenAIChatCompletionClient( model="deepseek-ai/DeepSeek-V4-Flash-0731", api_key=os.environ["DEEPINFRA_API_KEY"], base_url="https://api.deepinfra.com/v1/openai", ) agent = Agent(client=client, instructions="You are a concise, helpful assistant.") result = await agent.run("Explain what a GPU inference provider does in two sentences.") print(result.text) async for chunk in agent.run("Write a haiku about fast inference.", stream=True): if chunk.text: print(chunk.text, end="", flush=True) print() if __name__ == "__main__": asyncio.run(main()) ``` ```csharp .NET theme={null} using System.ClientModel; using Microsoft.Agents.AI; using OpenAI; var apiKey = Environment.GetEnvironmentVariable("DEEPINFRA_API_KEY") ?? throw new InvalidOperationException("DEEPINFRA_API_KEY is not set."); AIAgent agent = new OpenAIClient( new ApiKeyCredential(apiKey), new OpenAIClientOptions { Endpoint = new Uri("https://api.deepinfra.com/v1/openai") }) .GetChatClient("deepseek-ai/DeepSeek-V4-Flash-0731") .AsAIAgent(instructions: "You are a concise, helpful assistant."); Console.WriteLine(await agent.RunAsync("Explain what a GPU inference provider does in two sentences.")); await foreach (AgentResponseUpdate update in agent.RunStreamingAsync("Write a haiku about fast inference.")) { Console.Write(update); } Console.WriteLine(); ``` > [!WARNING] > > Use the Chat Completions client only: `OpenAIChatCompletionClient` in Python, `GetChatClient()` in .NET. `OpenAIChatClient` (Python) and `GetResponsesClient()` (.NET) target the OpenAI Responses API, which DeepInfra does not serve. Function tools, sessions, and structured output need no DeepInfra-specific setup. `response_format` with `json_schema` and `strict: true` is supported on [many models](https://deepinfra.com/models?q=json); see [Structured Outputs](/chat/structured-outputs). > [!NOTE] > > To pick a model, list the catalog with context windows and pricing via [`/v1/openai/models?filter=with_meta&sort_by=maf`](https://api.deepinfra.com/v1/openai/models?filter=with_meta\&sort_by=maf). Each entry's `metadata` block has `context_length`, `max_tokens`, and per-million-token `pricing`; the `id` is what you pass as `model`. > > ```bash theme={null} > curl -s "https://api.deepinfra.com/v1/openai/models?filter=with_meta&sort_by=maf" \ > | jq '.data[] | select(.metadata.tags | index("chat")) | {id, context_length: .metadata.context_length, pricing: .metadata.pricing}' > ``` ### Environment variables The Python clients fall back to `OPENAI_API_KEY`, `OPENAI_BASE_URL`, and `OPENAI_CHAT_COMPLETION_MODEL` when constructor arguments are omitted, so you can also configure them that way: ```bash theme={null} export OPENAI_API_KEY="" export OPENAI_BASE_URL="https://api.deepinfra.com/v1/openai" export OPENAI_CHAT_COMPLETION_MODEL="deepseek-ai/DeepSeek-V4-Flash-0731" ``` > [!NOTE] > > If a real OpenAI key is already set in `OPENAI_API_KEY`, the client will send it to DeepInfra. Passing `api_key` and `base_url` explicitly, as above, avoids that. ## Embeddings `OpenAIEmbeddingClient` takes the same arguments. Use any [embedding model](https://deepinfra.com/models/embeddings) from the catalog. ```python theme={null} from agent_framework.openai import OpenAIEmbeddingClient client = OpenAIEmbeddingClient( model="Qwen/Qwen3-Embedding-8B", api_key=os.environ["DEEPINFRA_API_KEY"], base_url="https://api.deepinfra.com/v1/openai", ) result = await client.get_embeddings(["DeepInfra serves open models.", "Agents call tools."]) for embedding in result: print(embedding.dimensions, embedding.vector[:5]) ``` ## Reasoning models DeepInfra returns the chain-of-thought of [reasoning models](/chat/reasoning) in a `reasoning_content` field, in both the final message and each streaming delta. Agent Framework does not read that field natively, but its `response_parser` hook lets you surface it as `text_reasoning` content. Set `reasoning_effort` through `options`; any key there is forwarded to the OpenAI SDK, and DeepInfra-only parameters can go in `options={"extra_body": {...}}`. ```python theme={null} from typing import Any from agent_framework import Agent, Content from agent_framework.openai import OpenAIChatCompletionClient def deepinfra_reasoning_parser(message: Any, contents: list[Content]) -> list[Content]: reasoning = getattr(message, "reasoning_content", None) if isinstance(reasoning, str) and reasoning: return [Content.from_text_reasoning(text=reasoning), *contents] return contents agent = Agent( client=OpenAIChatCompletionClient( model="deepseek-ai/DeepSeek-V4-Flash-0731", api_key=os.environ["DEEPINFRA_API_KEY"], base_url="https://api.deepinfra.com/v1/openai", response_parser=deepinfra_reasoning_parser, ), instructions="You are a helpful assistant.", ) result = await agent.run("What is 17 * 23?", options={"reasoning_effort": "high"}) for message in result.messages: for content in message.contents: if content.type == "text_reasoning": print("[reasoning]", content.text) print("[answer]", result.text) ``` DeepInfra does not require reasoning to be echoed back on later turns, so no `message_preparer` is needed; sessions and tool-call loops work with this parser as-is. ## Learn more > **[Agent Framework docs](https://learn.microsoft.com/agent-framework/user-guide/overview)** > > Agents, tools, sessions, structured output, workflows, and observability. > **[OpenAI-compatible endpoints](https://github.com/microsoft/agent-framework/blob/main/python/packages/openai/AGENTS.md)** > > Agent Framework's guidance on adapting its OpenAI client to other providers. > [!NOTE] > > Agent Framework is under active development and API names change between releases; see the [Python changelog](https://github.com/microsoft/agent-framework/blob/main/python/CHANGELOG.md). Tested with `agent-framework-openai` 1.14.2, `openai` 3.11.0, and `Microsoft.Agents.AI.OpenAI` 1.20.0. # NVIDIA OpenShell Source: https://docs.deepinfra.com/integrations/openshell Run agents in an NVIDIA OpenShell sandbox whose only allowed destination is DeepInfra inference. [NVIDIA OpenShell](https://docs.nvidia.com/openshell/latest) is an open-source runtime for AI agents. Each agent runs in a sandbox with default-deny networking, a per-binary Layer 7 policy that lists the hosts, methods and paths it may use, and a gateway that keeps API keys outside the sandbox. Every connection is either allowed by a rule you can read or denied and logged. DeepInfra is a first-party provider profile in the OpenShell repository, [`providers/deepinfra.yaml`](https://github.com/NVIDIA/OpenShell/blob/main/providers/deepinfra.yaml). With it attached, the agent's environment holds only an opaque placeholder in `DEEPINFRA_API_KEY`. The gateway swaps in your real key on requests to `api.deepinfra.com:443` and nowhere else, so the credential never enters the sandbox. The agent keeps the whole [model catalog](https://deepinfra.com/models/text-generation) through the OpenAI-compatible API; the rest of the internet stays closed unless you open it. ## Watch the walkthrough The recording below runs this guide end to end in under seven minutes: import the provider profile, start a sandbox, show that the key never enters it, pin the API surface, apply a base policy with one internal endpoint, and run Hermes Agent behind the same boundary. The sections that follow repeat every command with its output, so you can pause the video and copy from the page. ## Quick start You need a Linux host with Docker Engine 28 or later and your user in the `docker` group, OpenShell v0.1.1, and a DeepInfra API key from the [Dashboard](https://deepinfra.com/dash/api_keys). Pin the OpenShell version; commands change between releases. ```bash theme={null} curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | OPENSHELL_VERSION=v0.1.1 sh sudo loginctl enable-linger $USER # keep the gateway running after you log out export DEEPINFRA_API_KEY="" ``` A gateway starts with no profiles, so the first step for any provider is to import one. Four commands give you a sandbox that can reach DeepInfra and nothing else. `--from-existing` reads `DEEPINFRA_API_KEY` from your shell into the gateway's credential store; it is never passed into the sandbox. ```bash theme={null} curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/v0.1.1/providers/deepinfra.yaml -o deepinfra.yaml openshell profile import -f deepinfra.yaml openshell provider create --name deepinfra --type deepinfra --from-existing openshell sandbox create --name di-smoke --from curlimages/curl:8.22.0 --provider deepinfra --detach -- sleep infinity ``` ```text theme={null} Imported 1 provider profile. ✓ Created provider deepinfra Created sandbox: di-smoke ``` The profile allows one endpoint, `api.deepinfra.com:443`, for two client binaries, `/usr/bin/curl` and `/usr/local/bin/curl`. OpenShell's default image has no `curl`, so the sandbox uses the official [`curlimages/curl`](https://hub.docker.com/r/curlimages/curl) image, and `sleep infinity` keeps it alive for the checks below. Run the profile's own smoke test, the public model catalog with metadata in the curated agent order; `jq` runs on your host: ```bash theme={null} openshell sandbox exec -n di-smoke --no-login-shell -- curl -sS "https://api.deepinfra.com/v1/openai/models?filter=with_meta&sort_by=openshell" | jq -r '.data[:3][].id' ``` ```text theme={null} deepseek-ai/DeepSeek-V4-Flash Qwen/Qwen3.5-397B-A17B moonshotai/Kimi-K2.6 ``` > [!NOTE] > > `openshell profile lint -f deepinfra.yaml` checks a profile you edited. `openshell provider list-profiles` and the older `openshell provider profile import -f deepinfra.yaml --global` spelling still work in v0.1.1. Import is create-only; to change an imported profile, export it, edit the file and run `openshell profile update deepinfra -f deepinfra.yaml`. ## Verify the boundary Inside the sandbox, `DEEPINFRA_API_KEY` is a placeholder. A chat completion puts it in the `Authorization` header where the key would go, and the gateway resolves it on the way out because the destination is the one the profile binds the credential to: ```bash theme={null} openshell sandbox exec -n di-smoke --no-login-shell -- sh -c 'echo $DEEPINFRA_API_KEY' openshell sandbox exec -n di-smoke --no-login-shell -- sh -c ' curl -sS https://api.deepinfra.com/v1/openai/chat/completions \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -H "Content-Type: application/json" \ -d @-' <<'EOF' | jq -c '{model, reply: .choices[0].message.content, tokens: .usage.total_tokens}' {"model": "deepseek-ai/DeepSeek-V4-Flash", "messages": [{"role": "user", "content": "Say hello in one word."}]} EOF ``` ```text theme={null} openshell:resolve:env:v3721373755583287707_DEEPINFRA_API_KEY {"model":"deepseek-ai/DeepSeek-V4-Flash","reply":"Hello","tokens":12} ``` Now the negative test. Nothing in the effective policy names `example.com`, so the sandbox refuses the connection before it leaves, and the log records both decisions with the binary and the reason (trimmed to the policy lines here; `openshell logs di-smoke --tail` streams it live): ```bash theme={null} openshell sandbox exec -n di-smoke --no-login-shell -- curl -sS https://example.com openshell logs di-smoke --since 5m --source sandbox ``` ```text theme={null} curl: (7) Failed to connect to example.com:443 after 3 ms: Could not connect to server [1790509026.810] [sandbox] [OCSF ] [ocsf] NET:OPEN [INFO] ALLOWED /usr/bin/curl(0) -> api.deepinfra.com:443 [policy:_provider_deepinfra engine:opa] [1790509026.944] [sandbox] [OCSF ] [ocsf] HTTP:POST [INFO] ALLOWED POST http://api.deepinfra.com:443/v1/openai/chat/completions [policy:_provider_deepinfra engine:l7] [1790509027.341] [sandbox] [OCSF ] [ocsf] NET:REFUSE [MED] DENIED example.com [reason:policy_dns_ineligible] [1790509027.343] [sandbox] [OCSF ] [ocsf] NET:OPEN [MED] DENIED /usr/bin/curl(0) -> example.com:443 [reason:transparent_tcp_policy_denied] ``` OpenShell also drafts a rule proposal from every blocked connection and flushes it to the gateway about ten seconds later. A draft never changes the policy by itself; it waits for you: ```bash theme={null} openshell rule get di-smoke --status pending ``` ```text theme={null} Network Rules: (version 1, 1 chunk) Chunk: 4cc7a5cf-0bb8-4967-8843-19579e60f8f4 Status: pending Rule: allow_example_com_443 Binary: /usr/bin/curl Confidence: 65% Rationale: Allow curl to connect to example.com:443 (HTTPS). Prover: prover: no new findings Candidate: 961168893ad8 Endpoints: example.com:443 [L4] Binaries: /usr/bin/curl Hits: 3 (first seen 2026-09-27 11:37:07, last seen 2026-09-27 11:37:47) ``` Approve with `openshell rule approve di-smoke --chunk-id ` only when that access is intended; otherwise `openshell rule reject di-smoke --chunk-id --reason "..."`. ## Pin the API surface OpenShell v0.1.1 has no local inference endpoint and no model router; the managed inference route was removed before 0.1.0. The agent calls DeepInfra's native URL, `https://api.deepinfra.com/v1/openai`, exactly as it would outside the sandbox. What changes is what it holds: a placeholder instead of the key, and a policy that admits only that host. There is also no per-model allowlist. The profile does not select a model, and L7 rules inspect method, path and query string, not the JSON body where `model` lives. The model is whatever the agent's configuration says. What you can pin at the gateway is the API surface. Export the profile, replace its `access: read-write` preset with explicit rules, and apply the update; running sandboxes reload it within about ten seconds. ```bash theme={null} openshell profile export deepinfra -o yaml > deepinfra.yaml ``` ```yaml theme={null} endpoints: - host: api.deepinfra.com port: 443 protocol: rest enforcement: enforce rules: - allow: { method: GET, path: /v1/openai/models } - allow: { method: POST, path: /v1/openai/chat/completions } ``` ```bash theme={null} openshell profile update deepinfra -f deepinfra.yaml ``` Chat completions still work; any other path on the same host is denied at the request layer, with the rule named in the response: ```bash theme={null} openshell sandbox exec -n di-smoke --no-login-shell -- sh -c ' curl -sS https://api.deepinfra.com/v1/openai/embeddings \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -H "Content-Type: application/json" \ -d @-' <<'EOF' | jq -c '{error, policy, rule, detail}' {"model": "BAAI/bge-m3", "input": "hello"} EOF ``` ```text theme={null} {"error":"policy_denied","policy":"_provider_deepinfra","rule":"POST /v1/openai/embeddings","detail":"POST /v1/openai/embeddings not permitted by policy"} ``` When the agent needs [embeddings](/apis/embeddings), add `- allow: { method: POST, path: /v1/openai/embeddings }` to the list. ## Write the sandbox policy A sandbox enforces its effective policy: the base policy you set plus one layer per attached provider. So far the sandbox ran on OpenShell's restrictive default with the provider layer as its only network rule. This complete base policy states the same contract explicitly, `api.deepinfra.com:443` for `curl` and nothing else. The filesystem and landlock sections repeat what OpenShell applied at startup; a live replacement must keep them unchanged. ```yaml deepinfra-only.yaml theme={null} version: 1 filesystem_policy: include_workdir: true read_only: [/bin, /usr, /lib, /proc, /dev/urandom, /etc, /var/log] read_write: [/tmp, /dev/null] landlock: compatibility: best_effort network_policies: deepinfra_api: endpoints: - host: api.deepinfra.com port: 443 protocol: rest enforcement: enforce rules: - allow: { method: GET, path: /v1/openai/models } - allow: { method: POST, path: /v1/openai/chat/completions } binaries: - path: /usr/bin/curl ``` The rule mirrors the narrowed profile. With the stock profile, write `access: read-write` in place of `rules`, but do not mix the two: OpenShell unions overlapping rules, so a `read-write` base rule would reopen the paths the profile closed. Network rules reload live: ```bash theme={null} openshell policy set di-smoke --policy deepinfra-only.yaml --wait ``` ```text theme={null} ✓ Policy version 2 submitted (hash: afd3d673119e) ✓ Policy version 2 loaded (active version: 2) ``` `openshell policy get di-smoke --full` now lists both `deepinfra_api` and `_provider_deepinfra`; the chat completion still succeeds and `example.com` is still refused. ### Allow one internal endpoint The common production variant is DeepInfra plus one approved internal service, with the open internet still denied. Add a second rule to the same file and apply it again. The placeholder host `docs.internal.example` stands in for yours; `read-only` admits `GET`, `HEAD` and `OPTIONS` only. ```yaml deepinfra-plus-internal.yaml theme={null} internal_docs: endpoints: - host: docs.internal.example port: 443 protocol: rest enforcement: enforce access: read-only binaries: - path: /usr/bin/curl ``` ```bash theme={null} openshell policy set di-smoke --policy deepinfra-plus-internal.yaml --wait openshell sandbox exec -n di-smoke --no-login-shell -- curl -sS https://api.deepinfra.com/v1/openai/models | jq '.data | length' openshell sandbox exec -n di-smoke --no-login-shell -- curl -sS https://docs.internal.example/ openshell sandbox exec -n di-smoke --no-login-shell -- curl -sS https://example.com openshell logs di-smoke --since 2m --source sandbox ``` ```text theme={null} 187 curl: (6) Could not resolve host: docs.internal.example curl: (7) Failed to connect to example.com:443 after 3 ms: Could not connect to server [1790509066.564] [sandbox] [OCSF ] [ocsf] HTTP:GET [INFO] ALLOWED GET http://api.deepinfra.com:443/v1/openai/models [policy:_provider_deepinfra engine:l7] [1790509067.272] [sandbox] [OCSF ] [ocsf] NET:REFUSE [LOW] DENIED docs.internal.example [reason:policy_dns_upstream_nxdomain] [1790509067.357] [sandbox] [OCSF ] [ocsf] NET:OPEN [MED] DENIED /usr/bin/curl(0) -> example.com:443 [reason:transparent_tcp_policy_denied] ``` The three reasons tell the story: DeepInfra is allowed, the internal name is admitted by policy and fails only because the placeholder does not resolve upstream, and `example.com` is refused by policy. An internal name that resolves to a private address works with an exact hostname; a wildcard such as `*.internal.example` also needs `allowed_ips`, see OpenShell's [network rules](https://docs.nvidia.com/openshell/latest/how-it-works/policies/network-rules#restrict-destination-addresses). ## Run Hermes Agent in the sandbox `curl` proves the boundary; an agent is what you run behind it. [Hermes Agent](/integrations/hermes-agent) has DeepInfra built in, so its configuration is two lines, and the placeholder the sandbox holds in `DEEPINFRA_API_KEY` is exactly what the built-in provider reads. Two image facts need handling. Hermes is Python, so the process that opens connections is the interpreter, `/usr/bin/python3.13` in this image, and it must be in the profile's `binaries`. And OpenShell requires the image's working directory to be writable by the sandbox user and not an image volume, sets `HOME` to that directory, and does not pass the image's environment to processes, so a two-line derivative gives Hermes a home at `/workspace/.hermes`: ```dockerfile Dockerfile theme={null} FROM nousresearch/hermes-agent:v2026.9.24 RUN mkdir -p /workspace/.hermes && chown -R hermes:hermes /workspace USER hermes WORKDIR /workspace ``` ```bash theme={null} docker build -t hermes-openshell:local . ``` Add the interpreter to `binaries` in your exported profile and apply it: ```yaml theme={null} binaries: - /usr/bin/curl - /usr/local/bin/curl - /usr/bin/python3.13 ``` ```bash theme={null} openshell profile update deepinfra -f deepinfra.yaml ``` The policy adds Hermes's install tree read-only; `include_workdir` makes `/workspace` writable. ```yaml hermes-policy.yaml theme={null} version: 1 filesystem_policy: include_workdir: true read_only: [/bin, /usr, /lib, /proc, /dev/urandom, /etc, /var/log, /opt/hermes] read_write: [/tmp, /dev/null] landlock: compatibility: best_effort ``` ```yaml config.yaml theme={null} model: default: deepseek-ai/DeepSeek-V4-Flash provider: deepinfra ``` Create the sandbox, upload the configuration into Hermes's home, and ask a one-shot question: ```bash theme={null} openshell sandbox create --name hermes --from hermes-openshell:local --provider deepinfra --policy hermes-policy.yaml --detach -- sleep infinity openshell sandbox upload hermes ./config.yaml .hermes/config.yaml openshell sandbox exec -n hermes --no-login-shell -- hermes chat -Q -q "In one sentence, which model are you and who serves it?" openshell logs hermes --since 2m --source sandbox ``` ```text theme={null} ⚠ tirith security scanner enabled but not available — command scanning will use pattern matching only session_id: 20260927_120545_c0cb65 I'm DeepSeek-V4-Flash, served by DeepInfra. [1790510745.387] [sandbox] [OCSF ] [ocsf] NET:OPEN [MED] DENIED /usr/bin/python3.13(0) -> models.dev:443 [reason:transparent_tcp_policy_denied] [1790510748.091] [sandbox] [OCSF ] [ocsf] NET:OPEN [MED] DENIED /usr/bin/python3.13(0) -> github.com:443 [reason:transparent_tcp_policy_denied] [1790510750.746] [sandbox] [OCSF ] [ocsf] NET:OPEN [MED] DENIED /usr/local/bin/uv(0) -> pypi.org:443 [reason:transparent_tcp_policy_denied] [1790510761.163] [sandbox] [OCSF ] [ocsf] NET:OPEN [INFO] ALLOWED /usr/bin/python3.13(0) -> api.deepinfra.com:443 [policy:_provider_deepinfra engine:opa] [1790510763.770] [sandbox] [OCSF ] [ocsf] HTTP:POST [INFO] ALLOWED POST http://api.deepinfra.com:443/v1/openai/chat/completions [policy:_provider_deepinfra engine:l7] ``` The log is the point: Hermes reached DeepInfra through the provider rule, and everything else it tried on startup, model metadata from `models.dev`, GitHub, and its optional `tirith` command scanner from PyPI, was denied and recorded. For an interactive session, replace `sleep infinity` with `hermes` and drop `--detach`. > [!NOTE] > > The same pattern works for [Pi](/integrations/pi): build the image from OpenShell's [Run Pi with OpenRouter](https://docs.nvidia.com/openshell/v0.1.1/tutorials/run-pi-with-openrouter) tutorial, list `/usr/local/bin/node` in `binaries`, and use the provider block from our Pi page. > > [NemoClaw](https://docs.nvidia.com/nemoclaw/latest), NVIDIA's OpenClaw, Hermes and Deep Agents stack on OpenShell, routes inference through its own managed provider and rejects a direct `deepinfra` provider in `openclaw.json`. Choose the provider during NemoClaw onboarding, as described in its [provider guide](https://docs.nvidia.com/nemoclaw/user-guide/openclaw/inference/learn-and-choose/choose-inference-provider), instead of importing this profile there. ## What you configure and what DeepInfra provides | You configure | DeepInfra provides | | - | - | | The OpenShell host and gateway (Docker driver in this guide) | The maintained `deepinfra` provider profile in the OpenShell repository | | A provider holding your `DEEPINFRA_API_KEY` in the gateway's credential store | OpenAI-compatible chat completions and embeddings on `api.deepinfra.com` | | The sandbox policy: which binaries reach which hosts, methods and paths | The [model catalog](https://deepinfra.com/models) with context lengths and pricing | | The model in the agent's configuration | API keys, usage metering and billing | ## Choosing a model Any [chat model](https://deepinfra.com/models/text-generation) works through the gateway; the request is an ordinary chat completion. The models below were each run through the sandbox for this guide. Context lengths come from the [catalog](https://api.deepinfra.com/v1/openai/models?filter=with_meta\&sort_by=openshell), which is public and also carries current pricing. | Model | Context | Good for | | - | - | - | | `deepseek-ai/DeepSeek-V4-Flash` | 1M | Long-context agent loops at very low cost | | `Qwen/Qwen3.5-27B` | 262K | Fast, inexpensive default with vision and reasoning | | `zai-org/GLM-5.3` | 1M | Strong reasoning on complex multi-step tasks | | `moonshotai/Kimi-K2.6` | 262K | Agentic tool use and coding | | `meta-llama/Llama-4-Scout-17B-16E-Instruct` | 327K | Llama ecosystem, vision, wide language coverage | | `google/gemma-4-31B-it-turbo` | 262K | Cheap, capable generalist | > [!NOTE] > > To compare other models, list the catalog with context windows and pricing via [`/v1/openai/models?filter=with_meta&sort_by=openshell`](https://api.deepinfra.com/v1/openai/models?filter=with_meta\&sort_by=openshell). Each entry's `metadata` block has `context_length`, `max_tokens` and per-million-token `pricing`; the `id` is what you pass as `model`. Reasoning models spend output tokens on thinking, so give them a `max_tokens` well above the visible answer. > > ```bash theme={null} > curl -s "https://api.deepinfra.com/v1/openai/models?filter=with_meta&sort_by=openshell" \ > | jq '.data[] | select(.metadata.tags | index("chat")) | {id, context_length: .metadata.context_length, pricing: .metadata.pricing}' > ``` ## Troubleshooting * **`profile import` fails with "already exists".** Import is create-only. Export, edit and `openshell profile update deepinfra -f deepinfra.yaml`, keeping `resource_version`, or `openshell profile delete deepinfra` and import again. `profile lint` on an exported file reports the same error; `update` validates on its own. * **`provider create --from-existing` finds no credential.** `DEEPINFRA_API_KEY` is not exported in the shell running the CLI. * **DeepInfra is denied like any other host.** The provider is not attached: `openshell sandbox provider list ` is empty. Attach it with `openshell sandbox provider attach deepinfra --wait`, then start a new process; a running one keeps the environment it started with. * **DeepInfra returns `401`.** The placeholder reached DeepInfra unresolved. The calling binary is not in the profile's `binaries` (the sandbox log names the real path OpenShell saw, for example a Python interpreter), or the log shows `credential_endpoint_mismatch`, meaning a base-policy rule admitted a host the profile does not bind the credential to. Fix the profile, not the network rule. * **`404` on a path like `/v1/openai/v1/chat/completions`.** The client appended `/v1` to the base URL. Set it to exactly `https://api.deepinfra.com/v1/openai`; nothing in OpenShell rewrites it. * **Hermes says "Model '' isn't available on DeepInfra".** It did not find `config.yaml`. OpenShell sets `HOME` to the working directory and ignores the image's `HERMES_HOME`, so the file belongs at `/.hermes/config.yaml`. * **`policy set` is rejected with "landlock policy cannot be changed on a live sandbox".** The file changed a startup section. Start from `openshell policy get --base` and change only `network_policies`. * **`sandbox create` fails with "not writable by the sandbox identity" or "image-declared volume masks OCI WorkingDir".** The image's `WORKDIR` must be a plain directory writable by its user. Derive an image that sets one, as in the Hermes section. ## Learn more > **[OpenShell documentation](https://docs.nvidia.com/openshell/latest)** > > Installation, providers, policies and the policy schema reference. > **[DeepInfra provider profile](https://github.com/NVIDIA/OpenShell/pull/1902)** > > The pull request that added `providers/deepinfra.yaml` to OpenShell. > **[NemoClaw inference providers](https://docs.nvidia.com/nemoclaw/user-guide/openclaw/inference/learn-and-choose/choose-inference-provider)** > > Choosing an inference provider in NVIDIA's NemoClaw stack. > **[Sandboxes](/sandboxes/overview)** > > DeepInfra's own isolated microVMs for running agent-written code. > [!NOTE] > > Tested on 2026-09-27 with OpenShell v0.1.1 (Docker driver, Docker Engine 29.8.1, Ubuntu 24.04 x86\_64), `curlimages/curl:8.22.0` and `nousresearch/hermes-agent:v2026.9.24` (Hermes v0.21.5). OpenShell's CLI and policy schema are still moving; re-check this page when you upgrade. # Pi Source: https://docs.deepinfra.com/integrations/pi Use DeepInfra models with the Pi coding agent via a custom OpenAI-compatible provider. [Pi](https://github.com/earendil-works/pi) is a coding agent. It doesn't ship DeepInfra as a built-in provider, but it works with any OpenAI-compatible endpoint — so you can add DeepInfra as a custom provider and run any [LLM from our catalog](https://deepinfra.com/models/text-generation). ## Configure the provider Add a `deepinfra` provider to your Pi configuration. See Pi's [custom provider docs](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/custom-provider.md) for where the config file lives. ```json theme={null} { "providers": { "deepinfra": { "name": "DeepInfra", "baseUrl": "https://api.deepinfra.com/v1/openai", "apiKey": "$DEEPINFRA_API_KEY", "api": "openai-completions", "models": [ { "id": "deepseek-ai/DeepSeek-V4-Flash", "name": "DeepSeek V4 Flash", "reasoning": true, "input": ["text"], "cost": { "input": 0.09, "output": 0.18, "cacheRead": 0.018 }, "compat": { "supportsDeveloperRole": false, "thinkingFormat": "deepseek" }, "contextWindow": 1048576, "maxTokens": 65536 } ] } } } ``` Get your API key from the [Dashboard](https://deepinfra.com/dash/api_keys). ## Run it Set your API key in the environment and launch Pi against the model: ```bash theme={null} export DEEPINFRA_API_KEY= pi --model deepinfra/deepseek-ai/DeepSeek-V4-Flash ``` ## Notes on the values * **`id`** (`deepseek-ai/DeepSeek-V4-Flash`) must match exactly — it's case-sensitive and is passed straight through as the model name. * The **`compat`** block applies to DeepSeek models only — omit it for other models: * **`supportsDeveloperRole: false`** is required for DeepSeek models. Because they're marked `"reasoning": true`, Pi defaults to sending the system prompt with the OpenAI-only `developer` role, which is rejected with a `422` error. This setting makes Pi use the standard `system` role instead. * **`thinkingFormat: "deepseek"`** matches how DeepSeek models stream reasoning tokens (as `reasoning_content`), so the model's thinking renders properly in Pi. * **`contextWindow`** is the model's full 1M-token window. **`maxTokens`** (`65536`) is just a per-request output cap you can raise or lower to taste. * The **`cost`** block only drives Pi's local spend display; the numbers above are DeepInfra's current per-million-token rates for this model, but they don't affect requests either way. > [!NOTE] > > To get the current context window and pricing for this or any other model, fetch [`/v1/openai/models?filter=with_meta&sort_by=pi`](https://api.deepinfra.com/v1/openai/models?filter=with_meta\&sort_by=pi), or see the [model's page](https://deepinfra.com/deepseek-ai/DeepSeek-V4-Flash). Each entry's `metadata` block maps onto the config above: `context_length` → `contextWindow`, `pricing.input_tokens` / `output_tokens` / `cache_read_tokens` → `cost`. > > ```bash theme={null} > curl -s "https://api.deepinfra.com/v1/openai/models?filter=with_meta&sort_by=pi" \ > | jq '.data[] | select(.id == "deepseek-ai/DeepSeek-V4-Flash").metadata | {context_length, pricing}' > ``` ## Troubleshooting **`422` role validation error** — an error like: ``` Error: 422: {"message":"Input should be ", ..., "param":"messages.0...role"} ``` means Pi sent the system prompt with the `developer` role, which isn't accepted. This happens by default with DeepSeek models because they're configured with `"reasoning": true`. Fix it by adding `"compat": { "supportsDeveloperRole": false }` to the model entry, as shown in the config above. **`401` / `402` errors** — if requests fail even though the config looks right, confirm your API key and balance with a direct API call. This bypasses Pi entirely and tells you whether the issue is your DeepInfra account or the Pi config: ```bash theme={null} curl -s https://api.deepinfra.com/v1/openai/chat/completions \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "deepseek-ai/DeepSeek-V4-Flash", "messages": [{"role": "user", "content": "Say hello in one word."}] }' ``` If this returns a normal completion, your key and balance are good. A `401` means the key is wrong; a `402`/quota error means the account needs funding. ## Learn more > **[Custom providers](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/custom-provider.md)** > > Pi's guide to defining a custom provider. > **[Providers overview](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/providers.md)** > > How Pi handles providers and models. > **[Chat Completions](/chat/overview)** > > DeepInfra's OpenAI-compatible API. > **[Authentication](/account/authentication)** > > API keys and scoped JWTs. # Models Source: https://docs.deepinfra.com/models Browse 200+ open-source models available on DeepInfra. DeepInfra hosts a large number of the most popular machine learning models. You can find [the full list here](https://deepinfra.com/models), conveniently split into categories based on their functionality. We are constantly adding more. DeepInfra is usually amongst the first to add a new model once it is available, and offers the best prices for open-source model inference. ## Model categories * **[Text generation / LLMs](https://deepinfra.com/models/text-generation)** — DeepSeek, Qwen, GLM, Kimi, Gemma, Nemotron, and more * **[Embeddings](https://deepinfra.com/models/embeddings)** — Qwen3 Embedding, BAAI/bge, sentence-transformers, and more * **[Rerankers](https://deepinfra.com/models/reranker)** — Qwen3 Reranker and more, for RAG pipelines * **[Vision / multimodal](https://deepinfra.com/models/multimodal)** — Qwen3-VL, Gemma 4, Kimi K3, and more * **[OCR](https://deepinfra.com/models/ocr)** — Specialized models for document text extraction * **[Text to image](https://deepinfra.com/models/text-to-image)** — FLUX, Qwen-Image, Nano Banana, and more * **[Text to video](https://deepinfra.com/models/text-to-video)** — Generate video clips from text prompts * **[Text to speech](https://deepinfra.com/models/text-to-speech)** — Convert text to natural-sounding audio * **[Speech recognition](https://deepinfra.com/models/automatic-speech-recognition)** — Whisper and other ASR models ## Model pages Each model has a dedicated page where you can: * Try it out interactively * See its API documentation * Grab ready-to-use code examples ## Private models We also support deploying [custom models](/private-models/overview) on DeepInfra infrastructure. Run your own fine-tuned or trained-from-scratch LLM on dedicated A100/H100/H200/B200/B300 GPUs. ## Specifying model versions Some models have more than one version available. You can infer against a particular version using `{"model": "MODEL_NAME:VERSION", ...}` format. You can also infer against a `deploy_id` using `{"model": "deploy_id:DEPLOY_ID", ...}`. This is especially useful for [Custom LLMs](/private-models/custom-llms) — you can start inferring before the deployment finishes and before you have the model name + version pair. ## Model deprecation Due to the fast-paced AI world, newer and better models are released every day. Occasionally we have to deprecate older models to maintain quality and affordability. When a model is deprecated: * **You'll receive at least 1 week's advance notice** before the deprecation date * **Your applications won't break** — after deprecation, inference requests are automatically forwarded to a recommended replacement model * **You'll get an email** notifying recent users of the model, including the deprecation date You can browse the current list of available models at [deepinfra.com/models](https://deepinfra.com/models). ## Suggest a model If you think there is a model that we should run, let us know at [info@deepinfra.com](mailto:info@deepinfra.com). We read every email. # Custom LLMs Source: https://docs.deepinfra.com/private-models/custom-llms Deploy your own LLM on dedicated A100/H100/H200/B200/B300 GPUs with autoscaling and an OpenAI-compatible endpoint. Run a dedicated instance of your public or private LLM on DeepInfra infrastructure. Your model gets its own GPU allocation, autoscaling, and an OpenAI-compatible API endpoint. ## Overview **Benefits:** * Predictable response times (no sharing with other users) * Autoscaling support * Run your own fine-tuned or trained-from-scratch model * Full OpenAI API compatibility **Trade-offs:** * Billed per GPU-hour, not per token — you need sufficient load to justify the cost Public models like Mixtral are shared across many users, giving very competitive per-token pricing. A private deployment gives you full GPU access, so you pay for GPU uptime regardless of traffic. ## Deployment configuration A deployment has fixed parameters: | Parameter | Description | | - | - | | `model_name` | Name used for inference calls | | `gpu` | `A100-80GB`, `H100-80GB`, `H200-141GB`, `B200-180GB`, `B300-288GB` (and more) | | `num_gpus` | Number of GPUs (model weights must fit with room for KV cache) | | `max_batch_size` | Max parallel requests; additional requests are queued | | `weights` | Hugging Face repo (public or private) | And dynamic settings (can be changed while running): | Setting | Description | | - | - | | `min_instances` | Minimum running copies (0 = scale to zero) | | `max_instances` | Maximum copies during high load | ## Create a deployment ### Web UI Go to [Dashboard → New Deployment → Custom LLM](https://deepinfra.com/dash/deployments?new=custom-llm). ### HTTP API ```bash theme={null} curl -X POST https://api.deepinfra.com/deploy/llm \ -d '{ "model_name": "test-model", "gpu": "A100-80GB", "num_gpus": 2, "max_batch_size": 64, "hf": { "repo": "deepseek-ai/DeepSeek-V4-Flash-0731" }, "settings": { "min_instances": 0, "max_instances": 1 } }' \ -H 'Content-Type: application/json' \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" ``` The model's full name will be `YOUR_GITHUB_USERNAME/model-name`. ## Monitor your deployment Track status via the [Dashboard → Deployments](https://deepinfra.com/dash/deployments) or via HTTP: ```bash theme={null} curl https://api.deepinfra.com/deploy/list \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" ``` ## Use your deployment Once running, inference via: * **Web demo**: `https://deepinfra.com/FULLNAME` * **OpenAI ChatCompletions API** * **OpenAI Completions API** * **DeepInfra inference API** ```bash theme={null} curl "https://api.deepinfra.com/v1/openai/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "model": "YOUR_USERNAME/test-model", "messages": [{"role": "user", "content": "Hello!"}] }' ``` You can also use `deploy_id` before the model is running: ```json theme={null} {"model": "deploy_id:YOUR_DEPLOY_ID", ...} ``` ## Update scaling settings ```bash theme={null} curl -X PUT https://api.deepinfra.com/deploy/DEPLOY_ID \ -d '{"settings": {"min_instances": 2, "max_instances": 2}}' \ -H 'Content-Type: application/json' \ -H "Authorization: Bearer $DEPLOY_API_KEY" ``` ## Delete a deployment * Use the trash icon in [Dashboard → Deployments](https://deepinfra.com/dash/deployments) * Or: `DELETE https://api.deepinfra.com/deploy/DEPLOY_ID` ## Limitations * **4 GPU limit per user** (e.g., 4×1GPU or 1×4GPU). Contact us for more. * GPU availability is not guaranteed during scale-up — you're only billed for what runs * Billing happens weekly in a separate invoice * Quantization is not currently supported (in progress) * `deploy_id` may not be immediately available while the model is deploying > [!WARNING] > > Forgetting to shut down a deployment is a common mistake. For example, leaving 2 GPUs running over a weekend (64 hours) at $2/GPU-hour costs $256. Set spending limits in [billing settings](https://deepinfra.com/dash/billing). # LoRA Adapters Source: https://docs.deepinfra.com/private-models/lora Deploy LoRA fine-tuned language models on DeepInfra. Deploy LoRA adapter models on top of base models hosted at DeepInfra. Your adapter is loaded on a supported base model and served with the standard OpenAI-compatible API. ## Prerequisites 1. A LoRA adapter model hosted on Hugging Face 2. A base model that supports LoRA at DeepInfra (see supported base models in the upload form) 3. A Hugging Face token if your LoRA adapter is private 4. A DeepInfra account and API key ## Deploy a LoRA model 1. Go to [Dashboard](https://deepinfra.com/dash) 2. Click **New Deployment** 3. Click the **LoRA Model** tab 4. Fill in the form: * **LoRA model name** — name used to reference this deployment * **Hugging Face Model Name** — path to your LoRA adapter on Hugging Face * **Hugging Face Token** — optional, required for private repos ## Example Using the public adapter `askardeepinfra/llama-3.1-8B-rank-32-example-lora` (base: `meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo`): 1. Go to Dashboard → New Deployment → LoRA Model 2. Fill in: * **LoRA model name**: `asdf/lora-example` * **Hugging Face Model Name**: `askardeepinfra/llama-3.1-8B-rank-32-example-lora` 3. Click **Upload** The deployment appears in [Dashboard → Deployments](https://deepinfra.com/dash/deployments). Initial state is `Initializing` → `Deploying` → `Running`. Once running, your model page is at `https://deepinfra.com/asdf/lora-example`. ## Inference ```bash theme={null} curl "https://api.deepinfra.com/v1/openai/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "model": "asdf/lora-example", "messages": [ { "role": "user", "content": "Hello!" } ] }' ``` The LoRA model name is used directly in the `model` field — the same as any other model. # LoRA for Image Generation Source: https://docs.deepinfra.com/private-models/lora-image Deploy LoRA adapters for text-to-image generation using models from Civitai. Deploy LoRA adapters for image generation on top of supported base image models. Source your LoRA from [Civitai](https://civitai.com). ## Prerequisites 1. A public LoRA model from Civitai 2. A DeepInfra account ## Deploy an image LoRA 1. Go to [Dashboard](https://deepinfra.com/dash) 2. Click **New Deployment** 3. Click the **LoRA text to image** tab 4. Fill in the form: * **LoRA model name** — name used to reference this deployment * **Base Model** — select the base model for this LoRA * **Civitai URL** — URL of the LoRA model from civitai.com The deployment appears in [Dashboard → Deployments](https://deepinfra.com/dash/deployments) with state `Initializing`. Deployment time varies from 5 seconds to 1 minute depending on LoRA size. Once `Running`, the model is ready to use. ## Inference ### Direct API endpoint ```bash theme={null} curl "https://api.deepinfra.com/yourname/yourmodel" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "prompt": "A cat in anime style", "lora_scale": 0.7 }' ``` ### OpenAI-compatible endpoint ```bash theme={null} curl "https://api.deepinfra.com/v1/openai/images/generations" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "model": "yourname/yourmodel", "prompt": "A cat in anime style" }' ``` ## Notes * Only public Civitai models are supported * Base models support the same parameters as their original versions * `lora_scale` controls the strength of the LoRA adaptation (0.0–1.0) # Deploy Private Models Source: https://docs.deepinfra.com/private-models/overview Run your own LLMs and image models on dedicated GPU infrastructure with autoscaling. DeepInfra allows you to deploy your own models on dedicated infrastructure — your weights, your endpoint, your isolation. ## Why run private models? * **Compliance** — data stays on dedicated infrastructure, not shared with other users * **Custom weights** — deploy fine-tuned or trained-from-scratch models * **Predictable latency** — no sharing with other users means consistent response times * **Autoscaling** — scale from 0 to many instances automatically based on load * **Competitive GPU pricing** — some of the lowest per-GPU-hour rates available, with no lock-in * **Simple deployment** — up and running in just a couple of clicks from the dashboard ## What you can deploy > **[Custom LLMs](/private-models/custom-llms)** > > Deploy any Hugging Face LLM on A100/H100/H200/B200/B300 GPUs with the OpenAI-compatible API. > **[LoRA Adapters](/private-models/lora)** > > Deploy LoRA fine-tuned language models on top of supported base models. > **[LoRA Image Models](/private-models/lora-image)** > > Deploy LoRA adapters for image generation from Civitai. ## GPU options Private model deployments run on: * **A100-80GB** — proven workhorse for LLM inference, great value * **H100-80GB** — fast and widely supported * **H200-141GB** — large HBM3e memory, ideal for big models * **B200-180GB** — NVIDIA Blackwell, significantly faster for inference workloads * **B300-288GB** — latest NVIDIA Blackwell Ultra, highest performance available ## Pricing model Unlike shared inference (pay per token), private deployments are billed per GPU-hour. You pay for the time your GPUs are running, regardless of traffic. > [!WARNING] > > Leaving a custom deployment running by mistake can rack up costs quickly. For example, forgetting to shut down a 2-GPU deployment over a weekend (64 hours) costs \~\$256 USD. Always set spending limits in [payment settings](https://deepinfra.com/dash/billing). ## Getting started 1. Go to [Dashboard → Deployments](https://deepinfra.com/dash/deployments) 2. Click **New Deployment** 3. Choose your deployment type (Custom LLM, LoRA, or LoRA Image) 4. Fill in the configuration and deploy See the specific guides for each deployment type: * [Custom LLMs](/private-models/custom-llms) * [LoRA Adapters](/private-models/lora) * [LoRA Image Models](/private-models/lora-image) # Quickstart Source: https://docs.deepinfra.com/quickstart Make your first API call in 60 seconds — no installation required. You don't need to install anything to do your first inference. You only need [your access token](https://deepinfra.com/dash/api_keys). DeepInfra gives you access to 100+ open-source models at the best prices available. ## Step 1: Get your API key Go to the [Dashboard](https://deepinfra.com/dash/api_keys) and create an API key (new to DeepInfra? See [Signing In](/account/signing-in)). Copy it when it is shown and export it, so the examples below can read it: ```bash theme={null} export DEEPINFRA_API_KEY= ``` ## Step 2: Make your first API call ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/openai/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{ "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "messages": [ { "role": "user", "content": "Hello!" } ] }' ``` ```python Python theme={null} import os from openai import OpenAI client = OpenAI( api_key=os.environ["DEEPINFRA_API_KEY"], base_url="https://api.deepinfra.com/v1/openai", ) response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash-0731", messages=[{"role": "user", "content": "Hello!"}], ) print(response.choices[0].message.content) ``` ```javascript JavaScript theme={null} import OpenAI from "openai"; const openai = new OpenAI({ apiKey: process.env.DEEPINFRA_API_KEY, baseURL: "https://api.deepinfra.com/v1/openai", }); const response = await openai.chat.completions.create({ model: "deepseek-ai/DeepSeek-V4-Flash-0731", messages: [{ role: "user", content: "Hello!" }], }); console.log(response.choices[0].message.content); ``` The response looks like this: ```json theme={null} { "id": "chatcmpl-guMTxWgpFf", "object": "chat.completion", "created": 1694623155, "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "Hello! It's nice to meet you. Is there something I can help you with?" }, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 15, "completion_tokens": 16, "total_tokens": 31, "estimated_cost": 0.0000268 } } ``` ## That's it You're using the [OpenAI Chat Completions API](/chat/overview) — the same interface you already know. The only changes are: * **Base URL**: `https://api.deepinfra.com/v1/openai` * **API key**: your DeepInfra token * **Model**: any model from [our catalog](https://deepinfra.com/models) The official OpenAI Python and Node.js libraries work out of the box. ## Install the SDK (optional) ```bash Python theme={null} pip install openai ``` ```bash JavaScript theme={null} npm install openai ``` ## Next steps > **[Chat Completions](/chat/overview)** > > Learn about the full chat completions API. > **[Streaming](/chat/streaming)** > > Stream responses token by token. > **[Tool Calling](/chat/tool-calling)** > > Give models access to external functions. > **[Model Catalog](/models)** > > Browse 100+ available models. # Request Compression Source: https://docs.deepinfra.com/request-compression Compress large request bodies with gzip to cut upload bandwidth and latency. DeepInfra accepts **gzip-compressed request bodies**. For large payloads — long chat histories, big embeddings batches, sizable inputs — compressing the request shrinks the bytes you upload (JSON and text typically compress 4–8×), which lowers your egress and can speed up requests over slower or metered connections. To use it, gzip the request body and set the `Content-Encoding: gzip` header. It works on **every** endpoint (chat completions, embeddings, and the rest), including requests whose response streams back — the request body is a single upload regardless of how the response is returned. ```bash cURL theme={null} # Pipe the JSON body through gzip and send it with Content-Encoding: gzip echo '{ "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "messages": [{"role": "user", "content": "Hello!"}] }' | gzip | curl "https://api.deepinfra.com/v1/openai/chat/completions" \ -H "Content-Type: application/json" \ -H "Content-Encoding: gzip" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ --data-binary @- ``` ```python Python theme={null} import gzip import json import requests payload = json.dumps({ "model": "deepseek-ai/DeepSeek-V4-Flash-0731", "messages": [{"role": "user", "content": "Hello!"}], }).encode("utf-8") response = requests.post( "https://api.deepinfra.com/v1/openai/chat/completions", data=gzip.compress(payload), headers={ "Content-Type": "application/json", "Content-Encoding": "gzip", "Authorization": "Bearer $DEEPINFRA_API_KEY", }, ) print(response.json()) ``` ```javascript JavaScript theme={null} import { gzipSync } from "node:zlib"; const payload = JSON.stringify({ model: "deepseek-ai/DeepSeek-V4-Flash-0731", messages: [{ role: "user", content: "Hello!" }], }); const response = await fetch("https://api.deepinfra.com/v1/openai/chat/completions", { method: "POST", headers: { "Content-Type": "application/json", "Content-Encoding": "gzip", Authorization: "Bearer $DEEPINFRA_API_KEY", }, body: gzipSync(payload), }); console.log(await response.json()); ``` ## When it helps Request compression pays off in proportion to how large your request body is: * **Worth it** for large bodies — long conversations, large embeddings batches, big inputs — where trimming the upload meaningfully cuts bandwidth and transfer time. * **Not worth it** for small requests (a short prompt). gzip adds a small header and a little CPU on both ends, so for tiny bodies there's nothing to gain. > [!NOTE] > > **gzip only.** `Content-Encoding: gzip` is the only supported request compression. Other > codings (`deflate`, `br`, `zstd`) are not decompressed — a body compressed with those will > fail to parse. Use `gzip`. # Sandboxes Source: https://docs.deepinfra.com/sandboxes/overview Isolated Linux microVMs for running untrusted code — create one with a single API call, run commands, move files in and out, and tear it down when you're done. Sandboxes give you an isolated Linux microVM on demand, ready to run code the moment it boots and torn down the moment you're done with it. They're built for agents and pipelines that need to execute arbitrary or untrusted code without you having to build and operate that infrastructure yourself. Sandboxes are ephemeral by design: they're short-lived, and data inside them is not backed up or guaranteed to persist while they run. Treat a sandbox as scratch space — write anything you want to keep to external storage before it shuts down. Manage sandboxes at [Dashboard → Sandboxes](https://deepinfra.com/dash/sandboxes), or drive them entirely from the [Python SDK](#python-sdk) or [HTTP API](#http-api). > [!NOTE] > > This page focuses on the details that aren't obvious from the API surface alone — what actually survives a restart, how long a sandbox lives, and where the sharp edges are. If you only read one section, read [Filesystem and persistence](#filesystem-and-persistence). ## Quickstart ```python theme={null} from deepinfra import Sandbox sb = Sandbox.create(plan="medium", timeout="10m") # blocks until running r = sb.exec("bash", "-c", "pip install pandas && python -c 'import pandas; print(pandas.__version__)'") print(r.stdout, r.stderr, r.returncode) out = sb.run_python("print(21 * 2)").check() # .check() raises on non-zero exit print(out.stdout) # "42" sb.fs.write("/workspace/in.csv", b"a,b\n1,2\n") # fs.read/write only reach inside /workspace data = sb.fs.read("/workspace/in.csv") sb.stop() # frees compute, keeps /workspace sb.start() # resumes with /workspace intact, everything else reset sb.terminate() # deletes the sandbox, including /workspace ``` That's the whole lifecycle: create, run, move files, stop or terminate. ## Plans and pricing Sandboxes come in multiple plan sizes, from a quick script to a heavier data-processing run. If you don't pass `plan`, you get `medium` — not the cheapest tier. Disk scales with plan. For current plan specs (vCPU, RAM, disk) and hourly pricing, don't hardcode numbers — check the live catalog: * Dashboard: [deepinfra.com/dash/sandboxes#catalog](https://deepinfra.com/dash/sandboxes#catalog) * API: `GET /v1/sandboxes/catalog` (see [HTTP API](#http-api)), or `Sandbox.catalog()` in the Python SDK Billing is per-second with no minimum, and only runs while a sandbox is `creating`, `starting`, `running`, or `stopping` — a `stopped` sandbox costs nothing. Because the meter starts while the microVM is still booting and keeps running for the few seconds it takes to save your disk on `stop()`, per-call cost is "wall clock the sandbox occupied capacity," not strictly "wall clock you could run commands." ## Lifecycle and timeouts | State | Description | | - | - | | `creating` | The sandbox is being provisioned | | `starting` | A fresh container is booting from the base image | | `running` | The sandbox is active and accepting commands | | `stopping` | The sandbox is shutting down; `/workspace` is being saved | | `stopped` | Shut down — not billed, `/workspace` preserved | | `failed` | Hit an unrecoverable error — **not** recoverable, see below | | `deleted` | Permanently removed (via `terminate()`, or automatic cleanup) | Operations are tied to state: `exec()` and `fs.read()`/`fs.write()` require `running`; `stop()` requires `running` or `starting`; `start()` requires `stopped`. Calling one from the wrong state returns a `409`/`ConflictError`. A sandbox is subject to three independent clocks: * **Idle timeout** — configurable per sandbox with `timeout` at creation (an hour, by default, if you don't pass one, and there's currently no way to disable it). Idle time is measured from when your **last call finished**, not when it started, so a single command that runs longer than the idle timeout can have its sandbox stopped out from under it mid-execution. If a job might take a while, set `timeout` generously (up to the 30-minute per-command cap) rather than relying on the default. * **24-hour running limit** — a sandbox auto-stops after 24 continuous hours of `running`. This resets every time you `start()` it, so stopping and restarting doesn't carry over — only the current run counts toward the 24h cap. You can `stop()`/`start()` the same sandbox indefinitely; there's no need to `terminate()` and recreate it just to reset the clock. * **Retention after stop** — a `stopped` sandbox has up to **7 days** to be `start()`-ed again before it's permanently deleted, `/workspace` included. Independently, each `exec()` command has its own timeout: 60 seconds by default, up to a hard cap of 30 minutes. > [!WARNING] > > A `failed` sandbox is not something to recover from — it is automatically deleted within about **5 minutes**, and unlike a clean `stop()`, its `/workspace` is **not** preserved. Treat `failed` as data loss, not a state to `start()` your way out of. `GET`/list responses include a `fail_reason` field with why it failed — useful for debugging, though the Python SDK doesn't expose it as an attribute yet, so read it from the raw HTTP response if you need it. ## Filesystem and persistence The persistence model is narrower than "it's a VM, everything sticks around" — this is the part worth reading closely. **Only `/workspace` survives a `stop()` / `start()` cycle.** Everything else — packages installed outside it, environment changes, background processes, anything written to `/tmp`, `/root`, or elsewhere — is gone the moment you `start()` again. That's because `start()` always boots a **fresh container from the base image**; it never resumes a suspended VM. Only `/workspace`'s contents come back. The rest of the container's own disk is small (a couple GB) regardless of plan, so treat `/workspace` as the one place anything you need to keep should live — don't count on `pip install`-ed packages surviving a stop. `fs.read()` and `fs.write()` enforce this directly: paths outside `/workspace` are rejected with `400` (`BadRequestError`), not silently redirected. Reading a path that doesn't exist returns `404`; reading something that isn't a regular file (a directory, for instance) also returns `400`. Both directions are capped at 100 MiB per call (`413` / `ContentTooLargeError`) — the same limit applies whether you're writing or reading. `exec()` isn't restricted the same way — a shell command can write anywhere on disk — but only what lands under `/workspace` will be there the next time you `start()`. **Persistence is also conditional on a clean stop.** If a sandbox crashes or is marked unhealthy — a node issue, an out-of-memory kill, anything that isn't you calling `stop()` — it goes to `failed`, and its `/workspace` is **not** preserved; the sandbox is torn down within a few minutes and any unsaved work in it is gone. Only an explicit `stop()` guarantees your data comes back. ## Isolation and networking Every sandbox boots as its own microVM — via Kata Containers running on QEMU/KVM — with its own kernel and its own virtualized hardware boundary, not a namespaced slice of a host shared with other tenants. Networking is locked down to match: * A sandbox can reach the public internet (so `pip install` works), but accepts **no inbound connections** at all. * It can't reach other sandboxes or DeepInfra's internal infrastructure. * Egress and ingress are each capped at **200 Mbit/s**. * Outbound SMTP (ports 25, 465, 587) is blocked. ## Limits and quotas * **Active sandboxes** — up to **25 non-stopped sandboxes per account** at a time. `creating`, `starting`, `running`, and `stopping` all count against this; `stopped` doesn't. Going over it returns a `429` (`RateLimitError`, aliased as `TooManySandboxesError` in the SDK) instead of a silent failure. * **Fleet capacity** — sandboxes run on a shared fleet, so a capacity crunch across all customers can occasionally return a `503` (`CapacityError`) on creation even when you're well under your own limit. Retry with backoff. * **File transfers** — both `fs.write()` and `fs.read()` are capped at 100 MiB per call (see above). * **Disk** — scales with plan (see the [catalog](#plans-and-pricing) for exact numbers); the container's own disk outside `/workspace` is only a couple GB regardless of plan. ## Tags Attach your own string key/value tags at creation time for bookkeeping: ```python theme={null} sb = Sandbox.create(plan="small", tags={"job": "etl-42"}) ``` Tags come back on every lookup (`GET`, list) and can be used to filter client-side: ```python theme={null} etl_boxes = Sandbox.list(tags={"job": "etl-42"}) ``` Tags are set once at creation — there's no endpoint to update them afterward. ## Errors | Cause | HTTP status | SDK exception | | - | - | - | | Malformed request (bad/outside-`/workspace` path, empty command, timeout out of range) | `400` | `BadRequestError` | | Missing/invalid API key | `401` | `AuthenticationError` | | Account suspended | `402` | *(none yet — generic `APIStatusError`)* | | Sandbox not found (or not yours) | `404` | `NotFoundError` | | Wrong state for this operation | `409` | `ConflictError` | | File transfer body over 100 MiB (either direction) | `413` | `ContentTooLargeError` | | Over the 25-active-sandboxes cap | `429` | `RateLimitError` / `TooManySandboxesError` | | Shared fleet capacity exhausted, or creation temporarily disabled | `503` | `CapacityError` | | Internal error | `5xx` | `InternalServerError` | The Python SDK adds a few client-side exceptions for conditions that aren't a single HTTP response: `SandboxTimeoutError` (waiting for a state transition took too long), `SandboxFailedError` (the sandbox went to `failed` while you were waiting on it), `SandboxExecError` (the exec stream ended without a return code), and `CommandFailedError` (raised by `.check()` on a non-zero exit). ## Python SDK ```bash theme={null} pip install deepinfra ``` Set `DEEPINFRA_API_KEY` (the same key you use for inference, from [deepinfra.com/dash/api\_keys](https://deepinfra.com/dash/api_keys)) and you're executing code in an isolated microVM in a few lines — see the [Quickstart](#quickstart) above. A few things worth knowing beyond the basic example: * `timeout` accepts plain seconds (`600`) or a duration string (`"90s"`, `"10m"`, `"2h"`, `"1h30m"`). * `Sandbox.create()` blocks until the sandbox is `running` by default. Pass `wait=False` to get it back immediately in whatever state it's in, or `wait_timeout=` to change how long to wait. * `Sandbox.from_id("sb-...")` reattaches to a sandbox by ID from anywhere in your code — you're not stuck driving it from the process that created it. * Every network method has an `a`-prefixed async twin (`acreate`, `aexec`, `aterminate`, and so on), so orchestrating many sandboxes at once is a normal `asyncio` program: ```python theme={null} import asyncio from deepinfra import Sandbox async def main(): sb = await Sandbox.acreate(plan="small") r = await sb.aexec("uname", "-a") print(r.stdout) await sb.aterminate() asyncio.run(main()) ``` * A sandbox terminates automatically when used as a context manager: ```python theme={null} with Sandbox.create(plan="small") as sb: sb.run_python("open('/workspace/out.txt', 'w').write('hi')") print(sb.fs.read("/workspace/out.txt")) # b'hi' -- fs.read() returns raw bytes, not str ``` * For large scripts, write the file and run it rather than passing code inline: ```python theme={null} sb.fs.write("/workspace/script.py", open("script.py").read()) sb.exec("python3", "/workspace/script.py", timeout="30m") ``` **Coming soon:** `exec_stream()` for live command output, `snapshot()` / `Sandbox.from_snapshot()` for point-in-time snapshots you control, `expose_port()`, and `fs.upload_dir()`. ## HTTP API Everything above is also available directly over HTTP. Authenticate with your [API key](/account/authentication). Create a sandbox: ```bash theme={null} curl -X POST https://api.deepinfra.com/v1/sandboxes \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "plan": "medium", "tags": {"job": "etl-42"}, "timeout_seconds": 600 }' ``` ```json theme={null} {"sandbox_id": "sb-xxxxxxxxxxxxxxxx"} ``` Get or list sandboxes: ```bash theme={null} curl https://api.deepinfra.com/v1/sandboxes/{sandbox_id} \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" curl https://api.deepinfra.com/v1/sandboxes \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" ``` Run a command — the response is streamed as `application/x-ndjson`, one JSON object per line, ending in exactly one terminal line (`{"returncode": N}` on completion, or `{"error": "..."}` if the command timed out or otherwise couldn't finish): ```bash theme={null} curl -N -X POST https://api.deepinfra.com/v1/sandboxes/{sandbox_id}/exec \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -H "Content-Type: application/json" \ -d '{"command": ["bash", "-c", "echo hi"], "timeout_seconds": 60}' ``` ``` {"stdout": "hi\n"} {"returncode": 0} ``` > [!NOTE] > > This streamed response is always HTTP `200` once it starts — a mid-command failure (timeout, oversized output, and so on) shows up as `{"error": ...}` in the terminal line, not as an HTTP error status. Check the last line, not just the status code. Move files in and out — both are scoped to `/workspace` and capped at 100 MiB per call: ```bash theme={null} curl -X PUT "https://api.deepinfra.com/v1/sandboxes/{sandbox_id}/fs/content?path=/workspace/in.csv" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -H "Content-Type: application/octet-stream" \ --data-binary $'a,b\n1,2\n' curl "https://api.deepinfra.com/v1/sandboxes/{sandbox_id}/fs/content?path=/workspace/in.csv" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" ``` Stop, start, or terminate: ```bash theme={null} curl -X POST https://api.deepinfra.com/v1/sandboxes/{sandbox_id}/stop \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" curl -X POST https://api.deepinfra.com/v1/sandboxes/{sandbox_id}/start \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" curl -X DELETE https://api.deepinfra.com/v1/sandboxes/{sandbox_id} \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" ``` List available plans and current pricing: ```bash theme={null} curl https://api.deepinfra.com/v1/sandboxes/catalog \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" ``` See the [API Reference](https://docs.deepinfra.com/api-reference) for full schemas. ## FAQ **What happens to my data when I stop a sandbox?** Nothing under `/workspace` is lost — it's preserved and restored when you `start()` again. Everything else in the container (installed packages, environment changes, `/tmp`, `/root`) is reset, because `start()` boots a fresh container rather than resuming the old one. **How long does a sandbox actually live?** Three independent clocks apply: it idles out after its configured `timeout` (1 hour by default) with no activity, it auto-stops after 24 continuous hours of `running` (this resets on every `start()`, so repeated stop/start cycles don't add up), and once stopped it's permanently deleted after 7 days if you don't start it again. **What happens if my sandbox crashes?** It moves to `failed` and is deleted within about 5 minutes — `/workspace` is **not** preserved in this case. Only a clean `stop()` guarantees your data survives. **Can a sandbox reach other sandboxes, or my own infrastructure?** No. It can reach the public internet, but not other sandboxes and not DeepInfra's internal infrastructure. **Why did my `fs.read()`/`fs.write()` call fail?** Most likely the path was outside `/workspace`, which comes back as `400`. A `404` means the file doesn't exist; `413` means the file is over the 100 MiB transfer cap. **What happens if my account is suspended?** Sandbox creation, `start()`, and any exec/file-transfer calls are blocked (`402`) while your account is suspended — for example, when your balance runs out. `stop()` and `terminate()` still work, so you can shut down a sandbox (and its billing) even while suspended. Consider [Automatic Top-Up](https://deepinfra.com/dash/billing) if you're running unattended jobs. **Where do I see what my sandboxes are costing me?** On the [Usage](https://deepinfra.com/dash/usage) page. # Stable Diffusion Source: https://docs.deepinfra.com/tutorials/stable-diffusion Generate images with Stable Diffusion and SDXL on DeepInfra. DeepInfra supports a wide variety of text-to-image models. This page walks through SDXL Turbo; the catalog also carries the FLUX, Qwen-Image and Nano Banana families. Browse [all text-to-image models](https://deepinfra.com/models/text-to-image). ## Quick start This example uses `stabilityai/sdxl-turbo`: ```javascript JavaScript theme={null} import { TextToImage } from "deepinfra"; import { createWriteStream } from "fs"; import { Readable } from "stream"; const DEEPINFRA_API_KEY = process.env.DEEPINFRA_API_KEY; const model = new TextToImage("stabilityai/sdxl-turbo", DEEPINFRA_API_KEY); const response = await model.generate({ prompt: "a burger with a funny hat on the beach", }); const result = await fetch(response.images[0]); if (result.ok && result.body) { Readable.fromWeb(result.body).pipe(createWriteStream("image.jpg")); } ``` ```bash cURL theme={null} curl "https://api.deepinfra.com/v1/inference/stabilityai/sdxl-turbo" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -d '{"prompt": "a burger with a funny hat on the beach"}' ``` `images[0]` is a `data:` URI containing the encoded image, not an HTTP link — `fetch()` accepts it directly, as above. ## Advanced options Each model has its own set of parameters including negative prompts, number of inference steps, guidance scale, seed, and more. Check the model's page for all available options: * [stabilityai/sdxl-turbo](https://deepinfra.com/stabilityai/sdxl-turbo) ## Using the OpenAI-compatible API You can also use the OpenAI images API for image generation — see [Image Generation](/apis/image-generation). ## Custom LoRA adapters To use custom style LoRA adapters for image generation, see [LoRA for Image Generation](/private-models/lora-image). # Whisper Speech Recognition Source: https://docs.deepinfra.com/tutorials/whisper Transcribe audio to text with OpenAI Whisper on DeepInfra. [Whisper](https://github.com/openai/whisper) is OpenAI's speech recognition model. Given an audio file, it produces transcribed text with per-sentence timestamps. DeepInfra hosts both Whisper and several other speech recognition models, all reachable through the same API. Browse [all speech recognition models](https://deepinfra.com/models/automatic-speech-recognition). ## Models | Model | Notes | | - | - | | `openai/whisper-large-v3` | Best accuracy | | `openai/whisper-large-v3-turbo` | Faster, near-large-v3 quality | | `Qwen/Qwen3-ASR-1.7B` | Small and fast | | `Qwen/Qwen3-ASR-0.6B` | Smallest | | `nvidia/Nemotron-3.5-ASR-Streaming-Multilingual-0.6b` | Streaming, multilingual | | `mistralai/Voxtral-Small-24B-2507` | Audio-understanding alternative | | `mistralai/Voxtral-Mini-3B-2507` | Smaller audio-understanding alternative | Whisper produces per-sentence timestamp segmentation. ## Example ```javascript JavaScript theme={null} import { AutomaticSpeechRecognition } from "deepinfra"; import path from "path"; import { fileURLToPath } from "url"; const __filename = fileURLToPath(import.meta.url); const __dirname = path.dirname(__filename); const client = new AutomaticSpeechRecognition( "openai/whisper-large-v3", process.env.DEEPINFRA_API_KEY ); const response = await client.generate({ audio: path.join(__dirname, "audio.mp3"), }); console.log(response.text); ``` ```bash cURL theme={null} curl -X POST \ -H "Authorization: Bearer $DEEPINFRA_API_KEY" \ -F audio=@audio.mp3 \ 'https://api.deepinfra.com/v1/inference/openai/whisper-large-v3' ``` ## Supported formats * `mp3` * `wav` ## Additional parameters Each model supports parameters like `language`, `task` (transcribe vs. translate), and more. Check the model's documentation page for the full list: * [openai/whisper-large-v3](https://deepinfra.com/openai/whisper-large-v3)