## Choose how you want to call a model Create an API key, send a small LLM request, and verify the response yourself. Ask an agent to help connect to Novita AI, then verify the real API response. ## Before you run * Sign in or [create a Novita account](https://novita.ai/user/register). * Create or manage an [API key](https://novita.ai/settings/key-management). * Confirm that your account has enough [credit](https://novita.ai/billing) for the product you want to use. **No install — paste this into your agent chat:** ```text theme={"system"} Read https://raw.githubusercontent.com/novitalabs/novita-skills/main/skills/novita-ai/SKILL.md and use Novita AI for this task. ``` **Optional — install once if your agent supports skills:** ```bash theme={"system"} npx skills add novitalabs/novita-skills --skill novita-ai ``` The `novita-ai` skill covers model APIs, GPUs, Agent Sandbox, integrations, and troubleshooting. Your agent still needs a valid API key and available credit when it makes a request. [View the skill on GitHub →](https://github.com/novitalabs/novita-skills) ## Explore Novita AI Explore hosted models and API capabilities. Run and manage your own models on dedicated GPUs. Look up endpoints, parameters, authentication, and responses. Find causes and fixes for common API errors. Create, store, and configure API keys. Check credit, usage, and payment settings. # LangChain Source: https://docs.novita.ai/guides/langchain Integrate Novita AI with LangChain to build intelligent, language model-driven applications. Learn setup, API usage, and function calling workflows. This guide will walk you through the process of integrating Novita AI with LangChain. You’ll be able to use Novita AI’s powerful language models with LangChain’s robust tools for building language model-driven applications. ## What is LangChain? LangChain is a framework for developing applications powered by language models. It enables applications that: * **Are context-aware**: LangChain connects a language model to sources of context (such as prompt instructions, few-shot examples, content to ground its response in, etc.). * **Reason**: LangChain allows language models to reason—whether it’s determining how to answer based on provided context or deciding what actions to take. With LangChain, you can build complex workflows, enhance model behavior with external knowledge, and create intelligent systems that can interact dynamically with users and data sources. ## Prerequisites Before you start, make sure you have the following: * **Novita AI LLM API Key**: * Visit [Novita AI’s website](https://novita.ai/) and create an account. * After logging in, go to the [**Key Management**](https://novita.ai/settings/key-management) page to generate your **API Key**. This key is required to connect Novita AI’s models to LangChain. ![Novita AI key management](https://mintlify.s3.us-west-1.amazonaws.com/novitaai/images/third-party/dify-1.png) * A basic understanding of Node.js, JavaScript, and how to use environment variables. ## Integration ### Step 1: Set the API Key For most environments, set the environment variable `NOVITA_API_KEY` as follows: ```bash theme={"system"} export NOVITA_API_KEY="your-api-key" ``` Make sure to replace `your-api-key` with the actual key you got from Novita AI. ### Step 2: Install the Required Packages To integrate Novita AI with LangChain, you need to install the `@langchain-community` package, which includes the Novita AI integration. Choose one of the following commands to install the necessary packages: **Using npm:** ```bash theme={"system"} npm install @langchain/community @langchain/core ``` **Using yarn:** ```bash theme={"system"} yarn add @langchain/community @langchain/core ``` **Using pnpm:** ```bash theme={"system"} pnpm add @langchain/community @langchain/core ``` ### Step 3: Instantiate the Novita AI Model Once you’ve installed the necessary packages, you can instantiate the Novita AI model using the `ChatNovitaAI` class. Here’s an example that demonstrates how to do that: ```javascript theme={"system"} import { ChatNovitaAI } from "@langchain/community/chat_models/novita"; const llm = new ChatNovitaAI({ model: "deepseek/deepseek-v4.1-flash", // You can choose the model you want to use temperature: 0, // Optional: Controls randomness. 0 is deterministic. // other parameters can be set here... }); ``` ### Step 4: Invoke the Model for Chat Completion Once the model is instantiated, you can use it to generate chat completions by invoking it with a message. Here's an example of how to send a message and get a response: ```javascript theme={"system"} const aiMsg = await llm.invoke([ { role: "system", content: "You are a helpful assistant that translates English to French. Translate the user sentence.", }, { role: "human", content: "I love programming.", }, ]); console.log(aiMsg.content); // The model’s response will be printed here ``` ### Step 5: Chain Model with Prompt Templates LangChain allows you to create powerful workflows by chaining models together with prompt templates. This can be especially useful when you want to reuse the same format for multiple inputs. Here’s an example where we chain the Novita AI model with a custom prompt template for translating between languages: ```javascript theme={"system"} import { ChatPromptTemplate } from "@langchain/core/prompts"; // Create a template for translating languages const prompt = ChatPromptTemplate.fromMessages([ [ "system", "You are a helpful assistant that translates {input_language} to {output_language}.", ], ["human", "{input}"], ]); // Chain the prompt with the model const chain = prompt.pipe(llm); // Invoke the chain with inputs for translation const result = await chain.invoke({ input_language: "English", output_language: "German", input: "I love programming.", }); console.log(result.content); // The translated text will be printed here ``` ### Step 6: Customize the Workflow You can modify the temperature, add more messages, or tweak other parameters depending on your use case. LangChain is highly flexible, allowing you to design complex interactions by chaining multiple prompts, adding conditional logic, or working with different models. ## Function Calling with Novita AI and LangChain To implement function calling (or tool usage) with Novita AI's LLM API, LangChain can serve as a convenient framework. In this example, we’ll create a simple math application that allows the model to perform addition and multiplication operations via function calls. 💡 While this guide uses LangChain for convenience, implementing function calling doesn’t require any specific framework. The key is designing the right prompts to make the model understand and correctly invoke functions. LangChain is used here simply to streamline the implementation. ### Prerequisites First, install the required packages: ```bash theme={"system"} pip install langchain-openai python-dotenv ``` ### Setting Up the Environment Create a `.env` file in your project root and add your Novita AI API key: ``` NOVITA_API_KEY=your_api_key_here ``` ### Implementation Steps 1. **Define the Tools** First, let’s create two simple mathematical tools using LangChain's `@tool` decorator: ```python theme={"system"} from langchain_core.tools import tool @tool def multiply(x: float, y: float) -> float: """Multiply two numbers together.""" return x * y @tool def add(x: int, y: int) -> int: """Add two numbers.""" return x + y tools = [multiply, add] ``` 2. **Create the Tool Execution Function** Next, implement a function to execute the tools: ```python theme={"system"} from typing import Any, Dict, Optional, TypedDict from langchain_core.runnables import RunnableConfig class ToolCallRequest(TypedDict): name: str arguments: Dict[str, Any] def invoke_tool( tool_call_request: ToolCallRequest, config: Optional[RunnableConfig] = None ): """Execute the specified tool with given arguments.""" tool_name_to_tool = {tool.name: tool for tool in tools} name = tool_call_request["name"] requested_tool = tool_name_to_tool[name] return requested_tool.invoke(tool_call_request["arguments"], config=config) ``` 3. **Set Up the LangChain Pipeline** Create a chain that uses Novita AI's LLM to select and prepare tool calls: ```python theme={"system"} from langchain_openai import ChatOpenAI from langchain_core.output_parsers import JsonOutputParser from langchain_core.prompts import ChatPromptTemplate from langchain_core.tools import render_text_description import os def create_chain(): """Create a chain that uses the specified LLM model to select and prepare tool calls.""" model = ChatOpenAI( model="meta-llama/llama-3.3-70b-instruct", api_key=os.getenv("NOVITA_API_KEY"), base_url="https://api.novita.ai/openai", ) rendered_tools = render_text_description(tools) system_prompt = f"""\ You are an assistant that has access to the following set of tools. Here are the names and descriptions for each tool: {rendered_tools} Given the user input, return the name and input of the tool to use. Return your response as a JSON blob with 'name' and 'arguments' keys. The `arguments` should be a dictionary, with keys corresponding to the argument names and the values corresponding to the requested values. """ prompt = ChatPromptTemplate.from_messages( [("system", system_prompt), ("user", "{input}")] ) return prompt | model | JsonOutputParser() ``` 4. **Create the Main Processing Function** Implement the main function that processes mathematical queries: ```python theme={"system"} def process_math_query(query: str): """Process a mathematical query by using an LLM to select the appropriate tool and execute it.""" chain = create_chain() message = chain.invoke({"input": query}) result = invoke_tool(message, config=None) return message, result ``` 5. **Usage Example** Here’s how to use the implementation: ```python theme={"system"} if __name__ == "__main__": message, result = process_math_query( "meta-llama/llama-3.3-70b-instruct", "what's 3 plus 1132" ) print(result) # Output: 1135 ``` # Langflow Source: https://docs.novita.ai/guides/langflow Six steps to achieve Novita AI's LLM API integration with Langflow, enabling faster application deployment and simplified workflows. The integration of Langflow and Novita AI creates a powerful development environment that streamlines the creation of multi-agent and RAG applications. This combination leverages Langflow's intuitive visual flow builder for rapid prototyping while enabling seamless workflow design and testing capabilities. Novita AI enhances this ecosystem with its advanced Model APIs, featuring industry-leading language models including Llama, DeepSeek, and Mistral. This step-by-step guide will help you access Novita AI LLM API on Langflow and start building AI-powered applications. ## **Accessing Novita AI LLM API on Langflow** ### Step 1: **Expand the "Models" Tag** * Find and expand the "Models" section on Langflow’s sidebar. Expand the Models tag on Langflow

### **Step 2: Drag Novita AI to Your Canvas** * Select and position the "Novita AI "component from the sidebar onto the Langflow canvas area.

Drag Novita AI to Your Canvas on Langflow

### **Step 3: Create a Global Variable for the Novita AI API Key** * Set up Langflow authentication instantly by adding your Novita AI API Key as a global variable.

Create a Global Variable for the Novita AI API Key on Langflow

### Step 4: **Get your Novita AI API Key** * Generate Your API Keys Instantly on the [Key Management page](https://novita.ai/settings/key-management). ### Step 5: **Select a Model** * Choose from Novita AI's powerful language models: DeepSeek, Llama, and Mistral and then select the perfect AI model for your project needs. Select a Model on Langflow

### **Step 6: Start Building Your AI App** * Create advanced AI functionality by connecting Novita AI components to your Langflow workflow system. # Langfuse Source: https://docs.novita.ai/guides/langfuse Step-by-step guide to set up Langfuse with Novita AI to develop, monitor, evaluate, and debug AI applications. With Langfuse, your team can collaboratively debug, analyze, and iterate on their LLM applications built with Novita AI. Its fully integrated features streamline the development workflow, enhancing efficiency and accelerating progress. This guide shows you how to integrate Novita AI with Langfuse. Novita AI's API endpoints for chat, language and code are fully compatible with OpenAI's API. This allows us to use the Langfuse OpenAI drop-in replacement to trace all parts of your application. # Prerequisites Before you begin, make sure you have the following: * Novita AI LLM API Key: * Visit Novita AI’s website to create an account. * Log in and go to the Key Management page to generate your API Key, which is essential for integrating Novita AI’s models with Langfuse. * Langfuse Account: * Sign up for a Langfuse account on the official [Langfuse](https://langfuse.com)website to start building powerful AI applications. # Integration Steps ## Step 1: Install Dependencies Ensure you have installed all the required Python packages: ```shell theme={"system"} pip install openai langfuse ``` ## Step 2: Set Up Environment Variables ```python theme={"system"} import os # Get keys for your project from the project settings page # https://cloud.langfuse.com os.environ["LANGFUSE_PUBLIC_KEY"] = "pk-..." # DOCS EXAMPLE KEYS os.environ["LANGFUSE_SECRET_KEY"] = "sk-..." # DOCS EXAMPLE KEYS os.environ["LANGFUSE_HOST"] = "https://cloud.langfuse.com" # 🇪🇺 EU region # os.environ["LANGFUSE_HOST"] = "https://us.cloud.langfuse.com" # 🇺🇸 US region # Get your Novita AI API key from the project settings page os.environ["NOVITA_API_KEY"] = "..." ``` ## Step 3: Langfuse OpenAI drop-in Replacement In this step we use the native [OpenAI drop-in replacement](https://langfuse.com/docs/integrations/openai/python/get-started) by importing `from langfuse.openai import openai`. To start using Novita AI with OpenAI's client libraries, pass in your Novita AI API key to the `api_key` option, and change the `base_url` to [`https://api.novita.ai/openai:`](https://api.novita.ai/openai) ```python theme={"system"} # instead of import openai: from langfuse.openai import openai client = openai.OpenAI( api_key=os.environ.get("NOVITA_API_KEY"), base_url="https://api.novita.ai/openai", ) ``` **Note:** The OpenAI drop-in replacement is fully compatible with the [Langfuse Python SDK](https://langfuse.com/docs/observability/sdk/overview) and [`@observe()` instrumentation](https://langfuse.com/docs/observability/sdk/instrumentation) to trace all parts of your application. ## Step 4: Run An Example The following code cell shows how to use the traced OpenAI client to call Novita AI's chat model. All API calls will be seamlessly traced by Langfuse. ```python theme={"system"} client = openai.OpenAI( api_key=os.environ.get("NOVITA_API_KEY"), base_url="https://api.novita.ai/openai", ) response = client.chat.completions.create( model="meta-llama/llama-3.3-70b-instruct", messages=[ {"role": "system", "content": "Act like you are a helpful assistant."}, {"role": "user", "content": "What are the famous attractions in San Francisco?"}, ] ) print(response.choices[0].message.content) San Francisco, one of the most iconic cities in the world, is home to a plethora of famous attractions that cater to all interests and ages. Here are some of the most popular attractions in San Francisco: 1. **Golden Gate Bridge**: An engineering marvel and a symbol of San Francisco, the Golden Gate Bridge is a must-visit attraction. Take a walk or bike ride across the bridge for spectacular views of the city and the Bay. 2. **Alcatraz Island**: Explore the infamous former prison turned national park, which once housed notorious inmates like Al Capone. Take a ferry to the island and enjoy a guided tour of the prison and its surroundings. 3. **Fisherman's Wharf**: A bustling waterfront district, Fisherman's Wharf is famous for its seafood restaurants, street performers, and stunning views of the Bay Bridge and Alcatraz Island. Don't miss the sea lions at Pier 39! 4. **Chinatown**: San Francisco's Chinatown is one of the largest and oldest in the United States. Explore the colorful streets, try some authentic Chinese cuisine, and shop for unique souvenirs. 5. **Golden Gate Park**: A sprawling urban park that's home to several attractions, including the de Young Museum, the California Academy of Sciences, and the Japanese Tea Garden. 6. **Cable Cars**: A classic San Francisco experience, the cable cars offer a fun and historic way to explore the city. Take a ride on the Powell-Mason line to Fisherman's Wharf, or the Powell-Hyde line to Lombard Street. 7. **Lombard Street**: Known as the "crookedest street in the world," Lombard Street is a scenic and winding road that offers stunning views of the city. 8. **Union Square**: A vibrant public square in the heart of the city, Union Square is surrounded by shopping, dining, and entertainment options. Catch a show at the historic Curran Theatre or take a stroll through the square. 9. **The Painted Ladies**: A row of colorful Victorian houses on Alamo Square, the Painted Ladies are a iconic symbol of San Francisco's architecture. Take a photo in front of these stunning homes. 10. **The Exploratorium**: A museum of science, art, and human perception, the Exploratorium is a great place to learn and have fun. With interactive exhibits and stunning views of the Bay, it's a must-visit for families and science enthusiasts. 11. **Pier 39**: A popular shopping and dining destination, Pier 39 offers stunning views of the Bay Bridge, Alcatraz Island, and the sea lions that call the pier home. 12. **The de Young Museum**: A fine arts museum located in Golden Gate Park, the de Young Museum features a diverse collection of art and cultural exhibitions from around the world. These are just a few of the many famous attractions in San Francisco. Whether you're interested in history, culture, science, or entertainment, San Francisco has something for everyone. ``` ## Step 5: See Traces in Langfuse After running the example model call, you can view the traces in Langfuse. These traces provide detailed information about your Novita AI API calls, including: * Request parameters (model, messages, temperature, etc.) * Response content * Token usage statistics * Latency metrics [Public example trace link in Langfuse](https://cloud.langfuse.com/project/cm7ua5l6e05wlad07qr6ce2wn/traces/039cc8b2-dba0-479f-9cd6-63672bc08c71?timestamp=2025-03-06T02%3A15%3A15.184Z). # LiteLLM Source: https://docs.novita.ai/guides/litellm Supercharge Your AI Applications with Novita AI and LiteLLM. LiteLLM is an open-source Python library and proxy server that provides access, spend tracking, and fallbacks to over 100 LLMs through a unified interface in the OpenAI format. By leveraging Novita AI's cutting-edge models, the integration with LiteLLM empowers your AI applications with seamless model switching, dependable fallbacks, and intelligent request routing—all through a standardized completion API that ensures compatibility across multiple providers. This guide will show you how to quickly get started with integrating Novita AI and LiteLLM, enabling you to set up this powerful combination and streamline your workflow with ease. ## **How to Integrate Novita AI with LiteLLM** ### Step 1: Install LiteLLM * Install the LiteLLM library using pip to create a unified interface for working with different language models. ```bash theme={"system"} pip install litellm ``` ### Step 2: Set Up Your API Credentials * Log in to [the key management page](https://novita.ai/settings/key-management) in Novita AI and click `Add New Key`to generate your API key. Visitthekeymanagementpageandselect Add New Keytogenerateyour AP Ikey Pn

Visitthekeymanagementpageandselect Add New Keytogenerateyour AP Ikey Pn

### Step 3: Structure Your Basic API Call * Create a completion request to Novita AI's models through LiteLLM's standardized interface. ```python theme={"system"} from litellm import completion import os ## set ENV variables. Visit https://novita.ai/settings/key-management to get your API key os.environ["NOVITA_API_KEY"] = "novita-api-key" response = completion( model="novita/deepseek/deepseek-v4.1-flash", messages=[{ "content": "Hello, how are you?","role": "user"}] ) ``` ### Step 4: Implement Streaming for Better User Experience * Enable streaming mode for more interactive applications or when handling longer responses. ```python theme={"system"} from litellm import completion import os ## set ENV variables. Visit https://novita.ai/settings/key-management to get your API key os.environ["NOVITA_API_KEY"] = "novita_api_key" response = completion( model="novita/deepseek/deepseek-v4.1-flash", messages = [{ "content": "Hello, how are you?","role": "user"}], stream=True, ) ``` # LlamaIndex Source: https://docs.novita.ai/guides/llamaindex Effortlessly integrate Novita AI with LlamaIndex to build intelligent, data-powered applications. Designed for optimal indexing and retrieval, LlamaIndex excels in delivering high efficiency for applications requiring precise and fast data access. By combining [Novita AI](https://novita.ai/) with LlamaIndex, you will unlock key benefits such as superior data retrieval accuracy, unmatched scalability, and cost-effective performance. This guide will walk you through how to use LlamaIndex with Novita AI based on the OpenAl APl, offering smarter, scalable, and highly efficient AI solutions that drive innovation and deliver exceptional results for developers. ## **How to Integrate Novita AI API with LlamaIndex** Step 1: Visit [Model Library](https://novita.ai/llm-api) on Novita AI and select a model of interest. images/Step1VisitModelLibraryonNovitaAIandselectamodelofinterest.png

images/Step1VisitModelLibraryonNovitaAIandselectamodelofinterest.png

Step 2: Navigate to the demo page of the chosen model and click the `Code` button on the right. images/Step2NavigatetothedemopageofthechosenmodelandclicktheCodebuttonontheright.png

images/Step2NavigatetothedemopageofthechosenmodelandclicktheCodebuttonontheright.png

Step 3: Copy the model’s name and make a note of it. Copy the model name and make a note of it

Step 4: [Log in ](https://novita.ai/user/login)to the Novita platform. images/Step4LogintotheNovitaplatform.png

Step 5: After logging in, go to the platform’s [settings page](https://novita.ai/settings). Open the platform settings page after logging in

Step 6: Create a new [API key](https://novita.ai/settings/key-management) and copy it for service authentication. images/Step6CreateanewAPIkeyandcopyitforserviceauthentication.png

images/Step6CreateanewAPIkeyandcopyitforserviceauthentication.png

Step 7: Install `llama_index` and related Python libraries by running: images/Step7Installllama_indexandrelatedPythonlibrariesbyrunning.png

images/Step7Installllama_indexandrelatedPythonlibrariesbyrunning.png

Step 8: Write Python code and set the model name and API key as parameters in the NovitaAI class. images/Step8WritePythoncodeandsetthemodelnameandAPIkeyasparametersintheNovitaAIclass.png

images/Step8WritePythoncodeandsetthemodelnameandAPIkeyasparametersintheNovitaAIclass.png

Step 9: Run the code to get the output. images/Step9Runthecodetogettheoutput.png

For more examples, refer to the documentation: [llama\_index/llama-index-integrations/llms/llama-index-llms-novita at main · run-llama/llama\_index](https://github.com/run-llama/llama_index/tree/main/llama-index-integrations/llms/llama-index-llms-novita). # Anthropic SDK Compatibility Source: https://docs.novita.ai/guides/llm-anthropic-compatibility Novita AI provides a compatibility API that allows you to use the Anthropic SDK with Novita AI models. This is useful if you are already using the Anthropic SDK and want to switch to Novita AI models. ## Supported Models Currently, only the following models are compatible with the Anthropic SDK: ## Quick Start Guide This guide demonstrates how to use the Anthropic SDK with the Novita AI models step by step. ### 1. Install the Anthropic SDK ```bash Python icon="python" theme={"system"} pip install anthropic ``` ```bash TypeScript icon="js" theme={"system"} npm install @anthropic-ai/sdk ``` ### 2. Initialize the Client The Anthropic SDKs are designed to pull the API key and base URL from the environmental variables: `ANTHROPIC_API_KEY` and `ANTHROPIC_BASE_URL`. Also, you can supply the parameters to the Anthropic client when initializing it. You can view and manage your API keys on the [settings page](https://novita.ai/settings/key-management?utm_source=getstarted). * Using Environment Variables ```bash Bash icon="terminal" theme={"system"} export ANTHROPIC_BASE_URL="https://api.novita.ai/anthropic" export ANTHROPIC_API_KEY="" ``` * Set the parameters while initializing the Anthropic client ```python Python icon="python" theme={"system"} import anthropic client = anthropic.Anthropic( base_url="https://api.novita.ai/anthropic", api_key="", # Rewrite header default_headers={ "Content-Type": "application/json", "Authorization": "Bearer ", } ) ``` ```typescript TypeScript icon="js" theme={"system"} import Anthropic from "@anthropic-ai/sdk"; const anthropic = new Anthropic({ baseURL: "https://api.novita.ai/anthropic", apiKey: "", // Rewrite header defaultHeaders: { "Content-Type": "application/json", Authorization: `Bearer `, } }); ``` ### 3. Call the API ```python Python icon="python" theme={"system"} import anthropic # Initialize the client, if you already set `ANTHROPIC_BASE_URL` and `ANTHROPIC_API_KEY` # in the environment variables, you can omit the `api_key` and `base_url` parameters. client = anthropic.Anthropic( base_url="https://api.novita.ai/anthropic", api_key="", # Rewrite header default_headers={ "Content-Type": "application/json", "Authorization": "Bearer ", } ) message = client.messages.create( model="moonshotai/kimi-k2-instruct", max_tokens=1000, temperature=1, system=[ { "type": "text", "text": "You are a world-class poet. Respond only with short poems." } ], messages=[ { "role": "user", "content": [ { "type": "text", "text": "Why is the ocean salty?" } ] } ] ) print(message.content) ``` ```typescript TypeScript icon="js" theme={"system"} import Anthropic from "@anthropic-ai/sdk"; // Initialize the client, if you already set `ANTHROPIC_BASE_URL` and `ANTHROPIC_API_KEY` // in the environment variables, you can omit the `baseURL` and `apiKey` parameters. const anthropic = new Anthropic({ baseURL: "https://api.novita.ai/anthropic", apiKey: "", // Rewrite header defaultHeaders: { "Content-Type": "application/json", Authorization: `Bearer `, } }); const msg = await anthropic.messages.create({ model: "moonshotai/kimi-k2-instruct", max_tokens: 1000, temperature: 1, system="You are a world-class poet. Respond only with short poems.", messages: [ { role: "user", content: [ { type: "text", text: "Why is the ocean salty?" } ] } ] }); console.log(msg); ``` # Large Language Models Source: https://docs.novita.ai/guides/llm-api ## Model Capabilities Large Language Models (LLMs) are types of AI model built using deep learning and natural language processing techniques. After being trained on massive amounts of text data, they can understand, generate, and process human language. LLMs have the following main capabilities: * **Text Generation** LLMs can generate logically coherent content based on context and adjust output style as needed. * **Language Understanding** LLMs can accurately understand the meaning of input text and support context-aware conversation. * **Text Translation** LLMs are capable of cross-lingual understanding and generation, enabling translation between different languages. * **Knowledge-based Q\&A** With extensive knowledge basis, LLMs can answer questions across domains such as culture, science, and history. * **Code Understanding & Generation** LLMs can understand and generate code (Python, Java, C++ and so on), detect code errors, and offer suggestions. * **Text Classification & Summarization** LLMs can understand complex sentences, categorize and extract information, and summarize key points. ## Model Selection On the [LLM Service Page](https://novita.ai/models), you can browse the list of supported models, learn model basics and pricing. Click on a specific model, and you can open its detail page for an online demo. After trying different models with your task, you can compare and choose the most suitable one. ## API Integration Novita AI provides fast, reliable APIs for open-source models, combining the ease and reliability of top-tier LLM APIs with the flexibility and cost-efficiency of open source LLMs. * [ChatCompletion](https://platform.openai.com/docs/guides/text-generation/chat-completions-api): Supports both streaming and non-streaming modes. * [Completion](https://platform.openai.com/docs/guides/text-generation/completions-api): Also supports streaming and non-streaming modes. If you're already using OpenAI's `ChatCompletion` or `Completion` API, simply set your base URL to: api.novita.ai/openai , get and set your API key, update the model name as needed to start using LLM services. For how to create, store, and use an API key, see [API Keys](/guides/llm-api-keys). ## Code Examples ### Python ```python ChatCompletion theme={"system"} from openai import OpenAI client = OpenAI( base_url="https://api.novita.ai/openai", api_key="", ) model = "deepseek/deepseek-v4.1-flash" stream = True # or False max_tokens = 512 chat_completion_res = client.chat.completions.create( model=model, messages=[ { "role": "system", "content": "you are a professional AI helper.", }, { "role": "user", "content": "Where can the example of GPU provided by novita ai be adapted?", } ], stream=stream, max_tokens=max_tokens, ) if stream: for chunk in chat_completion_res: print(chunk.choices[0].delta.content or "", end="") else: print(chat_completion_res.choices[0].message.content) ``` ```python Completion theme={"system"} from openai import OpenAI client = OpenAI( base_url="https://api.novita.ai/openai", api_key="", ) model = "deepseek/deepseek-v4.1-flash" stream = True # or False max_tokens = 512 completion_res = client.completions.create( model=model, prompt="Where can the example of GPU provided by novita ai be adapted?", stream=stream, max_tokens=max_tokens, ) if stream: for chunk in completion_res: print(chunk.choices[0].text or "", end="") else: print(completion_res.choices[0].text) ``` ### Curl ```bash ChatCompletion theme={"system"} export API_KEY="" curl "https://api.novita.ai/openai/v1/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${API_KEY}" \ -d '{ "model": "deepseek/deepseek-v4.1-flash", "messages": [ { "role": "system", "content": "you are a professional AI helper." }, { "role": "user", "content": "Where can the example of GPU provided by novita ai be adapted?" } ], "max_tokens": 512 }' ``` ```bash Completion theme={"system"} export API_KEY="" curl "https://api.novita.ai/openai/v1/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${API_KEY}" \ -d '{ "model": "deepseek/deepseek-v4.1-flash", "prompt": "Where can the example of GPU provided by novita ai be adapted?", "max_tokens": 512 }' ``` ## Key Parameters ### Base Parameters `model`: The target LLM. See the full list on the [LLM Service Page](https://novita.ai/models). ### Messages > ChatCompletion Only `messages`: Inputs and outputs when interacting with LLM. Each message belongs to one character. Messages can help you get better outputs, you can try different methods to get better results. * `content`: message content. * `role`: Define the speaker: * `system`: Set the AI's role, telling model its role or behavior. * `user`: The user's input. * `assistant`: The AI's response. Users can give examples of output in advance, telling model how to answer current request. * `name` (optional): separating message writers with same role. ### Prompt > ChatCompletion Only `prompt`: Instruction text for generating a completion written by user to LLM to tell the model their task clearly. ### Output Control Different combination of parameters can help the model to generate more satisfying content. **Diversity of Text** > 'temperature' and 'top\_p' can control the diversity of generated text, we advice you to set only one of them. The bigger the value you set, The more diverse the text will be. The smaller the value you set, The more accurate the text will be. * `temperature`: Controls randomness. Higher = more creative. * `top_p`: Nucleus sampling. Controls cumulative probability. * `top_k`: Limits candidate token count. **Repetition Penalties** * `presence_penalty`: Controls repeated tokens of the texts. If one token has already existed in the text, penalty will come, this results in more token in the text. * `frequency_penalty`: Control token frequency of the texts. Every time the same token exist in the text, penalty will come, which results in less same token in the future in the text. * `repetition_penalty`: Penalizes or encourages repetition. ### Output Limits * `max_tokens`: Max tokens in one response. Content after this value in the text will be truncated. * `stream`: Whether to use streaming. For those with long outputs, we advice you to use streaming mode to prevent timeout error: * `true`: Streamed, partial results as they are generated. * `false`: Full result after processing. * `stop`: Strings that will terminate generation when encountered. # API Keys Source: https://docs.novita.ai/guides/llm-api-keys An API key authenticates requests to Novita AI. This page explains how to create and store a key, use it across environments, and configure its access controls. Use this page to: * Authenticate requests to Novita AI with a Bearer API key. * Create an API key in the console and store it safely. * Set the key as an environment variable on Linux, macOS, or Windows. * Check how long a key stays valid and which key operations the OpenAPI supports. * Configure model and network access policies for a key. ## Authentication Novita AI uses Bearer authentication for API access. Send your API key in the `Authorization` request header: ``` Authorization: Bearer ``` An example request: ```bash theme={"system"} curl "https://api.novita.ai/openai/v1/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $NOVITA_API_KEY" \ -d '{ "model": "deepseek/deepseek-v4.1-flash", "messages": [{"role": "user", "content": "Hello"}] }' ``` ## Create an API key Go to [Key Management](https://novita.ai/settings/key-management?utm_source=getstarted) in the console. Select **Create API Key**, then give the key a name that reflects its purpose, such as `production` or `local-testing`. The full key is shown only once, when it is created. Copy it immediately and store it in a secure location, such as a secrets manager or an environment variable. If you lose it, you cannot recover it; create a new key instead. ## Configure access controls Access controls are optional and configured per API key: * [Model Access for API Keys](/guides/llm-model-access) controls which models a key can call within your team's enabled model range. * [Network Access for API Keys](/guides/llm-network-access) controls which source IPs can use a key for model invocation. The dedicated guides explain policy behavior, permissions, configuration, and rejection errors. ## Key format and validity * Every key begins with the `sk_` prefix. * A key is shown in full only once, when it is created. After that, the console shows a masked form. * Each account can create up to **10** API keys. When you create a key, choose one of four expiration options: **Permanent / 90 days / 30 days / 24 hours**. You set the expiration when you create the key. It **cannot be changed** afterward: you cannot renew a key or expire it immediately. **Permanent** keys remain valid indefinitely, so a leaked key remains usable for longer. Use permanent keys only when appropriate. Expiration is configured in the console only; the OpenAPI does not create keys or set, change, or remove their expiration. An expired key is rejected when used, but its configuration remains visible and the delete action is still available. Expiration is different from disabling a key. To restore access, create a new key. To disable a key before it expires, delete it in the console; the deletion takes effect immediately. ## What the OpenAPI covers You create and delete API keys in the console only. The Novita OpenAPI has no endpoints for creating or deleting keys, and it does not support configuring a key's expiration. The key-related OpenAPI endpoints let you list keys and manage their model and network access policies: * [List API Keys](/api-reference/key-list-with-model-access) — list the keys on your team, with an optional model access summary. * [Get API Key Model Access Policy](/api-reference/key-get-model-access-policy) — read a single key's model access policy. * [Set API Key Model Access Policy](/api-reference/key-put-model-access-policy) — set or update a key's model access policy. * [Reset API Key Model Access Policy](/api-reference/key-delete-model-access-policy) — reset a key's model access policy to the default. This resets the policy only; it does not delete the key itself. * [Get API Key IP Access Policy](/api-reference/key-get-ip-access-policy) — read a single key's network access policy. * [Set API Key IP Access Policy](/api-reference/key-put-ip-access-policy) — set or update a key's allowed source IPs. * [Delete API Key IP Access Policy](/api-reference/key-delete-ip-access-policy) — clear a key's network access policy, restoring no source-IP restriction. ## Store your key as an environment variable Hardcoding a key in source code can expose it, for example when you commit the file. Reading it from an environment variable such as `NOVITA_API_KEY` keeps it out of your source code. ### Temporary vs. permanent A key set with `export` (Linux/macOS) or `set` (Windows) lasts only for the current terminal session and is gone when you close it. This is suitable for a quick test. To keep the key across sessions, set it permanently as shown below, then open a new terminal for the change to take effect. ```bash Linux theme={"system"} # Temporary: current session only export NOVITA_API_KEY="" # Permanent: append to your shell profile, then reload echo 'export NOVITA_API_KEY=""' >> ~/.bashrc source ~/.bashrc ``` ```bash macOS theme={"system"} # Temporary: current session only export NOVITA_API_KEY="" # Permanent: append to your shell profile, then reload # Newer macOS uses zsh (~/.zshrc); older setups use bash (~/.bash_profile) echo 'export NOVITA_API_KEY=""' >> ~/.zshrc source ~/.zshrc ``` ```powershell Windows theme={"system"} # Temporary: current PowerShell session only $env:NOVITA_API_KEY = "" # Permanent: persist for the current user, then open a new terminal setx NOVITA_API_KEY "" ``` Read the key from the environment in your code: ```python Python theme={"system"} import os from openai import OpenAI client = OpenAI( base_url="https://api.novita.ai/openai", api_key=os.environ.get("NOVITA_API_KEY"), ) ``` ```javascript Node.js theme={"system"} import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://api.novita.ai/openai", apiKey: process.env.NOVITA_API_KEY, }); ``` ### The variable is set but the code still can't find it A key set with `export` or `$env:` exists only in the terminal session where you ran the command. A new terminal or tab does not inherit it. Set the key permanently (`>> ~/.zshrc`, `setx`), or run the `export`/`$env:` command again in the session you are using. A permanent change (a shell profile or `setx`) applies to terminals started after the change. Open a new terminal, then restart your IDE or editor so it can read the updated environment. On Windows, `setx` does not affect terminals that are already open. Processes launched by `systemd`, `supervisor`, Docker, or a CI runner do not read your interactive shell profile. Set the variable in the service's own configuration, such as a `systemd` unit's `Environment=`, a `docker run -e` flag, or the CI project's secrets, rather than in `~/.bashrc`. `sudo` does not pass your environment through by default, so a variable exported by your user is not visible to the elevated process. Use `sudo -E` to preserve the environment, or set the variable in the elevated context. ## Related * [Common Error Codes](/guides/error) — resolve `401` and `403` responses related to keys. * [Rate limits](/guides/llm-rate-limits) — request and token limits that apply to your account. # Batch Inference Source: https://docs.novita.ai/guides/llm-batch-api The Batch API for Large Language Models enables asynchronous processing of numerous inference requests and is fully compatible with the OpenAI API standard. The Batch API is a cost-effective solution when immediate inference results are unnecessary. It provides a higher rate limits than online calls, ensuring results are delivered within a reasonable timeframe of 24 hours. This API is ideal for: * Conducting evaluations and data analysis. * Classifying extensive datasets. * Generating document summaries in an offline mode. Supported Models: ## Quick Start ### 1. Preparing Batch Files The Batch API uses .jsonl format files as input, with each line representing the details of an API inference request. Available endpoints include `/v1/chat/completions` and `/v1/completions`. Set the `endpoint` parameter to `/v1/chat/completions` or `/v1/completions` for OpenAI API compatibility. Each request must include a unique `custom_id` to locate inference results in the output file after batch completion. Parameters in the `body` field of each line are sent as actual inference request parameters to the endpoint. All requests within a single batch JSONL file must target the same model. Do not mix requests for different models in one batch. Below is an example input file containing 2 requests: ```JSON theme={"system"} {"custom_id": "request-1", "body": {"model": "deepseek/deepseek-r1-0528", "messages": [{"role": "user", "content": "Hello, world!"}], "max_tokens": 400}} {"custom_id": "request-2", "body": {"model": "deepseek/deepseek-r1-0528", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}} ``` ### 2. Upload Batch Input File Upload the batch input file to ensure it can be accurately referenced when creating a batch. Use the Files API to upload your .jsonl file and set the purpose to `batch`. Note that the file will be retained for 15 days. For how to get API key, refer to the [API Key Management](/api-reference/basic-authentication). Code Example **Python** ```python theme={"system"} from openai import OpenAI client = OpenAI( base_url="https://api.novita.ai/openai/v1", api_key="", ) batch_input_file = client.files.create( file=open("batch_input.jsonl", "rb"), purpose="batch", ) print(batch_input_file) ``` **Curl** ```bash theme={"system"} export API_KEY="" curl --request POST \ --url https://api.novita.ai/openai/v1/files \ --header 'Authorization: Bearer ${API_KEY}' \ --form 'file=@"/your/batch_input.jsonl"' \ --form 'purpose="batch"' ``` Sample response upon successful file upload: ``` { "id": "file_d2co***as73c0cjd0", "object": "file", "bytes": 238, "filename": "batch_input.jsonl", "created_at": 1754894162, "purpose": "batch", "metadata": { "total_requests": 2 } } ``` ### 3. Creating a Batch Once the input file is successfully uploaded, you can initiate a batch using the ID of the uploaded File object. The completion window is fixed at `24h` and is currently non-adjustable. Code Example **Python** ```python theme={"system"} from openai import OpenAI client = OpenAI( base_url="https://api.novita.ai/openai/v1", api_key="", ) batch = client.batches.create( input_file_id="file_d2cor0es1cas73c0cj60", endpoint="/v1/chat/completions", completion_window="24h" ) print(batch) ``` **Curl** ```bash theme={"system"} export API_KEY="" curl --request POST \ --url https://api.novita.ai/openai/v1/batches \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ${API_KEY}' \ --data '{ "input_file_id": "file_d2co***as73c0cjd0", "endpoint": "/v1/chat/completions", "completion_window": "24h" }' ``` This request will yield a Batch object that includes metadata about your batch, as illustrated in the example below: ```JSON theme={"system"} { "id": "batch_d2cq***73a68lu0", "object": "batch", "endpoint": "/v1/chat/completions", "input_file_id": "file_d2co***as73c0cjd0", "output_file_id": "", "error_file_id": "", "completion_window": "24h", "in_progress_at": null, "expires_at": null, "finalizing_at": null, "completed_at": null, "failed_at": null, "expired_at": null, "cancelling_at": null, "cancelled_at": null, "status": "validating", "errors": "", "version": 0, "created_at": "2025-08-11T16:31:52.949816948+08:00", "updated_at": null, "created_by": "8f242aa1-f725-4a67-8***9-cb68025e0976", "created_by_key_id": "key_cc19f96c***e7390644a37da21", "remark": "", "total": 0, "completed": 0, "failed": 0, "metadata": null, "request_counts": { "total": 0, "completed": 0, "failed": 0 } } ``` ### 4. Check the Status of a Batch You can check a batch's status at any time to receive the latest batch information. The status enumeration values of the Batch object are as follows:

Status	Description
VALIDATING	The input file is being validated before the batch can begin
PROGRESS	Batch is in progress
COMPLETED	Batch processing completed successfully
FAILED	Batch processing failed
EXPIRED	Batch exceeded deadline
CANCELLING	Batch is being cancelled
CANCELLED	Batch was cancelled

Code Example **Python** ```python theme={"system"} from openai import OpenAI client = OpenAI( base_url="https://api.novita.ai/openai/v1", api_key="", ) batch = client.batches.retrieve("batch_d2cq***73a68lu0") print(batch) ``` **Curl** ```bash theme={"system"} export API_KEY="" curl --request GET \ --url https://api.novita.ai/openai/v1/batches/{batch_id} \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ${API_KEY}' ``` ### 5. Retrieve the Results Once the batch inference is complete, you can download the result output file using the `output_file_id` field from the Batch object. The result output file will be deleted 30 days after the batch inference concludes, so please retrieve it promptly via the interface. Code Example **Python** ```python theme={"system"} from openai import OpenAI client = OpenAI( base_url="https://api.novita.ai/openai/v1", api_key="", ) content = client.files.content("example-250811-1") print(content.read()) ``` **Curl** ```bash theme={"system"} export API_KEY="" curl --request GET \ --url https://api.novita.ai/openai/v1/files/{file_id}/content \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ${API_KEY}' ``` The response returns the raw file content. For batch output files, each line contains a response like this: ```json theme={"system"} { "custom_id": "request-2589", "error": null, "id": "batch_req_task_d2c", "response": { "body": { "id": "29e1432c-edfb-44a4-b531-c23c600abfae", "object": "chat.completion", "created": 1754902266, "model": "deepseek-test", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "Hello! 👋 How can I assist you today? 😊" }, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 5, "completion_tokens": 13, "total_tokens": 18 } }, "request_id": "request-2589", "status_code": 200 } } ``` ## Instructions ### Limitations 1. Each batch can contain up to 50,000 requests.
2. The maximum input file size per batch is 100MB. ### Error Handling Errors encountered during batch processing are recorded in a separate error file, accessible via the error\_file\_id field. Common error codes include:

Error Code	Description	Solution
400	Invalid request format	Check JSONL syntax and required fields
401	Authentication failed	Verify API key
404	Batch not found	Check batch ID
429	Rate limit exceeded	Reduce request frequency
500	Server error	Contact us

### Batch Expiration Batches not completed within 24 hours will transition to an EXPIRED state. Unfinished requests will be canceled, while completed requests will be provided through an output file. You only pay for tokens consumed by completed requests. The batch makes every effort to complete within 24 hours. ## All Batch API 1. [Create batch](/api-reference/model-apis-llm-create-batch) 2. [Retrieve batch](/api-reference/model-apis-llm-retrieve-batch) 3. [Cancel batch](/api-reference/model-apis-llm-cancel-batch) 4. [List batch](/api-reference/model-apis-llm-list-batches) 5. [Upload file](/api-reference/model-apis-llm-upload-batch-input-file) 6. [List files](/api-reference/model-apis-llm-list-files) 7. [Retrieve file](/api-reference/model-apis-llm-query-file) 8. [Delete file](/api-reference/model-apis-llm-delete-file) 9. [Retrieve file content](/api-reference/model-apis-llm-retrieve-file-content) # LLM API Billing Source: https://docs.novita.ai/guides/llm-billing Understand how novita AI LLM API billing works, including token-based pricing, HTTP status code billing rules, and the 499 client disconnection billing policy. ## Pricing novita AI LLM API is billed per token. Each request incurs charges for: * **Input Tokens** — tokens consumed by the prompt * **Output Tokens** — tokens generated by the model Total Cost = Input Tokens × Input Rate + Output Tokens × Output Rate Rates vary by model. Refer to the [Pricing Page](https://novita.ai/pricing) for details. ## HTTP Status Codes | HTTP Status | Name | Description | Charged | | :- | :- | :- | :- | | 200 | Success | Request completed successfully | **Yes** | | 400 | Bad Request | Invalid request parameters | No | | 401 | Unauthorized | Invalid or missing API key | No | | 403 | Forbidden | Insufficient permissions | No | | 429 | Rate Limited | TPM or RPM limit exceeded | No | | 499 | Client Disconnected | Client closed the connection before completion | **Yes** | | 500 | Internal Server Error | Server-side error | No | | 503 | Service Unavailable | Service temporarily unavailable | No | | 504 | Gateway Timeout | Upstream response timed out | No | **Billing principle:** * Requests rejected before reaching the model (400/401/403/429): not charged * Error due to platform issues (500/503/504): not charged * Requests where the model has begun inference (200/499): charged ## 499 — Client Disconnected Once a request reaches the model, inference begins immediately on the server side. If the client disconnects before receiving the full response, compute resources have already been consumed. The request is charged based on actual token usage. | Request Mode | Billing | | :- | :- | | Non-Streaming | Charged for actual token usage regardless of disconnect timing | | Streaming | Charged for actual token usage regardless of disconnect timing | **Recommendations** * Use `max_tokens` to limit output length at the request level * Set client timeout to ≥ 60s to avoid unintended disconnections * Prefer controlling generation via `max_tokens` rather than closing the connection ## FAQ Inference begins as soon as the request reaches the model. Disconnecting does not halt computation already in progress. Set the `max_tokens` parameter to control output length before sending the request. In your usage dashboard, alongside standard requests, with actual token consumption recorded. # Novita Deployments User Guide Source: https://docs.novita.ai/guides/llm-dedicated-endpoint > **Navigation**: Models Console → Deployments > **Applies to**: Current production version > **Last updated**: 2026-04-20 *** ## Table of Contents 1. [What are Deployments](#1-what-are-deployments) 2. [Quick Start (Go live in 5 minutes)](#2-quick-start-go-live-in-5-minutes) 3. [Creating a Deployment](#3-creating-a-deployment) * 3.1 [Naming](#31-naming) * 3.2 [Selecting a Model](#32-selecting-a-model) * 3.3 [Selecting a GPU Instance](#33-selecting-a-gpu-instance) * 3.4 [Configuring Autoscaling](#34-configuring-autoscaling) * 3.5 [Engine Settings (Advanced)](#35-engine-settings-advanced) 4. [Deployment Lifecycle & Status](#4-deployment-lifecycle--status) 5. [Autoscaling In Depth](#5-autoscaling-in-depth) 6. [LoRA Adapter Support](#6-lora-adapter-support) 7. [Billing](#7-billing) 8. [FAQ & Troubleshooting](#8-faq--troubleshooting) *** ## 1. What are Deployments A Deployment is Novita's **dedicated AI inference endpoint** product. Unlike Serverless endpoints that share compute resources with other users, each Deployment gives you: * **Exclusive GPU**: All compute resources are yours alone — no noisy neighbors * **Predictable Performance SLA**: Dedicated compute means consistent, foreseeable inference latency * **Flexible Model Sources**: Deploy any model from Hugging Face or the Novita model catalog * **OpenAI-Compatible Chat API**: For pure text inference, simply swap `base_url` and `model` to migrate existing OpenAI integrations * **Per-second Billing**: You are only charged while the endpoint is active. Billing pauses automatically when Scale-to-Zero kicks in **When to use Deployments:** | Use Case | Why it fits | | - | - | | Production API services | Stable latency, fully isolated from other users | | Private or fine-tuned model serving | Deploy any custom HuggingFace model | | High-concurrency inference | Scale to multiple replicas automatically | | Cost-sensitive workloads | Scale-to-Zero stops billing during idle periods | *** ## 2. Quick Start (Go live in 5 minutes) **Step 1 — Navigate to Deployments** Log in to Novita → left sidebar → **Models Console** → **Models APIs** → **Deployments** **Step 2 — Create a Deployment** Click **+ New Deployment** and fill in: * A Deployment name (e.g. `my-llama3-endpoint`) * Model source (the Novita model catalog is recommended for the fastest setup) * GPU instance (the system auto-recommends a suitable spec for your model) * Autoscaling settings **Step 3 — Wait for the Deployment to start** Startup time varies with model size, typically **5–60 minutes**, progressing through three phases: 1. Requesting GPU 2. Downloading Model 3. Engine Initializing Once the status shows **RUNNING**, the endpoint is ready to receive requests. **Step 4 — Call the API** Go to the Deployment detail page → **Quick Start** panel → copy the ready-to-run code snippet. > Manage your API Keys under **Settings → API Keys**. *** ## 3. Creating a Deployment Click **+ New Deployment** to open the creation form, which has four configuration sections. ### 3.1 Naming Recommended naming format: `{model}-{environment}-{purpose}` — e.g. `llama3-prod-chatbot`. ### 3.2 Selecting a Model Two model sources are supported: #### Novita Model Catalog (Recommended) Choose from Novita's hosted model list — no token required, **works out of the box**. Covers all major open-source models (Llama 3, Qwen, DeepSeek, Mistral, and more). > Novita pre-validates model compatibility and applies engine optimizations, resulting in faster startup and higher stability. #### Hugging Face Models Enter a HuggingFace repository ID (e.g. `meta-llama/Meta-Llama-3-8B-Instruct`). * **Public models**: No token needed, deploy directly * **Private or Gated models**: A HuggingFace Access Token must be linked first **How to link your HF Token:** 1. Go to [HuggingFace → Settings → Access Tokens](https://huggingface.co/settings/tokens) and create a token 2. In the Model field on the Create Deployment form, click **Integrate HF Token** 3. Paste and save the token > If your token expires or is revoked, active Deployments that rely on it will fail to re-pull the model. Keep your token up to date. #### LoRA Adapter (Optional) After selecting a Base Model, you can attach one or more LoRA Adapters from HuggingFace. Multiple adapters can run on the same Deployment without requiring additional GPU resources. See [Section 6 — LoRA Adapter Support](#6-lora-adapter-support) for details. **Model file format requirements (for custom HuggingFace models):** ### 3.3 Selecting a GPU Instance The system automatically recommends a GPU configuration based on your model size. > **TIGHT MEMORY warning**: If the selected GPU has limited VRAM for the chosen model, the system shows a `TIGHT MEMORY` warning. Increase the GPU count or contact Novita support. > GPU type **cannot be changed** after a Deployment is created. To switch GPU type, delete and recreate the Deployment. *** ### 3.4 Configuring Autoscaling Autoscaling controls how many replicas run in response to traffic. #### Enable Autoscaling (Recommended) Use the dual-handle slider to set the replica range: | Parameter | Description | Default | | - | - | - | | Min Replicas | Minimum active replicas at all times. Set to 0 to enable Scale-to-Zero | 1 | | Max Replicas | Maximum replicas during peak traffic | 3 | | Scale-down Delay | Seconds to wait after traffic drops before scaling down (prevents flapping) | 300s (minimum) | **Scale-to-Zero (Min Replicas = 0):** * After idling for longer than the Scale-down Delay, the Deployment enters **SLEEPING** status and billing pauses * The first incoming request wakes it up automatically * Cold start time: typically 5 minutes depending on model size * ⚠️ Best suited for dev/test or low-frequency workloads. For production, keep Min Replicas ≥ 1 #### Disable Autoscaling Runs a fixed number of replicas. Best for workloads with strict latency SLAs that cannot tolerate any scaling delay. ### 3.5 Engine Settings (Advanced) Novita supports two inference engines — **vLLM** and **SGLang** — matched automatically to your model. These settings are hidden by default during Deployment creation. #### Max Concurrency per Replica Controls how many requests a single replica handles simultaneously. | Setting | Effect | | - | - | | Below recommended | Lower latency, but limited throughput | | Equal to recommended | Optimal balance of throughput and latency (recommended) | | Above recommended | Higher throughput, but increased per-request latency | > The system calculates a recommended value based on your GPU instance. Default is 16. #### Suffix Decoding N-gram based speculative decoding that pre-generates future tokens to speed up inference. * Most effective for **highly predictable output formats** (e.g. code generation, structured JSON) * Provides limited benefit for free-form conversation; excessively high values may actually increase latency *** ## 4. Deployment Lifecycle & Status ### State Transition Diagram ```text theme={"system"} Create │ ▼ PENDING ──── Waiting for GPU resource allocation │ ▼ DEPLOYING ── Three sub-phases: │ ├─ Requesting GPU │ ├─ Downloading Model │ └─ Engine Initializing │ ├──────────────── FAILED (deployment failed) │ ▼ RUNNING ──── Live and accepting requests │ ├─ Zero traffic + Scale-to-Zero enabled ──► SLEEPING │ │ │ First request ──► DEPLOYING ──► RUNNING │ ├─ Config update ──► ROLLING (zero-downtime rolling update) │ ├─ Traffic change ──► SCALING (autoscaling in progress) │ └─ Manual terminate ──► TERMINATING ──► TERMINATED (can be redeployed or deleted) ``` > **When billing starts**: Only running replicas are billed. Instances still deploying and replicas still scaling up do not count toward charges. *** ## 5. Autoscaling In Depth ### How It Works Novita autoscaling monitors live traffic and dynamically adjusts replica count within the Min–Max range: * **Scale-Up**: Request queue backlog detected → add replicas → more GPUs handle requests in parallel * **Scale-Down**: Traffic drops → wait for Scale-down Delay to expire → reduce replicas * **Scale-to-Zero**: When Min Replicas = 0 and the Deployment has been idle past the delay, it enters SLEEPING and billing stops ### Cost vs. Availability Trade-off | Configuration | Cost | Availability | Best for | | - | - | - | - | | Min=0, Max=N | Lowest (no charge when idle) | Cold start delay (5 min) | Dev/test, low-frequency workloads | | Min=1, Max=N | Medium | Always available, scales on demand | Most production workloads ✅ | | Min=N, Max=N (fixed) | Highest | No scaling delay at all | Ultra-low latency SLA requirements | ### Per-Replica Cost Each additional replica adds cost at the same GPU rate as the base replica. Example: a 2× H100 Deployment that scales to 2 replicas doubles the GPU cost. ### Best Practices * Set **Min Replicas = 1** in production to avoid cold starts impacting end users * The default Scale-down Delay of 300s (5 minutes) works well for most cases; increase it if your traffic is highly variable * Set Max Replicas to no more than 1.5× your expected (peak QPS / per-replica QPS) to avoid unexpected cost spikes *** ## 6. LoRA Adapter Support ### What is LoRA LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique that adds lightweight adapter layers on top of a Base Model to customize it for specific tasks — without retraining the full model. ### Using LoRA in Novita Deployments **Adding adapters at creation time:** In Create Deployment → Model field → after selecting a Base Model, click **+ Add Adapter** and enter the LoRA adapter's HuggingFace repository ID. **Viewing adapters at runtime:** In the Engine Configuration panel, a `+N LoRA` badge appears next to the Model ID. Hover over it to see the full list of attached adapters. ### Multi-LoRA: Multiple Adapters on One Deployment A single Deployment can run multiple LoRA adapters simultaneously. Specify which adapter to use per request via the `model` field: > Multi-LoRA requires no extra GPU resources. All adapters share a single copy of the Base Model weights in memory. *** ## 7. Billing ### Billing Unit Charged by **GPU-second**: number of GPUs × seconds running × unit price. ### When Billing Starts and Stops | Event | Details | | - | - | | **Billing starts** | After GPU allocation completes during DEPLOYING (i.e. when Downloading Model begins) | | **Billing stops** | When the Deployment enters SLEEPING or TERMINATED status | | **Continuous billing** | A RUNNING Deployment is billed even when it receives zero API requests | ### GPU Pricing > For the latest pricing, refer to the [Novita pricing page](https://novita.ai/pricing). ### Billing Example **Scenario**: A customer deploys model instance X on a single RTX 4090 (priced at \$0.61/GPU/hour), with autoscaling set to Min=0, Max=5. Usage and charges for 9:00–10:00: 1. **9:00:00 – 9:15:40** — Instance is SLEEPING. Charge: **\$0.00** 2. **9:15:41 – 9:16:45** — 1 running replica serving traffic (65 seconds). Charge: ($0.61 ÷ 3600) × 1 replica × 65s = **$0.011\*\* 3. **9:16:46 – 10:00:00** — 2 running replicas serving traffic (1,994 seconds). Charge: ($0.61 ÷ 3600) × 2 replicas × 1,994s = **$0.676\*\* **Total for 9:00–10:00: $0 + $0.011 + $0.676 = $0.687** ### Cost Control Tips 1. **Enable Scale-to-Zero** (Min Replicas = 0) for low-frequency workloads — zero cost when idle 2. **Audit your Deployment list regularly** and delete unused Deployments 3. **Cap Max Replicas conservatively** to prevent unexpected cost spikes from runaway autoscaling 4. **TERMINATED status costs nothing** — terminate and redeploy on demand *** ## 8. FAQ & Troubleshooting ### Deployment Issues **Q: My Deployment has been stuck in DEPLOYING for a long time — what should I do?** * `Requesting GPU`: GPU resources may be constrained. Wait 5–10 minutes, or try a different GPU type * `Downloading Model`: Large models (70B+) can take 10+ minutes to download * `Engine Initializing`: Should complete within 5 minutes under normal conditions **Q: My Deployment shows FAILED — what are the common causes?** * Model is not in `.safetensors` format (`.bin` is not supported) * HuggingFace Token is invalid or lacks access to a gated model * Insufficient GPU VRAM for the model (TIGHT MEMORY configuration) * Model architecture is not yet supported Debugging steps: check the change log in the Settings Tab → verify model file format → validate the HF Token → increase GPU count and recreate the Deployment. **Q: My Deployment is SLEEPING — how do I wake it up?** Send any API request to it. The Deployment wakes up automatically. The first request waits for the cold start to complete before receiving a response. *** ### API Issues **Q: What do the common HTTP error codes mean?** | Code | Cause | Resolution | | - | - | - | | `400` | Malformed request | Validate your request JSON; ensure all required fields are present | | `401` | Missing or invalid API Key | Include a valid key in `Authorization: Bearer ` | | `403` | API Key lacks access to this endpoint | Confirm the key belongs to the same account that owns the Deployment | | `404` | Wrong Endpoint URL or Model ID | Re-copy the URL and Model ID from the Quick Start panel | | `422` | Invalid parameter value (e.g. max\_tokens too large) | Adjust the parameter — try reducing max\_tokens | | `429` | Rate limit exceeded | Reduce request frequency, or contact Novita to raise your limit | | `500` | Internal server error | Retry after a short wait; if it persists, contact Novita support | **Q: Where do I find my API Key?** Go to **Settings → API Keys** to create or manage keys. A key is only shown once at creation — save it immediately. *** ### Billing Issues **Q: Why am I being charged when there are no requests?** A RUNNING Deployment continuously occupies GPU resources regardless of request volume. **Fix**: Enable Autoscaling and set Min Replicas = 0. The Deployment will automatically sleep and stop billing when idle. **Q: How do I stop all charges completely?** Two options: * **Scale-to-Zero**: Let autoscaling trigger naturally (requires Autoscaling on with Min = 0) * **Terminate**: Click **Terminate** on the Deployment detail page to release the GPU immediately *** ## Appendix: Glossary | Term | Definition | | - | - | | Deployment | Novita's dedicated inference endpoint product | | Replica | A single running instance of the inference service; multiple replicas run in parallel | | Scale-to-Zero | Setting Min Replicas to 0 so the endpoint sleeps when idle and billing stops | | Scale-down Delay | Wait period before scaling down, preventing flapping on variable traffic | | LoRA Adapter | Lightweight fine-tuning plugin layered on top of a Base Model | | Endpoint URL | The API access address for this Deployment | | Endpoint ID | Unique identifier for this Deployment | | Base Model | The underlying foundation model being served | | Max Concurrency | Maximum simultaneous requests a single replica handles | | Suffix Decoding | N-gram speculative decoding to accelerate inference on predictable outputs | | GPU-second | Billing unit: 1 GPU running for 1 second | *** *For support, contact the Novita team at: [support@novita.ai](mailto:support@novita.ai)* # DeepSeek-OCR Usage Source: https://docs.novita.ai/guides/llm-deepseek-ocr ## Overview DeepSeek-OCR is designed for document recognition and image-to-text scenarios, pushing the limits of visual and textual compression. The model can render long texts into highly compressed images, achieving an OCR accuracy of 97% at a lossless compression ratio of 10x and around 60% accuracy at 20x compression. This model currently supports only single-turn, independent recognition tasks. It does not support multi-turn conversations. Only one image can be uploaded per request, and it is strongly recommended to use preset prompts for optimal performance. ## Recommended Preset Prompts ```bash theme={"system"} # Convert the document contents to markdown format <|grounding|>Convert the document to markdown. # Perform text recognition on this image <|grounding|>OCR this image. # Extract all text without layout consideration Free OCR. # Parse any figures or tables in the document Parse the figure. # Provide a detailed description of the image content Describe this image in detail. # Locate the position of <|ref|>xxxx<|/ref|> in the image Locate <|ref|>xxxx<|/ref|> in the image. ``` ## Usage Example This example uses the `<|grounding|>OCR this image.` preset prompt to perform image text recognition. ```python theme={"system"} from openai import OpenAI client = OpenAI( base_url="https://api.novita.ai/openai", api_key="", ) response = client.chat.completions.create( model="deepseek/deepseek-ocr-2", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://example.com/image.png" } }, { "type": "text", "text": "<|grounding|>OCR this image." } ] } ], stream=False, max_tokens=4096 ) content = response.choices[0].message.content print(content) ``` Example input image: OCR Example Image

Example output: ``` <|/ref|><|det|>[[37, 48, 279, 140]]<|/det|> <|ref|>Deploy open-source and specialized models<|/ref|><|det|>[[42, 48, 857, 133]]<|/det|> <|ref|>smarterandfasterwithsimpleApls.Accessthe<|/ref|><|det|>[[44, 185, 902, 246]]<|/det|> <|ref|>latest chat, code, image, audio, video models and<|/ref|><|det|>[[41, 291, 945, 370]]<|/det|> <|ref|>more,ready for production with built-in<|/ref|><|det|>[[40, 407, 756, 488]]<|/det|> <|ref|>scalability.<|/ref|><|det|>[[39, 515, 232, 606]]<|/det|> <|ref|>Explore<|/ref|><|det|>[[87, 813, 266, 879]]<|/det|> <|ref|>Models<|/ref|><|det|>[[289, 816, 432, 878]]<|/det|> ``` # Function Calling Source: https://docs.novita.ai/guides/llm-function-calling `Function Calling` empowers AI models to interact with external tools and APIs, enabling them to perform specific actions and access real-time information. This capability extends the functionality of AI models beyond simple text generation, allowing for more dynamic and practical applications. ## Supported Models The following models support `Function Calling`: ## Quick Start Guide This guide demonstrates how to use Function Calling to retrieve current weather information for a user's specified location. We will walk through a complete Python code example. For the specific API format of Function Calling, please refer to the API reference [Create Chat Completion](/api-reference/model-apis-llm-create-chat-completion). ### 1. Initialize the Client First, you need to initialize the client with your Novita API key. ```python theme={"system"} from openai import OpenAI import json client = OpenAI( base_url="https://api.novita.ai/openai", # Get the Novita AI API Key from: https://novita.ai/settings/key-management. api_key="", ) model = "deepseek/deepseek_v3" ``` ### 2. Define the Function to Be Called Next, define the Python function that the model can call. In this example, it's a function to get weather information. ```python theme={"system"} # Example function to simulate fetching weather data. def get_weather(location): """Retrieves the current weather for a given location.""" print("Calling get_weather function with location: ", location) # In a real application, you would call an external weather API here. # This is a simplified example returning hardcoded data. return json.dumps({"location": location, "temperature": "60 degrees Fahrenheit"}) ``` ### 3. Construct the API Request with Tools and User Message Now, create the API request to the Novita endpoint. This request includes the `tools` parameter, defining the functions the model can use, and the user's message. ```python theme={"system"} tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get weather of an location, the user shoud supply a location first", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA", } }, "required": ["location"] }, } }, ] messages = [ { "role": "user", "content": "What is the weather in San Francisco?" } ] # Let's send the request and print the response. response = client.chat.completions.create( model=model, messages=messages, tools=tools, ) # Please check if the response contains tool calls if in production. tool_call = response.choices[0].message.tool_calls[0] print(tool_call.model_dump()) ``` **Output:** ```js theme={"system"} {'id': '0', 'function': {'arguments': '{"location": "San Francisco, CA"}', 'name': 'get_weather'}, 'type': 'function'} ``` ### 4. Respond with the Function Call Result and Get the Final Answer The next step is to process the function call, execute the `get_weather` function, and send the result back to the model to generate the final response to the user. ```python theme={"system"} # Ensure tool_call is defined from the previous step if tool_call: # Extend conversation history with the assistant's tool call message messages.append(response.choices[0].message) function_name = tool_call.function.name if function_name == "get_weather": function_args = json.loads(tool_call.function.arguments) # Execute the function and get the response function_response = get_weather( location=function_args.get("location")) # Append the function response to the messages messages.append( { "tool_call_id": tool_call.id, "role": "tool", "content": function_response, } ) # Get the final response from the model, now with the function result answer_response = client.chat.completions.create( model=model, messages=messages, # Note: Do not include tools parameter here. ) print(answer_response.choices[0].message) ``` **Output:** ``` ChatCompletionMessage(content="The weather in San Francisco, CA is currently **60 degrees Fahrenheit**. For more detailed information, such as specific conditions (e.g., sunny, cloudy, rainy), you might want to check a local weather app or website. Let me know if you'd like help with anything else!", refusal=None, role='assistant', function_call=None, tool_calls=None) ``` ### The Complete Code ```python theme={"system"} from openai import OpenAI import json client = OpenAI( base_url="https://api.novita.ai/openai", # Get the Novita AI API Key from: https://novita.ai/settings/key-management. api_key="", ) model = "deepseek/deepseek_v3" # Example function to simulate fetching weather data. def get_weather(location): """Retrieves the current weather for a given location.""" print("Calling get_weather function with location: ", location) # In a real application, you would call an external weather API here. # This is a simplified example returning hardcoded data. return json.dumps({"location": location, "temperature": "60 degrees Fahrenheit"}) tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get weather of an location, the user shoud supply a location first", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA", } }, "required": ["location"] }, } }, ] messages = [ { "role": "user", "content": "What is the weather in San Francisco?" } ] # Let's send the request and print the response. response = client.chat.completions.create( model=model, messages=messages, tools=tools, ) # Please check if the response contains tool calls if in production. tool_call = response.choices[0].message.tool_calls[0] print(tool_call.model_dump()) # Ensure tool_call is defined from the previous step if tool_call: # Extend conversation history with the assistant's tool call message messages.append(response.choices[0].message) function_name = tool_call.function.name if function_name == "get_weather": function_args = json.loads(tool_call.function.arguments) # Execute the function and get the response function_response = get_weather( location=function_args.get("location")) # Append the function response to the messages messages.append( { "tool_call_id": tool_call.id, "role": "tool", "content": function_response, } ) # Get the final response from the model, now with the function result answer_response = client.chat.completions.create( model=model, messages=messages, # Note: Do not include tools parameter here ) print(answer_response.choices[0].message) ``` # Interleaved Thinking Support Source: https://docs.novita.ai/guides/llm-interleaved-thinking > **Last Updated**: 2025-12-03
> **Status**: Supported (OpenAI-Compatible) ## 1. Overview **Interleaved Thinking** is an advanced reasoning framework that enables models to perform explicit reasoning steps between tool calls. Models with Interleaved Thinking can: * Reflect on the current environment and tool outputs * Decide the next action based on updated reasoning * Maintain a continuous reasoning chain across multiple tool invocations * Provide transparent, inspectable multi-step thinking through `reasoning_details` or `reasoning_content` This capability transforms traditional function-calling into **agent-level tool use**, making complex workflows more accurate, reliable, and context-aware. Novita fully supports Interleaved Thinking for all models that natively expose reasoning streams (e.g., MiniMax-M3 and other OpenAI-compatible reasoning models). ## 2. Key Concepts ### 2.1 Interleaving Instead of executing a single reasoning phase followed by a tool call, the model performs: ```BASH theme={"system"} Reason → Tool Call → Observe → Reason → Tool Call → ... ``` This allows the model to adjust its strategy dynamically based on previous tool outputs. ### 2.2 Reasoning Details (`reasoning_details`) For some models, the content of the model's thinking will be returned in the form of a separate structure: ```JSON theme={"system"} "reasoning_details": [ { "type": "reasoning.text", "format": "openai-responses-v1", "text": "Model’s step-by-step reasoning..." } ] ``` For such models, Novita supports returning this field in both streaming and non-streaming modes. ### 2.3 Conversation Memory Requirement To maintain reasoning continuity: You **must append the model’s full response** including `reasoning_details`, `tool_calls`, and `content` to subsequent `messages`. Failing to preserve the chain may result in: * Incorrect tool use * Lost reasoning context * Repeated or circular tool calls * Reduced reliability This requirement mirrors OpenAI’s reasoning APIs. ## 3. API Behavior ### 3.1 Request Format No changes are required on the user side. Interleaved Thinking works with the standard OpenAI-compatible Chat Completions API. ### 3.2 Response Format The model may return the following fields: * `reasoning_content`: original thinking content * `reasoning_details`: structured reasoning segments, this field is optioanl * `tool_calls`: tool invocation plan * `content`: natural language output These extend the standard OpenAI format. ## 4. Example Request (MiniMax-M3) ```JSON theme={"system"} { "model": "minimax/minimax-m3", "messages": [ { "role": "user", "content": "How's the weather in San Francisco?" }, { "role": "assistant", "name": "MiniMax AI", "content": "", "tool_calls": [ { "id": "call_function_asqvfevfc8af_1", "type": "function", "function": { "name": "get_weather", "arguments": "{\"location\": \"San Francisco, US\"}" } } ], "reasoning_details": [ { "type": "reasoning.text", "id": "reasoning-text-1", "format": "openai-responses-v1", "index": 0, "text": "The user is asking about the weather in San Francisco..." } ] }, { "role": "tool", "tool_call_id": "call_function_asqvfevfc8af_1", "content": "24℃, sunny" } ], "stream": true, "reasoning_split": true, "max_tokens": 1024, "temperature": 0.7, "tools": [ { "type": "function", "function": { "name": "get_weather", "description": "Get weather for a specific location", "parameters": { "type": "object", "properties": { "location": { "type": "string" } }, "required": ["location"] } } } ] } ``` ## 5. Example Response (Non-Streaming) ```JSON theme={"system"} { "id": "07a4dedfdb1498b045498dfd42497639", "object": "chat.completion", "created": 1764303147, "model": "MiniMax", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "", "name": "MiniMax", "tool_calls": [ { "index": 0, "id": "call_function_9w7wq1j9zmpl_1", "type": "function", "function": { "name": "get_weather", "arguments": "{\"location\": \"ShangHai\"}" } } ], "reasoning_content": "The user asked for Shanghai weather...", "reasoning_details": [ { "type": "reasoning.text", "text": "The user is asking about the weather in Shanghai...", "id": "reasoning-text-1", "format": "openai-responses-v1", "index": 0 } ] }, "finish_reason": "tool_calls" } ] } ``` ## 6. Streaming Response Example ```JSON theme={"system"} { "id": "664c5ad870c1888fbcbd267d9829e354", "object": "chat.completion.chunk", "created": 1764303504, "model": "minimax/minimax-m3", "choices": [ { "index": 0, "delta": { "role": "assistant", "reasoning_content": "...\n\nThe user has specifically asked...", "reasoning_details": [ { "type": "reasoning.text", "text": ".\n\nThe user has specifically asked...", "id": "reasoning-text-1", "format": "openai-responses-v1", "index": 0 } ] }, "finish_reason": null } ] } ``` ## 7. Developer Notes ### 7.1 Models That Support Interleaved Thinking All models exposing `reasoning_details` through OpenAI-compatible APIs, including: * MiniMax-M3 * (Upcoming) Novita Reasoning Series * Other reasoning-enabled partner models ### 7.2 Pricing Billing is based on reasoning tokens, following the model’s pricing rules. `reasoning_details` will increase token usage. ### 7.3 Error Handling You may encounter: * Missing tool parameters * Recursive or repeated tool calls * Incorrect assumptions in the reasoning phase Ensure your application validates tool arguments and handles model errors gracefully. ## 8. Best Practices **✓ Always include full model messages in the next request** Include: * `content` * `tool_calls` * `reasoning_details` **✓ Enable streaming for long-chain reasoning** Streaming allows the client to: * Monitor the reasoning process * Detect incorrect tool plans early * Provide faster user feedback **✓ Combine with server-side state machines for stability** For production systems, we recommend pairing Interleaved Thinking with deterministic guardrails, e.g.: * Parameter validators * Execution sandbox * Maximum recursion safeguards ## 9. Summary Interleaved Thinking significantly enhances multi-step reasoning and tool-use reliability: * Transparent and inspectable step-wise reasoning * Adaptive planning between tool invocations * Stronger context retention across long workflows * Fully compatible with OpenAI-style Chat Completions API Novita will continue expanding support for advanced reasoning models, bringing agent-level intelligence to the API layer. # Model Access for API Keys Source: https://docs.novita.ai/guides/llm-model-access Model access controls which models an individual API key can call. It does not change your team's enabled model range. Instead, it limits the models that a key can call within that range. Calls to models outside the key's model access policy are rejected. This capability covers models in the LLM product line. Standalone image and video multimodal capabilities are not currently included in model access configuration. Examples: * **Separate keys by purpose**: let production keys call only stable models and experimental keys call only low-cost models. * **Limit unexpected costs**: if a key makes unexpected calls to high-cost models, restrict it to a smaller set of models. * **Limit shared keys**: for a key shared with a contractor or partner, allow only the agreed models. ## Access modes Each API key has one of two access modes: | Access mode | Meaning | | :- | :- | | All enabled models | Default mode. The key can access all models your team has enabled, including models enabled in the future. You can exclude specific models. Future enabled models remain accessible unless you exclude them. | | Selected models only | The key can access only the models you select. Models your team enables later are not added automatically; you must add them yourself. | In both modes, the accessible models never exceed your team's enabled model range and never include models that the platform has taken offline. * All enabled models: accessible models = team's enabled models (including future ones) − excluded models. * Selected models only: accessible models = selected models (intersected with the team's enabled models). ## Configure model access Only team admins can configure model access for an API key. Developers and Basic members see a read-only summary for keys they can access but cannot edit it. Billing members do not see model access. Go to [Key Management](https://novita.ai/settings/key-management) and find the API key you want to configure. While creating or editing the key, open the **Model Access** section. Select **All enabled models** or **Selected models only**: * For **All enabled models**, add any models you want to block under **Excluded models**. * For **Selected models only**, add at least one model under **Accessible models**. After you save, the new model access settings may take some time to take effect. When creating or editing a key, you can reuse another key's settings within the same team with **Copy config** and **Paste config**. After pasting, save the settings or adjust them first. ## When a model is taken offline A model being taken offline and an API key access restriction are different situations. Once a model is offline, no API key can call it. This is not a model access configuration problem with a particular key. If the platform takes a model offline, it is removed from the set of available models: * Calls to that model immediately return a "model unavailable" response, even if the key previously had access to it. * If a key's saved configuration still contains the offline model, the model remains visible but is greyed out. Remove it before saving; otherwise, the configuration cannot be saved. ## When a call is rejected An offline model returns a "model unavailable" response. This is not a key permission denial and does not return `model_access_denied`. When an API key calls a model outside its model access policy, the call returns HTTP 403 with the error code `model_access_denied`. Contact your team admin to adjust the policy. See also [Common Error Codes](/guides/error). ## Related * [API Keys](/guides/llm-api-keys) — create, store, and manage API keys, including expiration and OpenAPI boundaries. * [Network Access for API Keys](/guides/llm-network-access) — restrict which source IPs can use a key. * [Get API Key Model Access Policy](/api-reference/key-get-model-access-policy) — read a key's model access policy programmatically. # LLM Monitoring Source: https://docs.novita.ai/guides/llm-monitor Description of your new file. Novita AI provides comprehensive monitoring metrics for the usage of large language model (LLM) APIs. These metrics help you gain deep insights into the availability and performance of your LLM API requests. You can view the monitoring metrics on the [LLM Monitoring Page](https://novita.ai/models-console/llm-metrics). ## Metric Descriptions All metrics are broken down by **model** and sampled at the **minute level**. However, depending on your selected time interval, samples may not be shown for every minute. In such cases, values will be averaged across the selected time range. * **Requests Per Minute (RPM)** Displays the number of API requests sent per minute. This helps you understand usage patterns and the level of API concurrency. * **Request Success Rate** Shows the percentage of successful API responses per minute (non-5xx status codes), reflecting the API's availability. * **Average Token Count per Request** Displays the average number of input and output tokens per request per minute, which helps you monitor token consumption patterns. * **End-to-End (E2E) Latency** Shows the total time required for the model to generate a full response in each request per minute. Metrics include the 99th percentile, 95th percentile, and average latency. * **Time to First Token (TTFT)** Only tracked for streaming requests where stream=true. Displays the time it takes to process the prompt and generate the first output token in each request per minute. Includes the 99th percentile, 95th percentile, and average latency. * **Time Per Output Token (TPOT)** Only tracked for streaming requests where stream=true. Displays the average time between consecutive output tokens in each request per minute. Includes the 99th percentile, 95th percentile, and average latency. # Network Access for API Keys Source: https://docs.novita.ai/guides/llm-network-access A network access policy restricts which source IPs can use an individual API key. Once configured, only IPs in the allowed list can use the key to invoke models; requests from other sources are rejected. A network access policy is configured per key, and policies are independent. An empty allowed list does not restrict the source IP. Network access applies only to model invocation. Management endpoints, including billing queries and key policy read/write operations, are not restricted by source IP. The policy limits model calls from sources outside the allowed list; it does not prevent an insider from routing traffic through an allowed source. ## Allowed IP rules * Supports IPv4 addresses (for example, `203.0.113.5`) and IPv4 CIDR ranges (for example, `198.51.100.0/24`). * **IPv6 is not supported yet.** If your egress is IPv6, or a dual-stack setup routes through IPv6, the call is rejected. * Up to 100 entries per key. * An empty allowed list means the source IP is not restricted; this is the default. Confirm your own egress IP before configuring the policy. When the list is non-empty, every source outside it, including your testing egress, is rejected. With dynamic public IPs, multi-datacenter egress, NAT gateways, or serverless functions, the egress IP can change or include multiple addresses. Use a CIDR range to cover the whole block, or confirm and add all egress IPs. ## Configure network access Team admins can configure network access for any key in the team; developers can configure keys they own. Basic members can view the configuration of their own keys but cannot edit it. Billing members have read-only access. Go to [Key Management](https://novita.ai/settings/key-management) and find the API key you want to configure. While creating or editing the key, open the **Network Access** section. Add allowed IPv4 addresses or CIDR ranges one by one. Leave the list empty to keep the source IP unrestricted. After you save, the new network access settings apply to subsequent calls. ## When a call is rejected When an API key invokes a model from a source IP outside the allowed range, the call returns HTTP 403 with the error code `ip_access_denied`. Confirm the request's public egress IP and add it to the key's allowed list, or contact your team admin to adjust the network access policy. See also [Common Error Codes](/guides/error). ## Related * [API Keys](/guides/llm-api-keys) — create, store, and manage API keys, including expiration and OpenAPI boundaries. * [Model Access for API Keys](/guides/llm-model-access) — restrict which models a key can call. * [Get API Key IP Access Policy](/api-reference/key-get-ip-access-policy) — read a key's network access policy programmatically. # Interactive Playground Source: https://docs.novita.ai/guides/llm-playgrounds To help users better understand and test the behavior of large language models, **Novita** provides a flexible interactive playground. Users can fine-tune a series of parameters to precisely control the model's generation logic, content style, and response structure, enabling more targeted output. ## Response Format Users can choose between two modes: * **Custom JSON Output Structure**: Define a JSON Schema to guide the model in generating output that follows a predefined format, ideal for structured tasks or API integration scenarios. * **Free-form Generation**: Allow the model to freely organize answers based on context, suitable for open-ended Q\&A and creative writing applications. ## System Prompt By setting a system prompt, users can define the model's "persona" and behavioral context. For example, if you set the prompt as “You are a professional travel planner,” the model will respond from that perspective, enhancing consistency and contextual coherence. ## Parameter Configuration The playground supports the adjustment of the following core parameters: * **max\_tokens**: Sets the maximum number of tokens the model can generate. Lower values improve response speed, while higher values are suitable for complex, lengthy outputs. * **temperature**: Controls the creativity and randomness of the output. Higher values lead to more diverse and imaginative responses, while lower values yield more conservative and deterministic outputs, ideal for precise tasks. * **top\_p (nucleus sampling)**: Dynamically adjusts the token sampling range per step to improve diversity and fluency. Recommended to use alongside temperature. * **top\_k**: Limits the model to select from the top K highest probability tokens at each step, which enhances efficiency and reduces noise. * **min\_p**: An alternative to top\_p and top\_k, used to filter out tokens with probabilities below a certain threshold, suitable for accuracy-critical tasks. * **presence\_penalty**: Encourages the model to introduce new topics in its output. Positive values increase the likelihood of expanding to new content. * **frequency\_penalty**: Suppresses the generation of repeated tokens. Positive values reduce repetition and help maintain originality. * **repetition\_penalty**: Further reduces the risk of loops or repetitive prompts, enhancing output robustness. ## Usage Recommendations * **For Creative Writing**: Increase temperature, top\_p, and presence\_penalty to encourage diverse and imaginative outputs. * **For Structured Tasks and Tool Invocation**: Use a fixed JSON schema with lower temperature and top\_k to ensure consistent and reliable outputs. * **For Testing Interactions**: Compare different parameter combinations to observe changes in generation style, information density, and contextual understanding, aiding in faster prompt iteration. *** To replicate the behavior from the playground in API calls, refer to the corresponding parameter documentation in the platform guide. # Prompt Cache Source: https://docs.novita.ai/guides/llm-prompt-cache Prompt Cache is a cost-optimization feature provided by the Novita inference engine. When a request matches a previous Prompt, the system returns a cached result and charges only a minimal cache token fee — significantly reducing cost and improving response latency. ## 1. Benefits With Prompt Cache enabled, you gain: * **Lower Cost** Repeated Prompts no longer require full inference. Only minimal cache token fees are charged. * **Lower Latency** Cached results are returned instantly without running the model. * **Higher Throughput** In high-QPS scenarios, Prompt Cache reduces compute load and improves overall system capacity. * **Transparent to Your Application** no additional logic or system changes are required. ## 2. Supported Models Several serverless open-source models currently support prompt cache billing, including: For pricing details regarding the prompt cache feature of supported models, please refer to: [https://novita.ai/pricing](https://novita.ai/pricing) (see "Cache" section). ## 3. Use Cases Prompt Cache is highly effective in workloads with **frequent repeated Prompts**, including but not limited to: * **Template-based Generation** * Fixed-format summaries * Template-driven rewriting * Prompts reused across tasks * **Text Classification & Field Extraction** * Content type classification * Tag or key information extraction * **Content Moderation** * Review of comments, ads, or titles * Many moderation prompts repeat across users and time * **Repeated System Prompts in Chat Applications** * Chatbot persona definitions * Global conversation rules * Background information reused across multiple turns * **Workflow / Assistant-style Prompts** * SQL generation assistants * Code repair assistants * Summary assistants with fixed output formats These scenarios naturally achieve high cache hit rates, reducing inference cost significantly. ## 4. Response Examples When the cache is hit, no inference is performed, resulting in significantly lower cost and faster responses. If the model supports prompt caching, your API calls require no modifications. Below is a sample response when hitting the cached result: ```JSON theme={"system"} { "prompt_tokens": 3295, "completion_tokens": 137, "total_tokens": 3432, "prompt_tokens_details": { "audio_tokens": 0, "cached_tokens": 448, "cache_creation_Prompt_tokens": 0, "cache_read_Prompt_tokens": 0 } } ``` # Rate limits Source: https://docs.novita.ai/guides/llm-rate-limits Rate limits control how frequently users can make requests to our LLM API within specific time periods. Understanding and working within these limits is essential for optimal API usage. New to the API? See [API Keys](/guides/llm-api-keys) for how to create and authenticate with a key first. ## 1. Understanding Rate Limits ### What are Rate Limits? Rate limits restrict the number of API requests that can be made within defined time periods. They help: * Prevent API abuse and misuse; * Ensure fair resource distribution among users; * Maintain consistent API performance and reliability; * Protect the stability of our services. ### Default Rate Limits Each account has a default rate limit for model calls, measured in RPM (requests per model per minute) and TPM (tokens per model per minute). Rate limits vary by account tier, as outlined in the tables below.

Tier	How to reach
T1	Monthly top-ups did not exceed \$50 in any of the last 3 calendar months.
T2	Monthly top-ups were at least \$50 but did not exceed \$500 in any of the last 3 calendar months.
T3	Monthly top-ups were at least \$500 but did not exceed \$3,000 in any of the last 3 calendar months.
T4	Monthly top-ups were at least \$3,000 but did not exceed \$10,000 in any of the last 3 calendar months.
T5	Monthly top-ups were at least \$10,000 in at least one of the last 3 calendar months.

The last 3 calendar months refers to the current month and the two months before it. Default Rate Limit by Tier (RPM / TPM): ## 2. Handling Rate Limits ### How to Monitor Rate Limits? When you exceed the rate limit, the API will return: * HTTP Status Code: 429 (Too Many Requests); * A rate limit exceeded message in the response body. ### Best Practices To avoid hitting rate limits: 1. Implement request throttling in your application; 2. Add exponential backoff for retries; 3. Monitor your API usage patterns. ### When You Hit Rate Limits If you receive a 429 error, you can: 1. **Retry Later**: Wait a short period before retrying your request; 2. **Optimize Requests**: Reduce request frequency; 3. **Rate Limits Increase**: For higher rate limits, you can: * Contact us through Discord * or Book a call with our sales team *** **If you have any questions, [please reach out to us on Discord](https://discord.gg/YyPRAzwp7P).** # Reasoning Models Source: https://docs.novita.ai/guides/llm-reasoning ## Overview Reasoning models are advanced language models optimized for complex problem-solving tasks. By generating detailed reasoning steps (chain-of-thought), they improve the accuracy of answers in analytical scenarios. ### Typical Use Cases * **Complex Problem Solving**: Suitable for tasks requiring step-by-step logic, such as math or scientific reasoning. * **Decision Support Systems**: Helps explain the logic behind conclusions by providing detailed reasoning processes. * **Education and Training**: Assists learners in understanding complex concepts by presenting derivation processes clearly. *** ## Installation & Setup Before using reasoning models, make sure the latest OpenAI SDK is installed: ```bash theme={"system"} pip install -U openai ``` *** ## API Usage Use the `/chat/completions` endpoint to invoke reasoning models. ### Request Parameters * `max_tokens`: Sets the maximum number of tokens the model can return. * `temperature`: Recommended between 0.5 and 0.7 (suggested: 0.6) to balance creativity and logic. * `top_p`: Recommended value is 0.95. *** ### Example Code #### Streaming Response ```python theme={"system"} from openai import OpenAI client = OpenAI(api_key="YOUR_API_KEY", base_url="https://api.novita.ai/openai") messages = [ {"role": "user", "content": "Explain Newton's Second Law."} ] response = client.chat.completions.create( model="deepseek/deepseek-v4.1-flash", messages=messages, stream=True, max_tokens=4096 ) content = "" reasoning_content = "" for chunk in response: if chunk.choices[0].delta.content: content += chunk.choices[0].delta.content if chunk.choices[0].delta.reasoning_content: reasoning_content += chunk.choices[0].delta.reasoning_content print("Final Answer:", content) print("Reasoning Steps:", reasoning_content) ``` #### Non-Streaming Response ```python theme={"system"} response = client.chat.completions.create( model="deepseek/deepseek-v4.1-flash", messages=[ {"role": "user", "content": "What is the greenhouse effect? How can it be mitigated?"} ], stream=False, max_tokens=4096 ) content = response.choices[0].message.content reasoning_content = response.choices[0].message.reasoning_content print("Final Answer:", content) print("Reasoning Steps:", reasoning_content) ``` *** ## Context Management Reasoning outputs are not automatically carried over to the next round of dialogue. You must manually maintain the message history: ```python theme={"system"} messages.append({"role": "assistant", "content": content}) messages.append({"role": "user", "content": "Please continue explaining the solution."}) ``` *** ## Supported Models The following reasoning models are currently supported on the Novita platform: *** ## Billing * Billing is based on the number of tokens for both input and output. * Please refer to each model's pricing page for specific billing rules and token conversion details. *** ## Notes & Best Practices * Avoid placing reasoning instructions in the `system` message. Instead, make the intent explicit in the `user` message. * For mathematical tasks, clearly instruct the model, e.g., “Please reason step by step and provide a final answer.” * To prevent the model from skipping reasoning steps, consider asking for a newline before the final answer. # Recommended Models Source: https://docs.novita.ai/guides/llm-recommended A list of recommended open models for common use cases. ## Which open models should I use? Actually, there’s no single right answer! Here’s a curated list based on Novita internal testing, community feedback, and external benchmarks. We recommend using it as a starting point, and we will update it regularly as new models emerge. * Model sizes are marked as Small, Medium, or Large. For MoE models, the activated parameter size is what matters for latency. * For best latency, use small or medium models. For best quality, use large models or fine-tune medium or small models. * Model names below are exact model IDs — copy them directly into your API calls. Each links to its model page for live pricing and a playground. * You can explore all models in the [Novita Model Library](https://novita.ai/models/llm). ### Code generation & reasoning | Model ID | Size / Notes | Context Window | | - | - | - | | [`moonshotai/kimi-k2.7-code`](https://novita.ai/models/llm/moonshotai-kimi-k2.7-code) | Large | 262K | | [`zai-org/glm-5.2`](https://novita.ai/models/llm/zai-org-glm-5.2) | Large | 1M | | [`deepseek/deepseek-v4-pro`](https://novita.ai/models/llm/deepseek-deepseek-v4-pro) | Large | 1M | ### General reasoning & planning | Model ID | Size / Notes | Context Window | | - | - | - | | [`deepseek/deepseek-v4-pro`](https://novita.ai/models/llm/deepseek-deepseek-v4-pro) | Large | 1M | | [`deepseek/deepseek-v4.1-flash`](https://novita.ai/models/llm/deepseek-deepseek-v4.1-flash) | Large | 1M | | [`qwen/qwen3.5-397b-a17b`](https://novita.ai/models/llm/qwen-qwen3.5-397b-a17b) | Large, 17B active | 262K | ### Agents, function calling & tool use | Model ID | Size / Notes | Context Window | | - | - | - | | [`zai-org/glm-5.2`](https://novita.ai/models/llm/zai-org-glm-5.2) | Large | 1M | | [`moonshotai/kimi-k2.7-code`](https://novita.ai/models/llm/moonshotai-kimi-k2.7-code) | Large | 262K | | [`minimax/minimax-m3`](https://novita.ai/models/llm/minimax-minimax-m3) | Large | 1M | ### Long context & summarization | Model ID | Size / Notes | Context Window | | - | - | - | | [`zai-org/glm-5.2`](https://novita.ai/models/llm/zai-org-glm-5.2) | Large | 1M | | [`deepseek/deepseek-v4-pro`](https://novita.ai/models/llm/deepseek-deepseek-v4-pro) | Large | 1M | | [`deepseek/deepseek-v4.1-flash`](https://novita.ai/models/llm/deepseek-deepseek-v4.1-flash) | Large, cost-effective | 1M | ### Vision & document understanding | Model ID | Size / Notes | Context Window | | - | - | - | | [`moonshotai/kimi-k2.5`](https://novita.ai/models/llm/moonshotai-kimi-k2.5) | Large | 262K | | [`qwen/qwen3.5-397b-a17b`](https://novita.ai/models/llm/qwen-qwen3.5-397b-a17b) | Large, 17B active | 262K | | [`google/gemma-4-31b-it`](https://novita.ai/models/llm/google-gemma-4-31b-it) | Medium | 262K | ### Low-latency NLU & extraction | Model ID | Size / Notes | Context Window | | - | - | - | | [`inclusionai/ling-3.0-flash`](https://novita.ai/models/llm/inclusionai-ling-3.0-flash) | MoE, small activation | 262K | | [`meta-llama/llama-3.1-8b-instruct`](https://novita.ai/models/llm/meta-llama-llama-3.1-8b-instruct) | Small | 128K | | [`google/gemma-4-26b-a4b-it`](https://novita.ai/models/llm/google-gemma-4-26b-a4b-it) | Medium, 4B active | 262K | ## Migrating from older models If you are still calling one of the models below, switch to its recommended replacement: | If you use | Switch to | | - | - | | `deepseek/deepseek-r1-0528` | `deepseek/deepseek-v4-pro` | | `zai-org/glm-4.5` | `zai-org/glm-5.2` | | `moonshotai/kimi-k2-instruct` | `moonshotai/kimi-k2.7-code` | | `qwen/qwen3-coder-480b-a35b-instruct` | `qwen/qwen3.8-max` | | `meta-llama/llama-4-maverick` / `llama-4-scout`, `qwen/qwen2.5-vl-72b-instruct` | `moonshotai/kimi-k2.5`, `qwen/qwen3.5-397b-a17b`, or `google/gemma-4-31b-it` | # Structured Outputs Source: https://docs.novita.ai/guides/llm-structured-outputs `Structured Outputs` enables the model to generate responses that adhere to your supplied [JSON Schema](https://json-schema.org/specification). ## Supported Models The following models support `Structured Outputs`: ## Quick Start Guide This guide demonstrates how to use `Structured Outputs` ability to generate the JSON response via `json_schema` response format. We will walk through a complete Python code example. ### 1. Initialize the Client First, you need to initialize the client with your Novita API key. ```python theme={"system"} from openai import OpenAI client = OpenAI( base_url="https://api.novita.ai/openai", # Get the Novita AI API Key from: https://novita.ai/settings/key-management. api_key="", ) # Go to the [Models](https://novita.ai/models) page to see the models that support `Structured Outputs`. model = "qwen/qwen3.8-flash" ``` ### 2. Define the JSON schema This example creates a schema for extracting expense information from the user's input. ```python theme={"system"} # Define system prompt for expense tracking. system_prompt = """You are an expense tracking assistant. Extract expense information from the user's input and format it according to the provided schema.""" # Define JSON schema for structured response. response_format = { "type": "json_schema", "json_schema": { "name": "expense_tracking_schema", "schema": { "type": "object", "properties": { "expenses": { "type": "array", "items": { "type": "object", "properties": { "description": { "type": "string", "description": "Description of the expense" }, "amount": { "type": "number", "description": "Amount spent in dollars" }, "date": { "type": "string", "description": "When the expense occurred" }, "category": { "type": "string", "description": "Category of expense (e.g., food, office, travel)" } }, "required": [ "description", "amount" ] } }, "total": { "type": "number", "description": "Total amount of all expenses" } }, "required": [ "expenses", "total" ], }, }, } ``` ### 3. Request chat completion API Now, make the chat completion request to the Novita endpoint. This request includes the `response_format` parameter, defining the JSON schema we defined in the previous step. ```python theme={"system"} chat_completion = client.chat.completions.create( model=model, messages=[ { "role": "system", "content": system_prompt, }, { "role": "user", "content": """I spent $120 on dinner at an Italian restaurant last Friday with my colleagues. Also bought office supplies for $45 on Monday.""", }, ], max_tokens=1024, temperature=0.8, stream=False, response_format=response_format, ) response_content = chat_completion.choices[0].message.content # Parse and prettify the JSON try: json_response = json.loads(response_content) prettified_json = json.dumps(json_response, indent=2) print(prettified_json) except json.JSONDecodeError: print("Could not parse response as JSON. Raw response:") print(response_content) ``` **Output:** ```json theme={"system"} { "expenses": [ { "date": "2023-03-17", "description": "Dinner at Italian restaurant", "amount": 120, "category": "Food & Dining" }, { "date": "2023-03-13", "description": "Office supplies", "amount": 45, "category": "Office Supplies" } ], "total": 165 } ``` ### The Complete Code ```python theme={"system"} from openai import OpenAI import json client = OpenAI( base_url="https://api.novita.ai/openai", # Get the Novita AI API Key from: https://novita.ai/settings/key-management. api_key="", ) # Go to the [Models](https://novita.ai/models) page to see the models that support `Structured Outputs`. model = "qwen/qwen3.8-flash" # Example of using JSON Schema for structured output # This example creates a schema for extracting expense information # Define system prompt for expense tracking system_prompt = """You are an expense tracking assistant. Extract expense information from the user's input and format it according to the provided schema.""" # Define JSON schema for structured response response_format = { "type": "json_schema", "json_schema": { "name": "expense_tracking_schema", "schema": { "type": "object", "properties": { "expenses": { "type": "array", "items": { "type": "object", "properties": { "description": { "type": "string", "description": "Description of the expense" }, "amount": { "type": "number", "description": "Amount spent in dollars" }, "date": { "type": "string", "description": "When the expense occurred" }, "category": { "type": "string", "description": "Category of expense (e.g., food, office, travel)" } }, "required": [ "description", "amount" ] } }, "total": { "type": "number", "description": "Total amount of all expenses" } }, "required": [ "expenses", "total" ], }, }, } chat_completion = client.chat.completions.create( model=model, messages=[ { "role": "system", "content": system_prompt, }, { "role": "user", "content": """I spent $120 on dinner at an Italian restaurant last Friday with my colleagues. Also bought office supplies for $45 on Monday.""", }, ], max_tokens=1024, temperature=0.8, stream=False, response_format=response_format, ) response_content = chat_completion.choices[0].message.content # Parse and prettify the JSON try: json_response = json.loads(response_content) prettified_json = json.dumps(json_response, indent=2) print(prettified_json) except json.JSONDecodeError: print("Could not parse response as JSON. Raw response:") print(response_content) ``` # Vision Language Models Source: https://docs.novita.ai/guides/llm-vision ## Overview Vision-Language Models (VLMs) are a type of multimodal foundation model capable of processing both image and text inputs. These models understand visual content in conjunction with language instructions, and generate high-quality responses based on the combined context. They are widely used in scenarios involving image recognition, content interpretation, and intelligent visual Q\&A. ### Typical Use Cases * **Image Recognition and Description**: Automatically identifies objects, colors, scenes, and spatial relationships in images, and generates natural language descriptions. * **Multimodal Understanding**: Combines image input and contextual text for multi-turn dialogue and task completion. * **Visual Question Answering**: Acts as an advanced OCR tool by recognizing and interpreting text embedded in images. * **Emerging Applications**: Ideal for use in intelligent vision assistants, robot perception, AR interfaces, and more. *** ## API Usage Guide To invoke a Vision-Language Model, use the `/chat/completions` endpoint with both image and text inputs. ### Image Detail Parameter Use the `detail` field to control image resolution. The following modes are available: * `high`: High resolution, preserves more detail—ideal for precision tasks. * `low`: Low resolution, faster response—suitable for real-time usage. * `auto`: Automatically selects the appropriate mode. *** ### Example Message Format #### Image via URL ```json theme={"system"} { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://example.com/image.png", "detail": "high" } }, { "type": "text", "text": "Please describe the scene in the image." } ] } ``` #### Image via Base64 ```json theme={"system"} { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "data:image/jpeg;base64,{base64_image}", "detail": "low" } }, { "type": "text", "text": "What text is present in the image?" } ] } ``` *** ### Python Code: Encode Image to Base64 ```python theme={"system"} import base64 from PIL import Image import io def image_to_base64(image_path): with Image.open(image_path) as img: buffered = io.BytesIO() img.save(buffered, format="JPEG") return base64.b64encode(buffered.getvalue()).decode('utf-8') base64_image = image_to_base64("path/to/your/image.jpg") ``` *** ## Multi-Image Input The API supports sending multiple images alongside text input. For best results, we recommend sending no more than two images per request. ```json theme={"system"} { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://example.com/image1.png" } }, { "type": "image_url", "image_url": { "url": "data:image/jpeg;base64,{base64_image}" } }, { "type": "text", "text": "Compare the common features of these two images." } ] } ``` *** ## Supported Models The following Vision-Language Models are currently supported on the Novita platform: Visit the [Model Hub](https://novita.ai/models-console/library) for a complete and up-to-date list of available models. *** ## Billing Image input is tokenized and counted toward billing together with text. * Each model uses a different image-to-token conversion method. * Refer to each model’s pricing page for detailed billing and token policy. *** ## API Call Examples ### Single Image Description ```python theme={"system"} from openai import OpenAI client = OpenAI(api_key="YOUR_KEY", base_url="https://api.novita.ai/openai") response = client.chat.completions.create( model="zai-org/glm-4.5v", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://example.com/cityscape.jpg" } }, { "type": "text", "text": "Describe the main buildings in the image." } ] } ], stream=True ) for chunk in response: print(chunk.choices[0].delta.content or "", end="", flush=True) ``` ### Multi-Image Comparison ```python theme={"system"} response = client.chat.completions.create( model="zai-org/glm-4.5v", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://example.com/product1.jpg" } }, { "type": "image_url", "image_url": { "url": "https://example.com/product2.jpg" } }, { "type": "text", "text": "Please compare the key differences between these two products." } ] } ], stream=True ) for chunk in response: print(chunk.choices[0].delta.content or "", end="", flush=True) ``` *** ## Notes & Troubleshooting * Image resolution and clarity significantly affect model performance. Use high-quality sources where possible. * Base64-encoded images should ideally be under 1MB to avoid timeouts or errors. * For detailed usage, [book a call with our sales team](https://meet.brevo.com/novita-ai/contact-sales) or contact support if needed. # LobeChat Source: https://docs.novita.ai/guides/lobechat Integrate Novita AI with LobeChat to build advanced AI apps with multimodal capabilities, enhanced privacy, and customizable features. With the **Novita AI & LobeChat integration**, you gain seamless access to Novita AI’s comprehensive suite of LLM models, including Llama, DeepSeek, and more. This integration enables you to create and deploy advanced AI applications tailored to your needs. Whether it’s customer service, content creation, or personal assistance, LobeChat, and Novita AI provide the tools to build intelligent, dynamic, and privacy-focused AI solutions. This guide will walk you through integrating Novita AI’s LLMs with the LobeChat platform, enabling you to deploy AI applications with advanced capabilities like multimodal interaction, context-aware reasoning, and extensible plugins. ## **What Is LobeChat?** LobeChat is an open-source, modern UI/Framework designed for ChatGPT and other LLMs. It simplifies the development of AI chat applications by supporting speech synthesis, multi-modal inputs, and an extensible plugin ecosystem. With LobeChat, you can deploy your private chat application using models like OpenAI’s ChatGPT, Claude, Gemini, or Novita AI’s LLMs in just one click. ## **Prerequisites** Before you begin, ensure you have the following: * **Novita AI LLM API Key**: * Visit [Novita AI’s website](https://novita.ai/) and create an account. * After logging in, go to the [**Key Management**](https://novita.ai/settings/key-management) page to generate your **API Key**. This key is required to connect Novita AI’s models to LobeChat. ![Novita AI key management](https://mintlify.s3.us-west-1.amazonaws.com/novitaai/images/third-party/dify-1.png) * **LobeChat Account** * Sign up for a LobeChat account at [LobeChat](https://lobechat.com). ## **Integration Steps** ### **Step 1: Connect Novita AI to LobeChat**

To connect Novita AI’s models with LobeChat: * Log in to your **LobeChat account** and access the [**Settings**](https://lobechat.com/settings/common). * Navigate to the **AI Service Provider** tab. * Scroll down to find the **Novita AI** section and expand it. * Paste your **Novita AI API Key** into the provided field and click **Save**. ### **Step 2:** Select a Novita AI Model Once the API Key is configured, you can select a Novita AI LLM model to power your AI application: Choose a Novita AI Model

### Step 3: Launch Your First Chat Begin your conversation with Novita AI LLM on LobeChat. # LoLLMS WebUI Source: https://docs.novita.ai/guides/lollmswebui Easily integrate Novita AI with LoLLMS WebUI to enhance your productivity and simplify complex tasks. LoLLMS WebUI, a centralized platform designed for effortless interaction with Large Language Models (LLMs) and multimodal AI systems, offers an intuitive interface to unlock the full potential of AI. The integration between Novita AI and LoLLMS WebUI will unleash transformative power,enabling you to simplify complex tasks, find answers, and explore new possibilities effortlessly. By combining Novita AI with LoLLMS, you'll gain access to cutting-edge AI capabilities, making it your ultimate assistant for enhanced productivity. This guide provides a step-by-step walkthrough to help you maximize the benefits of this powerful integration. ## How to Leverage Novita AI with LoLLMS WebUI You can find the GitHub repository of LoLLMS WebUI here: [ParisNeo/lollms-webui](https://github.com/ParisNeo/lollms-webui). ### Obtain Your Novita AI API Key 1. [Log in](https://novita.ai/user/login) to your Novita AI account. 2. Navigate to [the Key Management page](https://novita.ai/settings/key-management). 3. Generate a new API Key and copy it. ### Install LoLLMS WebUI 1. Automatic Installation: * Download the installation script from [the scripts folder](https://github.com/ParisNeo/lollms-webui/tree/main/scripts) and run it: * `lollms_installer.bat` for Windows. * `lollms_installer.sh` for Linux. * `lollms_installer_macos.sh` for Mac. 2. Manual Installation: * Ensure Python 3.11 is installed. Check your version with `python --version`. * If needed, download Python 3.11 from [Python.org](https://www.python.org/downloads/release/python-3118/). Step 1: Clone the Repository ```bash theme={"system"} git clone --recursive https://github.com/ParisNeo/lollms-webui.git cd lollms-webui git submodule update --init --recursive ``` Step 2: Create and Activate a Virtual Environment * Create a Virtual Environment: ```bash theme={"system"} python -m venv venv ``` * Activate the Virtual Environment: * On Windows: ```bash theme={"system"} .\venv\Scripts\activate ``` * On Linux/Mac: ```bash theme={"system"} source venv/bin/activate ``` Step 3: Install Requirements ```bash theme={"system"} pip install -r requirements.txt pip install -e ./lollms_core ``` Step 4: Create global\_paths\_cfg.yaml ```bash theme={"system"} mkdir -p $HOME/.lollms_personal_data cat > global_paths_cfg.yaml << EOL lollms_path: $(pwd)/lollms_core/lollms lollms_personal_path: $HOME/.lollms_personal_data EOL ``` ### Configure Novita AI in LoLLMS WebUI 1. Install Novita AI Binding * Set environment variables for Novita AI binding. For example: * On Windows: ```bash theme={"system"} set NOVITA_AI_API_KEY="your_api_key_here" set NOVITA_AI_MODEL_NAME="your_model_name_here" ``` * On Linux/Mac: ```bash theme={"system"} export NOVITA_AI_API_KEY="your_api_key_here" export NOVITA_AI_MODEL_NAME="your_model_name_here" ``` * Run the script to finalize the setup: ```bash theme={"system"} python zoos/bindings_zoo/novita_ai/__init__.py ``` 2. Run the Server ```bash theme={"system"} python app.py ``` 3. Use Novita AI in LoLLMS WebUI * Open your browser and navigate to [http://localhost:9600](http://localhost:9600) (or the port shown in the terminal). * Select Novita AI from the available bindings. * Enter your Novita AI API key. * Select your desired Novita AI model. [******Here is an application: Vibe Coding Using Novita AI Bindings and Services on LoLLMS WebUI.******](https://www.youtube.com/watch?v=jyFaP4zTM9g) # MLflow Source: https://docs.novita.ai/guides/mlflow Trace Novita AI calls in MLflow with OpenAI-compatible SDK instrumentation for Python and JavaScript. This guide shows how to integrate **Novita AI** with MLflow Tracing. By using Novita AI's OpenAI-compatible endpoint (`https://api.novita.ai/openai`), you can capture prompts, responses, latency, token usage, and model metadata in MLflow. MLflow trace details and timeline

# Prerequisites Before you start, make sure you have: * Novita AI API key: create one in [Key Management](https://novita.ai/settings/key-management). * A running MLflow tracking server. You can use local default `http://localhost:5000`. * Python or JavaScript runtime. # Integration Steps ## Step 1: Install Dependencies ```bash Python icon="python" theme={"system"} pip install 'mlflow[genai]' openai ``` ```bash JavaScript / TypeScript icon="js" theme={"system"} npm install @mlflow/openai openai ``` ## Step 2: Start MLflow Server If you have a local Python environment >= 3.10, you can start MLflow with: ```bash theme={"system"} mlflow server ``` MLflow also provides a Docker Compose setup: ```bash theme={"system"} git clone --depth 1 --filter=blob:none --sparse https://github.com/mlflow/mlflow.git cd mlflow git sparse-checkout set docker-compose cd docker-compose cp .env.dev.example .env docker compose up -d ``` Then open `http://localhost:5000` to confirm the MLflow UI is accessible. ## Step 3: Enable Tracing and Call Novita AI ```python Python icon="python" theme={"system"} import openai import mlflow # Enable auto-tracing for OpenAI-compatible calls mlflow.openai.autolog() # Optional: set tracking target and experiment mlflow.set_tracking_uri("http://localhost:5000") mlflow.set_experiment("Novita AI") client = openai.OpenAI( base_url="https://api.novita.ai/openai", api_key="", ) response = client.chat.completions.create( model="deepseek/deepseek-v4.1-flash", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the capital of France?"}, ], ) print(response.choices[0].message.content) ``` ```ts JavaScript / TypeScript icon="js" theme={"system"} import { OpenAI } from "openai"; import { tracedOpenAI } from "@mlflow/openai"; const client = tracedOpenAI( new OpenAI({ baseURL: "https://api.novita.ai/openai", apiKey: "", }) ); const response = await client.chat.completions.create({ model: "deepseek/deepseek-v4.1-flash", messages: [ { role: "system", content: "You are a helpful assistant." }, { role: "user", content: "What is the capital of France?" }, ], temperature: 0.1, max_tokens: 100, }); console.log(response.choices[0].message?.content); ``` ## Step 4: View Traces in MLflow UI Open your MLflow UI (for example `http://localhost:5000`) and go to your configured experiment to inspect traces. You should see: * Prompt and completion content * Latency and token usage * Model and request metadata * Errors/exceptions (if any) ## Step 5: Advanced Tracing References ### Streaming and Async MLflow supports tracing for streaming and async Novita AI APIs. See: * [OpenAI Tracing](https://mlflow.org/docs/latest/genai/tracing/integrations/listing/openai/) ### Combine with frameworks or manual tracing ```python Python icon="python" theme={"system"} import json from openai import OpenAI import mlflow from mlflow.entities import SpanType # Initialize the OpenAI client with Novita AI API endpoint client = OpenAI( base_url="https://api.novita.ai/openai", api_key="", ) # Create a parent span for the Novita AI call @mlflow.trace(span_type=SpanType.CHAIN) def answer_question(question: str): messages = [{"role": "user", "content": question}] response = client.chat.completions.create( model="deepseek/deepseek-v4.1-flash", messages=messages, ) # Attach session/user metadata to the trace mlflow.update_current_trace( metadata={ "mlflow.trace.session": "session-12345", "mlflow.trace.user": "user-a", } ) return response.choices[0].message.content answer = answer_question("What is the capital of France?") ``` ```ts JavaScript / TypeScript icon="js" theme={"system"} import * as mlflow from "@mlflow/core"; import { OpenAI } from "openai"; import { tracedOpenAI } from "@mlflow/openai"; mlflow.init({ trackingUri: "http://localhost:5000", experimentId: "", }); // Wrap the OpenAI client and point to Novita AI endpoint const client = tracedOpenAI( new OpenAI({ baseURL: "https://api.novita.ai/openai", apiKey: "", }) ); // Create a traced function that wraps the Novita AI call const answerQuestion = mlflow.trace( async (question: string) => { const resp = await client.chat.completions.create({ model: "deepseek/deepseek-v4.1-flash", messages: [{ role: "user", content: question }], }); return resp.choices[0].message?.content; }, { name: "answer-question" } ); await answerQuestion("What is the capital of France?"); ``` For full upstream reference: * MLflow Novita AI integration page: [Tracing Novita AI](https://mlflow.org/docs/latest/genai/tracing/integrations/listing/novitaai/) * MLflow OpenAI tracing docs: [OpenAI Tracing](https://mlflow.org/docs/latest/genai/tracing/integrations/listing/openai/) For Novita model details and endpoint usage, see: * Novita LLM API guide: [LLM API](/guides/llm-api) # A Brief Introduction to Clip Skip Source: https://docs.novita.ai/guides/model-apis-clip-skip Clip Skip is a feature that literally skips part of the image generation process, resulting in slightly different outputs. This also leads to faster image rendering. ### But why would anyone want to skip a part of the diffusion process? A typical Stable Diffusion 1.5 base model image goes through 12 "clip" layers, which represent levels of refinement. The early layers are very broad, while the later layers produce images that are clearer and more specific. In the case of some base models, especially those based on Danbooru tags, trial and error has shown that skipping certain layers can lead to better images, as the broad clip layers may introduce unwanted noise. You can literally skip layers, saving GPU time and achieving better art. If you want a picture of "a cow", you might not care about the subcategories of "cow" that the text model might have. Especially since these can vary in quality. So if you want "a cow", you might not want "an Aberdeen Angus bull". (The full post is at the bottom of this page.) You can think of CLIP Skip as a setting for "how accurate you want the text model to be". You can test it out and see that each clip stage adds more definition in terms of description. For example, if you have a detailed prompt about a young man standing in a field, with lower clip stages you’d get an image of "a man standing", while deeper stages would yield "a young man standing" or "a young man standing in a forest", and so on. CLIP Skip becomes particularly effective when using models that are structured in a specific way, where the "1girl" tag can break down into many sub-tags connected to that main tag. ### Do I need it? It’s a minor optimization recommended only for hardcore, quality-obsessed enthusiasts. If you’re working on anime or semi-realism, it’s worth a try. ### Limitations **How many layers to skip** Generally speaking, skipping 1 or 2 layers can yield good results. However, skipping more than 2 layers may produce images that appear to have low guidance. **Inconsistent compatibility** Clip Skip has become one of those "wear your seatbelt" kinds of safe defaults, where many people prefer to set it and forget it. This approach isn’t wise. The feature can also yield unpredictable results when used with other technologies, such as LoRAs and Textual Inversions. Missing layers where layers are expected can degrade the image quality or have no effect at all. It’s often faster to simply try it out than to compare results. Just make sure you also lock the seed, guidance, concept, and sampler to accurately assess the differences. # Configure Custom AWS S3 Bucket Source: https://docs.novita.ai/guides/model-apis-configure-custom-s3-bucket By default, Novita AI temporarily stores output results in our private S3 bucket and returns the results to the user through a temporary authorized S3 link. However, you can set up a **Custom S3 Bucket** to allow us to save the results in your own bucket. Please follow the steps below to enable this. ## 1. Configure S3 Bucket Policy First, change your S3 Bucket Policy configuration to the following format (replace `${BucketName}` with your bucket name): ```json theme={"system"} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "CanonicalUser": "e98cde8d11ec7c03ac08688f1a933b08b0f0f7746b21c4f2e7b2c8202cc0532f" }, "Action": [ "s3:PutObject", "s3:PutObjectAcl" ], "Resource": "arn:aws:s3:::${BucketName}/*" } ] } ``` ## 2. Enable Custom Storage in V3 APIs For V3 API endpoints, Novita AI provides the `custom_storage` parameter in the request body, allowing you to configure your custom S3 bucket for storing generated images. Here's an example using the `txt2img` API endpoint: ```bash theme={"system"} curl --location 'https://api.novita.ai/v3/async/txt2img' \ --header 'Authorization: Bearer {{API Key}}' \ --header 'Content-Type: application/json' \ --data '{ "extra": { "response_image_type": "jpeg", "custom_storage": { "aws_s3": { "region": "us-east-2", "bucket": "test_bucket", "path": "/" } } }, "request": { "prompt": "a cute dog", "model_name": "realisticVisionV51_v51VAE_94301.safetensors", "negative_prompt": "", "width": 512, "height": 384, "image_num": 2, "steps": 20, "seed": 123, "clip_skip": 1, "sampler_name": "Euler a", "guidance_scale": 7.5 } }' ``` # Upload Custom LoRA Models Source: https://docs.novita.ai/guides/model-apis-custom-model * Currently, the Model Upload feature only supports LoRA models, with a limit of 5 uploads per user. We plan to gradually lift these restrictions in the future; * The Model Upload feature is temporarily available free of charge. However, we may introduce a fee for this service in the future. Please stay tuned for future announcements; * If you have any questions, please contact our customer service or technical team through the [Discord Channel](https://discord.gg/YyPRAzwp7P). 1. Log in to your account and go to the Model Management page. 2. Click the `Upload Model` button to upload your models. 3. Upload the LoRA model and fill in the form. Please note that the model name cannot be modified after submission. 4. After the model is uploaded successfully, you can use the model from the model list. **For example:** In the following example, the LoRA model name `test_sd_name_in_api.safetensors` can be found in the `MODEL NAME IN API` column in your private model list. ```bash theme={"system"} curl --location --request POST 'https://api.novita.ai/v3/async/txt2img' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer {{API Key}}' \ --data-raw '{ "model_name": "AnythingV5_v5PrtRE.safetensors", "width": 512, "height": 768, "prompt": "a body stood high, behind him, looking down.", "negative_prompt": "", "sampler_name": "DPM++ 2S a Karras", "guidance_scale": 7.5, "steps": 30, "image_num": 1, "clip_skip": 0, "seed": -1, "loras": [ {"model_name":"test_sd_name_in_api.safetensors", "strength":0.7} ] }' ``` # Overview Source: https://docs.novita.ai/guides/model-apis-overview Novita AI deploys state-of-the-art open-source models. Use our APIs to access them out of the box! Test prompts, explore our models, and compare results instantly, no coding required. Get started with Novita's OpenAI-compatible APIs. Try the Image, Audio and Video Generation Models in the Playground. Explore our API references. # Rate limits Source: https://docs.novita.ai/guides/model-apis-rate-limits ## 1. Understanding Rate Limits Rate limiting refers to the rules that govern how frequently a user’s API may access our platform services within a given time window. Its goals are: * **Prevent API abuse and misuse:** By setting an upper bound on request frequency, we limit abnormal or unintended traffic. * **Ensure fair resource allocation:** We prevent a small number of users from monopolizing system resources, so everyone can access the service on equal terms. * **Maintain API performance and reliability:** By smoothing out traffic spikes, we stabilize response times and reduce failure rates due to overload, improving overall service quality. * **Protect system stability:** Rate limits help absorb sudden bursts of traffic, preventing high-concurrency spikes from overwhelming our infrastructure. ## 2. Rate Limiting Metrics For our image and video models, we use two primary rate-limiting metrics: * **IPM (Images Per Minute):** The number of images an image model is allowed to generate per minute. * **RPM (Requests Per Minute):** The number of API requests a video model is allowed to handle per minute. ## 3. Default Rate Limits For different models, our platform applies differentiated rate-limiting strategies based on each model’s computational resource consumption. ### **IPM** The IPM (Images Per Minute) limit controls the number of images that can be generated per minute. The default IPM values for each model are listed in the table below. | Resource/Service | Model API | User Default Settings | | :- | :- | :- | | Text to Image | `txt2img_v3` | 20 | | Image to Image | `img2img_v3` | 10 | | Remove Background | `remove_background` | 10 | | Replace Background | `replace_background` | 10 | | Remove Text | `remove_text` | 10 | | Inpainting | `inpainting` | 10 | | Cleanup | `cleanup` | 10 | | Merge Face | `merge_face` | 10 | | FLUX.1 \[schnell] Text to Image | `flux-1-schnell` | 10 | | Upscale | `upscale_v3` | 20 | ### **RPM** The RPM (Requests Per Minute) limit controls the number of API requests allowed per minute. The default RPM values for each model are provided in the table below. | Resource/Service | Model API | User Default Settings | | :- | :- | :- | | Video Merge Face | `video_merge_face` | 10 | | Text to Video | `txt2video` | 2 | | Image to Video | `img2video` | 2 | | Wan 2.1 Text to Video | `wan_txt_to_video` | 20 | | Wan 2.1 Image to Video | `wan i2v` | 20 | | Hunyuan Video Fast | `hunyuan_video_fast` | 20 | | KLING V1.6 Image to Video | `Kling i2v` | 20 | | KLING V1.6 Text to Video | `Kling t2v` | 20 | ## 4. Handling Rate Limits ### How to Monitor Rate Limits? If your API requests exceed the allowed rate, the API will return: * **HTTP Status Code:** 429 Too Many Requests * **Response Body:** A message indicating that the rate limit has been exceeded. ### Best Practices **To avoid triggering rate limits, you can:** * **Implement client-side request throttling:** Respect the platform’s rate limits by controlling your application’s request rate, ensuring you don’t send too many calls in a short period. * **Use exponential backoff on retries:** When you receive a rate-limit response (e.g., HTTP 429), wait progressively longer between retry attempts instead of retrying immediately, reducing load on the service. * **Monitor your API usage:** Continuously track and log your request counts, frequencies, and any error responses so you can adjust your usage patterns proactively. ### When You Hit Rate Limits **If you receive an HTTP 429 ("Too Many Requests") response, you can:** * **Retry later:** Wait a short period before sending your request again. * **Optimize your requests:** Reduce the frequency of calls to stay within the platform’s rate limits. * **Request a higher rate limit:** If you require a higher rate limit, please fill out the contact form at the [Quotas\&Limits](https://novita.ai/quota-limits/image) to get in touch with us. # A Brief Introduction to Sampler Source: https://docs.novita.ai/guides/model-apis-sampler Stable Diffusion Art. (n.d.). Samplers. Retrieved from [https://stable-diffusion-art.com/samplers/.](https://stable-diffusion-art.com/samplers/) Many sampling methods are available in Novita AI, including Euler a, Heun, and DDIM. What are samplers? How do they work? What are the differences between them? Which one should you use? You will find the answers in this article. To produce an image, Stable Diffusion first generates a completely random image in the latent space. The noise predictor then estimates the noise of the image, which is subsequently subtracted from the image. This process is repeated multiple times, resulting in a clean image. This denoising process is called sampling because Stable Diffusion generates a new sample image at each step. The method used in sampling is referred to as the `sampler` or `sampling method`. Sampling is just one part of the Stable Diffusion model. Read the article "How Does Stable Diffusion Work?" if you want to understand the entire model. Below is a sampling process in action. The sampler gradually produces cleaner and cleaner images. While the framework is the same, there are many different ways to carry out this denoising process. It is often a trade-off between speed and accuracy. ### Samplers Overview As of this writing, there are 19 samplers available in Novita AI, and this number seems to be growing over time. What are the differences? You will learn about them in the later part of this article. The technical details can be overwhelming, so I will include a bird's-eye view in this section to help you get a general idea of what they are. #### Old-School ODE Solvers Let’s start with the easier ones. Some of the samplers on the list were invented more than a hundred years ago. They are old-school solvers for ordinary differential equations (ODE). * Euler – The simplest possible solver. * Heun – A more accurate but slower version of Euler. * LMS (Linear Multi-Step Method) – Same speed as Euler but supposedly more accurate. #### Ancestral Samplers Do you notice that some sampler names have a single letter "a"? * Euler a * DPM2 a * DPM++ 2S a * DPM++ 2S a Karras These are ancestral samplers. An ancestral sampler adds noise to the image at each sampling step. They are stochastic samplers because the sampling outcome contains some randomness. Be aware that many other samplers are also stochastic, even if their names do not include an "a". The drawback of using an ancestral sampler is that the image may not converge. Compare the images generated using Euler a and Euler below. Euler a does not converge. (sampling steps 2–40) Euler converges. (sampling steps 2-40) Images generated with Euler a do not converge at high sampling steps, while images from Euler converge well. For reproducibility, it is desirable to have the image converge. If you want to generate slight variations, you should use a variational seed. #### Karras Noise Schedule The samplers labeled "Karras" use the noise schedule recommended in the [Karras article](https://arxiv.org/abs/2206.00364). If you look closely, you will see that the noise step sizes are smaller near the end. This adjustment improves the quality of the images. #### DDIM and PLMS DDIM (Denoising Diffusion Implicit Model) and PLMS (Pseudo Linear Multi-Step Method) were the samplers included with the original Stable Diffusion v1. DDIM is one of the first samplers designed for diffusion models, while PLMS is a newer and faster alternative to DDIM. They are generally considered outdated and are not widely used anymore. #### DPM and DPM++ DPM (Diffusion Probabilistic Model Solver) and DPM++ are new samplers designed for diffusion models released in 2022. They represent a family of solvers with similar architecture. DPM and DPM2 are similar, with DPM2 being second-order (more accurate but slower). DPM++ is an improvement over DPM. DPM adaptive adjusts the step size adaptively. It can be slow since it doesn’t guarantee completion within the specified number of sampling steps. #### UniPC UniPC (Unified Predictor-Corrector) is a new sampler released in 2023. Inspired by the predictor-corrector method in ODE solvers, it can achieve high-quality image generation in 5-10 steps. ### How to Pick a Sampler In this section, I will provide some objective comparisons to help you decide. #### Image Convergence In this section, I will generate the same image using different samplers with up to 40 sampling steps. The last image at the 40th step will be used as a reference for evaluating how quickly the sampling converges, with the Euler method serving as the benchmark. > Euler, DDIM, PLMS, LMS Karras and Heun First, let’s look at Euler, DDIM, PLMS, LMS Karras, and Heun as a group since they represent old-school ODE solvers or original diffusion solvers. DDIM converges at about the same number of steps as Euler but with more variations due to the injection of random noise during its sampling steps. PLMS did not perform well in this test. LMS Karras seems to have difficulty converging and stabilizes at a higher baseline. Heun converges faster but is twice as slow since it is a second-order method. Therefore, we should compare Heun at 30 steps with Euler at 15 steps, for example. > Ancestral Samplers If a stable, reproducible image is your goal, you should avoid using ancestral samplers, as they all fail to converge. > DPM and DPM2 DPM fast did not converge well. DPM2 and DPM2 Karras performs better than Euler but again in the expense of being two times slower. DPM adaptive performs deceptively well because it uses its own adaptive sampling steps. It can be very slow. > DPM++ solvers DPM++ SDE and DPM++ SDE Karras suffer the same shortcoming as ancestral samplers. They not only don’t converge, but the images also fluctuate significantly as the number of steps changes. DPM++ 2M and DPM++ 2M Karras perform well. The Karras variant converges faster when the number of steps is high enough. > UniPC UniPC converges a bit slower than Euler, but not too bad. #### Speed Although DPM adaptive performs well in convergence, it is also the slowest. You may have noticed the rest of the rendering times fall into two groups, with the first group taking about the same time (about 1x), and the other group taking about twice as long (about 2x). This reflects the order of the solvers. 2nd order solvers, although more accurate, need to evaluate the denoising U-Net twice. So they are 2x slower. #### Quality Of course, speed and convergence mean nothing if the images look crappy. **Final images** Let’s first look at samples of the image. DPM++ fast failed pretty badly. Ancestral samples did not converge to the image that other samplers converged to. Ancestral samplers tend to converge to an image of a kitten, while the deterministic ones tend to converge to a cat. There are no correct answers as long as they look good to you. Perceptual quality An image can still look good even if it hasn’t converged. Let’s look at how quickly each sampler can produce a high-quality image. You will see perceptual quality measured with BRISQUE (Blind/Referenceless Image Spatial Quality Evaluator). It measures the quality of natural images. DDIM is doing surprisingly well here, capable of producing the highest quality image within the group in as few as 8 steps. ### So… which one is the best? Here are my recommendations: 1. If you want to use something fast, converging, new, and with decent quality, excellent choices are * DPM++ 2M Karras with 20 – 30 steps * UniPC with 20-30 steps. 2. If you want good quality images and don’t care about convergence, good choices are * DPM++ SDE Karras with 10-15 steps (Note: This is a slower sampler) * DDIM with 10-15 steps. 3. Avoid using any ancestral samplers if you prefer stable, reproducible images. 4. Euler and Heun are fine choices if you prefer something simple. Reduce the number of steps for Heun to save time. # SDKs for Model APIs Source: https://docs.novita.ai/guides/model-apis-sdks Novita AI provides official SDKs to help you easily integrate our services into your applications. ## JavaScript SDK ### Installation Install our JavaScript SDK using `npm`: ```bash theme={"system"} npm i novita-sdk ``` ### Quick Start Visit [https://github.com/novitalabs/javascript-sdk](https://github.com/novitalabs/javascript-sdk) for a quick start. ## Python SDK ### Installation Install our Python SDK using `pip`: ```bash theme={"system"} pip install novita-client ``` ### Quick Start Visit [https://github.com/novitalabs/python-sdk](https://github.com/novitalabs/python-sdk) for a quick start. # Training Image Caption Guidance Source: https://docs.novita.ai/guides/model-apis-training-guidance ### Preparation Depending on how and what you are training, you may need to crop the photos to a specific width and height. Other types of training may categorize images into various sizes and do not require cropping. Look into what is required for the training method you are using, the model you are training, and the program you are using to train your model. ### Captioning – General Notes #### Avoid Automated Captioning * BLIP and deepbooru are exciting, but it is still a bit early for them. * BLIP and deepbooru struggle with context and relative importance. * It is faster to manually caption than to fix mistakes made by BLIP or deepbooru and still have to manually caption. #### Caption in the Same Manner You Prompt * Captioning and prompting are related. * Recognize how you typically prompt. Do you use verbose sentences? Short descriptions? Vague or detailed prompts? * Caption in a similar style and verbosity to how you usually prompt. #### Follow a Set Structure per Concept * Following a structure will benefit the learning process. * You might have one structure for photographs and another for illustrations. However, try to avoid mixing and matching structures when captioning a single dataset. #### Captions are Like Variables You Can Use in Your Prompts Everything you describe in a caption can be thought of as a variable that you can manipulate in your prompt. This has two implications: 1. You want to describe as many details as possible about anything that isn’t the concept you are trying to implicitly teach. In other words, describe everything that you want to become a variable. For example, if you are teaching a specific face but want to be able to change the hair color, you should describe the hair color in each image so that "hair color" becomes one of your variables. 2. You don’t want to describe anything (beyond a class-level description) that you want to be implicitly taught. In other words, the thing you are trying to teach shouldn’t become a variable. For example, if you are teaching a specific face, you should not describe it as having a big nose. You don’t want the nose size to be variable because then it isn’t that specific face anymore. However, you can still caption "face" if you want, which provides context to the model you are training. This does have some implications described in the following point. #### Leveraging Classes as Tags * There are two concepts here. 1. Using generic class tags will bias that entire class toward your training data. This may or may not be desired depending on your goals. 2. Using generic class tags provides context to the learning process. Conceptually, it is easier to learn what a "face" is when the model already has a reasonable approximation of "face". * If you want to bias the entire class of your model toward your training images, use broad class tags rather than specific ones. For example, if you want to teach your model that every man should look like Brad Pitt, your captions should contain the tag "man" but should not be more specific than that. This influences your model to produce a Brad Pitt-looking man whenever you use the word "man" in your prompt. This also allows your model to draw on and leverage what it already knows about the concept of "man" while it is training. * If you want to reduce the impact of your training on the entire class, include specific tags and de-emphasize class tags. For example, if you want to teach your model that only "ohwxman" should look like Brad Pitt, and you don't want every "man" to look like Brad Pitt, you would not use "man" as a tag, only tagging it with "ohwxman". This reduces the impact of your training images on the tag "man" and strongly associates your training images with "ohwxman". Your model will draw on what it knows about "ohwxman", which is practically nothing (see note), thus building up knowledge almost solely from your training images, creating a very strong association. * **Note:** This is simplified for the sake of understanding. This would actually be tokenized into two tokens, "ohwx" and "man", but these tokens would be strongly correlated for training purposes, which should still reduce the impact on the overall class of "man" compared to training with "man" as a token in the caption. The mathematics involved is quite complex and well beyond the scope of this document. #### Consistent Captioning * Use consistent captions across all of your training. This will help you consistently invoke your concept when prompting. * Using inconsistent tags across your dataset will make the concept you are trying to teach harder for the model to grasp, as you are essentially forcing it to learn both the concept and the different phrasings for that concept. It’s much better to have it just learn the concept under a single term. * For example, you probably don’t want to have both "legs raised in air" and "raised legs" if you are trying to teach a single concept of a person with their legs up in the air. You want to be able to consistently invoke this pose in your prompt, so choose one way to caption it. #### Avoid Repetition * Try to avoid repetition wherever possible. Similar to prompting, repeating words increases the weighting of those words. * For example, if we repeat the word "background" too much, we may have three tags that say "background" (e.g., simple background, white background, lamp in background). Even though we want the background to have low weight, we have unintentionally increased the weighting significantly. It would be better to combine these or reword them (e.g., simple white background with a lamp). #### Take Note of Ordering * Again, just like with prompting, order matters for the relative weighting of tags. * Having a specific structure or order that you generally use for captions can help you maintain the relative weightings of tags between images in your dataset, which should benefit the training process. * Having a standardized ordering makes the whole captioning process faster as you become familiar with captioning in that structure. #### Use Your Model's Existing Knowledge to Your Advantage * Your model already produces decent results and reasonably understands what you are prompting. Take advantage of that by captioning with words that already work well in your prompts. * You want to use descriptive words, but if you use words that are too obscure or niche, you likely can't leverage much of the existing knowledge. For example, you could say "sarcastic" or "mordacious". The model has some idea of what "sarcastic" conveys, but it likely has no clue what "mordacious" means. * You can also look at this from the opposite perspective. If you were trying to teach the concept of "mordacious", you might have a dataset full of images that convey "sarcastic" and caption them with both the tags "sarcastic" and "mordacious" side by side (so that they are close in relative weighting). ### Captioning – Structure This is mainly for people or characters, so it might not be quite as applicable to something like fantasy landscapes, but perhaps it can provide some inspiration. #### General Format #### Loose Associations * This is where we put any final loose associations we have with the image. * This could be anything that pops up in our head, usually “feelings” that we feel when looking at the image or concepts we feel are portrayed, really anything goes here as long as it exists in the image. * Keep in mind this is for loose associations. If the image is very obviously portraying some feeling, we may want it tagged closer to the start of the caption for higher weighting. * For example: happy, sad, joyous, hopeful, lonely, sombre #### FULL EXAMPLE OF A SINGLE IMAGE This is an example of how we would caption a single image we picked off of safebooru. We will assume that I want to train the style of this image and associate it with the tag "ohwxStyle", and we will assume that we have many images in this style within our dataset. Sample Image: [https://safebooru.org/index.php?page=post\&s=view\&id=3887414](https://safebooru.org/index.php?page=post\&s=view\&id=3887414) * Globals: ohwxStyle * Type or Perspective Of a: anime, drawing, of a young woman, full body shot, from side * Action words: sitting, looking at viewer, smiling, head tilt, holding a phone, eyes closed * Subject description: short brown hair, pale pink dress with dark edges, stuffed animal in lap, brown slippers * Notable details: sunlight through windows as lighting source * Background or location: brown couch, red patterned fabric on couch, wooden floor, white water-stained paint on walls, refrigerator in background, coffee machine sitting on a countertop, table in front of couch, bananas and coffee pot on table, white board on wall, clock on wall, stuffed animal chicken on floor * Loose associations: dreary environment All together: ohwxStyle, anime, drawing, of a young woman, full body shot, from side, sitting, looking at viewer, smiling, head tilt, holding a phone, eyes closed, short brown hair, pale pink dress with dark edges, stuffed animal in lap, brown slippers, sunlight through windows as lighting source, brown couch, red patterned fabric on couch, wooden floor, white water-stained paint on walls, refrigerator in background, coffee machine sitting on a countertop, table in front of couch, bananas and coffee pot on table, white board on wall, clock on wall, stuffed animal chicken on floor, dreary environment # API V2 to V3 Migration Guide Source: https://docs.novita.ai/guides/model-apis-v2-to-v3-migration ## Text to Image ### Request Body Parameter Mapping

V2	V3	Description
extraobject	extraobject
enable\_nsfw\_detection boolean	enable\_nsfw\_detection boolean
nsfw\_detection\_level Enum: `0, 1, 2`	nsfw\_detection\_level Enum: `0, 1, 2`
enable\_progress\_info		Deprecated
response\_image\_type Enum: `png`, `jpeg`	response\_image\_type Enum: `png, webp, jpeg`	V3 adds support for `webp`image format
	requestobject	New Field All image generation parameters must be passed via the `request`in V3
promptstring \	promptstring	Moved Inside
	lorasobject\[]	Moved Inside Migrate LoRA usage: From `prompt` to `request.loras` parameter
	model\_namestring	New Field Name of lora, retrieve the corresponding sd\_name\_in\_api value by invoking the [Get Model API](https://novita.ai/docs/api-reference/model-apis-get-model) endpoint with filter.types=lora as the query parameter.
	strengthnumber(float32)	New Field The strength value of lora. The larger the value, the more biased the effect is towards lora, Range \[0, 1]
negative\_promptstring	negative\_promptstring	Moved Inside
sampler\_namestring	sampler\_namestring	Moved Inside
batch\_sizeinteger	image\_numinteger	Changed `num_images` → `request.image_num`
n\_iter		Deprecated
stepsstring	stepsstring	Moved Inside
cfg\_scaleinteger	guidance\_scale number(float32)	Changed `cfg_scale`→ `request.guidance_scale`
seedinteger	seedinteger	Moved Inside
heightinteger	heightinteger	Moved Inside Range Change: \[128, 2048].
widthinteger	widthinteger	Moved Inside Range Change: \[128, 2048].
model\_namestring	model\_namestring	Moved Inside This parameter specifies the name of the model checkpoint. Retrieve the corresponding sd\_name value by invoking the [Query Model](https://novita.ai/docs/api-reference/model-apis-get-model) API with filter.types=checkpoint as the query parameter.
restore\_facesbool	restore\_facesbool	Moved Inside
restore\_faces\_model		Deprecated
sd\_vaestring	sd\_vaestring	Moved Inside
clip\_skipinteger	clip\_skipinteger	Moved Inside
enable\_hrboolean	hires\_fixobject	Changed `enable_hr`→ `request.hires_fix`
hr\_upscaler Enum: `Latent`, `ESRGAN_4x`, `R-ESRGAN 4x+`, `R-ESRGAN 4x+ Anime6B`	upscaler Enum: `RealESRGAN_x4plus_anime_6B`, `RealESRNet_x4plus,Latent`	Changed `hr_upscaler`→`request.hires_fix.upscaler`
hr\_scalenumber		Deprecated
hr\_resize\_xinteger	target\_widthinteger	Changed `hr_resize_x`→ `request.hires_fix.target_width`
hr\_resize\_yinteger	target\_heightinteger	Changed `hr_resize_y`→ `request.hires_fix.target_height`
img\_expire\_ttlinteger		Deprecated Default 3600s
sd\_refinerobject	refinerobject	Changed `sd_refiner`→ `request.refiner`
checkpointstring		Deprecated
switch\_at number(float32)	switch\_at number(float32)	Changed `sd_refiner.switch_at`→ `request.refiner.switch_at`
controlnet\_unitsobject\[]		Deprecated `img2img` Only

### Response Parameter Mapping

V2	V3	Description
code		Deprecated
msg		Deprecated
data		Deprecated
task\_id	task\_id	Changed `data.task_id`→ `task_id`
warn		Deprecated

### Example * Request ```bash theme={"system"} curl --location "https://api.novita.ai/v2/txt2img" \ --header "Authorization: Bearer {{API Key}}" \ --header "Content-Type: application/json" \ --data '{ "extra": { "enable_nsfw_detection": false, "nsfw_detection_level": 0, "response_image_type": "jpeg" }, "prompt": "(masterpiece, best quality, ultra-detailed:1.3), , 1girl, mecha armor, futuristic city background, glowing neon lights, dynamic pose, cyberpunk style, intricate mechanical details, (floating holographic interface:1.2), sparkling blue eyes, wind-blown hair, (steam and energy particles:1.1), cinematic lighting", "negative_prompt":"(worst quality, low quality:1.4), (deformed, distorted:1.3), disfigured, blurry, bad anatomy, extra limbs, (mutated hands:1.2), (text, watermark:1.5), overexposed, underexposed, (cartoonish:1.2), (simplistic background:1.3), grainy, (traditional drawing:1.2), (realistic:1.4), (noisy:1.3)", "sampler_name":"DPM++ 2M Karras", "batch_size": 1, "n_iter": 1, "steps": 25, "cfg_scale": 7, "seed": 65531, "height": 512, "width": 512, "model_name": "sd_xl_base_1.0.safetensors", "restore_faces": false, "restore_faces_model": "", "sd_vae": "", "clip_skip": 1, "enable_hr": true, "hr_upscaler": "Latent", "hr_scale": 2, "hr_resize_x": 1024, "hr_resize_y": 1024, "img_expire_ttl": 3600 }' ``` * Response ``` { "code": 0, "msg": "", "data": { "task_id": "12905b6a-d436-4010-b199-d800130c0aab" } } ``` * Reqeust ```bash theme={"system"} curl --location 'https://api.novita.ai/v3/async/txt2img' \ --header 'Authorization: Bearer {{API Key}}' \ --header 'Content-Type: application/json' \ --data '{ "extra": { "enable_nsfw_detection": false, "nsfw_detection_level": 0, "response_image_type": "jpeg" }, "request": { "prompt": "(masterpiece, best quality, ultra-detailed:1.3), 1girl, mecha armor, futuristic city background, glowing neon lights, dynamic pose, cyberpunk style, intricate mechanical details,(floating holographic interface:1.2), sparkling blue eyes, wind-blown hair, (steam and energy particles:1.1), cinematic lighting", "negative_prompt": "(worst quality, low quality:1.4), (deformed, distorted:1.3), disfigured,blurry, bad anatomy, extra limbs, (mutated hands:1.2), (text, watermark:1.5),overexposed, underexposed,(cartoonish:1.2), (simplistic background:1.3), grainy, (traditional drawing:1.2),(realistic:1.4), (noisy:1.3)", "model_name": "sd_xl_base_1.0.safetensors", "width": 512, "height": 512, "image_num": 1, "steps": 25, "seed": 65531, "restore_faces": false, "clip_skip": 1, "sampler_name":"DPM++ 2M Karras", "guidance_scale": 7, "hires_fix": { "target_width": 1024, "target_height": 1024, "strength": 0.8, "upscaler": "Latent" }, "loras": [ { "model_name": "AnimeStyle_XL_v1_AutoRunMech_103883", "strength": 0.8 } ] } }' ``` * Repspone ```bash theme={"system"} { "task_id":"0696ce57-22df-4a89-9efc-80c386a82dbd" } ``` ## Image to Image ### Request Body Parameter Mapping

V2	V3	Description
extraobject	extraobject
enable\_nsfw\_detection boolean	enable\_nsfw\_detection boolean
nsfw\_detection\_level Enum: `0, 1, 2`	nsfw\_detection\_level Enum: `0, 1, 2`
enable\_progress\_info		Deprecated
response\_image\_type Enum: `png`, `jpeg`	response\_image\_type Enum: `png, webp, jpeg`	V3 adds support for `webp`image format
	requestobject	New field All image generation parameters must be passed via the `request`in V3
promptstring \	promptstring	Moved Inside
	lorasobject\[]	Moved Inside Migrate LoRA usage: From `prompt` to `request.loras` parameter
	model\_namestring	New Field Name of lora, retrieve the corresponding sd\_name\_in\_api value by invoking the [Get Model API](https://novita.ai/docs/api-reference/model-apis-get-model) endpoint with filter.types=lora as the query parameter.
	strength number(float32)	New Field The strength value of lora. The larger the value, the more biased the effect is towards lora, Range \[0, 1]
negative\_promptstring	negative\_prompt string	Moved Inside
sampler\_namestring	sampler\_namestring	Moved Inside
batch\_sizeinteger	image\_numinteger	Changed `batch_size`→ `request.image_num`
n\_iterinteger		Deprecated
stepsstring	stepsstring	Moved Inside
cfg\_scaleinteger	guidance\_scale number(float32)	Changed `cfg_scale`→ `request.guidance_scale`
seedinteger	seedinteger	Moved Inside
heightinteger	heightinteger	Moved Inside Range Change: \[128, 2048].
widthinteger	widthinteger	Moved Inside Range Change: \[128, 2048].
model\_namestring	model\_namestring	Moved Inside This parameter specifies the name of the model checkpoint. Retrieve the corresponding sd\_name value by invoking the [Query Model](https://novita.ai/docs/api-reference/model-apis-get-model) API with filter.types=checkpoint as the query parameter.
init\_imagesstring\[]	image\_base64string	Changed `init_images`→ `request.image_base64`
denoising\_strength number(float)	strength number(float)	Changed `denoising_strength`→ `request.strength`
restore\_facesbool		Deprecated
sd\_vaestring	sd\_vaestring	Moved Inside
clip\_skipinteger	clip\_skipinteger	Moved Inside
maskstring		Deprecated Recommendation: Use V3 Inpainting API
mask\_blurinteger		Deprecated Recommendation: Use V3 Inpainting API
resize\_modeinteger		Deprecated
image\_cfg\_scaleinteger		Deprecated
inpainting\_fillinteger		Deprecated Recommendation: Use V3 Inpainting API
inpaint\_full\_resinteger		Deprecated Recommendation: Use V3 Inpainting API
inpaint\_full\_res\_padding integer		Deprecated Recommendation: Use V3 Inpainting API
inpainting\_mask\_invert integer		Deprecated Recommendation: Use V3 Inpainting API
initial\_noise\_multiplier number(float32)		Deprecated
img\_expire\_ttlinteger		Deprecated Default 3600s
sd\_refinerobject	refinerobject	Changed `sd_refiner`→ `request.refiner`
checkpoint		Deprecated
switch\_at number(float32)	switch\_at number(float32)	Moved Inside
	controlnetobject	New Field
controlnet\_unitsobject\[]	unitsobject\[]	Changed `controlnet_units`→ `request.controlnet.units`
modelstring	model\_name string	Changed `controlnet_units.model`→ `request.controlnet.units.model_name`
weightnumber	strength number(float32)	Changed `controlnet_units.weight`→ `request.controlnet.units.strength`
input\_imagestring	image\_base64 string	Changed `controlnet_units.input_image`→ `request.controlnet.units.image_base64`
modulestring,Enum	preprocessor string,Enum	Changed `controlnet_units.module`→ `request.controlnet.units.preprocessor`
control\_mode		Deprecated
mask		Deprecated Recommendation: Use V3 Inpainting API
resize\_mode		Deprecated
processor\_res		Deprecated
threshold\_a		Deprecated
threshold\_b		Deprecated
guidance\_start number(float32)	guidance\_start number(float32)	Moved Inside
guidance\_end number(float32)	guidance\_end number(float32)	Moved Inside
pixel\_perfect		Deprecated

### Response Parameter Mapping

V2	V3	Description
code		Deprecated
msg		Deprecated
data		Deprecated
task\_id	task\_id	Changed `data.task_id`→ `task_id`
warn		Deprecated

### Example * Request ``` curl --request POST \ --url https://api.novita.ai/v2/img2img \ --header 'Authorization: ' \ --header 'Content-Type: application/json' \ --data '{ "extra": { "enable_nsfw_detection": false, "nsfw_detection_level": 0, "response_image_type": "png" }, "prompt": "realistic, photograph, (masterpiece), 8k quality, (detailed eyes:1.2), (highest quality:1.1), highly detailed, majestic, top quality, best quality, newest, ai-generated, (intricate details:1.1), extremely beautiful, elegant, majestic, immersive background+, (detailed face, perfect face), In the heart of a vibrant garden, a girl with red hair and brown eyes sits in contemplation of the nature around her.", "negative_prompt": "(worst quality:1.5), (low quality:1.5), (normal quality:1.5), anime, cartoon, painting, drawing, illustration, manga, sketch, nudity, young, child, hairband, headband, horns, lowres, bad anatomy, bad hands, multiple eyebrow, (cropped), extra limb, missing limbs, deformed hands, long neck, long body, long torso, (bad hands), signature, username, artist name, conjoined fingers, deformed fingers, ugly eyes, imperfect eyes, skewed eyes, unnatural face, unnatural body, error, grain, jpeg artifacts", "sampler_name":"DPM++ 2M Karras", "batch_size": 1, "n_iter": 1, "steps": 25, "cfg_scale": 7.5, "seed": 765824, "height": 768, "width": 512, "model_name": "cyberrealistic_v31_62396.safetensors", "init_images": [ "" ], "denoising_strength": 0.8, "restore_faces": false, "clip_skip": 1, "img_expire_ttl": 3600, "controlnet_units": { "model": "control_v11p_sd15_lineart", "weight": 1, "input_image": "", "module": "lineart", "guidance_start": 0, "guidance_end": 1 } }' ``` * Response ``` { "code": 0, "msg": "", "data": { "task_id": "a1b0d99b-2ef0-4b20-8eeb-c641ae84ed45" } } ``` * Request ``` curl --request POST \ --url https://api.novita.ai/v3/async/img2img \ --header 'Authorization: ' \ --header 'Content-Type: ' \ --data '{ "extra": { "enable_nsfw_detection": false, "nsfw_detection_level": 0, "response_image_type": "png" }, "request": { "prompt": "realistic, photograph, (masterpiece), 8k quality, (detailed eyes:1.2), (highest quality:1.1), highly detailed, majestic, top quality, best quality, newest, ai-generated, (intricate details:1.1), extremely beautiful, elegant, majestic, immersive background+, (detailed face, perfect face), In the heart of a vibrant garden, a girl with red hair and brown eyes sits in contemplation of the nature around her.", "negative_prompt": "(worst quality:1.5), (low quality:1.5), (normal quality:1.5), anime, cartoon, painting, drawing, illustration, manga, sketch, nudity, young, child, hairband, headband, horns, lowres, bad anatomy, bad hands, multiple eyebrow, (cropped), extra limb, missing limbs, deformed hands, long neck, long body, long torso, (bad hands), signature, username, artist name, conjoined fingers, deformed fingers, ugly eyes, imperfect eyes, skewed eyes, unnatural face, unnatural body, error, grain, jpeg artifacts", "sampler_name": "DPM++ 2M Karras", "image_num": 1, "steps": 25, "guidance_scale": 7.5, "seed": 765824, "height": 768, "width": 512, "model_name": "cyberrealistic_v31_62396.safetensors", "image_base64": "", "strength": 0.8, "clip_skip": 1, "controlnet": { "units": [ { "model_name": "control_v11p_sd15_lineart", "strength": 1, "image_base64": "", "preprocessor": "lineart", "guidance_start": 0, "guidance_end": 1 } ] } } }' ``` * Response ```bash theme={"system"} { "task_id": "9490dde7-cd28-493e-9db0-8a38a65bf5e6" } ``` # What's the Variational Autoencoder VAE? Source: https://docs.novita.ai/guides/model-apis-vae ### Introduction to Autoencoders Autoencoders are a class of neural networks primarily used for unsupervised learning and dimensionality reduction. The fundamental idea behind autoencoders is to encode input data into a lower-dimensional representation and then decode it back to the original data, aiming to minimize the reconstruction error. The basic architecture of an autoencoder consists of two main components: the encoder and the decoder. * **Encoder**: The encoder is responsible for transforming the input data into a compressed or latent representation. It typically consists of one or more layers of neurons that progressively reduce the dimensions of the input. * **Decoder**: The decoder, on the other hand, takes the compressed representation produced by the encoder and attempts to reconstruct the original input data. Like the encoder, it often consists of one or more layers, but in reverse order, gradually increasing the dimensions. ### Variational Autoencoders (VAEs) Overview Variational Autoencoders (VAEs) address some of the limitations of traditional autoencoders by introducing a probabilistic approach to encoding and decoding. The motivation behind VAEs lies in their ability to generate new data samples by sampling from a learned distribution in the latent space, rather than from a fixed latent vector as was the case with vanilla autoencoders. This makes them suitable for generative tasks. * **Probabilistic Nature**: Unlike deterministic autoencoders, VAEs model the latent space as a probability distribution. This produces a probability distribution function over the input encodings instead of just a single fixed vector, allowing for a more nuanced representation of uncertainty in the data. The decoder then samples from this probability distribution. * **Role of Latent Space**: The latent space in VAEs serves as a continuous, structured representation of the input data. Since it is continuous by design, this allows for easy interpolation. Each point in the latent space corresponds to a potential output, enabling smooth transitions between different data points and ensuring that points closer in the latent space lead to similar generations. The concept can be elucidated through a straightforward example, as presented below. Encoders within a neural network are tasked with acquiring a representation of input images in the form of a vector. This vector encapsulates various features such as a subject’s smile, hair color, gender, age, etc., denoted as a vector similar to \[0.4, 0.03, 0.032, …]. In this illustration, the focus is narrowed to a singular latent representation, specifically the attribute of a "smile". Autoencoders vs VAEs - Sciforce MediumIn the context of Vanilla Autoencoders (AE), the smile feature is encapsulated as a fixed, deterministic value. In contrast, Variational Autoencoders (VAEs) are deliberately crafted to encapsulate this feature as a probabilistic distribution. This design choice facilitates the introduction of variability in generated images by enabling the sampling of values from the specified probability distribution. In summary, VAEs go beyond mere data reconstruction; they generate new samples and provide a probabilistic framework for understanding latent representations. The inclusion of probabilistic elements in the model’s architecture sets VAEs apart from traditional autoencoders. Compared to traditional autoencoders, VAEs offer a richer understanding of the data distribution, making them particularly powerful for generative tasks. # Novita Connect Source: https://docs.novita.ai/guides/novita-connect Run Claude Code, Codex, OpenCode, Pi and DeepSeek Harness on Novita models with one command. Novita Connect plugs Novita models into the coding harness you already use. Keep your tools, workflow and shortcuts; switch the model underneath with one command, and switch back by simply running your harness as usual.

The CLI is called `novi`. It starts your harness with Novita settings injected for that session only, so your own configuration is left untouched. Novita Connect is currently in **Beta**. Commands and the default model lineup may change. It runs on macOS and Linux (amd64 and arm64). ## Requirements * A Novita API key from [Key Management](https://novita.ai/settings/key-management). * macOS or Linux with `curl`. * The harness you want to use. If it isn't installed, the `novi` menu shows installation instructions and an official download link. ## Quickstart ```bash theme={"system"} curl -fsSL https://novita.ai/connect/install.sh | sh ``` The installer puts `novi` in `~/.local/bin`, verifies the download against `SHA256SUMS`, and installs the latest release. ```bash theme={"system"} novi login ``` Paste a key from [Key Management](https://novita.ai/settings/key-management). You can also set `NOVITA_API_KEY` instead of saving a key. ```bash theme={"system"} novi claude ``` Claude Code starts on Novita models. Run `novi` on its own to open the interactive menu and pick a harness, browse models, or check your balance. ### Installer options | Variable | Effect | | - | - | | `NOVI_INSTALL_DIR` | Install somewhere other than `~/.local/bin` | | `NOVI_VERSION=x.y.z` | Pin a specific release instead of the latest | ```bash theme={"system"} curl -fsSL https://novita.ai/connect/install.sh | NOVI_VERSION=0.3.0 sh ``` ## Supported harnesses | Harness | How to start | Your config | | - | - | - | | Claude Code | `novi claude` | Untouched | | Codex | `novi codex` | Untouched | | OpenCode | `novi opencode` | Per-process provider | | Pi | `novi pi` | Temporary provider extension | | DeepSeek Harness | `novi dsh` | Temporary patch overlay | | Claude Desktop (Beta) | Menu → Launch desktop app | Long-term setup; Disconnect restores | | Codex Desktop (Beta) | Menu → Launch / One-time use | Long-term setup or isolated instance | `--model` and `--print-env` go right after the harness name; everything else is passed through to the harness: ```bash theme={"system"} novi codex --model zai-org/glm-5.3 novi claude --print-env # show what would be injected, secrets masked ``` OpenCode, Pi and DeepSeek Harness use Novita Chat Completions through the same local relay. OpenCode and Pi keep their original providers and model selections alongside Novita, and all generated provider files are removed on exit. `novi dsh` starts the web profile; use `novi dsh headless "task"` or `novi dsh --profile ` to choose another profile. Claude Code preserves an existing claude.ai login and custom model picker; otherwise it uses the relay token. Its default model and tier aliases point to Novita during the session. Preserving the login does not provide automatic Anthropic subscription passthrough. When Codex's `auth.json` contains an access token, its original models and Novita models share the picker. Original model requests go to OpenAI with the client's credential; Novita requests use only the Novita key. Without a ChatGPT login, `novi` uses a standalone Novita provider. A login stored only in a system keychain is not yet detected. DeepSeek Harness requires a version with `--patch` support (verified with 0.2.0-rc.2). Pi requires its provider extension API. OpenCode 2 runs with a private server so it inherits the session configuration. ## Models Every Novita model the harness can use appears in its own model picker (`/model`), with prices and context sizes. The default lineup: | Role | Model | | - | - | | Main | `moonshotai/kimi-k3` | | Mid | `zai-org/glm-5.3` | | Background tasks | `deepseek/deepseek-v4.1-flash` | Pick another model for a session with `--model `, and list everything available with prices: ```bash theme={"system"} novi models ``` Tools, MCP servers, subagents and web search keep working on Novita models, and so does Codex's tool search. Web search runs through [Novita AI Search](/guides/ai-search-quickstart) and is billed to your account; set `NOVI_NO_WEB_SEARCH=1` to turn it off. ## How your key stays safe * **Your key stays with `novi`.** Each session runs through a local relay on `127.0.0.1`. The harness only gets a session token that works there, so neither the harness nor anything it runs can read your Novita API key. * **Nothing is left behind.** CLI sessions inject everything into the harness process. Settings a harness saves on its own during the session (model, effort) are put back when it exits. ## Desktop apps (Beta) On macOS, `novi` can connect Claude Desktop and Codex Desktop to Novita. Open the `novi` menu and choose the app, or run: ```bash theme={"system"} novi codex-desktop # launch; first use saves long-term setup (macOS) novi claude-desktop # same launch behavior for Claude Desktop (macOS) ``` * **Launch desktop app** opens the app with a long-term Novita setup, and reuses an existing setup instead of reconfiguring every time. The setup keeps one background relay on `127.0.0.1:17861` for both apps. * **One-time use** (Codex Desktop only, experimental) starts an isolated instance with its own `CODEX_HOME` and app data. Your original configuration stays untouched; you may need to complete first-run setup. * **Disconnect / restore settings** (press **m** in the menu for connection management) puts back what `novi` changed and keeps your later edits. Every change is journaled. Claude Desktop runs in third-party gateway mode with a Novita profile. Cloud sessions and Remote Control are unavailable in this mode, and earlier Anthropic conversations stay in the original data directory. Use **Switch to official Claude** to go back while keeping the Novita profile for later. On Linux, use **Connect / start** in the menu, then quit and reopen the app yourself. Run `novi desktop status` for desktop app and connection diagnostics. ## Usage receipts CLI sessions print a receipt on stderr when they exit, without changing the agent's stdout. To summarize locally observed input, output and cache tokens by model: ```bash theme={"system"} novi usage --last 7d ``` Add `--json` for request-level records. Usage metadata is stored privately under `~/.config/novi/usage`; no prompts, response text, credentials or working directories are recorded. Totals are estimates based on the catalog price at request time, not your actual Novita bill, and exclude search and other charges. Check your [billing page](/guides/llm-billing) for actual charges. ## Commands | Command | Description | | - | - | | `novi` | Interactive menu | | `novi login` | Save your Novita API key (or set `NOVITA_API_KEY`) | | `novi claude [--model M] [args…]` | Start Claude Code on Novita | | `novi codex [--model M] [args…]` | Start Codex on Novita | | `novi opencode [--model M] [args…]` | Start OpenCode on Novita | | `novi pi [--model M] [args…]` | Start Pi on Novita | | `novi dsh [--model M] [args…]` | Start DeepSeek Harness on Novita | | `novi models` | Model list with prices | | `novi doctor` | Key, connectivity, harness versions and updates | | `novi usage [--last 7d] [--json]` | Local usage history (estimated model cost) | | `novi update --check` | Check for a new `novi` release | | `novi update` | Verified in-place update to the latest release | | `novi desktop status` | Desktop app and connection diagnostics | | `novi desktop disconnect --all` | Restore both desktop connections | | `novi uninstall [--yes]` | Remove `novi` and everything it added | | `novi --print-env` | Show what would be injected (secrets masked) | | `novi version` | Print the installed version | ## Environment variables | Variable | Effect | | - | - | | `NOVITA_API_KEY` | Key to use instead of the saved one | | `NOVI_NO_WEB_SEARCH=1` | Turn off web search | | `NOVI_NO_UPDATE_CHECK=1` | Don't look up the latest harness versions | | `NOVI_NPM_REGISTRY` | npm registry mirror for version lookups | | `NOVI_CODEX_APPLY_PATCH=freeform` | Experimental: Codex's freeform `apply_patch` tool | | `NOVI_NO_ANIMATION=1` / `NO_COLOR=1` | Static menu | | `NOVI_NOVITA_DIRECT=1` | Bypass your proxy for `api.novita.ai` only | | `NOVI_RELAY_DUMP=` | Debug: log relay requests (never credentials) | ## Update and uninstall ```bash theme={"system"} novi update ``` `novi update` downloads the latest release, verifies its checksum and version, then replaces `novi` in place. A running desktop relay is restarted on the new binary. To go back, reinstall a specific version with `NOVI_VERSION=x.y.z` and the installer. ```bash theme={"system"} novi uninstall ``` Uninstall restores desktop connections, then removes `novi` and `~/.config/novi/`. Your harness configs and session history stay. Usage history and recovery backups are preserved; remove them separately if you no longer need them. ## Known limitations * Tested with Claude Code 2.1.283–2.1.295 and Codex 0.153.4–0.162.0. `novi doctor` flags older versions. * Codex conversations started through `novi` can't be resumed with plain `codex resume`. * Codex's own sandbox needs unprivileged user namespaces, which Ubuntu 24.04 blocks by default. * Claude Desktop gateway mode does not use Anthropic sign-in; Remote Control and other features that require the official endpoint are unavailable. * macOS and Linux only. ## FAQ No. CLI sessions inject Novita settings into the harness process only, and settings the harness saves on its own during the session are put back when it exits. To use your harness without Novita, run it as usual (for example `claude` instead of `novi claude`). No. The harness talks to a local relay on `127.0.0.1` using a session token that only works there. Your API key stays with `novi`. Pass `--model ` right after the harness name, for example `novi claude --model zai-org/glm-5.3`, or switch inside the harness with `/model`. Run `novi models` to see every available model with prices. If direct access works, run `NOVI_NOVITA_DIRECT=1 novi` to bypass the proxy for `api.novita.ai` only. Set `"novita_direct": true` in `novi`'s `config.json` to persist this, including for the background desktop relay. Run `novi doctor`. It checks your key, connectivity, harness versions and updates, and includes desktop app paths, configuration paths and connection status. ## Related Configure Claude Code manually with Novita's Anthropic-compatible API. Configure Codex manually with Novita's OpenAI-compatible API. # NovitaClaw Overview Source: https://docs.novita.ai/guides/novitaclaw Deploy a persistent OpenClaw agent on Novita Sandbox with one command, then manage it from CLI, Web UI, or automation scripts. No session cap — your instance runs 24/7 for as long as you need it. *** ## What's New in v1.6.0 * **Gateway Config Export / Import** — Back up, restore, and migrate your OpenClaw configuration across sandboxes. Export to stdout or file, import with automatic JSON validation and Gateway restart. Supports cross-sandbox migration workflows. ```bash CLI icon="terminal" theme={"system"} # Export config novitaclaw gateway config export -o config.json # Import into another sandbox novitaclaw gateway config import config.json ``` * **ClawHub Skill Install** — Install OpenClaw extension skills from ClawHub directly into a sandbox. Skills are available immediately with no Gateway restart required. ```bash CLI icon="terminal" theme={"system"} novitaclaw skill install sonoscli ``` ### Previous Releases * **Telegram Channel (CLI)** — Polling and Webhook modes for connecting your agent to Telegram. * **Slack Channel** — Socket and HTTP modes for connecting your agent to Slack. * **Install Script Improvements** — Install directory changed to `~/.novitaclaw-cli`, improved `update` command with PEP 668 support and symlink recovery. * **On-Demand Mode** — Sandbox auto-pauses after idle, resumes on access. Zero billing while paused. * **Feishu Channel** — Connect your agent to Feishu (Lark) via Webhook or Event mode. * **Web Terminal** — Browser-based terminal access to your sandbox, no SSH required. * **File Manager** — Browse and manage workspace files directly from the browser. * **systemd Service Management** — Gateway, Web Terminal, and File Manager are fully managed by systemd with automatic restart on failure. * **Crash Auto-Recovery** — Automatic diagnostics, repair, and service restoration on repeated crashes. * **Config Auto-Backup** — Every config write creates an automatic backup; crash recovery restores from the latest known-good backup. *** ## Getting Started ### Prerequisites * Python installed * A Novita API key (create/manage keys in [Key Management](https://novita.ai/settings/key-management)) **How to install Python** **Windows:** Download the Python installer and run it. **Check "Add Python to PATH"** before clicking Install Now — skipping this is the most common reason beginners hit errors later. **macOS:** Open Terminal and run: ```bash CLI icon="terminal" theme={"system"} python3 --version ``` If you see Python 3.10 or higher, you're good. If Terminal prompts you to install Command Line Developer Tools, click Install and wait a few minutes. **Linux (Ubuntu / Debian):** ```bash CLI icon="terminal" theme={"system"} sudo apt update && sudo apt install python3 python3-pip -y ``` ### Step 1: Install NovitaClaw ```bash CLI icon="terminal" theme={"system"} sudo pip3 install novitaclaw ``` ```powershell PowerShell icon="terminal" theme={"system"} pip install novitaclaw ``` After installation, verify by running `novitaclaw --help`. If you see a list of commands, you're all set. ### Step 2: Set your API key ```bash CLI icon="terminal" theme={"system"} export NOVITA_API_KEY=sk_your_api_key ``` ```powershell PowerShell icon="terminal" theme={"system"} $env:NOVITA_API_KEY = "sk_your_api_key" ``` ### Step 3: Launch your instance ```bash CLI icon="terminal" theme={"system"} novitaclaw launch ``` On success, the CLI returns: * **Web UI URL** — Chat with your agent * **Gateway WebSocket URL & Token** — For programmatic access * **Web Terminal URL** — Terminal access to the sandbox in your browser * **File Manager URL** — Browse and manage workspace files * **Login credentials** — Username and password for Web Terminal & File Manager Open the returned **Web UI URL** and go to the `Chat` tab to start using your agent. ### Web Terminal The Web Terminal provides browser-based terminal access to your sandbox environment. Open the **Web Terminal URL** returned by the launch command and log in with the provided credentials. This gives you a full shell session — install packages, inspect logs, or debug your agent directly. ### File Manager The File Manager lets you browse, upload, download, and manage files in the sandbox workspace through a web interface. Open the **File Manager URL** and log in with the same credentials. Use it to view agent workspace files, edit configurations, or transfer files without needing CLI access. *** ## Pricing | What you use | What you pay | | - | - | | Sandbox runtime | Sandbox rates | | Novita-hosted LLMs | Model rates | | Third-party LLMs | Sandbox runtime only | # Command Reference Source: https://docs.novita.ai/guides/novitaclaw-commands Complete reference for all NovitaClaw CLI commands. ## `launch` Create a new sandbox instance. ```bash CLI icon="terminal" theme={"system"} novitaclaw launch [OPTIONS] ``` | Option | Description | | - | - | | `--api-key` | Novita API key | | `--gateway-token` | Custom gateway token (optional, auto-generated by default) | | `--timeout` | Creation timeout in seconds (default: `60`) | | `--type` | Sandbox type: `persistent` (default) or `on-demand` | | `--idle-timeout` | Idle timeout in seconds for on-demand mode (default: `300`, range: 60–86400) | ## `list` List active sandbox instances. ```bash CLI icon="terminal" theme={"system"} novitaclaw list --api-key YOUR_API_KEY ``` | Option | Description | | - | - | | `--api-key` | Novita API key | | `--quiet`, `-q` | Only print sandbox IDs, one per line (useful for scripting) | ## `status` Check a sandbox instance status. Displays webhook URLs for all configured channels. ```bash CLI icon="terminal" theme={"system"} novitaclaw status --api-key YOUR_API_KEY ``` | Option | Description | | - | - | | `--api-key` | Novita API key | | `--json` | Output in JSON format (useful for extracting webhook URLs) | ## `stop` Terminate a sandbox instance. ```bash CLI icon="terminal" theme={"system"} novitaclaw stop --api-key YOUR_API_KEY ``` | Option | Description | | - | - | | `--api-key` | Novita API key | | `--yes`, `-y` | Skip confirmation prompt | ## `pause` Pause a sandbox instance. Zero billing while paused. ```bash CLI icon="terminal" theme={"system"} novitaclaw pause --api-key YOUR_API_KEY ``` ## `resume` Resume a paused sandbox instance (\~1s to restore). ```bash CLI icon="terminal" theme={"system"} novitaclaw resume --api-key YOUR_API_KEY ``` ## `config set` Update sandbox runtime configuration. ```bash CLI icon="terminal" theme={"system"} # idle-timeout range: 60–86400 seconds novitaclaw config set idle-timeout 600 # timeout in seconds novitaclaw config set timeout 3600 ``` ## `tui` Connect to a sandbox via terminal UI. ```bash CLI icon="terminal" theme={"system"} novitaclaw tui --token YOUR_GATEWAY_TOKEN ``` ## `doctor` Run diagnostics and repair checks on a sandbox instance. ```bash CLI icon="terminal" theme={"system"} novitaclaw doctor --api-key YOUR_API_KEY ``` | Option | Description | | - | - | | `--api-key` | Novita API key | | `--deep` | Scan system services for extra gateway installs | | `--fix` | Apply recommended repairs (alias for `--repair`) | | `--force` | Apply aggressive repairs (overwrites custom service config) | | `--generate-gateway-token` | Generate and configure a gateway token | | `--no-workspace-suggestions` | Disable workspace memory system suggestions | | `--repair` | Apply recommended repairs without prompting | | `--yes` | Accept defaults without prompting | ## `gateway update` Update OpenClaw to the latest version inside a sandbox. ```bash CLI icon="terminal" theme={"system"} novitaclaw gateway update --api-key YOUR_API_KEY ``` | Option | Description | | - | - | | `--api-key` | Novita API key | | `--restart` | Restart gateway after update | ## `gateway restart` Restart the OpenClaw Gateway in a sandbox. ```bash CLI icon="terminal" theme={"system"} novitaclaw gateway restart --api-key YOUR_API_KEY ``` ## `services setup` Install and start Web Terminal and File Manager on an existing sandbox. Use this to add services to older sandboxes that were launched before these services were included. ```bash CLI icon="terminal" theme={"system"} novitaclaw services setup --api-key YOUR_API_KEY ``` ## `pair list` List pending pairing requests for a channel. ```bash CLI icon="terminal" theme={"system"} novitaclaw pair list --channel CHANNEL --api-key YOUR_API_KEY ``` ## `pair approve` Approve a pairing request for a channel. ```bash CLI icon="terminal" theme={"system"} novitaclaw pair approve --channel CHANNEL --code CODE --api-key YOUR_API_KEY ``` ## `pair feishu` Configure Feishu (Lark) as a messaging channel. ```bash CLI icon="terminal" theme={"system"} novitaclaw pair feishu \ --app-id cli_xxx \ --app-secret secret_xxx \ --mode webhook \ --verification-token tok_xxx \ --encrypt-key key_xxx ``` | Option | Description | | - | - | | `--app-id` | Feishu app ID | | `--app-secret` | Feishu app secret | | `--mode` | `event` (default) or `webhook` | | `--verification-token` | Webhook verification token (webhook mode only) | | `--encrypt-key` | Webhook encrypt key (webhook mode only) | | `--webhook-host` | Webhook bind address (default: `0.0.0.0`) | | `--webhook-port` | Webhook listen port (default: `3000`) | | `--webhook-path` | Webhook URL path (default: `/feishu/events`) | ## `pair telegram` Configure Telegram as a messaging channel. ```bash CLI icon="terminal" theme={"system"} # Polling mode (default) novitaclaw pair telegram --bot-token YOUR_BOT_TOKEN # Webhook mode novitaclaw pair telegram \ --bot-token YOUR_BOT_TOKEN \ --mode webhook \ --webhook-url https://example.com/webhook/telegram \ --webhook-secret YOUR_SECRET ``` | Option | Description | | - | - | | `--bot-token` | Telegram bot token from BotFather | | `--mode` | `polling` (default) or `webhook` | | `--webhook-url` | Public webhook URL (webhook mode only) | | `--webhook-secret` | Webhook secret (webhook mode only) | | `--webhook-host` | Webhook bind address (default: `0.0.0.0`) | | `--webhook-port` | Webhook listen port (default: `8787`) | | `--webhook-path` | Webhook URL path (default: `/webhook/telegram`) | | `--dm-policy` | DM policy (default: `pairing`) | ## `pair slack` Configure Slack as a messaging channel. ```bash CLI icon="terminal" theme={"system"} # Socket mode (default) novitaclaw pair slack --bot-token xoxb-... --app-token xapp-... # HTTP mode novitaclaw pair slack \ --bot-token xoxb-... \ --mode http \ --signing-secret YOUR_SIGNING_SECRET ``` | Option | Description | | - | - | | `--bot-token` | Slack bot token (`xoxb-...`) | | `--app-token` | Slack app token (`xapp-...`, socket mode only) | | `--mode` | `socket` (default) or `http` | | `--signing-secret` | Slack signing secret (HTTP mode only) | | `--webhook-host` | Webhook bind address (default: `0.0.0.0`) | | `--webhook-port` | Webhook listen port (default: `8788`) | | `--webhook-path` | Webhook URL path (default: `/webhook/slack`) | ## `update` Check for CLI updates and upgrade to the latest version. ```bash CLI icon="terminal" theme={"system"} novitaclaw update ``` *** ## Programmatic Usage Use JSON output for automation: ```bash CLI icon="terminal" theme={"system"} novitaclaw -j launch --api-key YOUR_API_KEY ``` Example response: ```json JSON icon="code" theme={"system"} { "success": true, "data": { "sandbox_id": "...", "webui": "https://...", "gateway_ws": "wss://...", "gateway_token": "...", "terminal_url": "https://...", "filemanager_url": "https://...", "services_username": "admin", "services_password": "..." } } ``` *** ## Environment Variables | Variable | Description | | - | - | | `NOVITA_API_KEY` | API key, eliminates the need for `--api-key` flag | | `OPENCLAW_GATEWAY_TOKEN` | Preset gateway token | *** ## Troubleshooting ### Command not found Try one of these: ```bash CLI icon="terminal" theme={"system"} # Linux ~/.local/bin/novitaclaw launch # macOS (typical path) ~/Library/Python/3.x/bin/novitaclaw launch ``` ### Connection timeout Increase launch timeout: ```bash CLI icon="terminal" theme={"system"} novitaclaw launch --timeout 180 --api-key YOUR_API_KEY ``` *** ## Frequently Asked Questions | Problem | Why it happens | What to do | | - | - | - | | "Command not found" / "Not recognized" | Python PATH not configured during install | Reinstall Python and check "Add to PATH", or add the bin directory manually | | "Permission Denied" | Insufficient permissions | Windows: Run Terminal as Administrator. Mac/Linux: prefix with `sudo` | | "Invalid API Key" | Key copied incorrectly | Check for extra spaces around the `sk_` string, or generate a new key | | Sandbox won't stop | Requires confirmation to prevent accidental deletion | Type `y` when prompted, or use `--yes` to skip confirmation | | Mac: `pip` command not found | macOS uses `pip3` by default | Use `pip3 install novitaclaw` instead of `pip` | # Configuration Source: https://docs.novita.ai/guides/novitaclaw-configuration Configure models, connect external messaging channels, set up on-demand mode, and understand the service reliability features of your NovitaClaw instance. ## On-Demand Mode On-demand mode automatically pauses your sandbox after a configurable idle period and resumes it on access. Ideal for low-traffic AI assistants, webhook-based IM integrations, and scheduled tasks. Zero billing while paused. ### Launching an On-Demand Sandbox ```bash CLI icon="terminal" theme={"system"} # Default idle timeout: 300 seconds novitaclaw launch --type on-demand # Custom idle timeout (60–86400 seconds) novitaclaw launch --type on-demand --idle-timeout 600 ``` On-demand mode cannot be combined with `--mode node`. ### Manual Pause and Resume ```bash CLI icon="terminal" theme={"system"} # Pause a sandbox (zero billing while paused) novitaclaw pause # Resume a sandbox (~1s to restore to pre-pause state) novitaclaw resume ``` ### Runtime Configuration ```bash CLI icon="terminal" theme={"system"} # Update idle timeout (60–86400 seconds) novitaclaw config set idle-timeout 600 # Update sandbox lifetime timeout (300–2592000 seconds) novitaclaw config set timeout 3600 ``` Changes take effect immediately. The in-sandbox agent picks up the new configuration on its next check cycle. ### How It Works * Idle detection — An agent daemon inside the sandbox periodically checks OpenClaw session activity and writes status to `/tmp/.novitaclaw-status.json`. * Auto-pause — The server-side idle monitor reads the agent status. After 2 consecutive idle checks, the sandbox is automatically paused. * Auto-resume — Incoming webhook requests or Web UI access automatically resume a paused sandbox. * Cron pre-wakeup — The scheduler scans paused sandboxes for upcoming cron schedules and resumes them \~120 seconds before the next job fires, ensuring cron jobs run on time. ### Checking Status ```bash CLI icon="terminal" theme={"system"} # View sandbox status (does not trigger resume when paused) novitaclaw status # List shows sandbox_type and state columns novitaclaw list ``` While paused, the `status` command still returns full URL information (read from the database without connecting to the sandbox), so scripts can store addresses without triggering a resume. ## Configuring Models Your instance is pre-configured with a Novita-hosted model out of the box. To change the models your agent uses, navigate to `Settings → Config`, click Raw to switch to Raw JSON5 view, then click the reveal button next to "secrets redacted" to show the full config. Update the following two sections: ### Step 1: Register the model under your provider Add a new object to the `models` array inside `models.providers.novita`: ```json JSON icon="code" theme={"system"} { "models": { "providers": { "novita": { "models": [ { "id": "model-id", "name": "display name", "reasoning": true, "input": ["text"], "contextWindow": 200000, "maxTokens": 50000 } ] } } } } ``` ### Step 2: Set it as primary or fallback Update the `model` field under `agents.defaults` to reference your model using the `provider/model-id` format: ```json JSON icon="code" theme={"system"} { "agents": { "defaults": { "model": { "primary": "novita/model-id", "fallbacks": ["novita/fallback-model-id"] } } } } ``` Click Update to save. Every LLM available on the [Novita platform](https://novita.ai/model-api/product/llm-api) is supported. Third-party providers can also be configured — when you bring your own LLM, you pay only for sandbox runtime, not Novita model usage. NovitaClaw model configuration

## Connecting Channels OpenClaw supports external messaging channels so your agent is reachable outside the Web UI. Channels are disabled by default and must be configured. ### Telegram Connect your agent to Telegram as a messaging channel. Two connection modes are supported: Polling (default, long-poll — no public URL needed) and Webhook (HTTP push — best for on-demand sandboxes). **Step 1: Create a Telegram Bot** * Open Telegram and find [@BotFather](https://t.me/BotFather). * Send `/newbot` and follow the prompts to name your bot. * Copy the bot token BotFather provides. #### Mode 1: Polling Polling mode uses long-poll connections. No public URL required — simplest to configure. Polling mode is not recommended for on-demand sandboxes. When the sandbox auto-pauses, the connection drops and incoming messages are lost. Use Webhook mode for on-demand sandboxes. ```bash CLI icon="terminal" theme={"system"} novitaclaw pair telegram \ --bot-token ``` #### Mode 2: Webhook Webhook mode requires the sandbox to expose an HTTP port for receiving Telegram push events. Best suited for on-demand sandboxes — incoming webhook requests automatically trigger resume. The `--webhook-url` is the public URL assigned to your sandbox. Run the following command to retrieve it: ```bash CLI icon="terminal" theme={"system"} novitaclaw status --json | jq -r '.telegram_webhook_url' ``` ```bash CLI icon="terminal" theme={"system"} novitaclaw pair telegram \ --bot-token \ --mode webhook \ --webhook-url \ --webhook-secret ``` #### Optional Webhook Parameters | Parameter | Default | Description | | - | - | - | | `--webhook-host` | `0.0.0.0` | Webhook server bind address | | `--webhook-port` | `8787` | Webhook listen port | | `--webhook-path` | `/webhook/telegram` | Webhook URL path | | `--dm-policy` | `pairing` | DM policy | #### Pairing Flow On first conversation, the bot replies with a pairing code: ```bash CLI icon="terminal" theme={"system"} # List pending pairing requests novitaclaw pair list --channel telegram # Approve a pairing request novitaclaw pair approve --channel telegram --code ``` #### Mode Comparison | | Polling | Webhook | | - | - | - | | Connection | Outbound (long-poll) | Inbound (HTTP push) | | Public port | Not required | Required | | Setup complexity | Low | Medium (additional secrets) | | On-demand auto-resume | Not supported (connection drops on pause) | Supported (webhook triggers resume) | | Recommended for | Always-on sandboxes, development | On-demand sandboxes, production IM | ### Slack Connect your agent to Slack as a messaging channel. Two connection modes are supported: Socket (default, WebSocket — no public URL needed) and HTTP (Events API webhook — best for on-demand sandboxes). #### Mode 1: Socket Socket mode uses a WebSocket connection. No public URL required — simplest to configure. Socket mode is not recommended for on-demand sandboxes. When the sandbox auto-pauses, the WebSocket connection drops and incoming messages are lost. Use HTTP mode for on-demand sandboxes. ```bash CLI icon="terminal" theme={"system"} novitaclaw pair slack \ --bot-token xoxb-... \ --app-token xapp-... ``` #### Mode 2: HTTP HTTP mode uses Slack Events API webhooks. Best suited for on-demand sandboxes — incoming webhook requests automatically trigger resume. ```bash CLI icon="terminal" theme={"system"} novitaclaw pair slack \ --bot-token xoxb-... \ --mode http \ --signing-secret ``` #### Optional HTTP Parameters | Parameter | Default | Description | | - | - | - | | `--webhook-host` | `0.0.0.0` | Webhook server bind address | | `--webhook-port` | `8788` | Webhook listen port | | `--webhook-path` | `/webhook/slack` | Webhook URL path | #### Pairing Flow On first conversation, the bot replies with a pairing code: ```bash CLI icon="terminal" theme={"system"} # List pending pairing requests novitaclaw pair list --channel slack # Approve a pairing request novitaclaw pair approve --channel slack --code ``` #### Mode Comparison | | Socket | HTTP | | - | - | - | | Connection | WebSocket (outbound) | HTTP push (inbound) | | Public port | Not required | Required | | Setup complexity | Low | Medium (additional secrets) | | On-demand auto-resume | Not supported (connection drops on pause) | Supported (webhook triggers resume) | | Recommended for | Always-on sandboxes, development | On-demand sandboxes, production IM | ### Channel Status `novitaclaw status` displays webhook URLs for all configured channels: ```bash CLI icon="terminal" theme={"system"} novitaclaw status # Shows: Feishu Webhook, Telegram Webhook, Slack Webhook (when configured) # JSON mode novitaclaw status --json | jq -r '.feishu_webhook_url' novitaclaw status --json | jq -r '.telegram_webhook_url' novitaclaw status --json | jq -r '.slack_webhook_url' ``` ### Feishu Connect your agent to Feishu (Lark) as a messaging channel. Two connection modes are supported: Webhook (HTTP push) and Event (WebSocket long-poll). #### Prerequisites: Create a Feishu App * Open the [Feishu Open Platform](https://open.feishu.cn/app), log in and click Create Custom App. * On the Credentials & Basic Info page, copy: App ID (format: `cli_xxx`) and App Secret. * Go to Permission Management, click Batch Import, and paste the following permissions: ```json JSON icon="code" theme={"system"} { "scopes": { "tenant": [ "im:message", "im:message:send_as_bot", "im:message:readonly", "im:message.p2p_msg:readonly", "im:message.group_at_msg:readonly", "im:resource", "im:chat.access_event.bot_p2p_chat:read", "im:chat.members:bot_access" ], "user": ["im:chat.access_event.bot_p2p_chat:read"] } } ``` * Go to App Capabilities > Bot and enable bot capability. * Create a version and publish the app. #### Mode 1: Webhook Webhook mode requires the sandbox to expose an HTTP port for receiving Feishu push events. Best suited for on-demand sandboxes — incoming webhook requests automatically trigger resume. Additional credentials: On the Feishu Open Platform, go to Development Configuration > Events & Callbacks > Encryption Strategy and copy: Verification Token and Encrypt Key. ```bash CLI icon="terminal" theme={"system"} novitaclaw pair feishu \ --app-id cli_xxx \ --app-secret secret_xxx \ --mode webhook \ --verification-token tok_xxx \ --encrypt-key key_xxx ``` On the Feishu Open Platform Event Subscription page: * Select Request URL Configuration * Enter the Webhook URL — get it via: ```bash CLI icon="terminal" theme={"system"} novitaclaw status --json | jq -r '.feishu_webhook_url' ``` * Add event: `im.message.receive_v1` #### Mode 2: Event Event mode uses a Feishu WebSocket long-poll connection. No public URL required — simplest to configure. Event mode is not recommended for on-demand sandboxes. When the sandbox auto-pauses, the WebSocket connection drops and incoming messages are lost. Use Webhook mode for on-demand sandboxes. ```bash CLI icon="terminal" theme={"system"} novitaclaw pair feishu \ --app-id cli_xxx \ --app-secret secret_xxx ``` After configuration, on the Feishu Open Platform Event Subscription page: * Select Use Long Connection to Receive Events * Add event: `im.message.receive_v1` Ensure the gateway is running (`novitaclaw status `) before saving, otherwise Feishu may fail to save the long connection configuration. #### Optional Webhook Parameters | Parameter | Default | Description | | - | - | - | | `--webhook-host` | `0.0.0.0` | Webhook server bind address | | `--webhook-port` | `3000` | Webhook listen port | | `--webhook-path` | `/feishu/events` | Webhook URL path | #### Pairing Flow Feishu uses a `pairing` strategy by default. On first conversation, the bot replies with a pairing code that must be approved via CLI: ```bash CLI icon="terminal" theme={"system"} # List pending pairing requests novitaclaw pair list --channel feishu # Approve a pairing request novitaclaw pair approve --channel feishu --code ``` #### Mode Comparison | | Event | Webhook | | - | - | - | | Connection | WebSocket long-poll (outbound) | HTTP push (inbound) | | Public port | Not required | Required (default 3000) | | Setup complexity | Low (App ID + Secret only) | Medium (additional Token + Key) | | On-demand auto-resume | Not supported | Supported (webhook triggers resume) | | Recommended for | Always-on sandboxes, development | On-demand sandboxes, production IM | ## Service Reliability All core services in the sandbox are managed by systemd for production-grade reliability: | Service | Description | Auto-Restart | | - | - | - | | OpenClaw Gateway | Agent runtime and WebSocket server | ✅ | | Web Terminal (ttyd) | Browser-based terminal access | ✅ | | File Manager (gohttpserver) | Web-based file management | ✅ | Crash auto-recovery: If the Gateway crashes repeatedly, the system automatically runs diagnostics, attempts repair, and restores the last known-good configuration from backup — no manual intervention required. Config auto-backup: Every configuration write creates an automatic backup. If a bad config causes a crash, the recovery process restores from the most recent valid backup. # OpenAI Agents SDK Source: https://docs.novita.ai/guides/openai-agents-sdk Seamlessly integrate Novita AI with OpenAI Agents SDK for building multi-agent workflows. The [OpenAI Agents SDK](https://github.com/openai/openai-agents-python) is a lightweight yet powerful framework for building multi-agent workflows. And the SDK is compatible with any model providers that support the OpenAI Chat Completions API format. This guide will walk you through how to use Novita LLM API with OpenAI Agents SDK. ## Get Started 1. Set up your Python environment and install the Agents SDK. ```bash theme={"system"} python -m venv env source env/bin/activate pip install openai-agents==0.0.7 ``` 2. Set up your Novita API key. Go to [Key Management](https://novita.ai/settings/key-management?utm_source=getstarted) and create a new API key. ## Hello world example If running this, ensure you set the NOVITA\_API\_KEY and MODEL\_NAME environment variables. ```python theme={"system"} import os from openai import AsyncOpenAI from agents import ( Agent, Runner, set_default_openai_api, set_default_openai_client, set_tracing_disabled, ) BASE_URL = "https://api.novita.ai/openai" API_KEY = os.getenv("NOVITA_API_KEY") MODEL_NAME = os.getenv("MODEL_NAME") # Because Novita not support the responses API so we use the chat completions API instead. set_default_openai_api("chat_completions") set_default_openai_client(AsyncOpenAI(base_url=BASE_URL, api_key=API_KEY)) # Disable tracing for this example # Refer to https://openai.github.io/openai-agents-python/tracing/#external-tracing-processors-list to use the custom spans. set_tracing_disabled(disabled=True) agent = Agent(name="Assistant", instructions="You are a helpful assistant", model=MODEL_NAME) result = Runner.run_sync( agent, "Write a haiku about recursion in programming.") print(result.final_output) # Code within the code, # Functions calling themselves, # Infinite loop's dance. ``` ## Handoffs example If running this, ensure you set the NOVITA\_API\_KEY and MODEL\_NAME environment variables. ```python theme={"system"} import os import asyncio from openai import AsyncOpenAI from agents import ( Agent, Runner, set_default_openai_api, set_default_openai_client, set_tracing_disabled, ) BASE_URL = "https://api.novita.ai/openai" API_KEY = os.getenv("NOVITA_API_KEY") MODEL_NAME = os.getenv("MODEL_NAME") # Because Novita not support the responses API so we use the chat completions API instead. set_default_openai_api("chat_completions") set_default_openai_client(AsyncOpenAI(base_url=BASE_URL, api_key=API_KEY)) # Disable tracing for this example # Refer to https://openai.github.io/openai-agents-python/tracing/#external-tracing-processors-list to use the custom spans. set_tracing_disabled(disabled=True) spanish_agent = Agent( name="Spanish agent", instructions="You only speak Spanish.", model=MODEL_NAME, ) english_agent = Agent( name="English agent", instructions="You only speak English", model=MODEL_NAME, ) triage_agent = Agent( name="Triage agent", instructions="Handoff to the appropriate agent based on the language of the request.", handoffs=[spanish_agent, english_agent], model=MODEL_NAME, ) async def main(): result = await Runner.run(triage_agent, input="Hola, ¿cómo estás?") print(result.final_output) # ¡Hola! Estoy bien, gracias por preguntar. ¿Y tú, cómo estás? if __name__ == "__main__": asyncio.run(main()) ``` ## Functions example If running this, ensure you set the NOVITA\_API\_KEY and MODEL\_NAME environment variables. ```python theme={"system"} import os import asyncio from openai import AsyncOpenAI from agents import ( Agent, Runner, set_default_openai_api, set_default_openai_client, set_tracing_disabled, function_tool, ) BASE_URL = "https://api.novita.ai/openai" API_KEY = os.getenv("NOVITA_API_KEY") MODEL_NAME = os.getenv("MODEL_NAME") # Because Novita not support the responses API so we use the chat completions API instead. set_default_openai_api("chat_completions") set_default_openai_client(AsyncOpenAI(base_url=BASE_URL, api_key=API_KEY)) # Disable tracing for this example # Refer to https://openai.github.io/openai-agents-python/tracing/#external-tracing-processors-list to use the custom spans. set_tracing_disabled(disabled=True) @function_tool def get_weather(city: str) -> str: return f"The weather in {city} is sunny." agent = Agent( name="Hello world", instructions="You are a helpful agent.", tools=[get_weather], model=MODEL_NAME, ) async def main(): result = await Runner.run(agent, input="What's the weather in Tokyo?") print(result.final_output) # The weather in Tokyo is sunny. if __name__ == "__main__": asyncio.run(main()) ``` # OpenCode Source: https://docs.novita.ai/guides/opencode Connect OpenCode to Novita's OpenAI-compatible LLM API endpoint. OpenCode is an AI coding tool that supports custom providers. You can configure OpenCode with Novita's OpenAI-compatible LLM API endpoint and use Novita models for coding, debugging, refactoring, and project Q\&A. ## Prerequisites Before you begin, prepare the following values: * **API key**: Create or copy an API key from [Key Management](https://novita.ai/settings/key-management). * **Base URL**: `https://api.novita.ai/openai/v1` * **Model ID**: Copy the model ID you want to use from the Model Hub, such as `deepseek/deepseek-v4.1-flash`. ## Install OpenCode OpenCode provides command-line and desktop options. Choose one installation method. ### Linux / macOS / WSL2 Use one of the following methods: ```bash theme={"system"} curl -fsSL https://opencode.ai/install | bash ``` ```bash theme={"system"} brew install anomalyco/tap/opencode ``` ```bash theme={"system"} npm i -g opencode-ai ``` ### Desktop You can also download the desktop app from: ```text theme={"system"} https://opencode.ai/download ``` ## Configure a custom provider Open the OpenCode settings page and find the provider settings. Depending on your OpenCode version, this is usually under **Settings** → **Providers** → **Custom Provider**. Fill in the following values: | Field | Value | | :- | :- | | Display name | `Novita` | | Base URL | `https://api.novita.ai/openai/v1` | | API key | Your Novita API key | | Model ID | For example, `deepseek/deepseek-v4.1-flash` | Make sure the Model ID exactly matches the value in Novita's Model Hub. Context length, pricing, and capabilities depend on the selected model. ## Verify the model After saving the provider, select your Novita model in OpenCode and send a test prompt: ```text theme={"system"} In one sentence, tell me what programming tasks you can help with. ``` If the model responds, the integration is working. If verification fails, check that: * Your API key is correct and your account has available balance. * The Base URL is `https://api.novita.ai/openai/v1`. * The Model ID is correct. * Your network can reach the Novita API endpoint. # OWL Source: https://docs.novita.ai/guides/owl Optimize Your Workflow with Novita AI and OWL. OWL is an open-source framework for multi-agent collaboration that specializes in real-world task automation, built on the CAMEL-AI Foundation. By integrating with various AI models, OWL combines dynamic agent interactions with powerful toolkits for online search, browser automation, document processing, and code execution. This framework enables more natural, efficient, and robust task automation across diverse domains, delivering actionable insights tailored to your specific needs. This guide will enable you to seamlessly integrate Novita AI with OWL and elevate your AI-driven processes for faster, smarter outcomes. ## **How to Integrate Novita AI with RAGFlow** ### **Step 1: Pre-Configuration Setup** 1. **Access the API Endpoint** * Use the official API endpoint: [https://api.novita.ai/openai](https://api.novita.ai/openai). 2. **Create an API Key** * Visit the [key management page](https://novita.ai/settings/key-management) and select `Add New Key` to generate your API key. Visitthekeymanagementpageandselect Add New Keytogenerateyour AP Ikey Pn

* Keep your API key safe. * **Note:** API keys are securely encrypted on the server. If you lose it, you’ll need to delete the old key and create a new one. Keepyour AP Ikeysafe 1 Pn

3. **Identify the Model ID** * meta-llama/llama-4-scout-17b-16e-instruct ### **Step 2: Install OWL** * To begin, download and install OWL by following the step-by-step instructions on the Github: [https://github.com/camel-ai/owl](https://github.com/camel-ai/owl). ### **Step 3: Configure OWL** * Prepare OWL for use by setting up environment variables and adding your API key. Run the following commands in your terminal: ```bash theme={"system"} cd owl cp .env_template .env ``` * Start OWL and specify your task by executing: ```bash theme={"system"} python examples/run_novita_ai.py ``` Step3 Configure OWL Pn

### **Step 4: View Results** 1. **Terminal Output** * Once launched, OWL will display the execution results directly in the terminal window. Terminal Output 1 Pn

2. **Web Interface** * For a more intuitive experience, use OWL’s web-based interface. Launch it by running: ```bash theme={"system"} cd owl python webapp.py ``` Web Interface 1 Pn

* Steps to use the Web UI: * Select `run_novita_ai` from the left-hand menu and go to the `Environment Variable Management` tab on the right and input your `NOVITA_API_KEY`. Stepstousethe Web UI 1 Pn

* Click `Run` to execute your task. Stepstousethe Web UI 2 Pn

* To perform a new task, update the input field and click `Run` again. Stepstousethe Web UI 3 Pn

* Results will be displayed in the terminal, or a new file will be generated in the root directory, depending on the task. Stepstousethe Web UI 5 Pn

* Alternatively, you can directly input your desired task into the content box and click the `Run` button to execute it. Stepstousethe Web UI 6 Pn

# Page Assist Source: https://docs.novita.ai/guides/pageassist Elevate Your Browsing Experience with Novita AI and Page Assist Integration. Page Assist offers a seamless browser extension that integrates AI capabilities directly into your web interactions. By combining Novita AI with Page Assist, you can unlock a powerful AI-enhanced browsing experience that enhances productivity and insight generation. This guide will walk you through the process of deploying and running Page Assist with Novita AI to supercharge your web interactions. ## **How to Integrate Novita AI with Page Assist** You can find the GitHub repository of Page Assist here: [n4ze3m/page-assist](https://github.com/n4ze3m/page-assist). ### Step 1: Prepare Your Environment * Install Bun or npm: Follow the installation guide for [Bun](https://bun.sh/) or use npm as an alternative. ### Step 2: Clone Page Assist Repository 1. Open your terminal and run: ```bash theme={"system"} git clone https://github.com/n4ze3m/page-assist.git cd page-assist ``` 2. Install dependencies using Bun or npm: ```bash theme={"system"} bun install ``` ### Step 3: Build Page Assist Extension * Build the extension for Chrome (default): ```bash theme={"system"} bun run build ``` * For Firefox, use: ```bash theme={"system"} bun build:firefox ``` ### Step 4: Load the Extension 1. For Chrome: * Navigate to `chrome://extensions`. * Enable Developer Mode. * Click `Load unpacked` and select the `build/chrome-xxx` (e.g. `build/chrome-mv3`) directory. 2. For Firefox: * Go to `about:addons`. * Click `Extensions` tab. * Click `Manage Your Extensions`. * Select `Load Temporary Add-on` and choose the `manifest.json` file from in the `build/firefox-xxx` (e.g. `build/firefox-mv3`) directory. ### Step 5: Configure Novita AI as OpenAI API Compatible Endpoint 1. Obtain Your Novita AI API Key: * [Log in](https://novita.ai/user/login) to your Novita AI account. * Navigate to [the Key Management page](https://novita.ai/settings/key-management). * Generate a new API Key and copy it. 2. Set Up Novita AI Endpoint: * In your Page Assist Settings, enter `Open Compatible API` to add provider. Inyour Page Assist Settings,enter Open Compatible AP Itoaddprovider Jpe

Inyour Page Assist Settings,enter Open Compatible AP Itoaddprovider Jpe

* Choose Novita from the `Custom list`, and use your API key for authentication. Choose Novitafromthe Customlist,anduseyour AP Ikeyforauthentication Jpe

Choose Novitafromthe Customlist,anduseyour AP Ikeyforauthentication Jpe

### Step 6: Choose your Model and Test Page Assist with Novita AI * Choose your model from the model list provided by Novita AI. Chooseyourmodelfromthemodellistprovidedby Novita AI Jpe

Chooseyourmodelfromthemodellistprovidedby Novita AI Jpe

* Interact with your Novita AI model by asking questions or analyzing web content. Interactwithyour Novita A Imodelbyaskingquestionsoranalyzingwebcontent Jpe

Interactwithyour Novita A Imodelbyaskingquestionsoranalyzingwebcontent Jpe

# Portkey Source: https://docs.novita.ai/guides/portkey Streamline AI development by using Portkey AI Gateway with Novita AI for fast, secure, and reliable performance. Portkey AI Gateway transforms how developers work with AI models like Novita AI, providing a unified interface for seamless access to multiple language models with fast, secure, and reliable routing. This integration simplifies AI development and improves application performance. This guide will walk you through setting up Portkey AI Gateway and then integrating Novita AI API with Portkey. ## How to Set Up Portkey AI Gateway Setting up Portkey AI Gateway is simple and efficient, requiring just three key steps: configuring the gateway, sending your first request, and optimizing routing and guardrails for seamless performance. ### Step 1: Setup your AI Gateway To run the gateway locally, ensure Node.js and npm are installed on your system. Once ready, execute the following command: ```jason theme={"system"} npx @portkey-ai/gateway ``` After the gateway starts, two key URLs will be displayed: * The Gateway: `http://localhost:8787/v1` * The Gateway Console: `http://localhost:8787/public/` ### Step 2: Make your first request Begin by installing the Portkey AI Python library: ```python theme={"system"} pip install -qU portkey-ai ``` Next, execute the following Python code to send your first request: ```python theme={"system"} from portkey_ai import Portkey # OpenAI compatible client client = Portkey( provider="openai", # or 'anthropic', 'bedrock', 'groq', etc Authorization="sk-***" # the provider API key ) # Make a request through your AI Gateway client.chat.completions.create( messages=[{"role": "user", "content": "What's the weather like?"}], model="gpt-4o-mini" ) ``` Effortlessly monitor all your local logs in one centralized location using the Gateway Console at: `http://localhost:8787/public/`. ### Step 3: Routing & Guardrails Portkey AI Gateway enables you to configure routing rules, add reliability features, and enforce guardrails. Below is an example configuration: ```python theme={"system"} config = { "retry": {"attempts": 5}, "output_guardrails": [{ "default.contains": {"operator": "none", "words": ["Apple"]}, "deny": True }] } # Attach the config to the client client = client.with_options(config=config) client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": "Reply randomly with Apple or Bat"}] ) # In this example, the guardrail denies all replies containing "Apple", so the response would always be "Bat". The retry configuration would attempt the request up to 5 times before giving up. ``` ## How to Integrate Novita AI API with Portkey To access the Novita AI API via the Portkey AI Gateway, follow these steps: ### Step 1: Install the Portkey SDK Integrate the Portkey SDK into your application to seamlessly interact with Novita AI’s API through Portkey’s gateway. **Node.JS** ```jason theme={"system"} npm install --save portkey-ai ``` **Python** ```python theme={"system"} pip install portkey-ai ``` ### Step 2: Initialize Portkey with the Virtual Key To integrate Novita AI with Portkey, retrieve your LLM API key from[ Novita AI ](https://novita.ai/settings/key-management)and add it to Portkey to generate the virtual key. **Node.JS SDK** ```jason theme={"system"} import Portkey from 'portkey-ai' const portkey = new Portkey({ apiKey: "PORTKEY_API_KEY", // Replace with your Portkey API key virtualKey: "VIRTUAL_KEY" // Replace with your virtual key for Novita AI }) ``` **Python SDK** ```python theme={"system"} from portkey_ai import Portkey portkey = Portkey( api_key="PORTKEY_API_KEY", # Replace with your Portkey API key virtual_key="VIRTUAL_KEY" # Replace with your virtual key for Novita AI ) ``` ### Step 3: Invoke Chat Completions with Novita AI Utilize the Portkey instance to send requests to Novita AI. If necessary, you can override the virtual key directly within the API call. **Node.JS SDK** ```jason theme={"system"} const chatCompletion = await portkey.chat.completions.create({ messages: [{ role: 'user', content: 'Say this is a test' }], model: 'Nous-Hermes-2-Mixtral-8x7B-DPO' }); console.log(chatCompletion.choices); ``` **Python SDK** ```python theme={"system"} completion = portkey.chat.completions.create( messages= [{ "role": 'user', "content": 'Say this is a test' }], model= 'reka-core' ) print(completion) ``` # Quickstart Source: https://docs.novita.ai/guides/quickstart Novita AI gives you hosted model APIs, agent sandboxes, and GPU infrastructure. Follow the path that matches your task and verify a working result before moving on to the deeper guides. If you have questions, please check our [FAQs](/guides/faq) first or [reach out to our support team on Discord](https://discord.gg/YyPRAzwp7P). ## 1. Sign in or create an account Sign in to [novita.ai](https://novita.ai/user/login). You can use Google or GitHub authentication, or [sign up with your email address](https://novita.ai/user/register). ## 2. Create an API key Novita AI authenticates API requests with a Bearer API key. Open [Key Management](https://novita.ai/settings/key-management) to create or manage a key, then follow [API Keys](/guides/llm-api-keys) to store it safely. Pass your API key only through the `NOVITA_API_KEY` environment variable in these examples. Do not put a real API key in source code, documentation, agent chats, analytics, screenshots, or logs. ## 3. Confirm available credit Check your available credit on [Billing and Payments](https://novita.ai/billing) before sending a request. Product credit and model access can affect whether a request succeeds. For longer-running use, see [Automatic Top-up](/guides/auto-top-up).

## 4. Send your first LLM request Complete the account, API key, and credit steps above before running this example. It uses the OpenAI-compatible chat completions endpoint and the `deepseek/deepseek-v4.1-flash` model: ```bash theme={"system"} export NOVITA_API_KEY="" curl "https://api.novita.ai/openai/v1/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${NOVITA_API_KEY}" \ -d '{ "model": "deepseek/deepseek-v4.1-flash", "messages": [ { "role": "user", "content": "Say hello in one sentence." } ], "max_tokens": 64 }' ``` A successful request returns HTTP `200` and a non-empty `choices[0].message.content` value, such as `"Hello!"`. The exact wording can vary by model. Add `--include` to the cURL command to inspect the HTTP status as well as the response body. An HTTP `200` alone does not confirm a valid model response. If the request fails, use the matching guide: * `401 Unauthorized`: check the key format and [API key configuration](/guides/llm-api-keys). * `403 Forbidden`: the Novita gateway may return `INVALID_API_KEY` for a missing or invalid key; check the key first. For a model access error, check [model access](/guides/llm-model-access) and whether identity verification is required. * `429 Too Many Requests`: check [rate limits](/guides/llm-rate-limits). * `5xx`: check [common error codes](/guides/error) and retry guidance. For Python, JavaScript, streaming, and additional request options, continue to the [LLM API guide](/guides/llm-api).

## 5. Use your agent Your agent can help you create the request and integrate Novita AI. Complete the account, API key, and credit steps above, and make `NOVITA_API_KEY` available in the environment where the agent runs the code. Ask it to read the Novita AI skill: ```text theme={"system"} Read https://raw.githubusercontent.com/novitalabs/novita-skills/main/skills/novita-ai/SKILL.md and use Novita AI for this task. ``` If your agent supports skills, you can install it once: ```bash theme={"system"} npx skills add novitalabs/novita-skills --skill novita-ai ``` [View the skill on GitHub →](https://github.com/novitalabs/novita-skills). After the agent writes the integration, run the request and verify HTTP `200` with non-empty `choices[0].message.content`. If the agent path does not work, return to [the manual request](#manual-api-request). ## 6. Troubleshooting If you cannot complete the first request, check these in order: 1. The `NOVITA_API_KEY` environment variable is set in the current shell. 2. The key is active and has access to `deepseek/deepseek-v4.1-flash`. 3. Your account has available credit and has completed any required verification. 4. The endpoint, model ID, and request body match the example. 5. The returned status code is covered by [Common Error Codes](/guides/error). For a complete list of endpoints and parameters, see the [API Reference](/api-reference/api-reference-overview). # Claude Code Agent Source: https://docs.novita.ai/guides/sandbox-coding-agent-claude-code Claude Code is Anthropic's coding agent. On Novita Sandbox, the `claude-code` template gives you a ready-to-use, isolated environment where Claude Code can read and edit code, run commands, and complete multi-step engineering tasks autonomously — without directly modifying files on your local machine. Typical use cases: * **Autonomous coding tasks** — hand Claude Code a prompt (e.g. "add error handling to all API endpoints") and let it implement changes end to end. * **Working on real repositories** — clone a Git repo into the sandbox and have Claude Code refactor, fix bugs, or add features. * **Safe, unattended automation** — run the agent fully automatically inside an isolated sandbox, so file and command actions never affect your own environment. * **Multi-step workflows** — start a session to plan, then resume it to carry out each step. The `claude-code` template comes with Claude Code pre-installed, so you can spin up a sandbox and drive Claude Code in just a few lines. The example below runs Claude Code in **headless mode** — a non-interactive, one-shot run that takes a prompt, produces its output, and exits, with no interactive terminal session — which is exactly what you want when driving it programmatically inside a sandbox. Create a sandbox from the template, run the `claude` CLI through `commands.run`, stream its output, then kill the sandbox. ## Quick start **`-p ""`** (`--print`): runs Claude Code in non-interactive mode — it processes the prompt, prints the result to stdout, and exits, instead of opening an interactive session. This is what makes it scriptable inside a sandbox. **`--dangerously-skip-permissions`:** skips the interactive approval prompts Claude Code normally shows before running tools (editing files, executing commands, etc.). It lets the agent act fully unattended, which is convenient in an isolated sandbox — but as the flag name warns, use it only in trusted, sandboxed environments, since the agent can take file and command actions without asking. ```python Python icon="python" theme={"system"} import os from novita_sandbox import Novita def main() -> None: novita = Novita(api_key=os.environ["NOVITA_API_KEY"]) sandbox = novita.sandbox.create( "claude-code", envs={"ANTHROPIC_API_KEY": os.environ["ANTHROPIC_API_KEY"]}, ) print("Sandbox created:", sandbox.sandbox_id) execution = sandbox.commands.run( 'claude --dangerously-skip-permissions -p "Hello"', on_stdout=lambda data: print(data), on_stderr=lambda data: print(data), ) print(execution) sandbox.kill() print("Sandbox killed") if __name__ == "__main__": main() ``` ```js JavaScript & TypeScript icon="js" theme={"system"} import { Novita } from 'novita-sandbox' const novita = new Novita({ apiKey: process.env.NOVITA_API_KEY }) const sandbox = await novita.sandbox.create('claude-code', { envs: { ANTHROPIC_API_KEY: process.env.ANTHROPIC_API_KEY }, }) console.log('Sandbox created:', sandbox.sandboxId) const execution = await sandbox.commands.run( 'claude --dangerously-skip-permissions -p "Hello"', { onStdout: (data) => console.log(data), onStderr: (data) => console.log(data), } ) console.log(execution) await sandbox.kill() console.log('Sandbox killed') ``` *** ## Custom Claude configuration To make Claude Code use a custom LLM (custom API token, base URL, or model), write the configuration to `~/.claude/settings.json` after creating the sandbox and before running `claude`. The `env` block in settings.json is injected into Claude Code as environment variables. ```python Python icon="python" theme={"system"} import os from novita_sandbox import Novita def main() -> None: novita = Novita(api_key=os.environ["NOVITA_API_KEY"]) sandbox = novita.sandbox.create("claude-code") print("Sandbox created:", sandbox.sandbox_id) sandbox.files.write( "~/.claude/settings.json", """{ "env": { "ANTHROPIC_AUTH_TOKEN": "", "ANTHROPIC_BASE_URL": "", "ANTHROPIC_MODEL": "", "ANTHROPIC_REASONING_MODEL": "", "ANTHROPIC_DEFAULT_HAIKU_MODEL": "", "ANTHROPIC_DEFAULT_SONNET_MODEL": "", "ANTHROPIC_DEFAULT_OPUS_MODEL": "" }, "model": "" }""", ) execution = sandbox.commands.run( 'claude --dangerously-skip-permissions -p "Hello"', on_stdout=lambda data: print(data), on_stderr=lambda data: print(data), ) print(execution) sandbox.kill() print("Sandbox killed") if __name__ == "__main__": main() ``` *** ## Work on a cloned repository A common workflow is to clone a Git repository into the sandbox and let Claude Code work on it. Use `sandbox.git.clone` to check out the repo (with credentials for private repos), then run `claude` from inside the cloned directory. ```python Python icon="python" theme={"system"} import os from novita_sandbox import Novita novita = Novita(api_key=os.environ["NOVITA_API_KEY"]) sandbox = novita.sandbox.create("claude-code") sandbox.git.clone( "https://github.com/your-org/your-repo.git", path="/home/user/repo", username="x-access-token", password=os.environ["GITHUB_TOKEN"], depth=1, ) result = sandbox.commands.run( 'cd /home/user/repo && claude --dangerously-skip-permissions -p "Add error handling to all API endpoints"', on_stdout=lambda data: print(data, end=""), ) print(result) sandbox.kill() ``` ```js JavaScript & TypeScript icon="js" theme={"system"} import { Novita } from 'novita-sandbox' const novita = new Novita({ apiKey: process.env.NOVITA_API_KEY }) const sandbox = await novita.sandbox.create('claude-code') await sandbox.git.clone('https://github.com/your-org/your-repo.git', { path: '/home/user/repo', username: 'x-access-token', password: process.env.GITHUB_TOKEN, depth: 1, }) const result = await sandbox.commands.run( 'cd /home/user/repo && claude --dangerously-skip-permissions -p "Add error handling to all API endpoints"', { onStdout: (data) => process.stdout.write(data) } ) console.log(result) await sandbox.kill() ``` *** ## Resume a session Claude Code can continue a previous session, so you can run a multi-step workflow across several invocations. Start a session with `--output-format json` to get a `session_id` in the response, then pass it to `--resume` on the next run to continue where you left off. ```python Python icon="python" theme={"system"} import os import json from novita_sandbox import Novita def main() -> None: novita = Novita(api_key=os.environ["NOVITA_API_KEY"]) sandbox = novita.sandbox.create( "claude-code", envs={"ANTHROPIC_API_KEY": os.environ["ANTHROPIC_API_KEY"]}, ) print("Sandbox created:", sandbox.sandbox_id) sandbox.git.clone( "https://github.com/your-org/your-repo.git", path="/home/user/repo", username="x-access-token", password=os.environ["GITHUB_TOKEN"], depth=1, ) # Start a new session with JSON output to capture the session ID initial = sandbox.commands.run( 'cd /home/user/repo && claude --dangerously-skip-permissions --output-format json -p "Analyze the codebase and create a refactoring plan"', ) session_id = json.loads(initial.stdout)["session_id"] print("Session ID:", session_id) # Resume the session with a follow-up task follow_up = sandbox.commands.run( f'cd /home/user/repo && claude --dangerously-skip-permissions --resume {session_id} -p "Now implement step 1 of the plan"', on_stdout=lambda data: print(data, end=""), ) print(follow_up) sandbox.kill() print("Sandbox killed") if __name__ == "__main__": main() ``` ```js JavaScript & TypeScript icon="js" theme={"system"} import { Novita } from 'novita-sandbox' const novita = new Novita({ apiKey: process.env.NOVITA_API_KEY }) const sandbox = await novita.sandbox.create('claude-code', { envs: { ANTHROPIC_API_KEY: process.env.ANTHROPIC_API_KEY }, }) console.log('Sandbox created:', sandbox.sandboxId) await sandbox.git.clone('https://github.com/your-org/your-repo.git', { path: '/home/user/repo', username: 'x-access-token', password: process.env.GITHUB_TOKEN, depth: 1, }) // Start a new session with JSON output to capture the session ID const initial = await sandbox.commands.run( 'cd /home/user/repo && claude --dangerously-skip-permissions --output-format json -p "Analyze the codebase and create a refactoring plan"', ) const sessionId = JSON.parse(initial.stdout).session_id console.log('Session ID:', sessionId) // Resume the session with a follow-up task const followUp = await sandbox.commands.run( `cd /home/user/repo && claude --dangerously-skip-permissions --resume ${sessionId} -p "Now implement step 1 of the plan"`, { onStdout: (data) => process.stdout.write(data) } ) console.log(followUp) await sandbox.kill() console.log('Sandbox killed') ``` # Codex Agent Source: https://docs.novita.ai/guides/sandbox-coding-agent-codex Codex is OpenAI's coding agent. On Novita Sandbox, the `codex` template gives you a ready-to-use, isolated environment where Codex can read and edit code, run commands, and complete multi-step engineering tasks autonomously — without directly modifying files on your local machine. Typical use cases: * **Autonomous coding tasks** — hand Codex a prompt (e.g. "add error handling to all API endpoints") and let it implement changes end to end. * **Working on real repositories** — clone a Git repo into the sandbox and have Codex refactor, fix bugs, or add features. * **Safe, unattended automation** — run the agent fully automatically inside an isolated sandbox, so file and command actions never affect your own environment. * **Multi-step workflows** — start a session to plan, then resume it to carry out each step. The `codex` template comes with Codex pre-installed, so you can spin up a sandbox and drive Codex in just a few lines. The examples below use the SDK's unified `Novita` client, create a sandbox from the `codex` template, run the `codex` CLI through `commands.run`, stream its output, then kill the sandbox. ## Quick start **`codex exec ""`:** the `exec` subcommand runs Codex non-interactively — it processes the prompt, prints the result, and exits, instead of opening an interactive session. This is what makes it scriptable inside a sandbox. **`--full-auto`:** runs Codex fully autonomously, automatically approving the file and command actions it would otherwise prompt for. Convenient in an isolated sandbox, but use it only in trusted, sandboxed environments. **`--skip-git-repo-check`** lets Codex run outside a Git repository (by default it expects to run inside one). ```python Python icon="python" theme={"system"} import os from novita_sandbox import Novita def main() -> None: novita = Novita(api_key=os.environ["NOVITA_API_KEY"]) sandbox = novita.sandbox.create( "codex", timeout=3600, envs={"CODEX_API_KEY": os.environ["CODEX_API_KEY"]}, ) print("Sandbox created:", sandbox.sandbox_id) try: execution = sandbox.commands.run( 'codex exec --full-auto --skip-git-repo-check "Hello"', on_stdout=lambda data: print(data, end=""), on_stderr=lambda data: print(data, end=""), timeout=0, ) print(execution) finally: sandbox.kill() print("Sandbox killed") if __name__ == "__main__": main() ``` *** ## Custom Codex configuration To make Codex use a custom LLM (custom API token, base URL, or model), write the credentials to `~/.codex/auth.json` and the provider settings to `~/.codex/config.toml` after creating the sandbox and before running `codex`. ```python Python icon="python" theme={"system"} import os from novita_sandbox import Novita def main() -> None: novita = Novita(api_key=os.environ["NOVITA_API_KEY"]) sandbox = novita.sandbox.create("codex", timeout=3600) print("Sandbox created:", sandbox.sandbox_id) try: sandbox.files.write( "~/.codex/auth.json", """{ "OPENAI_API_KEY": "" }""", ) sandbox.files.write( "~/.codex/config.toml", """model_provider = "custom" model = "" model_reasoning_effort = "high" disable_response_storage = true [model_providers] [model_providers.custom] name = "custom" wire_api = "responses" requires_openai_auth = true base_url = "" """, ) execution = sandbox.commands.run( 'codex exec --full-auto --skip-git-repo-check "Hello"', on_stdout=lambda data: print(data, end=""), on_stderr=lambda data: print(data, end=""), timeout=0, ) print(execution) finally: sandbox.kill() print("Sandbox killed") if __name__ == "__main__": main() ``` *** ## Work on a cloned repository A common workflow is to clone a Git repository into the sandbox and let Codex work on it. Use `sandbox.git.clone` to check out the repo (with credentials for private repos), then run `codex` from inside the cloned directory. ```python Python icon="python" theme={"system"} import os from novita_sandbox import Novita novita = Novita(api_key=os.environ["NOVITA_API_KEY"]) sandbox = novita.sandbox.create("codex", timeout=3600) try: sandbox.git.clone( "https://github.com/your-org/your-repo.git", path="/home/user/repo", username="x-access-token", password=os.environ["GITHUB_TOKEN"], depth=1, ) result = sandbox.commands.run( 'codex exec --full-auto --skip-git-repo-check "Add error handling to all API endpoints"', cwd="/home/user/repo", on_stdout=lambda data: print(data, end=""), timeout=0, ) print(result) finally: sandbox.kill() ``` ```js JavaScript & TypeScript icon="js" theme={"system"} import { Novita } from 'novita-sandbox' const novita = new Novita({ apiKey: process.env.NOVITA_API_KEY }) const sandbox = await novita.sandbox.create('codex', { timeoutMs: 3_600_000 }) try { await sandbox.git.clone('https://github.com/your-org/your-repo.git', { path: '/home/user/repo', username: 'x-access-token', password: process.env.GITHUB_TOKEN, depth: 1, }) const result = await sandbox.commands.run( 'codex exec --full-auto --skip-git-repo-check "Add error handling to all API endpoints"', { cwd: '/home/user/repo', onStdout: (data) => process.stdout.write(data), timeoutMs: 0, } ) console.log(result) } finally { await sandbox.kill() } ``` *** ## Resume a session Codex can continue a previous session, so you can run a multi-step workflow across several invocations. Start a session with `--json` to capture the `thread_id` from the first event (`thread.started`), then pass it to `codex exec resume ` to continue where you left off. ```python Python icon="python" theme={"system"} import json import os from novita_sandbox import Novita def main() -> None: novita = Novita(api_key=os.environ["NOVITA_API_KEY"]) sandbox = novita.sandbox.create( "codex", timeout=3600, envs={"CODEX_API_KEY": os.environ["CODEX_API_KEY"]}, ) print("Sandbox created:", sandbox.sandbox_id) try: # Start a new session with JSON output to capture the thread ID. initial = sandbox.commands.run( 'codex exec --full-auto --skip-git-repo-check --json "Create plan.md with a 3-step plan for a TODO CLI app"', timeout=0, ) # The first --json event is thread.started. thread_id = json.loads(initial.stdout.strip().splitlines()[0])["thread_id"] print("Thread ID:", thread_id) # Resume the session with a follow-up task. follow_up = sandbox.commands.run( f'codex exec resume {thread_id} --full-auto --skip-git-repo-check "Now implement step 1 of the plan"', on_stdout=lambda data: print(data, end=""), timeout=0, ) print(follow_up) finally: sandbox.kill() print("Sandbox killed") if __name__ == "__main__": main() ``` ```js JavaScript & TypeScript icon="js" theme={"system"} import { Novita } from 'novita-sandbox' const novita = new Novita({ apiKey: process.env.NOVITA_API_KEY }) const sandbox = await novita.sandbox.create('codex', { timeoutMs: 3_600_000, envs: { CODEX_API_KEY: process.env.CODEX_API_KEY }, }) console.log('Sandbox created:', sandbox.sandboxId) try { // Start a new session with JSON output to capture the thread ID. const initial = await sandbox.commands.run( 'codex exec --full-auto --skip-git-repo-check --json "Create plan.md with a 3-step plan for a TODO CLI app"', { timeoutMs: 0 } ) // The first --json event is thread.started. const threadId = JSON.parse(initial.stdout.trim().split('\n')[0]).thread_id console.log('Thread ID:', threadId) // Resume the session with a follow-up task. const followUp = await sandbox.commands.run( `codex exec resume ${threadId} --full-auto --skip-git-repo-check "Now implement step 1 of the plan"`, { onStdout: (data) => process.stdout.write(data), timeoutMs: 0, } ) console.log(followUp) } finally { await sandbox.kill() console.log('Sandbox killed') } ``` # OpenAI Agents API Source: https://docs.novita.ai/guides/sandbox-coding-agent-openai-agents-api The Agents API gives your application access to the Codex harness through an OpenAI-managed API. OpenAI manages sessions, orchestration, context compaction, and recovery while your application provides tools and chooses its execution environment. Agents can operate in a sandbox where they can execute code, edit files, connect to MCP servers, and produce artifacts. For the full API surface, see the [official Agents API documentation](https://developers.openai.com/api/docs/guides/agents-api/overview). This guide walks you through using Novita Sandbox as the execution environment for the OpenAI Agents API. *** ## Application managed ### Prerequisites * **OpenAI Agent Environment** — create one on the [Environments tab](https://platform.openai.com/agents?tab=environments) and save its API key as `OPENAI_ENVIRONMENT_KEY`. * **Novita Sandbox** — install the [SDK or CLI](/guides/sandbox-installation), then set `NOVITA_API_KEY` in your environment. ### Create an Agent Go to the [Agents page](https://platform.openai.com/agents) and create a new Agent. This Agent is what your application will invoke through the Agents API. Give the Agent a name and description, then pick a model under **Model**. The values shown below are the ones used throughout this guide; the model and reasoning settings are yours to choose, but keep the Agent's description focused on operating inside a sandbox. ### Start a Session On the [Sessions tab](https://platform.openai.com/agents?tab=sessions), start a new Session. Select **Self-hosted** as the execution mode — this tells OpenAI to connect to an environment that you host and manage, which is where Novita Sandbox comes in. Choose the Agent you just created and set the workspace directory (defaults to `/workspace`). This directory must exist inside your sandbox. #### Connect environment Once the Session is created, open it and click **Connect environment**. The dialog walks you through connecting your self-hosted environment to the Session, starting with the code block you'll run later. Save the **Connect an environment** command from that dialog — you will run it inside your sandbox in [Run the Executor](#run-the-executor). It looks like this: ```bash theme={"system"} CODEX_API_KEY="$OPENAI_ENVIRONMENT_KEY" \ codex exec-server \ --remote '' \ --environment-id '' ``` The `--remote` and `--environment-id` values are specific to this Session, so copy them from the dialog rather than from this page. ### Launch the Sandbox Only [v2 regions](/guides/sandbox-overview#regions) are supported. Next, start a sandbox from the [`openai-agents-api-executor`](https://github.com/novitalabs/openai-agents-api-executor) template with the Novita CLI. This template provides the self-hosted environment your session connects to. ```bash CLI icon="terminal" theme={"system"} novita sandbox create openai-agents-api-executor --long-running --timeout 720h ``` By default a sandbox lives for at most one hour. `--long-running` lifts that cap so the sandbox can stay alive for the `--timeout` you set — here 720 hours (30 days), which lets the Session stay connected to the same environment without you having to recreate it. A sandbox's own filesystem is tied to that sandbox, so it disappears with it. If you want your code and data to survive beyond the sandbox — and be readable by other agents — manage your code with Git, and/or mount a [volume](/guides/sandbox-volume) to keep the data outside the sandbox and share it across sandboxes. ### Run the Executor Start the Codex executor inside the sandbox, pointing it at your Session. This is the **Connect an environment** command you saved from [Connect environment](#connect-environment): ```bash theme={"system"} CODEX_API_KEY="$OPENAI_ENVIRONMENT_KEY" \ codex exec-server \ --remote '' \ --environment-id '' ``` `--remote` and `--environment-id` are specific to the Session you created — copy both values from its **Connect environment** dialog. `CODEX_API_KEY` is the API key of the OpenAI Agent Environment you created in the prerequisites. ### Confirm the Connection The Session is now wired up. In the Session page, send the agent a message asking it to report its runtime environment — for example: ```text theme={"system"} Check the runtime environment on my machine. ``` The agent inspects the workspace and reports back. If the setup is correct, it describes the environment **inside your sandbox** rather than your local machine — in the example below, Debian GNU/Linux 12 (Bookworm) on kernel Linux 6.1.158, x86\_64 — which confirms the Session is executing through the Novita Sandbox you launched. *** ## Webhook managed Novita Sandbox follows OpenAI's best practices and provides a long-running controller, [`openai-agents-api-executor`](https://github.com/novitalabs/openai-agents-api-executor), which manages the relationship between Sessions and sandboxes for you. Compared with [Application managed](#application-managed), there is no manual sandbox creation and no `codex exec-server` registration step — Sessions are connected to Novita-managed environments automatically as they run. ### Prerequisites #### Clone the executor repository Clone the executor repository, which holds the webhook controller and client you deploy in the steps below: ```bash theme={"system"} git clone git@github.com:novitalabs/openai-agents-api-executor.git ``` #### Envs Create a `.env` file from the repository's [`.env.example`](https://github.com/novitalabs/openai-agents-api-executor/blob/main/.env.example), then fill in the variables below: ```bash theme={"system"} cp .env.example .env ``` * **Novita Sandbox** — install the [SDK or CLI](/guides/sandbox-installation), then set `NOVITA_API_KEY` in your environment. * **OpenAI application key** — create one on the [API keys page](https://platform.openai.com/api-keys), then set `OPENAI_API_KEY` to it. This key needs the `api.agents.read`, `api.agents.write`, and `api.responses.write` scopes; it stays in your application and never enters the sandbox. * **OpenAI Agent Environment** — create one on the [Environments tab](https://platform.openai.com/agents?tab=environments) with every other permission set to `None`, and save its API key as `OPENAI_EXECUTOR_API_KEY`. This is the only key that enters the sandbox. * **Agent ID** — set `OPENAI_AGENT_ID` to the ID printed by `--create-agent` in [Create the Agent](#create-the-agent). The controller serves only that Agent and ignores Sessions belonging to other Agents. * **Webhook signing secret** — set `OPENAI_WEBHOOK_SECRET`, which you get when you [register the webhook](#register-the-webhook). Leave it empty for the first deploy, then fill it in and redeploy. ### Create the Agent Create the Agent the controller will serve: ```bash theme={"system"} npm run webhook:client -- --create-agent ``` It prints the new agent ID: ```json theme={"system"} {"agent_id":"agent_1c0960ecfe5e41bb8d289525a2446c72d2d44445f38647ff92"} ``` Put that ID in `.env` as `OPENAI_AGENT_ID`. Pass `--instructions` and `--model` to give the Agent its own role; `--update-agent ` edits them in place, which keeps the agent ID and so needs no redeploy. ### Get the Webhook URL Deploy the controller, which receives OpenAI webhooks and starts or resumes a worker sandbox per Session: ```bash theme={"system"} npm run webhook:deploy ``` It prints the webhook URL exposed by the controller's sandbox: ```text theme={"system"} Reusing controller sandbox ir8oymt3mp5hu1qcfkjyl Webhook: https://8000-ir8oymt3mp5hu1qcfkjyl.us-phx-1.sandbox.novita.ai/webhook Controller log: /app/controller.log (inside ir8oymt3mp5hu1qcfkjyl) Next: register that URL in the OpenAI platform under the project that owns OPENAI_API_KEY, for these events: - agent.session.action_required - agent.session.failed Then put the signing secret in OPENAI_WEBHOOK_SECRET and redeploy. Until then the controller answers 503 "Webhook not configured". ``` The controller's sandbox ID is saved to `.controller.json`. Rerunning `webhook:deploy` reconnects to the same controller and restarts it with fresh environment variables. If that sandbox is gone, a new one is created and the URL changes, so the webhook registered in the OpenAI platform has to be updated to match. ### Register the Webhook In the OpenAI platform, open **Project settings → Webhooks** and add a webhook pointing at the URL printed above. Subscribe it to these two events: * `agent.session.action_required` * `agent.session.failed` Register it in the project that owns `OPENAI_API_KEY` — a webhook registered in another project of the same organization is never delivered. ### Redeploy the Controller Add the signing secret to `.env` as `OPENAI_WEBHOOK_SECRET`, then redeploy so the controller picks it up: ```bash theme={"system"} npm run webhook:deploy ``` ### Send Input Send the Agent a message through the client, which talks to the Agent running in the worker sandbox: ```bash theme={"system"} npm run webhook:client -- --input "Check the runtime environment on my machine." ``` The agent inspects its workspace and reports the environment back. If the setup is correct, it describes the environment **inside the sandbox** — confirming the Session is executing through a Novita-managed worker, with no sandbox of your own to create or register. The first line of output is the session ID; pass it back with `--session-id` to continue in the same workspace: ```bash theme={"system"} npm run webhook:client -- --session-id "$SESSION_ID" --input "Read it again." ``` # Connect to sandbox Source: https://docs.novita.ai/guides/sandbox-connect You can use the `novita.sandbox.connect()` method to connect to a running or paused sandbox and continue controlling it with the SDK. This is useful for reusing the same sandbox instance across serverless function invocations, restoring a user's web app session, reconnecting to long-running work, or resuming a paused sandbox after a short period of inactivity. ## Connect to the sandbox Before connecting, you need the sandbox ID. You can obtain it with `novita.sandbox.list()`, then pass it to `novita.sandbox.connect()`. ```js JavaScript & TypeScript icon="js" theme={"system"} import { Novita, SandboxState } from 'novita-sandbox' const novita = new Novita() // --- Get the sandbox ID --- // Get all running sandboxes const runningSandboxesPaginator = await novita.sandbox.list({ query: { state: [SandboxState.RUNNING], }, }) const runningSandboxes = await runningSandboxesPaginator.nextItems() if (runningSandboxes.length === 0) { throw new Error('No running sandboxes found') } const runningSandboxId = runningSandboxes[0].sandboxId // --- Connect to the sandbox --- const sandbox = await novita.sandbox.connect(runningSandboxId) console.log('connected to sandbox: ', sandbox.sandboxId) // Now you can use the sandbox as usual // ... ``` ```python Python icon="python" theme={"system"} from novita_sandbox import Novita, SandboxQuery, SandboxState novita = Novita() # --- Get the sandbox ID --- # Get all running sandboxes running_sandboxes_paginator = novita.sandbox.list(query=SandboxQuery(state=[SandboxState.RUNNING])) running_sandboxes = running_sandboxes_paginator.next_items() if len(running_sandboxes) == 0: raise Exception("No running sandboxes found") running_sandbox_id = running_sandboxes[0].sandbox_id # --- Connect to the sandbox --- sandbox = novita.sandbox.connect(running_sandbox_id) print("connected to sandbox: ", sandbox.sandbox_id) # Now you can use the sandbox as usual # ... sandbox.kill() ``` ```bash CLI icon="terminal" theme={"system"} # Attach an interactive terminal to a running sandbox (alias: sandbox cn) novita sandbox connect ``` # Web Console Source: https://docs.novita.ai/guides/sandbox-console [Agent Sandbox console](https://novita.ai/zh/sandbox-console/view) helps you view and manage sandboxes, templates, and usage statistics. ## Sandbox Management * View currently running sandboxes. * Search for a sandbox by ID. * Filter sandboxes by creation time, template, vCPUs, memory, and other attributes.

## Template Management * Display the list of available templates, including official templates provided by Novita and your [custom templates](/guides/sandbox-template-build-operations). * Support searching for a template by ID. * Support filtering templates by visibility (public, private, or all), vCPUs, and memory. Template Management

## Usage Statistics The console provides daily resource usage and cost analysis to help you monitor and optimize your consumption. * **vCPU Hours**\ Displays the total vCPU runtime across all sandboxes for the current day (unit: hours). * This data is calculated with second-level precision and can help you evaluate compute usage. * **Memory Hours**\ Displays the total memory usage across all sandboxes for the current day (unit: GiB-hours). * The system calculates this based on each sandbox's allocated memory and runtime. * **Usage Costs**\ Displays the total resource cost for the current day (unit: USD). * This includes vCPU and memory costs. For detailed billing rules, see [Pricing](/guides/sandbox-pricing). * All resources are billed by the second and displayed at daily granularity. * The chart shows usage trends over the past 30 days and can be used to monitor changes in resource consumption. * Usage data may not include the most recent usage and may differ slightly from your final bill. Final charges are based on the [Billing Details](https://novita.ai/billing/details) page. Usage Statistics

# Create Sandbox Source: https://docs.novita.ai/guides/sandbox-create ## Create Sandbox The simplest way to create a sandbox is to call `novita.sandbox.create()` without any arguments. ```js JavaScript & TypeScript icon="js" theme={"system"} import 'dotenv/config' import { Novita } from 'novita-sandbox' const novita = new Novita() const sandbox = await novita.sandbox.create() await sandbox.kill() ``` ```python Python icon="python" theme={"system"} from dotenv import load_dotenv from novita_sandbox import Novita load_dotenv() novita = Novita() sandbox = novita.sandbox.create() sandbox.kill() ``` ```bash CLI icon="terminal" theme={"system"} # Create a sandbox from the base template and connect a terminal (alias: sandbox cr) novita sandbox create base # Create without connecting a terminal (detached), returns the sandbox ID novita sandbox create base -d ``` ## From Template You can create a sandbox from a template by passing the template name or ID. ```python Python icon="python" theme={"system"} from dotenv import load_dotenv from novita_sandbox import Novita load_dotenv() novita = Novita() # Create a sandbox from a template name or template ID. sandbox = novita.sandbox.create( "my-python-app", # template name or ID timeout=300, # seconds, optional metadata={"env": "demo"}, envs={"KEY": "value"}, ) print("Sandbox ID:", sandbox.sandbox_id) # Use it. result = sandbox.commands.run("python3 --version") print(result.stdout) sandbox.kill() ``` Use a context manager for automatic cleanup: ```python Python icon="python" theme={"system"} from novita_sandbox import Novita novita = Novita() with novita.sandbox.create("my-python-app") as sandbox: print(sandbox.commands.run("echo hello").stdout) # Automatically killed when exiting the with block. ``` ```js JavaScript & TypeScript icon="js" theme={"system"} import 'dotenv/config' import { Novita } from 'novita-sandbox' const novita = new Novita() // Create a sandbox from a template name or template ID. const sandbox = await novita.sandbox.create('my-python-app', { timeoutMs: 300_000, // milliseconds, optional metadata: { env: 'demo' }, envs: { KEY: 'value' }, }) console.log('Sandbox ID:', sandbox.sandboxId) // Use it. const result = await sandbox.commands.run('python3 --version') console.log(result.stdout) await sandbox.kill() ``` ```bash CLI icon="terminal" theme={"system"} # Create a sandbox from a specific template (by name or template ID) novita sandbox create

Error Name	Status Code	Description
INVALID\_REQUEST\_BODY	400	Request parameter validation failed
IMAGE\_FILE\_EXCEEDS\_MAX\_SIZE	400	Image size exceeds limit
INVALID\_IMAGE\_FORMAT	400	Image format does not meet requirements
IMAGE\_EXCEEDS\_MAX\_RESOLUTION	400	Image resolution exceeds limit
INVALID\_IMAGE\_SIZE	400	Image width or height exceeds limit
API\_NOT\_FOUND	404	API not found
IMAGE\_NO\_FACE\_DETECTED	400	No face detected
INVALID\_CUSTOM\_OUTPUT\_PATH	400	Invalid OSS path
ILLEGAL\_PROMPT	400	Prompt contains inappropriate content
ILLEGAL\_IMAGE\_CONTENT	400	Image contains inappropriate content
INVALID\_AUDIO\_FILE	400	Invalid audio input
BILLING\_FAILED	500	Billing service error
BILLING\_AUTH\_FAILED	403	Billing service authentication failed
BILLING\_BALANCE\_NOT\_ENOUGH	400	Insufficient balance
MISSING\_API\_KEY	400	API Key not provided
INVALID\_API\_KEY	403	API Key validation failed
FEATURE\_NOT\_ALLOWED	403	No permission to upload model
API\_NOT\_ALLOWED	403	No permission to use this API
RATE\_LIMIT\_EXCEEDED	429	Rate limit exceeded
NEED\_REAL\_NAME\_VERIFY	403	Enterprise verification not completed
CREATE\_TASK\_FAILED	500	Failed to create task
TASK\_NOT\_FOUND	404	Task not found
GET\_RESULT\_FAILED	500	Failed to get task result
TASK\_FAILED	500	Task execution failed

Error Name	Status Code	Description
INVALID\_API\_KEY	403	API Key not provided
MODEL\_NOT\_FOUND	404	Model not found
FAILED\_TO\_AUTH	401	Authentication failed
NOT\_ENOUGH\_BALANCE	403	Insufficient balance
INVALID\_REQUEST\_BODY	400	Request body format error, see message for details
RATE\_LIMIT\_EXCEEDED	429	Too many requests, please try again later
TOKEN\_LIMIT\_EXCEEDED	429	Token limit exceeded, please try again later
SERVICE\_NOT\_AVAILABLE	503	Service unavailable
ACCESS\_DENY	403	Access denied

Error Name	Status Code	Description
UNKNOWN	500	Unknown error
GET\_TOKEN\_FAILED	400	Failed to obtain token
FORBIDDEN	403	Access forbidden / No permission
UNAUTHORIZED	401	Unauthorized
USER\_ALREADY\_EXISTS	400	User already exists
INVALID\_USER\_OR\_PASSWORD	400	Invalid username or password
INVALID\_CODE	400	Invalid verification code
USER\_NOT\_FOUND	400	User not found
USER\_PHONE\_NOT\_CONSIST	400	User phone number mismatch
SEND\_CODE\_TOO\_FAST	429	Verification code sent too frequently
INVALID\_PUBLIC\_KEY	400	Invalid public key
USER\_NOT\_ACTIVATED	400	User not activated
USER\_ALREADY\_ACTIVATED	400	User already activated
INVALID\_USER\_TOKEN	400	Invalid user token
BANNED\_USER	400	Account has been banned
RATE\_LIMIT\_EXCEEDED	429	Request rate limit exceeded
RESOURCE\_NOT\_FOUND	400	Resource not found (e.g., container not found)
CONFLICT	400	Conflict (e.g., container conflict)
VALIDATOR\_PARAM	400	Parameter validation failed / Invalid parameter
REQUEST	400	Request error
OPERATION\_LIMIT	400	Operation limit reached
INSUFFICIENT\_RESOURCE	400	Insufficient resources
CLUSTER\_STATUS	400	Cluster status abnormal
NODE\_STATUS	400	Node status abnormal
DEPENDENT\_RESOURCE\_STATE	400	Dependent resource state abnormal
PREPAID\_INSTANCE\_NOT\_SUPPORT\_RELEASE	400	Prepaid instance does not support release
CREATING\_INSTANCE\_NOT\_SUPPORT\_RENEWAL	400	Instance in creation does not support renewal
INSTANCE\_LOCAL\_STORAGE\_NOT\_FOUND	400	Instance local storage not found
INVALID\_COMMAND\_PARAM	400	Invalid instance startup command parameter
GPU\_SPEC\_USED	400	GPU specification already in use
INCORRECT\_USER\_SYNCER\_REQUIRE\_PARAMS	400	Incorrect user syncer request parameters
MIGRATE\_INSUFFICIENT\_RESOURCE	400	Insufficient resources for migration
WALLET\_NOT\_FOUND	500	Wallet not found
WALLET\_UNSUPPORT\_RECHARGE\_METHOD	400	Unsupported wallet recharge method
BALANCE\_NOT\_ENOUGH	400	Insufficient balance
UNSUPPORTED\_BILLING\_MODE	400	Unsupported billing mode
EXPIRED\_OR\_BALANCE\_NOT\_ENOUGH	400	Expired or insufficient balance
ORDER\_NOT\_FOUND	400	Order not found
SAVING\_PLAN\_ALREADY\_EXISTS	400	Saving plan already exists
CREATE\_INSTANCE\_LIMIT	400	Instance creation limit reached, please recharge or delete other instances
NETWORK\_STORAGE\_TOO\_LARGE	400	Network storage size exceeds limit
CUR\_CLUSTER\_NETWORK\_STORAGE\_NOT\_SUPPORT	400	Network storage not supported in current region
NETWORK\_STORAGE\_IN\_USE	400	Network storage is in use
NETWORK\_STORAGE\_UNAVAILABLE	400	Network storage unavailable
NETWORK\_STORAGE\_NOT\_FOUND	400	Network storage not found
IMAGE\_NOT\_FOUND	400	Image not found
IMAGE\_AUTH\_IN\_USE	400	Image authentication in use
NETWORK\_NOT\_FOUND	400	Instance network not found
NETWORK\_IN\_USE	400	Instance network in use
NETWORK\_MAX\_LIMIT	400	Instance network creation limit exceeded
SEND\_MSG\_ERROR	400	Message sending error
JOB\_NOT\_FOUND	400	Instance job not found
SERVERLESS\_ENDPOINT\_NOT\_FOUND	400	Serverless endpoint not found
SERVERLESS\_WORKER\_NOT\_FOUND	400	Serverless worker not found
SERVERLESS\_PRODUCT\_NOT\_FOUND	400	Serverless product not found
SERVERLESS\_APP\_NAME\_IS\_EXIST	400	Serverless application name already exists
TEMPLATE\_IS\_PRIVATE	400	Template is private
TEMPLATE\_NOT\_FOUND	400	Template not found

Error Name	Status Code	Description
UNKNOWN	500	Unknown error, please contact us
LIST\_BILL\_TOO\_FAST	429	Requests are too frequent, please try again later
INVALID\_PRODUCT\_CATEGORY	400	Invalid productCategory parameter
INVALID\_BILL\_CYCLE	400	Invalid cycle parameter
LIST\_BILL\_ERROR	500	Query error, please contact us

Configuration Item	Description
Min Worker Count	The minimum number of worker instances to keep for the Endpoint. Setting a higher minimum helps reduce cold start time. If set to 0, there will be no idle workers when there are no requests, which may increase response latency for new requests. Use 0 with caution for latency-sensitive scenarios.
Max Worker Count	The maximum number of worker instances that the Endpoint can scale up to. When request volume increases, the platform automatically increases workers up to this maximum. This limit helps control costs.
Idle Timeout (seconds)	When a worker is about to be released due to scale-down, the platform keeps it for the configured idle timeout so it can respond quickly to new requests. You are charged for the worker during this period.
Max Concurrent Requests	The maximum number of concurrent requests handled by one worker. If this is exceeded, requests are routed to other workers. If all workers are fully occupied, excess requests are queued until execution is possible.
GPUs / Worker	Number of GPU cards allocated to each worker.
CUDA Version	CUDA version used by the worker.