131072 Context

$0.390 / 1M input tokens

$0.390 / 1M output tokens

Demo

API

Join Discord

Try out meta-llama/llama-3.3-70b-instruct

Model Configuration

Response format

System Prompt

max_tokens

temperature

top_p

min_p

top_k

presence_penalty

frequency_penalty

repetition_penalty

README

Introduction

Qwen2 is the new series of Qwen large language models. For Qwen2, we release a number of base language models and instruction-tuned language models ranging from 0.5 to 72 billion parameters, including a Mixture-of-Experts model. This repo contains the 72B Qwen2 base language model.

Compared with the state-of-the-art opensource language models, including the previous released Qwen1.5, Qwen2 has generally surpassed most opensource models and demonstrated competitiveness against proprietary models across a series of benchmarks targeting for language understanding, language generation, multilingual capability, coding, mathematics, reasoning, etc.

For more details, please refer to our blog, GitHub, and Documentation.

Model Details

Qwen2 is a language model series including decoder language models of different model sizes. For each size, we release the base language model and the aligned chat model. It is based on the Transformer architecture with SwiGLU activation, attention QKV bias, group query attention, etc. Additionally, we have an improved tokenizer adaptive to multiple natural languages and codes.

Performance

The evaluation of base models mainly focuses on the model performance of natural language understanding, general question answering, coding, mathematics, scientific knowledge, reasoning, multilingual capability, etc.

The datasets for evaluation include:

English Tasks : MMLU (5-shot), MMLU-Pro (5-shot), GPQA (5shot), Theorem QA (5-shot), BBH (3-shot), HellaSwag (10-shot), Winogrande (5-shot), TruthfulQA (0-shot), ARC-C (25-shot)

Coding Tasks : EvalPlus (0-shot) (HumanEval, MBPP, HumanEval+, MBPP+), MultiPL-E (0-shot) (Python, C++, JAVA, PHP, TypeScript, C#, Bash, JavaScript)

Math Tasks : GSM8K (4-shot), MATH (4-shot)

Chinese Tasks : C-Eval (5-shot), CMMLU (5-shot)

Multilingual Tasks : Multi-Exam (M3Exam 5-shot, IndoMMLU 3-shot, ruMMLU 5-shot, mMMLU 5-shot), Multi-Understanding (BELEBELE 5-shot, XCOPA 5-shot, XWinograd 5-shot, XStoryCloze 0-shot, PAWS-X 5-shot), Multi-Mathematics (MGSM 8-shot), Multi-Translation (Flores-101 5-shot)

How to use

You can choose 3 programming languages to access our qwen/qwen-2-72b-instruct model.

HTTP/cURL

We provide compatibility with the OpenAI API standard

The API Base URL

1https://api.novita.ai/v3/openai

Example of Using Chat Completions API

Generate a response using a list of messages from a conversation

1# Get the Novita AI API Key by referring to: https://novita.ai/docs/get-started/quickstart.html#_2-manage-api-key
2export API_KEY="{YOUR Novita AI API Key}"
3
4curl "https://api.novita.ai/v3/openai/chat/completions" \
5  -H "Content-Type: application/json" \
6  -H "Authorization: Bearer ${API_KEY}" \
7  -d '{
8    "model": "qwen/qwen-2-72b-instruct",
9    "messages": [
10        {
11            "role": "system",
12            "content": "Act like you are a helpful assistant."
13        },
14        {
15            "role": "user",
16            "content": "Hi there!"
17        }
18    ],
19    "max_tokens": 512
20}'

The response may look like this

1{
2  "id": "chat-5f461a9a23a44ef29dbd3124b891afc0",
3  "object": "chat.completion",
4  "created": 1731584707,
5  "model": "qwen/qwen-2-72b-instruct",
6  "choices": [
7    {
8      "index": 0,
9      "message": {
10        "role": "assistant",
11        "content": "Hello! It's nice to meet you. How can I assist you today? Do you have any questions or topics you'd like to discuss? I'm here to help with anything you need."
12      },
13      "finish_reason": "stop",
14      "content_filter_results": {
15        "hate": { "filtered": false },
16        "self_harm": { "filtered": false },
17        "sexual": { "filtered": false },
18        "violence": { "filtered": false },
19        "jailbreak": { "filtered": false, "detected": false },
20        "profanity": { "filtered": false, "detected": false }
21      }
22    }
23  ],
24  "usage": {
25    "prompt_tokens": 46,
26    "completion_tokens": 40,
27    "total_tokens": 86,
28    "prompt_tokens_details": null,
29    "completion_tokens_details": null
30  },
31  "system_fingerprint": ""
32}

If you want to receive a response via streaming, simply pass "stream": true in the request (see the difference on line 20). An example is provided.

1# Get the Novita AI API Key by referring to: https://novita.ai/docs/get-started/quickstart.html#_2-manage-api-key
2export API_KEY="{YOUR Novita AI API Key}"
3
4curl "https://api.novita.ai/v3/openai/chat/completions" \
5  -H "Content-Type: application/json" \
6  -H "Authorization: Bearer ${API_KEY}" \
7  -d '{
8    "model": "qwen/qwen-2-72b-instruct",
9    "messages": [
10        {
11            "role": "system",
12            "content": "Act like you are a helpful assistant."
13        },
14        {
15            "role": "user",
16            "content": "Hi there!"
17        }
18    ],
19    "max_tokens": 512,
20    "stream": true
21}'

The response may look like this

1data: {"id":"chat-d821b951d6ff43ab838d18137aef7d0a","object":"chat.completion.chunk","created":1731586102,"model":"meta-llama/llama-3.1-8b-instruct","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null,"content_filter_results":{"hate":{"filtered":false},"self_harm":{"filtered":false},"sexual":{"filtered":false},"violence":{"filtered":false},"jailbreak":{"filtered":false,"detected":false},"profanity":{"filtered":false,"detected":false}}}],"system_fingerprint":""}
2
3...
4
5data: {"id":"chat-d821b951d6ff43ab838d18137aef7d0a","object":"chat.completion.chunk","created":1731586102,"model":"meta-llama/llama-3.1-8b-instruct","choices":[{"index":0,"delta":{"content":"n, ne"},"finish_reason":null,"content_filter_results":{"hate":{"filtered":false},"self_harm":{"filtered":false},"sexual":{"filtered":false},"violence":{"filtered":false},"jailbreak":{"filtered":false,"detected":false},"profanity":{"filtered":false,"detected":false}}}],"system_fingerprint":""}
6
7data: {"id":"chat-d821b951d6ff43ab838d18137aef7d0a","object":"chat.completion.chunk","created":1731586102,"model":"meta-llama/llama-3.1-8b-instruct","choices":[{"index":0,"delta":{"content":"ed"},"finish_reason":null,"content_filter_results":{"hate":{"filtered":false},"self_harm":{"filtered":false},"sexual":{"filtered":false},"violence":{"filtered":false},"jailbreak":{"filtered":false,"detected":false},"profanity":{"filtered":false,"detected":false}}}],"system_fingerprint":""}
8
9data: {"id":"chat-d821b951d6ff43ab838d18137aef7d0a","object":"chat.completion.chunk","created":1731586102,"model":"meta-llama/llama-3.1-8b-instruct","choices":[{"index":0,"delta":{"content":" assi"},"finish_reason":null,"content_filter_results":{"hate":{"filtered":false},"self_harm":{"filtered":false},"sexual":{"filtered":false},"violence":{"filtered":false},"jailbreak":{"filtered":false,"detected":false},"profanity":{"filtered":false,"detected":false}}}],"system_fingerprint":""}
10
11data: {"id":"chat-d821b951d6ff43ab838d18137aef7d0a","object":"chat.completion.chunk","created":1731586102,"model":"meta-llama/llama-3.1-8b-instruct","choices":[{"index":0,"delta":{"content":"s"},"finish_reason":null,"content_filter_results":{"hate":{"filtered":false},"self_harm":{"filtered":false},"sexual":{"filtered":false},"violence":{"filtered":false},"jailbreak":{"filtered":false,"detected":false},"profanity":{"filtered":false,"detected":false}}}],"system_fingerprint":""}
12
13data: {"id":"chat-d821b951d6ff43ab838d18137aef7d0a","object":"chat.completion.chunk","created":1731586102,"model":"meta-llama/llama-3.1-8b-instruct","choices":[{"index":0,"delta":{"content":"tan"},"finish_reason":null,"content_filter_results":{"hate":{"filtered":false},"self_harm":{"filtered":false},"sexual":{"filtered":false},"violence":{"filtered":false},"jailbreak":{"filtered":false,"detected":false},"profanity":{"filtered":false,"detected":false}}}],"system_fingerprint":""}
14
15data: {"id":"chat-d821b951d6ff43ab838d18137aef7d0a","object":"chat.completion.chunk","created":1731586102,"model":"meta-llama/llama-3.1-8b-instruct","choices":[{"index":0,"delta":{"content":"ce wi"},"finish_reason":null,"content_filter_results":{"hate":{"filtered":false},"self_harm":{"filtered":false},"sexual":{"filtered":false},"violence":{"filtered":false},"jailbreak":{"filtered":false,"detected":false},"profanity":{"filtered":false,"detected":false}}}],"system_fingerprint":""}
16
17...
18
19data: {"id":"chat-d821b951d6ff43ab838d18137aef7d0a","object":"chat.completion.chunk","created":1731586102,"model":"meta-llama/llama-3.1-8b-instruct","choices":[{"index":0,"delta":{"content":" "},"finish_reason":null,"content_filter_results":{"hate":{"filtered":false},"self_harm":{"filtered":false},"sexual":{"filtered":false},"violence":{"filtered":false},"jailbreak":{"filtered":false,"detected":false},"profanity":{"filtered":false,"detected":false}}}],"system_fingerprint":""}
20
21data: {"id":"chat-d821b951d6ff43ab838d18137aef7d0a","object":"chat.completion.chunk","created":1731586102,"model":"meta-llama/llama-3.1-8b-instruct","choices":[{"index":0,"delta":{"content":"just want to chat?"},"finish_reason":"stop","content_filter_results":{"hate":{"filtered":false},"self_harm":{"filtered":false},"sexual":{"filtered":false},"violence":{"filtered":false},"jailbreak":{"filtered":false,"detected":false},"profanity":{"filtered":false,"detected":false}}}],"system_fingerprint":""}
22
23data: [DONE]

Model Parameters

Feel free to check out our documentation for more details.

Python

First, install the official OpenAI Python client

1pip install 'openai>=1.0.0'

and then you can run inferences with us

Example of Using Chat Completions API

Generate a response using a list of messages from a conversation

1from openai import OpenAI
2
3client = OpenAI(
4    base_url="https://api.novita.ai/v3/openai",
5    # Get the Novita AI API Key by referring to: https://novita.ai/docs/get-started/quickstart.html#_2-manage-api-key.
6    api_key="<YOUR Novita AI API Key>",
7)
8
9model = "qwen/qwen-2-72b-instruct"
10stream = True  # or False
11max_tokens = 512
12
13chat_completion_res = client.chat.completions.create(
14    model=model,
15    messages=[
16        {
17            "role": "system",
18            "content": "Act like you are a helpful assistant.",
19        },
20        {
21            "role": "user",
22            "content": "Hi there!",
23        }
24    ],
25    stream=stream,
26    max_tokens=max_tokens,
27)
28
29if stream:
30    for chunk in chat_completion_res:
31        print(chunk.choices[0].delta.content or "")
32else:
33    print(chat_completion_res.choices[0].message.content)

If you set stream: true (line 10), the print may look like this

1It'
2s
3 ni
4ce to
5meet you.
6Is
7 the
8re so
9meth
10ing I
11 can h
12e
13lp
14you wi
15th t
16oday,
17 or
18 woul
19d
20 you like to chat?

If you don't want to receive a response via streaming, simply set stream: false. The output will look like this

1How can I assist you today? Do you have any questions or topics you'd like to discuss?

Model Parameters

Feel free to check out our documentation for more details.

JavaScript

First, install the official OpenAI JavaScript client

1npm install openai

and then you can run inferences with us in the browser or in node.js

Example of Using Chat Completions API

Generate a response using a list of messages from a conversation

1import OpenAI from "openai";
2
3const openai = new OpenAI({
4  baseURL: "https://api.novita.ai/v3/openai",
5  apiKey: "<YOUR Novita AI API Key>",
6});
7const stream = true; // or false
8
9async function run() {
10  const completion = await openai.chat.completions.create({
11    messages: [
12      {
13        role: "system",
14        content: "Act like you are a helpful assistant.",
15      },
16      {
17        role: "user",
18        content: "Hi there!"
19      }
20    ],
21    model: "qwen/qwen-2-72b-instruct",
22    stream
23  });
24  
25  if (stream) {
26    for await (const chunk of completion) {
27      if (chunk.choices[0].finish_reason) {
28        console.log(chunk.choices[0].finish_reason);
29    } else {
30        console.log(chunk.choices[0].delta.content);
31      }
32    }
33  } else {
34    console.log(JSON.stringify(completion));
35  }
36}
37
38run();

If you set stream: true (line 7), the print may look like this

1It'
2s
3 nic
4e to
5 m
6eet you
7. Ho
8w can
9I
10 as
11sist
12 you
13toda
14y? Do you
15hav
16e any q
17uest
18io
19ns or
20 to
21pics you
22'
23d
24li
25ke to
26 di
27scuss
28stop

If you don't want to receive a response via streaming, simply set stream: false. The output will look like this

1{
2  "id": "chat-a3ff0e39b4c24abcbd258ab1a1f38db9",
3  "object": "chat.completion",
4  "created": 1731642457,
5  "model": "qwen/qwen-2-72b-instruct",
6  "choices": [
7    {
8      "index": 0,
9      "message": {
10        "role": "assistant",
11        "content": "How can I help you today? Would you like to talk about something specific or just have a chat? I'm here to assist you with any questions or information you might need."
12      },
13      "finish_reason": "stop",
14      "content_filter_results": {
15        "hate": { "filtered": false },
16        "self_harm": { "filtered": false },
17        "sexual": { "filtered": false },
18        "violence": { "filtered": false },
19        "jailbreak": { "filtered": false, "detected": false },
20        "profanity": { "filtered": false, "detected": false }
21      }
22    }
23  ],
24  "usage": {
25    "prompt_tokens": 46,
26    "completion_tokens": 37,
27    "total_tokens": 83,
28    "prompt_tokens_details": null,
29    "completion_tokens_details": null
30  },
31  "system_fingerprint": ""
32}

Model Parameters

Feel free to check out our documentation for more details.

Qwen2-72B performance

Datasets	DeepSeek-V2	Mixtral-8x22B	Llama-3-70B	Qwen1.5-72B	Qwen1.5-110B	Qwen2-72B
Architecture	MoE	MoE	Dense	Dense	Dense	Dense
#Activated Params	21B	39B	70B	72B	110B	72B
#Params	236B	140B	70B	72B	110B	72B
English
MMLU	78.5	77.8	79.5	77.5	80.4	84.2
MMLU-Pro	-	49.5	52.8	45.8	49.4	55.6
GPQA	-	34.3	36.3	36.3	35.9	37.9
Theorem QA	-	35.9	32.3	29.3	34.9	43.1
BBH	78.9	78.9	81	65.5	74.8	82.4
HellaSwag	87.8	88.7	88	86	87.5	87.6
WindoGrande	84.8	85	85.3	83	83.5	85.1
ARC-C	70	70.7	68.8	65.9	69.6	68.9
TruthfulQA	42.2	51	45.6	59.6	49.6	54.8
Coding
HumanEval	45.7	46.3	48.2	46.3	54.3	64.6
MBPP	73.9	71.7	70.4	66.9	70.9	76.9
EvalPlus	55	54.1	54.8	52.9	57.7	65.4
MultiPL-E	44.4	46.7	46.3	41.8	52.7	59.6
Mathematics
GSM8K	79.2	83.7	83	79.5	85.4	89.5
MATH	43.6	41.7	42.5	34.1	49.6	51.1
Chinese
C-Eval	81.7	54.6	65.2	84.1	89.1	91
CMMLU	84	53.4	67.2	83.5	88.3	90.1
Multilingual
Mulit-Exam	67.5	63.5	70	66.4	75.6	76.6
Multi-Understanding	77	77.7	79.9	78.2	78.2	80.7
Multi-Mathematics	58.8	62.9	67.1	61.7	64.4	76
Multi-Translation	36	23.3	38	35.6	36.2	37.8