GateLLM

GateLLM

AI Gateway — route, translate & observe LLM traffic

🔒 Get Your API Key

1
Contact your administrator to create a user and API key for you.
2
Copy your API key in the format sk-xxxxxxxx and keep it safe.
3
Include the key in your request headers:
// HTTP Header
Authorization: Bearer sk-your-api-key-here

🚀 Available Models

Retrieve all models currently supported by the gateway:

GET /v1/models List all models 
// Request
curl http://localhost:7890/v1/models \
  -H "Authorization: Bearer sk-your-api-key"

// Response
{
  "object": "list",
  "data": [
    { "id": "qwen3.6-plus", "object": "model" },
    { "id": "glm-5.1", "object": "model" }
  ]
}

⚡ Quick Start

# Send your first chat request
curl http://localhost:7890/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-your-api-key" \
  -d '{
    "model": "qwen3.6-plus",
    "messages": [
      { "role": "user", "content": "Hello!" }
    ]
  }'
📝 Replace sk-your-api-key with your actual API key and qwen3.6-plus with an available model name.

🔗 OpenAI Protocol

POST /v1/chat/completions Chat completions
POST /v1/completions Text completions
POST /v1/responses Responses API
POST /v1/images/generations Image generation
POST /v1/embeddings Embeddings
POST /v1/rerank Rerank (Jina v1 / Cohere v1)
POST /v2/rerank Rerank (Cohere v2)
POST /v1/audio/transcriptions Audio transcription
POST /v1/audio/speech Text-to-speech

🔗 Anthropic Protocol

POST /v1/messages Messages API

🔗 Gemini Protocol

POST /v1beta/models/{model}:generateContent
POST /v1beta/models/{model}:streamGenerateContent
POST /v1/models/{model}:generateContent v1 stable alias
POST /v1/models/{model}:streamGenerateContent v1 stable alias

🔗 DashScope Protocol

POST /v1/services/{*rest} All services (text/image/video/embedding/rerank/audio)
GET /v1/services/{model}/tasks/{task_id} Task status

🔄 Unified Proxy (/v3)

Protocol-agnostic reverse proxy — resolves the target protocol from model configuration, forwards requests as-is with auth injection. Works with any model protocol.

POST /v3/{model}/{*rest} Proxy with body
GET /v3/{model}/{*rest} Proxy without body 
# POST example — request body forwarded unchanged
# {*rest} = any upstream path, e.g. "chat/completions"
curl http://localhost:7890/v3/my-model/{chat/completions} \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-your-api-key" \
  -d '{ "key": "value" }'

# GET example — query params forwarded to upstream
curl http://localhost:7890/v3/my-model/{status}?verbose=true \
  -H "Authorization: Bearer sk-your-api-key"
📝 Replace {...} with the actual upstream path. The path is appended to the upstream base URL. Set direct_path = true on the model to use the base URL as-is instead.

🔧 MCP (Model Context Protocol)

All MCP servers share a single JSON-RPC 2.0 endpoint. Operations are distinguished by the method field in the request body:

POST /mcp JSON-RPC (all methods)
GET /mcp/sse SSE transport
Available methods: initialize · tools/list · tools/search · tools/call 
# 1. List all tools from all servers
curl -X POST http://localhost:7890/mcp \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-your-api-key" \
  -d '{ "jsonrpc": "2.0", "id": 1, "method": "tools/list" }'

# 2. Search tools across all servers (BM25 ranking)
curl -X POST http://localhost:7890/mcp \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-your-api-key" \
  -d '{
    "jsonrpc": "2.0", "id": 2, "method": "tools/search",
    "params": { "query": "pull request", "limit": 5 }
  }'

# 3. Call a specific tool (name format: server_name.tool_name)
curl -X POST http://localhost:7890/mcp \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-your-api-key" \
  -d '{
    "jsonrpc": "2.0", "id": 3, "method": "tools/call",
    "params": {
      "name": "github.get_repo_info",
      "arguments": { "owner": "octocat", "repo": "hello-world" }
    }
  }'
📝 Tool names use server_name.tool_name format. Access is controlled by your API key's MCP ACL settings configured in the Console.
🛠 Management Console