OpenAI SDK, AI SDK and LangChain
Call your model from the OpenAI SDK, the Vercel AI SDK or LangChain through your project's OpenAI-compatible endpoint.
Every deployed project also answers in the OpenAI Chat Completions format. Point the OpenAI SDK, the Vercel AI SDK or LangChain at your project, and your model's tool calls come back as tool_calls, the same as they would from OpenAI.
Connect an SDK
Each SDK needs three settings:
| Setting | Value |
|---|---|
| Base URL | https://fernfly.com/api/p/<project-id>/v1 |
| API key | Your project API key |
| Model | fernfly. Any name works: the project's newest deployment answers. |
The project id is the one in your stable endpoint, under Active endpoint on the project's Overview. See Call your model. Like the stable endpoint, this one always answers with the newest deployment.
Keep the key on your server. To call your model from a browser, use the intent bar or the native endpoint with keyless calls.
You don't send your tools. The model already knows your project's tools, and the endpoint ignores any tools in the request.
The examples below use a model trained on the twelve tools of chart-command-bar, a chart you change by typing a sentence. To build the same model, see Try it with the example model.
OpenAI SDK
import OpenAI from 'openai'
const fernfly = new OpenAI({
baseURL: 'https://fernfly.com/api/p/<project-id>/v1',
apiKey: process.env.FERNFLY_KEY,
})
const completion = await fernfly.chat.completions.create({
model: 'fernfly',
messages: [{ role: 'user', content: 'the small numbers are invisible' }],
})
for (const call of completion.choices[0].message.tool_calls ?? []) {
console.log(call.function.name, JSON.parse(call.function.arguments))
}
// set_axis_scale { axis: 'y', scale: 'logarithmic' }Vercel AI SDK
Install ai and @ai-sdk/openai-compatible. Give generateText the tools your code runs, under the same names as your project's tools, and the SDK runs the one the model picks.
import { createOpenAICompatible } from '@ai-sdk/openai-compatible'
import { generateText, jsonSchema, tool } from 'ai'
import toolSpecs from './tools.json' with { type: 'json' }
import { handlers } from './chart.js' // { set_axis_scale: (args) => …, … }
const fernfly = createOpenAICompatible({
name: 'fernfly',
baseURL: 'https://fernfly.com/api/p/<project-id>/v1',
apiKey: process.env.FERNFLY_KEY,
})
const tools = Object.fromEntries(toolSpecs.map(spec => [spec.name, tool({
description: spec.description,
inputSchema: jsonSchema(spec.parameters),
execute: args => handlers[spec.name](args),
})]))
const { toolCalls, toolResults } = await generateText({
model: fernfly('fernfly'),
tools,
prompt: 'the small numbers are invisible',
})toolCalls holds what the model chose and toolResults what your handlers returned. A call to a tool you didn't pass comes back marked invalid and isn't run.
streamText works the same way. With stopWhen, the SDK sends the tool results back to the endpoint, which ends the turn with an empty reply without calling your model, so the loop stops after one round.
LangChain
import os
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
model="fernfly",
base_url="https://fernfly.com/api/p/<project-id>/v1",
api_key=os.environ["FERNFLY_KEY"],
)
message = llm.invoke("the small numbers are invisible")
print(message.tool_calls)
# [{'name': 'set_axis_scale', 'args': {'axis': 'y', 'scale': 'logarithmic'}, 'id': 'call_…', 'type': 'tool_call'}]tool_calls comes back without bind_tools. Give the model to an agent with your tools, and the agent runs the call, sends back the result, and finishes when the endpoint ends the turn.
The request
| Field | What the endpoint does with it |
|---|---|
messages | Reads the last message. If the user sent it, its text is what the model reads. If it's a tool result, the endpoint ends the turn with an empty reply and doesn't call your model. Earlier messages are ignored, because the model reads one request at a time. |
model | Returns it unchanged in the response. The project's newest deployment answers whatever the name. |
stream | true returns server-sent events. The whole answer arrives in one chunk. |
metadata | Stores it with the request, as meta on the native endpoint. |
tools, tool_choice, temperature and other fields | Ignored. |
The response
| What comes back | When |
|---|---|
tool_calls, with finish_reason "tool_calls" | The model chose one or more calls. |
Text in content, with finish_reason "stop" | The request has nothing to do with your tools. The text is the reply to show the user. |
Empty content, with finish_reason "stop" | The model couldn't produce a call and the AI fallback didn't produce one either. Also the answer to a tool result. |
A response also carries a fernfly object with the fields of the native endpoint's response that Chat Completions has no place for: log_ref, latency_ms, out_of_scope, needs_fallback, quota_exceeded and meta. Read it as completion.fernfly in Node.js and completion.model_extra["fernfly"] in Python. There is no usage: each request counts once toward your daily requests, whatever its length. See Usage limits.
Check each call before you run it, as you would with the native endpoint: that the tool exists in your app, and that the user may do what it asks.
Errors
Errors use OpenAI's format, so each SDK raises its usual error, with the same message as the native endpoint.
| Status | error.type | error.code | Cause |
|---|---|---|---|
| 400 | invalid_request_error | The body isn't valid JSON, has no messages, or its last message is neither from the user nor a tool result. | |
| 401 | invalid_request_error | invalid_api_key | The key is missing or wrong, and keyless calls are off. |
| 403 | permission_error | origin_not_allowed | A browser on a site that isn't in the project's allowed sites made the call. |
| 404 | invalid_request_error | not_found | The project id is wrong, or the project has no deployed model yet. |
| 429 | rate_limit_error | rate_limit_exceeded | A keyless caller went over 20 requests a minute. |
| 429 | insufficient_quota | insufficient_quota | Your plan's daily inference requests are used up. The limit resets at 00:00 UTC, and the OpenAI SDKs don't retry this error. |
| 502 | api_error | model_unavailable | The model couldn't be reached. The SDKs retry it. |
Try it with the example model
chart-command-bar is open source and ships its tools and training pairs, so you can train the model these examples use.
Download tools.json and pairs.jsonl from the repository.
Choose New project, then From scratch. Choose Route Manifest as the Source type, then Continue.
Paste the contents of tools.json into Tools JSON and choose Extract routes. All twelve tools are selected. Choose Continue.
On the Generate step, choose Import JSONL and pick pairs.jsonl, then Continue.
On the Train step, open Advanced mode and choose Fern Pinna (1.5B) as the Base model. Then choose Start training.
When training finishes, save the API key the Deploy step shows, and copy the project id from the Endpoint.
Run any example above with your project id and key. The chart-command-bar README shows how to run the chart itself, which calls the native endpoint from the browser.
Revised