Docs

OpenAI SDK, AI SDK and LangChain

Call your model from the OpenAI SDK, the Vercel AI SDK or LangChain through your project's OpenAI-compatible endpoint.

Every deployed project also answers in the OpenAI Chat Completions format. Point the OpenAI SDK, the Vercel AI SDK or LangChain at your project, and your model's tool calls come back as tool_calls, the same as they would from OpenAI.

Connect an SDK

Each SDK needs three settings:

SettingValue
Base URLhttps://fernfly.com/api/p/<project-id>/v1
API keyYour project API key
Modelfernfly. Any name works: the project's newest deployment answers.

The project id is the one in your stable endpoint, under Active endpoint on the project's Overview. See Call your model. Like the stable endpoint, this one always answers with the newest deployment.

Keep the key on your server. To call your model from a browser, use the intent bar or the native endpoint with keyless calls.

You don't send your tools. The model already knows your project's tools, and the endpoint ignores any tools in the request.

The examples below use a model trained on the twelve tools of chart-command-bar, a chart you change by typing a sentence. To build the same model, see Try it with the example model.

OpenAI SDK

import OpenAI from 'openai'

const fernfly = new OpenAI({
  baseURL: 'https://fernfly.com/api/p/<project-id>/v1',
  apiKey: process.env.FERNFLY_KEY,
})

const completion = await fernfly.chat.completions.create({
  model: 'fernfly',
  messages: [{ role: 'user', content: 'the small numbers are invisible' }],
})

for (const call of completion.choices[0].message.tool_calls ?? []) {
  console.log(call.function.name, JSON.parse(call.function.arguments))
}
// set_axis_scale { axis: 'y', scale: 'logarithmic' }

Vercel AI SDK

Install ai and @ai-sdk/openai-compatible. Give generateText the tools your code runs, under the same names as your project's tools, and the SDK runs the one the model picks.

import { createOpenAICompatible } from '@ai-sdk/openai-compatible'
import { generateText, jsonSchema, tool } from 'ai'
import toolSpecs from './tools.json' with { type: 'json' }
import { handlers } from './chart.js' // { set_axis_scale: (args) => …, … }

const fernfly = createOpenAICompatible({
  name: 'fernfly',
  baseURL: 'https://fernfly.com/api/p/<project-id>/v1',
  apiKey: process.env.FERNFLY_KEY,
})

const tools = Object.fromEntries(toolSpecs.map(spec => [spec.name, tool({
  description: spec.description,
  inputSchema: jsonSchema(spec.parameters),
  execute: args => handlers[spec.name](args),
})]))

const { toolCalls, toolResults } = await generateText({
  model: fernfly('fernfly'),
  tools,
  prompt: 'the small numbers are invisible',
})

toolCalls holds what the model chose and toolResults what your handlers returned. A call to a tool you didn't pass comes back marked invalid and isn't run.

streamText works the same way. With stopWhen, the SDK sends the tool results back to the endpoint, which ends the turn with an empty reply without calling your model, so the loop stops after one round.

LangChain

import os
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="fernfly",
    base_url="https://fernfly.com/api/p/<project-id>/v1",
    api_key=os.environ["FERNFLY_KEY"],
)

message = llm.invoke("the small numbers are invisible")
print(message.tool_calls)
# [{'name': 'set_axis_scale', 'args': {'axis': 'y', 'scale': 'logarithmic'}, 'id': 'call_…', 'type': 'tool_call'}]

tool_calls comes back without bind_tools. Give the model to an agent with your tools, and the agent runs the call, sends back the result, and finishes when the endpoint ends the turn.

The request

FieldWhat the endpoint does with it
messagesReads the last message. If the user sent it, its text is what the model reads. If it's a tool result, the endpoint ends the turn with an empty reply and doesn't call your model. Earlier messages are ignored, because the model reads one request at a time.
modelReturns it unchanged in the response. The project's newest deployment answers whatever the name.
streamtrue returns server-sent events. The whole answer arrives in one chunk.
metadataStores it with the request, as meta on the native endpoint.
tools, tool_choice, temperature and other fieldsIgnored.

The response

What comes backWhen
tool_calls, with finish_reason "tool_calls"The model chose one or more calls.
Text in content, with finish_reason "stop"The request has nothing to do with your tools. The text is the reply to show the user.
Empty content, with finish_reason "stop"The model couldn't produce a call and the AI fallback didn't produce one either. Also the answer to a tool result.

A response also carries a fernfly object with the fields of the native endpoint's response that Chat Completions has no place for: log_ref, latency_ms, out_of_scope, needs_fallback, quota_exceeded and meta. Read it as completion.fernfly in Node.js and completion.model_extra["fernfly"] in Python. There is no usage: each request counts once toward your daily requests, whatever its length. See Usage limits.

Check each call before you run it, as you would with the native endpoint: that the tool exists in your app, and that the user may do what it asks.

Errors

Errors use OpenAI's format, so each SDK raises its usual error, with the same message as the native endpoint.

Statuserror.typeerror.codeCause
400invalid_request_errorThe body isn't valid JSON, has no messages, or its last message is neither from the user nor a tool result.
401invalid_request_errorinvalid_api_keyThe key is missing or wrong, and keyless calls are off.
403permission_errororigin_not_allowedA browser on a site that isn't in the project's allowed sites made the call.
404invalid_request_errornot_foundThe project id is wrong, or the project has no deployed model yet.
429rate_limit_errorrate_limit_exceededA keyless caller went over 20 requests a minute.
429insufficient_quotainsufficient_quotaYour plan's daily inference requests are used up. The limit resets at 00:00 UTC, and the OpenAI SDKs don't retry this error.
502api_errormodel_unavailableThe model couldn't be reached. The SDKs retry it.

Try it with the example model

chart-command-bar is open source and ships its tools and training pairs, so you can train the model these examples use.

Download tools.json and pairs.jsonl from the repository.

Choose New project, then From scratch. Choose Route Manifest as the Source type, then Continue.

Paste the contents of tools.json into Tools JSON and choose Extract routes. All twelve tools are selected. Choose Continue.

On the Generate step, choose Import JSONL and pick pairs.jsonl, then Continue.

On the Train step, open Advanced mode and choose Fern Pinna (1.5B) as the Base model. Then choose Start training.

When training finishes, save the API key the Deploy step shows, and copy the project id from the Endpoint.

Run any example above with your project id and key. The chart-command-bar README shows how to run the chart itself, which calls the native endpoint from the browser.

Revised

Was this page helpful?