Docs

Training best practices

How to describe your tools, shape your training data and test a model so it answers your users accurately.

Most of a model's accuracy is decided before training starts: by how clearly the tools are described and how well the training pairs match what users actually type. This page lists what makes the biggest difference.

Describe the tools well

The model learns from each tool's name, description and arguments. Studio writes them from your docs, and you can improve them in your API docs or in a route manifest.

  • One tool per action. list_orders and refund_order are easier to tell apart than one orders tool with an action argument.
  • Names that say what the tool does. list_overdue_invoices is clearer than get_inv_2.
  • Descriptions in your users' words. "Lists invoices, filtered by status, customer, amount or date" tells the model which requests belong to the tool.
  • Allowed values as an enum. If status can only be paid, unpaid or void, list them. The model then chooses from the list instead of inventing a value.
  • Ranges as two arguments. min_total and max_total are easier to fill from "between $200 and $500" than one total_range string.
  • Only the tools users will ask for. Leave out admin, internal and rarely used operations. Each extra tool is another one the model could pick by mistake.

For an intent bar, keep to 30 tools or fewer. The intent bar start enforces this.

Shape the training data

  • Choose the style that matches your input. Use Terse for a command bar where users type short phrases, and Verbose where they type full sentences. See Create a model.
  • Use enough pairs. In verbose mode, 25 pairs per tool is a good default, and 50 handles unusual phrasings better. Ten pairs per tool works only when users phrase things the way the examples do.
  • Read a sample before you train. Look through the pairs on the Generate step or the Data tab. Fix or remove any pair with the wrong call, since the model will learn it.
  • Cover the words your users use. If your users have their own terms ("open" for unpaid, "PO" for purchase order), make sure the pairs include them.
  • Keep the examples of what the model can't do. Studio adds out-of-scope and related-but-unsupported examples. Removing them makes the model guess a tool for requests it should decline.
  • Balance the tools. A tool with far fewer pairs than the others gets chosen less often, including when it is the right one.

Choose the model size

Start with Fern Bud (12M) while you work on the tools and data, because its runs take minutes. Move to Fern Pinnule (160M) or Fern Pinna (1.5B) when the data is right and you need more accuracy. The base model is fixed by the first run, so a different size means a new project. See Fern models.

Leave Epochs, Batch size and Learning rate at the values Studio fills in. They are chosen from your number of pairs and the base model. Change them only to fix a specific problem, such as a loss curve that is still falling steeply when the run ends.

Test before you rely on it

  • Automated tests on the Overview runs up to 20 of your training pairs against the live model. Expect a high score here: these are sentences the model has seen.
  • Try it with sentences you wrote yourself, in the words your users would use. These are a better guide to real accuracy.
  • Keep a list of 20 to 50 real requests, with the call each should produce, and never add them to the training data. Run them after every training run. Because the model has never seen them, the score shows whether a change helped.

Improve it from real traffic

Once users are sending requests, the requests the model missed are the best training data you have.

  • History and Analytics show what people ask and which requests got no tool call.
  • Retrain turns those requests into training pairs after you correct them. Fixes your users made in the intent bar appear there too.

See Retrain.

Revised

Was this page helpful?