Docs

Train and deploy

How training runs work, what to do when one stops or doesn't progress, and how deployments and versions behave.

The Training jobs tab lists every training run in a project, newest first. You can start a run there whenever the project has training pairs and no run in progress.

Starting a run

When a project has pairs but no trained model, Training jobs shows Training data ready. Choose the hardware and Start fine-tuning.

A run trains the project's base model on its approved pairs, with the settings fixed by the first run. See Create a model for what the settings mean.

You can have up to three runs in progress across all your projects, and one per project.

Run statuses

StatusMeaning
PendingWaiting to start.
RunningTraining. The log and loss curve update as it goes.
StalledTraining hasn't made progress for at least five minutes. See When training doesn't progress.
DoneFinished. Its model can be deployed.
StoppedStopped early by you. Its partial model is waiting for you to keep or discard it.
FailedEnded with an error. The log says why.
CancelledEnded without a model, after you stopped it while it wasn't progressing or discarded a partial model.

Each run shows its log, its loss curve and, once it finishes, its scores. The loss is a measure of how far the model's answers are from your training pairs: a curve that falls and levels off is a run that learned.

Stopping a run

Stop ends a run early and keeps the model trained so far. The run is then marked Stopped, with two buttons:

  • Keep treats the partial model as finished, so you can deploy it.
  • Discard throws it away. The project goes back to having training pairs and no model, and you can start again.

A partial model has learned less than a finished one, so test it with Try it before relying on it.

When training doesn't progress

If a run makes no progress for five minutes, its status becomes Stalled and a notice reads Training hasn't made progress in N minutes. Two things can happen:

  • It recovers. The run goes back to Running and carries on. You don't need to do anything.
  • You stop it. Stop on a stalled run ends it at once, and it is recorded as Cancelled. Train again whenever you're ready.

A stalled run still counts as in progress, so it blocks another run on the same project until it recovers or you stop it. If you stop it, its training minutes are counted only up to its last progress, so the stalled time doesn't use your daily allowance.

Deploying

A project's first model is deployed as soon as its run finishes. After that, deploying is your choice: a finished run that isn't deployed shows Deploy, and the project's Overview shows Model ready to deploy.

Deploying only changes which model answers requests. It costs nothing and doesn't retrain anything.

Versions and endpoints

Each deployment is a version, and a project has two kinds of endpoint:

EndpointURLAnswers with
Stable/api/p/<project-id>/inferThe newest deployment. Deploying a new model switches it over, with no change to your code.
Versioned/api/deployments/<deployment-id>/inferThat one deployment's model, for as long as it exists.

The deployment that the stable endpoint uses is marked live on Training jobs.

Use the stable endpoint in your app. Use a versioned endpoint when you want to keep traffic on a known model while you test a new one, then switch your app to the stable endpoint once you are satisfied.

Older deployments keep answering on their versioned endpoints. Studio has no control to make an older deployment live again. To go back to an older model, point your app at that deployment's versioned endpoint, or contact us.

The project API key

The first deployment creates the project's API key for the stable endpoint and shows it once, on the wizard's Deploy step or under Save your project API key on Training jobs. Copy it then. See API keys and keyless calls.

Revised

Was this page helpful?