Running ML experiments with coding agents
Coding agents can now write the training script, launch the run, read the curves and decide what to try next. What they can't do on their own is remember. This guide shows a setup where every experiment an agent runs is recorded with the question it was testing, so the results add up instead of disappearing with the session.
1. Give your agent a place to record
Tell your coding agent:
Install Probe on this machine and its coding agents. Read https://research.prbe.ai/install.md and follow it.
It signs you up if you're new, installs Probe for the coding agents you use, and asks you to sign in once in the browser. By default a second agent, the Probe daemon, runs beside your coding agent and records the work as it happens. From then on, /probe on records and reads, /probe read only searches, and /probe off keeps Probe out of a conversation.
2. Wire the training code once
Ask the agent to run /instrument-code on your training script, eval or sweep. It adds a few lines like these:
import probe
probe.init(project="folding", experiment="dockq-sweep", question="does temperature 0.7 win?")
probe.log({"train/loss": 0.42, "eval/dockq": 0.71}, step=42)
probe.finish()
The first run in an experiment states the question it is testing; later runs in the same experiment reuse it. Logging never blocks the training loop. Trainers that already have a tracker, and RL trainers such as Miles, TRL, verl, SkyRL and ART, are covered by the same skill. See the SDK quickstart.
3. Let the agent read before it runs
Before proposing a run, the agent can ask what your team already tried: past experiments, their questions, metrics and notes. With the Probe daemon recording, it asks with probe ask; when your agent records its own work, it uses the Probe MCP server's six read-only tools. This is what stops an agent from repeating a run a teammate ruled out last week. See The Probe daemon and MCP server.
4. Keep the reasoning with the result
- One question per experiment. Make the agent state what it expects before it launches, so a failed run still tells you something.
- Relationships, not copies. A retry, a fork at a training step and a run in a group are recorded as such, so the history reads as a tree. See Runs.
- Summaries people read. The Probe daemon keeps each project's and experiment's summary current (or your agent's
/write-summaryskill, when it records its own work), so a teammate can catch up without reading every run. See The Probe daemon.
5. Scale out to several agents
When several agents run experiments in parallel, they write to the same record. The agents doing the work set each project's headline metric and pick the best experiment, branches are ranked on that metric with their agent-hours and cost, and probe session send messages one of your own running sessions that the Probe daemon serves. See Agent swarms.
What changes for you
- You'll always know how any result was made.
- Your agent won't repeat what your team already tried.
- You can pick up any experiment where it left off, or hand it off.