跳转到内容

任务结构

my-task/
├── instruction.md # 给 agent 的指令
├── task.toml # 声明:超时、资源、验证器配置
├── environment/
│ ├── Dockerfile # 也支持 docker-compose.yaml / Apptainer.def
│ └── ...
├── solution/
│ └── solve.sh # 参考解,用于 oracle 验证
└── tests/
├── test.sh # 验证入口
└── ...

只写给 agent 看的部分,不包含评测细节。示例(官方 hello-world):

Create a file called hello.txt with "Hello, world!" as the content.

真实字段(schema_version = "1.4"):

schema_version = "1.4"
[task]
name = "harbor/hello-world"
version = "1.0.0"
authors = []
keywords = []
[metadata]
difficulty = "easy"
category = "programming"
tags = ["trivial"]
[verifier]
timeout_sec = 120.0
[agent]
timeout_sec = 120.0
[environment]
build_timeout_sec = 600.0
cpus = 1
memory_mb = 2048
storage_mb = 10240
gpus = 0
mcp_servers = []
[verifier.env]
[solution.env]
  • 指令是否只依赖环境里存在的东西
  • tests/ 是否能独立判断成功/失败,不需要人看
  • solution/solve.sh 跑完是否必然拿满分(oracle 基线)
  • 是否存在绕过测试的捷径(reward hacking)
  • 资源与超时是否写死在 task.toml