任务结构
my-task/├── instruction.md # 给 agent 的指令├── task.toml # 声明:超时、资源、验证器配置├── environment/│ ├── Dockerfile # 也支持 docker-compose.yaml / Apptainer.def│ └── ...├── solution/│ └── solve.sh # 参考解,用于 oracle 验证└── tests/ ├── test.sh # 验证入口 └── ...instruction.md
Section titled “instruction.md”只写给 agent 看的部分,不包含评测细节。示例(官方 hello-world):
Create a file called hello.txt with "Hello, world!" as the content.task.toml
Section titled “task.toml”真实字段(schema_version = "1.4"):
schema_version = "1.4"
[task]name = "harbor/hello-world"version = "1.0.0"authors = []keywords = []
[metadata]difficulty = "easy"category = "programming"tags = ["trivial"]
[verifier]timeout_sec = 120.0
[agent]timeout_sec = 120.0
[environment]build_timeout_sec = 600.0cpus = 1memory_mb = 2048storage_mb = 10240gpus = 0mcp_servers = []
[verifier.env]
[solution.env]一份任务的可验证性检查清单
Section titled “一份任务的可验证性检查清单”- 指令是否只依赖环境里存在的东西
-
tests/是否能独立判断成功/失败,不需要人看 -
solution/solve.sh跑完是否必然拿满分(oracle 基线) - 是否存在绕过测试的捷径(reward hacking)
- 资源与超时是否写死在
task.toml