DS-Lighting: Making Agent Harnesses Explicit for Data-Science Automation

arXiv:2608.28590v1 announces DS-Lighting, a unified harness toolkit that makes data-science automation harnesses explicit. It decomposes the harness into data, workflow, execution, and evaluation layers, represents agents as executable operator programs, and unifies benchmarks in an MLE-Bench-style task format to improve reproducibility, comparability, and reliability while reducing system-level failures. Code is provided at the linked repository.

DS-Lighting introduces an explicit, four-layer harness (data, workflow, execution, evaluation) to standardize how data-science tasks are represented and executed by LLM agents. This design supports both predefined pipelines and adaptive search, aiming to improve end-to-end robustness.

By integrating multiple open-source benchmarks into a shared MLE-Bench-like task format, the framework enables controlled comparisons under a common interface, sandboxed runtime, and metric protocol. The result is improved reproducibility, comparability, and reliability across heterogeneous tasks and models.

The work includes empirical evaluations across agents, harness configurations, models, and ablations, and provides a public codebase at the linked repository for reproducibility and adoption.

Source: cs.AI updates on arXiv.org — https://arxiv.org/abs/2608.28590