Stories about SWE-chat
1 related stories
RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests
AI InsightRealSWE compares real user requests against SWE-bench benchmark tasks, finding that 88% of real prompts contain only a problem statement or limited context versus just 7% of benchmark problems; 87% of real prompts are casual while 94% of benchmark problems are formal. This indicates a significant distribution shift in coding agent evaluation, suggesting current benchmarks overestimate model performance on realistic short requests.Key TakeawayReveals a large distribution gap between benchmarks and real requests: 91% format mismatch on problem statements.Why It MattersBenchmarks reflect idealized long problems while real user requests are shorter and more casual, so evaluation scores may not represent real-world deployment performance.Who's Affected- AI ResearchersNeed to re-examine validity of current coding benchmarks and develop new eval sets matching real distribution.
- DevelopersShould include short, casual prompts when evaluating coding agents to avoid overestimating capability.
- EnterprisesShould consider performance on real user inputs rather than benchmark scores when selecting agents.
What's NextWatch for whether RealSWE releases a benchmark set based on real distribution and whether agent rankings shift; also observe if it spurs more ecologically valid evaluation research.Importance 76/100