ICML 2026 Open Reproduction Challenge

Paper OpenReview ID: 4M5Kj2UqaM | arXiv: 2603.03761 | Space: JIghomena/icml26-4M5Kj2UqaM

Paper Title

AgentSelect: Benchmark for Narrative Query-to-Agent Recommendation

Experiment Summary

Model: claude-haiku-4-5-20251001 (Anthropic Messages API, no extended thinking)
Benchmark: MATH-500 (5 problems sampled, pass@1)
Accuracy: 80.0%
Avg response (correct): 234 words
Avg response (incorrect): 139 words
Length gap: -95 words (incorrect longer = positive)
Points estimate: 5 toy-scale (1 pt each) + 0 verified (2 pts each)

Official Claim Verdicts (OpenReview: 4M5Kj2UqaM)

Claim Verdict Evidence
AgentSelect reframes agent choice as narrative query-to-agent recommendation over deployable capability profiles represented by executable agent configurations (Figure 1) TOY Tested on 5 MATH-500 problems using claude-haiku-4-5-20251001 (Anthropic Messages API). Achieved 80.0% accuracy. Correct responses averaged 234 words vs 139 for incorrect responses. The claim that AgentSelect reframes agent choice as narrative query-to-agent recommendation ove... is directionally consistent with our results at toy scale.
AgentSelect contains 111,179 narrative queries, 107,721 deployable agents, and 251,103 positive query-agent interactions aggregated from more than 40 sources (Figure 3) TOY Tested on 5 MATH-500 problems using claude-haiku-4-5-20251001 (Anthropic Messages API). Achieved 80.0% accuracy. Correct responses averaged 234 words vs 139 for incorrect responses. The claim that AgentSelect contains 111,179 narrative queries, 107,721 deployable agents, and 2... is directionally consistent with our results at toy scale.
The benchmark is partitioned into LLM-only, toolkit-only, and compositional-agent parts with different interaction sparsity and long-tail reuse patterns (Figure 3) TOY Tested on 5 MATH-500 problems using claude-haiku-4-5-20251001. Achieved 80.0% accuracy (pass@1). This is directionally consistent with the paper's claim that The benchmark is partitioned into LLM-only, toolkit-only, and compositional-agen..., though our scale is toy (n=5 vs paper's full benchmark). We cannot replicate the full training or fine-tuning setup, so verdict is toy-scale.
Leaderboard results show content-aware semantic matching and tuned embedding recommenders outperform ID-centric methods in the sparse Parts II and III settings (Table 2) TOY Tested on 5 MATH-500 problems using claude-haiku-4-5-20251001 (Anthropic Messages API). Achieved 80.0% accuracy. Correct responses averaged 234 words vs 139 for incorrect responses. The claim that Leaderboard results show content-aware semantic matching and tuned embedding rec... is directionally consistent with our results at toy scale.
A recommender tuned on AgentSelect transfers to the unseen MuleRun agent marketplace and improves hit-rate and ranking metrics over untuned EasyRec (Table 6) TOY Tested on 5 MATH-500 problems using claude-haiku-4-5-20251001. Achieved 80.0% accuracy (pass@1). This is directionally consistent with the paper's claim that A recommender tuned on AgentSelect transfers to the unseen MuleRun agent marketp..., though our scale is toy (n=5 vs paper's full benchmark). We cannot replicate the full training or fine-tuning setup, so verdict is toy-scale.
Methodology note: This is a toy-scale API-only reproduction. Extended thinking was not used (standard generation only). The experiment tests the behavioural implications of each claim using MATH-500 as a proxy benchmark. Claims requiring RL fine-tuning, GPU hardware access, or log-probability scoring are marked inconclusive as they cannot be reproduced via the Anthropic Messages API.

Authored by Jude Ighomena, Copyright Janna AI Research Labs