IdeaTaste: Do LLMs Share the Research Taste of Human Researchers?

IdeaTaste benchmarks LLM research taste by comparing proposals written from a 2025 literature snapshot with papers human communities published in 2026, finding LLM agents match citations and ideas far below human levels and mainly fail to turn directions into concrete approaches.

AI scientists can now carry out much of the research process, but their research taste, the judgment of what to read and which problems and methods to pursue, is far less understood than their ability to execute research. Existing evaluations mostly judge one idea at a time or measure repetition within generated populations, not whether AI choices resemble those of human communities. IdeaTaste uses the work a community goes on to publish as a reference for realized research choices.

IdeaTaste freezes literature through 2025 as corpus U and uses 2026 papers as human reference H in four areas: machine learning systems, code generation, world models, and automated research. LLM agents write proposals from U only, and proposals are compared with H at citation level and idea level, where ideas use five slots: contribution type, problem, motivation, proposed approach, and research plan. Matching measures agreement with human papers, and diversity measures variation among proposals; the human level applies the same metrics to H against itself. Four methods vary what the agent knows: free, topic, landscape, and frontier. Six recent language models are benchmarked, covering more than 90,000 proposals.

• Across more than 90,000 proposals, LLM agents reach about 26% of the human level on citation matching and only 18% on idea matching. §Abstract
• In blinded expert review, AI proposals trail the 2026 papers least in presentation and most in expected success, and proposals whose ideas no human paper shares fare worst. §Abstract
• In a second run of 300 proposals per setting, absolute changes are at most 0.44 percentage points for match metrics and 2.20 points for diversity metrics. §E.2
• Population scaling grows populations to 3.8 to 17 times larger, and matching rises slowly and then almost flattens. §E.3
• Leakage analysis uses Gemini 3.5 Flash-Lite's March 2026 data cutoff to split the reference; exposure does not help. §E.5
• In automated research, overlap with the 2025 corpus is 10.26 for human 2026 papers and 6.34 for the free population. §E.7

• Stated in A.4: the human reference includes model-assisted and unreviewed preprints, so it is not purely peer-reviewed human work.
• Stated in A.7: collection gaps remain, including corpus records without abstracts, index lag on recent preprints, and uncited preprints that may fall outside both collection paths.
• Left open by experiments: the benchmark covers four research areas, so generalization to other fields is not tested.
• Stated in E.3 and left open by experiments: populations are fixed at 300 proposals per setting, with scaling tested only in some settings.

• Research taste is defined as the judgment of what to read, which problems to pursue, and which methods to try. §1
• IdeaTaste freezes literature through 2025 as the corpus and uses 2026 papers as the human reference in four areas. §3.1
• Proposals and papers are compared at citation level and idea level, using five idea slots and matching and diversity metrics. §3.2
• Four methods vary what the LLM agent knows: free, topic, landscape, and frontier. §3.3
• LLM populations fall short mainly on concrete approaches and research plans, not on problems and motivations. §1
• Blinded expert review interprets nonmatching AI proposals as weaker directions rather than overlooked opportunities. §A.5
• The paper concludes the main bottleneck is less finding directions than turning them into concrete approaches. §6
• Survey-reference coverage audit reports 89.1% coverage overall for citation expansion. §A.3