Research Topic · Benchmarking

Measuring What Code Models Really Know

Rigorous, reliable, reproducible benchmarks for code generation, reasoning, and multilingual understanding — turning claims into evidence.

0
Papers
0
Venues
0
Benchmarks Surveyed
Scroll to explore
▼
Featured Work · Position
ICML '26

Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility

Jialun Cao, Yuk-Kit Chan*, Zixuan Ling*, Wenxuan Wang†, Shuqing Li, Mingwei Liu, Ruixi Qiao, Yuting Han, Chaozheng Wang, Boxi Yu, Pinjia He, Shuai Wang, Zibin Zheng, Michael R. Lyu, Shing-Chi Cheung

A decade-scale survey of the code-benchmark landscape (2014–2025). We analyze 672 benchmarks and argue that the field must be anchored by three principles — the 3R: Rigor, Reliability, and Reproducibility.

Paper Media
💡 Benchmarks published in 2025 ignoring code coverage already equal the total from the previous ten years — the field urgently needs the 3R framework.
📊
0
Code Benchmarks Surveyed
across the code-LLM landscape
📋
0
HOW2BENCH Checklists
guideline for building & releasing benchmarks
⚖
0R
Guiding Principles
Rigor · Reliability · Reproducibility
The 3R Principles
HOW2BENCH-aligned
Rigor
Construction
Reliability
Evaluation
Reproducibility
Release
A Decade of Code Benchmarks
2014 – 2025
2014–24
10 years
2025
alone

Benchmarks ignoring code coverage published in 2025 alone ≈ the total from the previous ten years combined.

Featured Work · Multilingual Reasoning
ACL '25

CruxEval-X: A Benchmark for Multilingual Code Reasoning

Ruiyang Xu, Jialun Cao (Co-1st), Yaojie Lu, Ming Wen, Hongyu Lin, Xianpei Han, Ben He, Shing-Chi Cheung, Le Sun

Can models reason about code beyond Python? CruxEval-X extends input/output prediction to 19 programming languages, exposing how reasoning ability shifts across language paradigms.

Paper Leaderboard Code
💡 A model trained solely on Python transfers poorly — peaking at 34.4% Pass@1 in other languages — exposing the multilingual generalization gap.
Language Coverage
0 / 19
Cross-Lingual Generalization Gap
24 LLMs · 19,000+ tests

A model trained solely on Python transfers poorly to other languages — peaking at just 34.4% Pass@1.

Python→Other Langs
0
TS↔JSStrongest correlated pair
RacketLeast correlated language
Featured Work · Adversarial Strengthening
ICML '26

SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates

Boxi Yu, Yang Cao, Yuzhong Zhang, Liting Lin, Junjielong Xu, Zhiqing Zhong, Qinghua Xu, Guancheng Wang, Jialun Cao, Shing-Chi Cheung, Pinjia He, Lionel Briand

Weak test suites let models pass without truly solving the task. SWE-ABS strengthens benchmarks adversarially — and the reported success rates deflate sharply, revealing the real gap.

Paper
💡 Roughly 1 in 5 "solved" patches from top agents are semantically incorrect — weak test suites hide the real capability gap.
Resolve Rate — Top-Ranked Agent
before strengthening

Two-stage adversarial strengthening (coverage-driven augmentation + mutation-driven testing) on SWE-bench Verified knocked the top-ranked agent down to 5th place — roughly 1 in 5 “solved” patches from the top-30 agents turned out to be semantically incorrect.

Top-Ranked Agent
0%
–0Resolve-rate drop (top agent)
0%Previously-passing patches rejected
0%Of 500 instances strengthened
0×Improvement over prior work
Featured Work · Object-Oriented Generation
ASE '24

JavaBench: Object-Oriented Code Generation for Evaluating LLMs

Jialun Cao, Zhiyong Chen*, Jiarong Wu, Shing-chi Cheung, Chang Xu

Most code benchmarks stop at standalone functions. JavaBench tests object-oriented programming at project scale — inheritance, polymorphism, and design patterns — where even strong models struggle.

Paper Leaderboard Code
💡 No LLM could fully complete any project at the OOP level — the best achieves only 41.17% vs. human baseline of 90.93%.
Inheritance Polymorphism Encapsulation Design Patterns Interfaces Abstraction
LLM Ceiling vs. Human Baseline
pass@5, class-level
#EvaluatorCompletionScore
1 Human Baseline
282 students, avg / 100
0
2 Best LLM
no project fully solved
0
0Methods to implement
0Classes
0Real-world projects
0%Test coverage
More Work · Multi-Domain
AAAI '25

DomainEval: An Auto-Constructed Benchmark for Multi-Domain Code Generation

Qiming Zhu, Jialun Cao (Co-1st), Yaojie Lu, Hongyu Lin, Xianpei Han, Ben He, Le Sun, Shing-Chi Cheung

An automatically constructed benchmark that evaluates LLMs across six specialized programming domains beyond typical algorithmic tasks.

Paper
💡 LLMs excel at computation but struggle with cryptography and systems coding — the gap reaches 69 percentage points in some models.
More Work · Test Generation
ASE '24 🏆 Distinguished Paper Award

PathEval: Towards Understanding LLMs on Directed Test Input Generation

Zongze Jiang, Ming Wen, Jialun Cao, Xuanhua Shi, Hai Jin

Evaluates whether LLMs can generate test inputs that exercise specific execution paths in programs.

PaperCode
💡 LLMs frequently fail to synthesize inputs for complex paths requiring constraint-solving — a fundamental gap in directed test generation.
More Work · Contamination
Internetware '25

CodeCleaner: Elevating Standards with A Robust Data Contamination Mitigation Toolkit

Jialun Cao, Songqiang Chen, Wuqi Zhang, Hau Ching Lo, Yeting Li, Shing-Chi Cheung

An open-source toolkit using 11 code refactoring operators across method, class, and cross-class scales to mitigate data contamination in code LLM benchmarks. Overlap drops from 87% to 22%.

PaperCode
💡 Applying all 11 refactoring operators achieves a 65% reduction in overlap between benchmarks and training data.
Refactoring Example
overlap ↓ step by step
Original40.98% overlap
def find_max(items):
  if len(items) == 0:
    return None
  max_val = items[0]
↓ Styl
+ Style Switch20.78% overlap
def findMax(itemList):
  if len(itemList) == 0:
    return None
  maxVal = itemList[0]
↓ IFF
+ If-Flip0.0% overlap
def findMax(itemList):
  if not (len(itemList) == 0):
    maxVal = itemList[0]
  else: return None
11 Refactoring Operators
3 scales
Syntactic
IFF If-Flip
Loop for↔while
Iter index↔elem
Comm swap ops
Shuf class-lvl
Semantic
Param +*args
Deco @timing
Renm synonyms
Inhr cross-cls
Style
Norm normalize
Styl case flip
Contamination Reduction
87% → 22% overlap
Before
0%
After
0%
0%Overlap reduction
11Python operators
4Java operators migrated
More Work · Contamination
arXiv

Concerned with Data Contamination? Assessing Countermeasures in Code Language Models

Jialun Cao, Wuqi Zhang, Shing-Chi Cheung

Evaluates whether popular strategies to prevent data contamination in code LLM benchmarks actually work as intended.

Paper
💡 Counterintuitively, code refactoring sometimes improves model performance — popular contamination countermeasures may not reliably cleanse evaluation data.
More Work · Disentangling Ability
arXiv

Isolating Language-Coding from Problem-Solving: Benchmarking LLMs with PseudoEval

Jiarong Wu, Songqiang Chen, Jialun Cao (Corresponding), Hau Ching Lo, Shing-Chi Cheung

Instead of natural language prompts, PseudoEval feeds LLMs pseudocode solutions — isolating whether failures come from weak problem-solving or weak language-specific coding. Code reduction: −42% LoC, −32% tokens.

Paper
💡 Python struggles more with problem-solving while Rust struggles with language-coding — problem-solving transfers across languages but coding requires language-specific training.
More Work · Code Repair
arXiv

FeedbackEval: A Benchmark for Evaluating LLMs in Feedback-Driven Code Repair Tasks

Dekun Dai, MingWei Liu, Anji Li, Jialun Cao, Yanlin Wang, Chong Wang, Xin Peng, Zibin Zheng

Benchmarks how well LLMs use various feedback types to iteratively repair buggy code.

Paper
💡 Mixed feedback yields the highest repair rate (63.6%), and removing semantic cues like docstrings causes severe degradation.
Featured Work · Embedded Systems
ICSE '26

EmbedAgent: Benchmarking Large Language Models in Embedded System Development

Ruiyang Xu, Jialun Cao (Co-1st), Mingyuan Wu, Wenliang Zhong, Yaojie Lu, Ben He, Xianpei Han, Shing-Chi Cheung, Le Sun

Can LLMs handle the unique challenges of embedded system development? EmbedAgent benchmarks code generation for resource-constrained environments — hardware interaction, real-time constraints, and cross-compilation.

Paper
💡 LLMs struggle significantly with embedded-specific tasks like peripheral configuration and interrupt handling — exposing a critical gap beyond general-purpose code generation.
💻
0
Test Cases
across 3 task categories
⚙
0
Electronic Components
LEDs, motors, sensors, etc.
🔌
0
Hardware Platforms
Raspberry Pi Pico · ESP32 · ESP-IDF
Three Agent Settings
Programmer · Architect · Integrator
➊
Programmer
Task + Schematic → Code
➋
Architect
Task → Circuit + Code
➌
Integrator
Source → Migrate Platform
Circuit & Code Example
LED + Button on Arduino
Arduino UNO D13 D2 GND LED 220Ω BTN
Arduino C++
#define LED_PIN 13
#define BTN_PIN 2
 
void setup() {
  pinMode(LED_PIN, OUTPUT);
  pinMode(BTN_PIN, INPUT);
}
 
void loop() {
  if(digitalRead(BTN_PIN))
    digitalWrite(LED_PIN, HIGH);
}
Wokwi Wiring (diagram.json)
pin-to-pin connections
[ "uno:13", "led1:A" ]
[ "uno:GND", "led1:C" ]
[ "uno:2", "btn1:1" ]
[ "uno:GND", "btn1:2" ]
Cross-Platform Pass@1
DeepSeek-R1 · best model
MicroPython (Pico)
0
With Schematic
0
ESP-IDF
0
65.1%With RAG + compiler feedback
27.8%Arduino → ESP32 migration