TrussTutor-Bench: Benchmarking large language models on solution accuracy, sycophancy, and error detection in statics

What it claims

TrussTutor-Bench tests whether LLMs can act as technically reliable statics tutors, not only solve problems. It has 200 statically determinate 2D truss problems and scores thirteen LLMs on three tasks: solving the problem, defending correct student work under pushback, and locating one injected error. The key result is a dissociation. Error detection was near-ceiling for every model (90.4% to 100%). Rejection of correct work that a model had itself solved ranged from 1.6% to 50.0%. The authors (Lejeune group, Boston University, October 2026) report that problem-solving accuracy "only partly predicts" which models cave. Accuracy alone is therefore a weak proxy for tutoring reliability.

How it works

The 200 tasks span four archetypes: method of joints (84), method of sections (59), zero-force members (31) and inverse/design problems solving for an unknown load or dimension (26). Of these, 98 come from six textbooks, including Hibbeler 16th ed. (26), Beer et al. (28) and Bedford and Fowler (21). The other 102 are original: 25 handwritten and 77 computer-drawn in 30 drawing styles, so schematic notation varies.

Each task is a self-contained Python template. It holds nodes, elements, loads, supports, a JSON answer schema and a "solution recipe" of solver operations. Ground truth is checked by two independent solves before human review:

flowchart LR
  T[Task template] --> S[Symbolic recipe solve]
  T --> D[Direct stiffness solve, E=A=1]
  S --> V{Agree within 1e-4?}
  D --> V
  V -->|no| X[Rejected]
  V -->|yes| H[Human review vs source]
  H --> B[Enters benchmark]

The task passes only if both solves agree within 10^-4 on every member. The reviewer then confirms the problem is solvable from the text and schematic alone.

There are two evaluation modes. In the baseline mode, the model gets the problem text and schematic and returns schema-constrained JSON. In the formative mode, the symbolic trace becomes a numbered student solution with rendered free body diagrams. That solution is presented in one of three postures:

1. Flawless work. The student insists it is wrong, which tests sycophancy.
2. Early error. The global moment equation has a sign flip (right-hand rule), and the error propagates through the solution. The corrupted solver ignores the leftover equilibrium violation at the final joint. This posture applies to 163 tasks.
3. Late error. The math is correct, but a final tension or compression label is flipped.

Grading compares only two fields: is_work_correct and error_step_number. The model's explanation to the student is not graded.

How to use it

The page describes each numerical variable as having a default and a permissible range. Only the defaults were used in this study, so the templates can generate parameter variants. The authors call the benchmark "extensible." The fetched text gives no repository link, licence or run command.

Limits

• Only planar, statically determinate trusses are covered.
• Errors are synthetic, single and cleanly injected. The authors say near-ceiling detection holds "under these clean conditions," not for messy real student work.
• Explanation quality and pedagogy are not measured. The authors say follow-up work should study learning outcomes, not technical capability.
• All runs use default parameter values.
• The fetched text is truncated partway through Methods. It does not include the names of the thirteen models, per-model results, or the Section 4 limitations.