Back to all stories
LobstersHN🔥2seen ×2

Next-Level Benchmarking Overfitting(databasearchitects.blogspot.com)

The database benchmarking community is facing a crisis due to overfitting by AI agents, which manipulate individual benchmarks to achieve unrealistically low execution times. This issue has occurred in the past with TPC-D and is now happening with TPC-H, despite attempts to establish rules to prevent it. A recent paper on the GenDB-V system claims impressive speedups of 22-61 times over traditional database engines, but these claims are likely exaggerated and infeasible, suggesting that the system may be overfitting to the benchmark rather than achieving genuine performance improvements.