When Can LLMs Replace Humans in A/B Tests?
Spotify engineers explored using LLMs to replace human outcomes in A/B tests, finding that LLM predictions can be a valid proxy if two conditions are met: surrogacy and comparability. With proper calibration, LLM outputs can recover the human treatment effect, but the method matters, as they found that machine learning models performed better than linear calibration. However, the limitations of LLM surrogacy lie in its inability to guarantee validity for new treatments, making human experiments indispensible for true product innovation.
MusicScale