Researchers and practitioners have embraced factorial experiments to simultaneously test multiple treatments, each with different levels. With the rise of technologies like Generative AI, factorial experimentation has become even more accessible: it is easier than ever to generate different versions of potential treatments. Typically, in large-scale factorial experiments, the primary objective is to identify the treatment with the largest causal effect. This is especially true for experiments that suffer from measurement error, attrition, non-compliance, and censoring: point estimates are unreliable, but — as we show — the asymmetry in the largest treatment effect makes it possible to identify the most impactful treatment even when point estimates are biased. To exploit this asymmetry, we propose using a Fisher randomization test as a general non-parametric approach for inference, which we apply to an existing field experiment that measured intern performance at a large financial firm. We show that the earliest possible intervention has an immediate and enduring impact: performance improves in the week of the intervention and in future weeks, sometimes even to a greater extent than interventions in those future weeks. The takeaway — intervene early — has important consequences across the many contexts of workplace programs.
Winner Take All: Exploiting Asymmetry in Factorial Designs / Matthew DosSantos DiSorbo, Iavor Bojinov, Fiammetta Menchetti. - ELETTRONICO. - (2024).
Winner Take All: Exploiting Asymmetry in Factorial Designs
Fiammetta Menchetti
2024
Abstract
Researchers and practitioners have embraced factorial experiments to simultaneously test multiple treatments, each with different levels. With the rise of technologies like Generative AI, factorial experimentation has become even more accessible: it is easier than ever to generate different versions of potential treatments. Typically, in large-scale factorial experiments, the primary objective is to identify the treatment with the largest causal effect. This is especially true for experiments that suffer from measurement error, attrition, non-compliance, and censoring: point estimates are unreliable, but — as we show — the asymmetry in the largest treatment effect makes it possible to identify the most impactful treatment even when point estimates are biased. To exploit this asymmetry, we propose using a Fisher randomization test as a general non-parametric approach for inference, which we apply to an existing field experiment that measured intern performance at a large financial firm. We show that the earliest possible intervention has an immediate and enduring impact: performance improves in the week of the intervention and in future weeks, sometimes even to a greater extent than interventions in those future weeks. The takeaway — intervene early — has important consequences across the many contexts of workplace programs.File | Dimensione | Formato | |
---|---|---|---|
Factorial_WP.pdf
Accesso chiuso
Descrizione: Working paper
Tipologia:
Preprint (Submitted version)
Licenza:
Solo lettura
Dimensione
624.37 kB
Formato
Adobe PDF
|
624.37 kB | Adobe PDF | Richiedi una copia |
I documenti in FLORE sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.