Synthetic Replication Games
The Synthetic Replication Games are a series of one-day events organised in the context of a large-scale replication project exploring the potential of large language models (LLMs) to augment human samples in social science experiments. During each event, researchers of all backgrounds and career stages are invited to collaborate in adapting experimental studies from top-tier journals into a standardised format and replicate them using LLM-powered synthetic participants. The material generated during each event will be consolidated in a battery of synthetic replication packages, which will serve three main purposes: (1) benchmarking the ability of LLMs to simulate human participants in social science experiments; (2) facilitating the use of experimental data to fine tune LLMs; and (3) developing a standardised format for the replication material of synthetic experiments.
Participants will be recognized as co-authors on the paper that will present the aggregated results of multiple synthetic replications using a variety of LLMs.
Games Structure
Researchers participating in the event are split into small teams of 3-5 members based on their research interests and skills. Each team engages in the synthetic replication of one or more experiments published in a top-tier journal. Each synthetic replication involves:
- Replicating the original results using the analysis script of the original study.
- Converting the original data collection instrument into a set of standardised experimental prompts.
- Introducing into the data collection instrument – and documenting – the prompt adaptations required to conduct the experiment with a synthetic sample.
- Running a pilot synthetic replication and benchmarking the LLM predictions with the results of the original study.
During the event, participants are provided with various tools to facilitate their work. These tools include a replication package of the original study, a standardised prompt template, and a Python-based platform to streamline the conduct of synthetic experiments.
Registration
Here you can look at past events and register for one of the upcoming games.
21 May 2025 – IMEBESS 2025, Valencia, Spain (in-person)
TOPICS
Laboratory and online experiments on diverse topics selected from large-scale replication efforts with human samples, namely:
- Camerer, C.F., Dreber, A., Holzmeister, F. et al. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nat Hum Behav 2, 637–644 (2018). https://doi.org/10.1038/s41562-018-0399-z
- Holzmeister, F., Johannesson, M., Camerer, C.F. et al. Examining the replicability of online experiments selected by a decision market. Nat Hum Behav 9, 316–330 (2025). https://doi.org/10.1038/s41562-024-02062-9
WHERE
European University, Valencia
WHEN
09:00 – 18:00 UTC+02:00
FORMAT
This is an in-person only event.
REGISTRATION DEADLINE
9 May 2025
24 October 2025 – Oxford, United Kingdom (hybrid)
TOPICS
Randomised controlled trials focused on microfinance or testing behavioural interventions in health, sustainability, and tax compliance.
WHERE
Nuffield College, University of Oxford, Oxford
WHEN
09:00 – 18:00 UTC+01:00
FORMAT
This is a hybrid event.
Participants can attend either in person or virtually.
REGISTRATION DEADLINE
13 October 2025
REGISTRATION FORM
18 November 2025 – VIRTUAL
TOPICS
Laboratory and online experiments in behavioural economics, covering classical experimental games (e.g., Dictator, Trust, Public Good Games) and their variations. The experiments will focus on various research themes – e.g., social norms, cooperation, corruption, and competition.
WHERE
Online
WHEN
09:00 – 18:00 UTC+01:00
FORMAT
This is a virtual only event.
REGISTRATION DEADLINE
7 November 2025
REGISTRATION FORM
16 December 2025 – VIRTUAL
TOPICS
Laboratory and online experiments in political sciences, covering a variety of design (e.g., conjoint and vignette studies) and research themes (e.g., affective polarization, inequality and redistribution, misinformation, opinion dynamics).
WHERE
Online
WHEN
09:00 – 18:00 UTC+01:00
FORMAT
This is a virtual only event.
REGISTRATION DEADLINE
5 December 2025