ASIST exists to advance the science of human-agent teaming (HAT), especially in operational contexts where measuring performance and team processes can be difficult. Hundreds of performers from academia, industry, and government have worked to develop multiple virtual environments, AI-enabled agents, and analytic components. All research and development conducted within the ASIST program is open-source and available to the public.
The Defense Advanced Research Projects Agency sponsors ASIST to help organizations “understand the most important component of the environments in which they operate: humans.”
ASIST Timeline
Fall 2020: Study 1 (“Falcon” Testbed)
Single player, no interventions
This experiment was designed to obtain measurements with which ASIST performers could (1) develop and assess claims concerning the social intelligence of Artificial Intelligence (agents) and (2) develop and test hypotheses concerning Theory of Mind in human-AI teams.
Summer 2021: Study 2 (“Saturn” Testbed)
Three player teams with exchangeable roles, no interventions
This study was designed to (1) support development of agents that implement artificial social intelligence (ASI) for predicting and inferring in a team search and rescue scenario in the Minecraft world; (2) support research concerning ASI and theory of mind (ToM); (3) develop a sample of interventions (advice) that may inform future research; and (4) support an evaluation of ASI agents relative to ground truth, human observers, and each other.
This study was also intended to (5) accelerate data collection by simplifying or automating experiment administration and shortening experimental sessions; and (6) accelerate research and development by performers by building upon the preceding experiment of Fall 2020.
Summer 2022: Study 3 (“Saturn+” Testbed)
Three player teams with fixed roles, with interventions
This study was designed to (1) support development of agents that implement artificial social intelligence (ASI), to intervene smartly in teams in ways that improve teamwork and its effects, or analytic components (ACs) that provide measures based in social science that drive ASI or assess its impact; (2) support research concerning ASI and Machine theory of Teams (MToT); and (3) support an evaluation of ASI advisors relative to ground truth, human observers, and each other. This study was also intended to (4) accelerate data collection by simplifying or automating experiment administration and shortening experimental sessions; and (5) accelerate research and development by performers by building upon Studies 1 and 2.
Summer 2023: Study 4 (“Dragon” Testbed; currently underway)
Three player teams with purchasable tools, with interventions
Study 4 has multiple research purposes. It is designed to (1) support development of agents that implement artificial social intelligence (ASI) to provide impactful interventions to teams in ways that improve teamwork and its effects, (2) support development of analytical components (ACs) that provide measures based in social science that drive ASI or assess its impact; (3) support research concerning ASI and Machine theory of Teams (MToT); and (4) support training and evaluating ASI advisors relative to ground truth and each other, (5) accelerate data collection for training and evaluation by simplifying or automating experiment administration and shortening experimental sessions to collect large scale human subject data for advanced machine learning modeling; (6) accelerate research and development by performers by building upon Studies 1-3, and (7) test the generalizability between findings from different experimental settings.
To achieve the purposes above, Study 4 allows for repeated participants without limit on how many trials they play. The task environment is designed to elicit teamwork (coordination and communication) and create ASI intervention opportunities while reducing training time and “barrier to entry” for players. The introduction of a Shop is designed primarily to provide opportunities for players to receive ASI interventions in an environment without time pressure, and also provides opportunities for team planning and negotiation.
