Testbed

Developed by Aptima and Arizona State University, the ASIST testbed for studying interaction between artificial social intelligence and human teams. It uses Microsoft Minecraft as its task environment for human subjects experiments, and supports the variety ASI and teamwork measurement agents developed by the performers above.

ASIST Minecraft Testbed

The current ASIST Minecraft Testbed (“ASIST Dragon”) is designed for large-scale public data collection of human-AI interaction in teams. It is available as an open-source project at this GitLab repository.

It includes the following features:

  • Agent and experiment support
    • Microservice architecture, with agents deployed as Docker containers
    • Unified message bus with a generalizable message set
    • Monitoring and validation of experimental sessions
  • Extended participant interaction
    • Custom Minecraft Mod to instantiate team tasks and enable rich logging of activity
    • Web-based “ClientMap” interface to enhance player interaction opportunities
    • Integrated survey administration
  • “Admin-less” data collection
    • Participant self-registration and login
    • Automated team formation in a waiting room
    • Guidance and instructions for setup and gameplay
    • Automated scaling to support play by multiple simultaneous teams
    • Automated data processing and export
  • Updated “Bomb Disposal” scenario
    • Incorporation of a required team planning phase
    • Structured communication methods in planning and mission execution
    • Flexible team member inter-dependencies

Testbed Technical Specifications

ASIST Administration VM

1 instance per testbed deployment

  • VM type: n2-standard-4
  • 4 vCPU
  • 16 GB Memory
  • 600 GB Storage
Admin-less Testbed VM Instances

1 instance per team (dynamically created/destroyed as teams form)

  • VM type: n2-standard-32  (Currently*)
  • 32 vCPU
  • 128 GB Memory
  • 600 GB Storage

Admin-less Testbed Instance memory may be reduced based on agent resource needs.

Admin-less Instance Architecture

Testbed User Flow

  1. Participant registers and signs in
  2. Participant fills out surveys and is assigned to a team in the waiting room
  3. When a team is formed, the waiting room creates a Testbed VM for that team through Google Cloud.
  4. Each player on the team is transferred to the testbed VM, with their browser getting redirected from the Waiting Room on the Admin VM to the ClientMap on the new Testbed VM
  5. The team plays as many games as they want on the testbed VM. Once the team decides to stop playing, the VM automatically exports data to Google Cloud, and the testbed VM is destroyed. Players are returned to the Waiting Room.

ASIST Study 3 Testbed (“ASIST Saturn+")

The testbed used for ASIST Study-3 is available as an open-source project at this GitLab repository.

Technical support for use of ASIST Saturn+ is limited.