Automatic Generation of Scenarios for System-level Simulation-based Verification of Autonomous Driving Systems

Supplementary web page for the submission to https://fmasworkshop.github.io/FMAS2023/

We provide a docker image containing the entire VIVAS workflow.
The user needs to set-up the AI-based agent (Interfuser repositiory) locally.
An interface using docker-pipe is provided to run the Vivas workflow from the docker-container on Carla-Interfuser.

InterFuser repository

Download the InterFuser repository Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer
Setup the environment as per the instructions in the repository. It also includes building carla, dowloading pre-trained neural network model checkpoints, and installing python packages.
In particuar, follow the instructions in https://github.com/opendilab/InterFuser/README.md for only the steps 1. Setup and 5. Evaluation.

Changes to the InterFuser repository

Download the Files_Interfuser.tar.gz archive, contaning some modifications to InterFuser to let it run with VIVAS.
Move the following files in the tarball in the respective paths.

  1. ego_route.xml (InterFuser/leaderboard/data)
  2. scenariotriggervivas.json (InterFuser/leaderboard/data/scenarios)
  3. routescenario.py (InterFuser/leaderboard/leaderboard/scenarios): to interface with vivasscenario
  4. setnewscenarios.py (InterFuser/leaderboard/leaderboard/scenarios): to interface with vivas_scenario
  5. leaderboardevaluator.py (InterFuser/leaderboard/leaderboard): to interface with metricsmanager of scenario runner
  6. run_evaluation.sh (InterFuser/leaderboard/scripts): to run interfuser agent on carla. Change necessary paths.
  7. basic_metric.py (InterFuser/scenario_runner/srunner/metrics/examples): to create metric log files
  8. vivas_metric.py (InterFuser/scenario_runner/srunner/metrics/examples): save simulation results into a json file
  9. metrics_manager.py (InterFuser/scenario_runner)

Make an empty folder results_vivas in the root of InterFuser repository.

VIVAS Docker

We provide a Docker image with the necessary tools to run the VIVAS loop as a tarball: vivas-docker.tar.gz

It can be imported with the following command:

docker load < vivas-docker.tar.gz

The source code of the VIVAS loop is provided separately from the Docker image for convenience (so e.g. the code be explored without running Docker): vivas-toplevel.tar.gz

Run Vivas loop on Carla-InterFuser

Download the archive with additional scripts for the connection between Carla-InterFuser and VIVAS: Files_vivas-docker.tar.gz.

Change necessary paths in the files vivas_pipe.sh and carla_env_init.py before running the following steps.

vivas_pipe.sh should be installed on user's computer locally. You can put it anywhere and run the following command to make it executable.

chmod +x docker_pipe.sh
  1. Start the carla server
    In the first terminal, in the root of vivas-docker run:
sh vivas_pipe.sh

A carla window will open.

It allows the user to pipe commands from the docker container to the host, thus allowing simulations on the host to be started from within the docker.

  1. Run Vivas Docker container
    In the second terminal, start vivas-docker container:
docker run --net=host -it -e DISPLAY=unix$DISPLAY -v /tmp/.X11-unix/X0:/tmp/.X11-unix/X0 -v /home/srajan/vivas-docker/submodules/vivas-toplevel/:/vivas-toplevel/ --device /dev/snd --device /dev/dri/card0 --device /dev/dri/ -v /home/srajan/docker_pipe:/docker_pipe -v /home/srajan/InterFuser/scenario_runner/srunner/scenarios/:/shared/scenarios/ -v /home/srajan/InterFuser/results_vivas/:/shared/carla_results/ -v $XAUTHORITY:/root/.Xauthority:ro vivas bash

Note: Update the paths to the vivas-toplevel, docker-pipe and InterFuser repositories in the command above.

Once inside the container, run:

cd /vivas-toplevel
python3 vivas.py tests/config_carla.json tmp/1

Check coverage results

Check the coverage results shown in the paper (saved in the folder /home/srajan/InterFuser/results_vivas/):

python3 results_parser.py

Replay a simulation scenario

Replays a Carla simulation using an existing concrete scenario without rerunning the complete Vivas workflow. Parameters are passed as CLI arguments fetched through sys.argv.

An example of running a saved scenario is shown below. The scenario is saved in the folder results_vivas/fmas23-new/ and the scenario file is results_vivas/fmas23-new/concretizer/scenario_00-01.py. The configuration file is config_carla.json.

python3 replay.py results_vivas/fmas23-new config_carla.json replay scenario_00-01.py