XFC 2021

Explainable Fuzzy AI Challenge (XFC 2021)

Introduction to the Challenge

One of the biggest issues facing widespread adoption of Artificial Intelligence (AI) in industry is the transparency of the decision-making process. Many AI systems are being used with the sole objective of maximizing both accuracy in the prediction and computational efficiency. However, the value of the explainability of these systems is often underestimated. Explainability, in part, refers to the human interpretability of the processes and predictions given by the system. The more explainable a certain algorithm is, the simpler it will be for a human to understand or explain the underlying reasoning. When it comes to critical decisions, humans expect the basis of those decisions to be such that they are easy to verify and corroborate.

In this challenge, teams of 2-6 students constructed an explainable AI system based on Fuzzy Inference Systems to play the Python arcade game “Asteroid Smasher”. The challenge took place throughout the 2021 Spring semester and culminated in a virtual competition where student-created AIs played game scenarios created by the organizers. The AIs/teams were scored on several criteria including performance during competition, explainability metrics, etc. Winners  received monetary prizes after the competition (details below).

What teams learned

  • The importance of an Explainable AI algorithm
  • Different techniques for gradient-free optimization, such as Genetic Algorithms
  • Application of all of these towards development of an AI agent that can play the game “Asteroid Smasher”

The Challenge

Screenshot of the game

The teams created fully autonomous XAI algorithms, in Python, that were able to play the Python Arcade Game “Asteroid Smasher”. In the game, a 2-dimensional spacecraft moves to avoid collisions with numerous asteroids that appear. The asteroids have different shapes, sizes and velocities. The spacecraft also has a weapon that can shoot straight ahead. If the projectiles emitted reach any of the target asteroids, they break into smaller pieces. The smallest asteroid pieces disappear after being hit by a projectile. A control system must consider all the different features of the system and determine the movement and shooting decisions of the spacecraft. 

The teams' code had three main parts:

  • The platform (main python script) for the Asteroids Arcade Game. This is given to the teams. It refers to all the different background commands and functions that the game needs to work. It has different sections empty that teams filled with their own code in order to make it work. 
  • The Fuzzy Inferencing System (FIS). This is the primary ‘brain’ of the XAI, a set of if-then rules that decide how the spacecraft moves. The organizers held a session to teach how to code a simple FIS for a different problem, so the teams had the knowledge needed to develop their own FIS and integrate it in the challenge.
  • The Optimization Process. This refers to the way in which the FIS is trained. AI requires a preliminary process called “training”, where the algorithm is exposed to training data/scenarios and its parameters are modified in order to achieve the best “score” possible according to some objective function. One particularly powerful type of optimization algorithm is Genetic Algorithms (GA). GAs are based on Darwinian principles of evolution and survival of the fittest. Through the competition, a free lecture was provided to the participants on this topic. The teams  used GAs or other applicable types of optimization algorithms in order to train their Fuzzy Systems to learn to play the game. There were no restrictions on the learning methods including manual tuning.

 

Who was able to participate?

The challenge was open to all undergraduate and graduate university students. The format of the challenge was such that it complemented the skills of students enrolled in STEM degrees, but is was not limited only to STEM majors.  

Timeline

1/19/2021: Competition Kick-Off

Online meeting where we will go through the challenge and have 2 guest speakers from industry discussing the importance of explainability and the future of Fuzzy AI. Details on how to join the meeting and the time will be sent to the participants after registration.

1/21/2021: Seminar: Fuzzy Inference Systems

This will be a recorded online session where we go through all the necessary steps to understand how to build a Fuzzy AI system. More information will be sent closer to the event by email.

1/31/2021: First Deliverable Submission Due Date

The teams will need to use the concepts learnt in the first seminar to build their own solution of a simplified real-world problem using Fuzzy AI. The details of the problem will be given during the first seminar. Guidance in the development to those teams that request it will also be granted. This milestone will contribute to final team scores.

2/1/2021: Seminar: Genetic Algorithms and other gradient-free optimization algorithms

In a similar format to the first class, a lecturer will walk you through the mechanics and implementation of a Genetic Algorithm (GA). We will also discuss the use of a GA in the optimization of a FIS. Other gradient-free optimization algorithms will also be discussed. The students will then need to apply these concepts in their second problem. 

2/14/2021: Second Deliverable Submission Due Date

The teams will need to use the concepts learnt in the Optimization seminar to optimize a Fuzzy Inference System. Details on the problem will be given at the end of the seminar. Guidance can be provided as requested by teams and submissions will count towards final evaluation scores.

2/15/2021: Introduction to the Asteroid Smasher game

In this meeting the Asteroid Smasher game will be introduced including constraints and details on controller development. Final details on the competition will also be given at this time including scoring metrics, expectations, and other final deliverables/requirements.

4/4/2021: Final Deliverable – Asteroid Smasher Fuzzy XAI Submission Due Date

This will be the submission date of the final version of your Fuzzy XAI controller for the Asteroid Smasher game. Details on submission format and requirement will be given to you beforehand by the organizers. The final week between the submission date and the competition date will be used to ensure any issues are resolved. Submissions must be received by 11:59 PM EST. 

4/10/2021: Competition Livestream Event

Finally, after all the hard work, the students and advisors will be recognized in this final event that will take place virtually. Teams will have the opportunity to describe what they did and their approach to creating their Fuzzy XAI systems. Then they will see their creations be put to the test in a series of scenarios created by the organizers. The teams will be scored and awards will be given out based on numerous criteria. 

Evaluation Criteria:

  • Fuzzy System deliverable
  • Optimization deliverable
  • Final Asteroid Smasher Fuzzy XAI performance: Agents will be evaluated against a portfolio of scenarios with a custom fitness function
  • Explainability of the created AI
  • Creativity in the solutions

Awards

There were several awards for the winners and for those who show creativity in their submissions! Monetary awards were given to the top 3 teams based on their overall competition score:

  • First place: $1400
  • Second place:$900
  • Third place $400

Additionally, the following awards were granted ($100 each): 

  • Most Explainable Solution
  • Most Innovative Solution
  • Most Computationally Efficient AI

Judges

Thank you to our Judges!

  • Barnabas Bede PhD, Associate Professor; Program Director Bachelor of Science in Computer Science in Machine Learning; Department Chair of Mathematics; DigiPen Institute of Technology
  • Alex Gonzalez, Research Assistant at Purdue Computer and Information Technology
  • Xiaonan(Shannon) Jing. PhD student major in Computer and Information Technology at Purdue University

Sponsors

References