Challenge Rules

MABe CHALLENGE 2022 OFFICIAL RULES

PLEASE READ THESE OFFICIAL RULES CAREFULLY. ENTRY INTO THIS CHALLENGE CONSTITUTES YOUR ACCEPTANCE OF THESE OFFICIAL RULES. IF YOU DO NOT AGREE TO ANY PART OF THESE OFFICIAL RULES, PLEASE DO NOT ENTER THIS CHALLENGE.

NO PURCHASE IS NECESSARY TO ENTER OR WIN. A PURCHASE OF ANY KIND WILL NOT INCREASE YOUR CHANCES OF WINNING. VOID WHERE PROHIBITED.

1. Challenge Description

The MABe Challenge is a competition to facilitate the progress of multi-agent behavior modeling. Whereas the 2021 MABe Challenge focused on accurately detecting human-defined behaviors of interest from animal tracking data, our goal for 2022 is to highlight unsupervised approaches to behavior analysis. Numerous groups have proposed unsupervised methods to identify types of animal behavior in a data-driven manner. However, due to a lack of benchmark datasets or metrics in behavioral neuroscience, each lab typically develops and evaluates methods on their own in-house data; there is no consensus in the field as to what constitutes a “good” unsupervised representation of animal pose and movement.  The goal of this year’s challenge is for participants to develop and test unsupervised behavioral embeddings set of benchmark datasets for the evaluation of unsupervised behavior representations. For this year’s challenge, we release a new trio of datasets from behavioral neuroscience, including social interactions in mice, flies, and ant-beetle pairs. Teams will be tasked with creating low-dimensional representations, or “embeddings”, that characterize animals’ actions in these three datasets, given either keypoint-based estimates of animals’ poses or raw videos of their interactions. We will evaluate the quality of these representations on a set of experimentally relevant benchmark tasks, such as detecting social behaviors, identifying differences between animal strains, or distinguishing effects of optogenetic stimulation. To prevent teams from fine-tuning their embedding to the evaluation tasks, only a small number (~2) of tasks will be known to the teams- all other tasks will be hidden for the duration of the challenge.

Your contribution will help improve our depth of understanding of animal behaviors in real datasets from neuroscience; these improvements can help accelerate behavioral experiments, and help us better understand the relationship between brains and behavior. Eligible winners of the challenge will be invited to speak at our Multi-Agent Behavior Workshop at CVPR2022; participation in the workshop is not required to win.

This Challenge will be run in accordance with these Official Rules (“Rules”).

 

2. Sponsors

The Challenge is organized and sponsored by Northwestern University and Janelia, with additional prize sponsorship by Google and Amazon (additional sponsors pending.) Northwestern and Janelia collectively are referred to as the “Organizers.” 

 

3. Organizers Admins

“Organizers Admins” are any companies or organizations authorized by Organizers to aid them with the administration or execution of this Challenge, including but not limited to, AIcrowd SA.

 

4. Challenge Start and End Dates

The challenge starts on February 7th and ends on May 10th. All prizes will be awarded by July 31st.

 

5. Am I Eligible to Enter the Challenge?

You are eligible to enter this Challenge if you (and each member of your Team) meet all of the following requirements as of the time and date of entry:

• You are an individual;
• You are 18 years of age or older but in no event less than the age of majority in your place of residence;
• You have Internet Access, an Email Account, and access to a personal computer;

The organizers will not be able to transfer prize money to accounts of any of the following countries or regions. (Please note that residents of these countries or regions are still allowed to participate in the challenge.)

• The Crimea region of Ukraine
• Cuba
• Iran
• North Korea
• Sudan
• Syria
• Quebec, Canada
• Brazil
• Italy

Please Note: it is entirely your responsibility to review and understand your employer’s, state’s, and country’s laws and/or policies about your eligibility to participate in this Challenge. If you participate in violation of your employer’s, state’s, or country’s laws and/or policies, you and your Entry may be disqualified from the Challenge. Organizers disclaim any and all liability or responsibility with respect to disputes arising between an employer and such employer’s employee or between a country and its resident in relation to this matter.

6. Is the Entry an Eligible Entry?

To be eligible to be considered for a prize, as solely determined by the Organizers:

The Entry MUST:

• be compatible with the official submission format;
• be in English;
• be the Team’s own original work;
• not have been submitted previously in any promotion of any kind;
• not contain material or content that: is inappropriate, indecent, obscene, offensive, sexually explicit, pornographic, hateful, tortious, defamatory, or slanderous or libelous; or promotes bigotry, racism, hatred or harm against any group or individual or promotes discrimination based on race, gender, ethnicity, religion, nationality, disability, sexual orientation, or age; or promotes alcohol, illegal drugs, or tobacco; or violates or infringes another’s rights, including but not limited to rights of privacy, publicity, or intellectual property; or is inconsistent with the message, brand, or image of Organizers, is unlawful; or is in violation of or contrary to the laws or regulations of the United States or any jurisdiction in which the Entry is created; and

The Team members MUST:

• ensure the Team has obtained any and all consents, approvals, or licenses required for submission of the Entry;
• obtain any consents necessary from all members of the Team with respect to the sharing of such member’s personal information as outlined herein;
• obtain the agreement of all members of the Team to these Rules;
• not generate the Entry by any means which violate these Rules, the Organizers’ Terms of Service or the Organizers Privacy Policy;
• not engage in false, fraudulent, or deceptive acts at any phase during participation in the Challenge; and
• not tamper or abuse any aspect of this Challenge.

7. Disqualification.

If you, any Team member, or the Entry is found to be ineligible for any reason, including but not limited to conflicts within Teams and noncompliance with Sections 5 and 6 of these Rules, the Organizers reserve the right to disqualify the Entry and/or you and/or your Team members from this Challenge and any other contest or promotional activity sponsored or administered in any way by the Organizers.

A participant is not allowed to create more than one account to participate in the challenge. A participant can only be in one team. Violating this provision will result in disqualification from the challenge.

8. How may the Entry be used?

The Entry may be used in a few different ways. Organizers do not claim to own your Team’s Entry, however, by submitting the Entry you and each member of your Team:

• hereby grant to Organizers and Organizers Admins a non-exclusive, irrevocable, royalty-free, world-wide right and license to (i) review and analyze the Entry in relation to this Challenge; and (ii) use the Entry or parts of your Entry in any media for any non-commercial or commercial purpose in connection with the marketing, sale, or promotion of Organizers, Organizers Admins and their respective products and services;
• agree that each member will execute any necessary paperwork for Organizers and Organizers Admins to use the rights and licenses granted hereunder;
• acknowledge and agree that there is no compensation for the use of the Entry as described in these Rules;
• acknowledge and agree that there may not be credit or attribution to the Team members for the use of the Entry as described in these Rules;
• acknowledge that the Organizers or Organizers Admins may have developed or commissioned materials similar to the Entry and waive any claims resulting from any similarities to the Entry;
• understand that the Entry may be posted on a public website or social media channel and that Organizers are not responsible for any unauthorized use of the Entry by visitors to such sites; and
• understand and acknowledge that, subject to provision of Prizes, Organizers are not obligated to use the Entry in any way, even if the Entry is selected as a winning Entry.

Personal data you submit in relation to this Challenge will be used by Organizers and Organizer Admins in accordance to Section 14 of these Rules.

 

9. How will Winners be Selected and Notified?

Entries will be judged via an algorithm that will generate a score based upon which Entries will be ranked. The rankings will be displayed on the AIcrowd Site’s Challenge specific leaderboard (“Leaderboard”).

In each task the algorithm will rank your Entry as follows:

The Challenge Organizers have constructed a hidden set of labels for a variety of subtasks (such as detecting occurrence of behaviors or determining the strain of the animal.) Following Entry submission, an automated script will split the Entry into training and test sets using 3 random seeds, and pair these sets with their associated hidden labels. The script will then train a single-layer neural network for each behavioral subtask in a supervised fashion, and evaluate network performance on the held-out test set to produce either an F1 score (for binary labels) or a mean squared error (MSE, for continuously valued labels) for each subtask.

To translate F1 and MSE scores to leaderboard ranking, we will use the following system. For each subtask, all Entries will be ranked according to their F1 (higher is better) or MSE (lower is better) scores. We will then average rankings across subtasks, to determine an overall Average Rank. The overall leaderboard ranking will then be determined by this Average Rank. There will be a separate leaderboard for each of the four tasks.

Tied Entries

If two or more participating Teams have the same score, the higher ranking will be awarded to the team having the greatest number of first-place rankings across subtasks.

If scores and error are identical, prizes will be shared evenly among the Teams.

Potential winners will be contacted within one week of each Deadline via the email address associated with the AIcrowd.com account through which the Entry was submitted. If a potential winner cannot be contacted, does not respond as directed, refuses the prize, or is found to be ineligible for any reason, such prize may be forfeited and awarded to an alternate winner. Only one alternate winner will be selected per each prize package, after which prizes will remain unawarded.

To the extent that there is any dispute as to the identity of the potential winner, the official account holder of the email address associated with the AIcrowd account through which the Entry was first submitted will be deemed the official potential winner by Organizers.

10. Your Odds of Winning

THE ODDS OF WINNING A PRIZE ARE SUBJECT TO THE TOTAL NUMBER OF ELIGIBLE ENTRIES RECEIVED, AND HOW YOUR ENTRY SCORES IN ACCORDANCE TO THE JUDGING CRITERIA. The total number of participants in the previous year of this challenge was 271.

11. Prizes

The prize pool is approximately $12000 USD in total. It will be assigned as follows:

There will be four tasks and a leaderboard for each task. The top team in each leaderboard is assigned $1500, the second place team in each leaderboard is assigned $1000, and the third place team in each leaderboard is assigned $500. The winning individuals may incur tax liability. 

12. When will prizes be awarded?

The prizes will be awarded within a commercially reasonable time frame. All members of a Team may be required to complete and sign additional documentation, such as non-disclosures, representations and warranties, liability, consent for use of the members’ names and publicity releases (unless prohibited by applicable law), and tax documents, or other similar documentation in order for the potentially winning team to claim the prize. If each Team member does not agree to complete and sign the additional documentation within a reasonable time, the Organizers may choose another Entry by another Team. Organizers will in no way be involved in any dispute with respect to receipt of a prize by any other members of a Team.

13. Winner List

A list of all winners of this Challenge will be posted on AIcrowd Site and may be announced at Organizers’ discretion via Organizers’ Twitter, Facebook, Blog, or Website, or at an Organizer or Organizer Admins sponsored or hosted event.

14. Your Personal Data and Privacy

Organizers may use cookies and/or collect IP addresses for the purpose of implementing or exercising its rights or obligations under the Rules, for information purposes, identifying your location, including without limitation for the purpose of redirecting you to the appropriate geographic website, if applicable, or for any other lawful purpose in accordance with the Privacy Policy.

Organizers may use the personal data you provide via your participation in this Challenge.

• to contact you in relation to the Challenge;
• to confirm the details of your Entry;
• to administer and execute this Challenge, including sharing it with Organizer Admins;
• at Organizers’ discretion, to credit you and/or your Team for the Entry, identify you and/or your Team as a Winner, or other similar notice; and
• as otherwise noted in these Rules or as necessary for Organizers to meet their obligations under these Rules or applicable law.

Organizers only require name and email address to be submitted for you to participate in this Challenge for its uses as outlined in this Section 15. 

If you reside within the European Union or the European Economic Area, you may have additional rights and protections under the General Data Protection Regulation. Please read the terms and conditions of the AIcrowd Site carefully to understand how your data may be used by AIcrowd SA.

15. Additional Terms and Conditions

If Organizers determine, in their sole discretion, that any portion of this Challenge is compromised by virus, bugs, unauthorized human intervention, or any other causes beyond its control, that in the sole opinion of Organizers corrupts, or impairs the administration, security, fairness or proper participation in/of the Challenge, Organizers reserve the right to (a) cancel the Challenge; (b) pause the Challenge until these issues are resolved; or (c) consider only those Entries submitted prior to the when the Challenge was compromised for the prizes.

To the fullest extent permitted by applicable law, you agree that Organizers and Organizer Admins, and each of their directors, officers, employees, agents and assigns, will not be liable for personal injuries, death, damages, expenses or costs or losses of any kind resulting from participation or inability to participate in this Challenge or acceptance of or use or inability to use a prize or parts thereof including, without limitation, claims, suits, injuries, losses and damages related to personal injuries, death, damage to or destruction of property, rights of publicity or privacy, defamation or portrayal in a false light (whether intentional or unintentional), whether under a theory of contract, tort (including negligence), warranty or other theory.

Your use of any other products and services required by these Rules, whether required by these Rules or not, are subject to the terms and conditions associated with such products or services, including the AIcrowd site and services.

---