Challenge Summary: Snake Species
By
debalb
Following is a summary of the Snake Species Identification challenge organized by the Institute of Global Health (IGH) on AIcrowd since Jan’19.
CHALLENGE SUMMARY 🐍
Snakebite is recognized by leading health organizations across the world (including WHO) as a “dramatic humanitarian crisis in global health”, affecting some of the poor and rural communities in developing countries, causing over hundreds of thousands of human deaths and disability around the world every year.
Correct administration of life saving antivenom depends primarily on the correct taxonomic identification (i.e. family, genus, species) of the biting snake which is a challenging task due to high species diversity, incomplete or misleading information of incidents and general lack of knowledge or resources in healthcare professionals.
Keeping this in perspective, the Institute of Global Health started exploring how Machine Learning can help with snake identification, in order to potentially reduce erroneous and delayed healthcare actions, using a dataset of RGB images of snakes, their corresponding species (class) and geographic location (continent, country).
Through crowdsourcing, the Institute of Global Health hopes to obtain the best classification model, trained using the available data, that can be utilized by health agents across the globe to prevent deaths due to snakebites.
Talking about the motivation to set up this challenge, following is what we learnt from Rafael, the organizer of this challenge:
“I very much would like to congratulate the whole concept and the whole idea behind AIcrowd. I think I've been selling the whole concept of crowd sourcing solutions globally and this is an extremely powerful idea. And some of the solutions that we've been crowdsourcing come from some of the countries that are most affected by the problem that we're proposing, in our case snakebites. For example the winner of round-2 was an Indian computer scientist and as you know India is one of the most affected countries in the world by snakebite if not the number one. So having an Indian student coming up with the best AI solution makes the whole concept of AIcrowd, the whole idea, extremely relevant from our own global health perspective. I mean the question behind the challenge is quite interesting and the fact that ultimately the solutions will have an impact in the field and you can dream of the fact that some of this solution could save lives one day. That's the ultimate goal of this work, that we have the potential to save lives in the field.”
TIMELINE ⏲
An interesting aspect about this challenge was that it was introduced in the format of an ongoing benchmark with no defined end date. As of Jan’20, 3 rounds have been completed as below:
21Jan'19 - Competition Open
31May'19 - Qualifying Round 1 submission deadline
31Jul'19 - Qualifying Round 2 submission deadline
17Jan'20 - Qualifying Round 3 submission deadline
PRIZES OFFERED 💰
There weren’t any cash prizes offered by the organizers. Instead, the participants were competing for an invitation to the Geneva Health Forum 2020 and also an opportunity to co-author the paper releasing the dataset and describing the top solution(s) as baselines.
It is usually interesting to see what attracts different people to participate in such challenges and solve some of the toughest problems. While for some it could be the promise of a reward or reputation, there are some who find a more personal motivation.
As shared by Gokul, winner of round-2 and also runners up in round-3:
“I was actually looking out for challenges at AIcrowd and I have a friend who's a wildlife photographer and he has been to the Sundarbans, in north-eastern part of India, where he has actually seen the impact of snake bites on real life. It actually affects the poor people in developing nations. So that motivated me, he also motivated me and said try doing something that is for the good of the people and it's also a neglected area.”
ANALYZING THE DATA 🔬🕵
At the core of the problem is understanding the dataset, which comprises 187,720 images of varying sizes, split into 85 species in round-1. As the rounds progress, the organizers keep adding more images and species to this dataset. To get a measure of the vastness of the entire dataset, snakes are classified into 24 families, containing 528 genera and 3,709 species.
Another level of difficulty is added by several aspects of snake morphology, which bring in a high intraclass variance for certain classes and a low inter-class variance among others. More specifically:
- Some species have patterns that vary depending on their age
- Some species have patterns that vary depending on their location
- Two species might look very similar, with one being venomous and the other not
A typical problem with classification challenges, imbalanced dataset, is found here as well, meaning that the distribution of images between classes is not equal for all classes, e.g. the class with the most images has 17,749, while the class with the fewest images has 552.
As we learn from the organizers:
“We have a lot of images, we are close to half a million images of snakes covering a large diversity of snakes. So there are 3,700 classes. That's the total number of classes in the world, species of snakes. And we have about 2,500 species of snakes that are covered in our dataset. But of course, the dataset is very biased. There are classes that are very well represented but other classes which are not so well represented, which is often the case in this type of dataset. The problem so far is, if we only limit the dataset to those classes where we have many images, we have a geographical bias as well in the sense that many of our images are very well represented for countries like Europe or the United States or well-developed countries but not necessarily some of the countries that are actually more affected by snakebites.”
LEADERBOARD RANKINGS 🥇
With all these problems at hand, the participant’s task was cut out for them right from the very first round.
Round-1 ended with the leading solution scoring F1-score of 0.846, closely followed by 2 more submissions both scoring above 0.80.
Round-2 saw a new winner emerge with an improved F1-score of 0.861 followed by 2 more submissions all scoring above 0.80.
Round-3 results were even more exciting with a new winner emerging yet again, and a new benchmark achieved with F1-score of 0.90
WINNING STRATEGIES 🎯
Solving a multi faceted challenge such as this often requires an "out of the box" thinking. While some tried developing deep neural networks, and others attempted to create novel ways of handling an unbalanced dataset, sometimes it is the simple things that can help get you closer to the goal.
For Gokul, this was to locate the actual snake in different images and isolate them from the rest of the noisy background. In his own words:
“The other main technique that was crucial in winning the challenge was figuring out where the snakes were, because they were all over the image. Most of the images had only one snake but they were all over the place. So I built a small ResNet-50 model which would predict the bounding boxes where the snake was and I could actually crop out that snake. So instead of feeding the entire image into the model I was only feeding the part where the snake was actually there.”
FUTURE OUTLOOK 🔮
Looking at the results from the latest round, the organizers feel confident that they will get to see more solutions with even higher accuracy and robustness allowing them to open up almost the entire dataset available on snake species for model training.
As shared by Rafael:
“The challenge is working very well, not only on AIcrowd, but beyond AIcrowd as well and the whole idea of crowdsourcing. I mean the whole idea of coming up with an AI solution to support healthcare providers, is quite powerful and we have big expectations from the World Health Organization, big expectation from Médecins Sans Frontières (“Doctors Without Borders”) and others who see great interest in the work that we're doing. What was always a challenge for us is how far can we go in the threshold? What is the lower limit, in terms of images per class, that we can give to the participants? Now what I heard from the winners of round-3, they really said that this limit could be pushed further down. So apparently they said that they can even work with 10 images per class, which I find is very exciting because we have a lot of classes which have only a few images. So if we can push the limit to that level, that will be amazing because it will almost give us the opportunity to open the whole dataset.”
Among other things that the organizers would like to try out in future rounds is to encourage participants to collaborate with expert zoologists specializing in snake related research.
Rafael adds further:
“That's an interesting aspect actually. Thinking about maybe doing a tandem, where you have a snake biologist together with an AI expert. That sounds like a great idea. How could we do this? We could reach out more to the pathological community. We could find a way of rewarding interdisciplinary teams. One option is, presenting this project in pathological conferences and presenting this a bit more.”
It is an exciting prospect indeed for the participants as well, as Gokul also agrees:
“That would actually make the challenge much more interesting, because I would have this larger insight. I think inter-class variances were very confusing at times, like let's say that there were these few classes/species which the system really found very hard to differentiate. If there was a hepatologist for example, they could actually lead me to think that this is how the species are actually being classified. Because I have no idea why these images are classified and even if I wanted to hand label it, I would not be able to do that. So a biologist could lead me to knowing that this is why this is called this species and these are the differences that they have.”
Surely, the future looks promising and it seems only a matter of time that the organizers would be able to deploy this solution in the field where it would be able to realize it’s end goal - saving lives.
Comments
You must login before you can post a comment.
You may also like...
debalb
marcel
snehananavati