LifeCLEF 2018 Expert

Note: Do not forget to read the Rules section on this page

Usage scenario

Automated identification of plants and animals has improved considerably in the last few years. In the scope of LifeCLEF 2017 in particular, we measured impressive identification performance achieved thanks to recent deep learning models (e.g. up to 90% classification accuracy over 10K species). This raises the question of how far automated systems are from the human expertise and of whether there is a upper bound that can not be exceeded. A picture actually contains only a partial information about the observed plant and it is often not sufficient to determine the right species with certainty. For instance, a decisive organ such as the flower or the fruit, might not be visible at the time a plant was observed. Or some of the discriminant patterns might be very hard or unlikely to be observed in a picture such as the presence of pills or latex, or the morphology of the root. As a consequence, even the best experts can be confused and/or disagree between each others when attempting to identify a plant from a set of pictures. Similar issues arise for most living organisms including fishes, birds, insects, etc. Quantifying this intrinsic data uncertainty and comparing it to the performance of the best automated systems is of high interest for both computer scientists and expert naturalists.

Challenge description

The goal of the task will be to return the most likely species for each observation of the test set. More practically, the run file to be submitted has to contain as much lines as the number of predictions, each prediction being composed of an ObservationId (the identifier of a specimen that can be itself composed of several images), a ClassId, a Probability and a Rank (used in case of equal probabilities). Each line should have the following format: <ObservationId;ClassId;Probability;Rank>

Here is a short fake run example respecting this format for only 3 observations: fake_run

The small fraction of the test set identified by the pool of experts will then be used to conduct the experts vs. machines evaluation.

Data

To conduct a valuable experts vs. machines experiment, we collected image-based identifications from the best experts in the plant domain. Therefore, we created sets of observations that were identified in the field by other experts (in order to have a near-perfect golden standard). These pictures will be immersed in a much larger test set that will have to be processed by the participating systems. As for training data, the datasets of the previous LifeCLEF campaigns will be made available to the participants and might be extended with new contents. It will contain between 1M and 2M pictures.

Submission instructions

As soon as the submission is open, you will find a “Create Submission” button on this page (just next to the tabs)

Evaluation criteria

The two main evaluation metrics will be the top-1 accuracy on 1) the fraction of the test set identified by the pool of experts, 2) on the whole test set.

Resources

Contact us

• Technical issues : https://gitter.im/crowdAI/lifeclef-2018-expert
• Discussion Forum : https://www.crowdai.org/challenges/lifeclef-2018-expert/topics

We strongly encourage you to use the public channels mentioned above for communications between the participants and the organizers. In extreme cases, if there are any queries or comments that you would like to make using a private communication channel, then you can send us an email at :

• Sharada Prasanna Mohanty: sharada.mohanty@epfl.ch
• Hervé Goëau: herve[DOT]goeau[AT]cirad[DOT]fr
• Alexis Joly: alexis[DOT]joly[AT]inria[DOT]fr
• Ivan Eggel: ivan[DOT]eggel[AT]hevs[DOT]ch

More information

You can find additional information on the challenge here: http://imageclef.org/node/231

Results (tables and figures)

(Official round during the LifeCLEF 2018 campaign)

Prizes

LifeCLEF 2018 is an evaluation campaign that is being organized as part of the CLEF initiative labs. The campaign offers several research tasks that welcome participation from teams around the world. The results of the campaign appear in the working notes proceedings, published by CEUR Workshop Proceedings (CEUR-WS.org). Selected contributions among the participants, will be invited for publication in the following year in the Springer Lecture Notes in Computer Science (LNCS) together with the annual lab overviews.

Datasets License