About the meta-kdd-cup-24-staging category

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

• Why should people use this category? What is it for?

• How exactly is this different than the other categories we already have?

• What should topics in this category generally contain?

• Do we need this category? Can we merge with another category, or subcategory?

View post 0

Updates to Task 1 Metrics

@saidinesh_pola I can’t share all the details about the GPT metric, but we do manage it for cases it’s not a valid score. For final submissions, each team will get to select any 2 successful submissions, doesn’t matter if it’s GPU track or not.

View post 1

Updates to Task 1 Metrics

@unnikrishnan.r I don’t see why gpt3.5 would naturally score prompt track submissions higher. Yes it is what is currently occuring but there is no natural reason for it. The metrics were decided based on actual human evaluations done on a blindly selected subset of Round 1 conversations, and gpt3.5 scores were the most correlated with the human evaluations.

View post 0

Updates to Task 1 Metrics

@saidinesh_pola , Yes GPT3.5 score is GPT-3.5-turbo generating a score based on a modified prompt similar to G-eval. It scores every utterance with the conversation history as context. And it generates a score.

View post 0

Updates to Task 1 Metrics

We are updating the metrics used to compute the leaderboards during the challenge.

Noting that the final prizes and standings will be decided based on the outcomes of the human evaluations, we explored how closely the current metrics correlated with overall score of human evaluation results.

After a thorough investigation, we noticed that World Level F1 and 4-gram BLEU did not accurately reflect the performance of the submitted models.

In light of the said observation, and with the motivation to provide a more accurate feedback to the teams about their performance, we have decided to incorporate three new metrics: CPDScore, USEScore, and BERTScore. CPDScore and USEScore have demonstrated superior accuracy compared to the previously used metrics. Moving forward, CPDScore will be the primary metric for leaderboard rankings, while BERTScore will serve as an additional metric for reference due to its widespread use in automatic evaluation benchmarks.

CPDScore is a LLM based metric that uses a prompt similar to G-EVAL. The metric focuses on “Humanness” whose criteria are described in the prompt. For Round 2, it employs GPT-3.5-turbo-0125, while for the final leaderboard we might use a stronger model.
USEScore calculates similarity using the Universal Sentence Encoder.
BERTScore is a commonly used benchmark in the evaluation of automatic metrics.

We have added the scores to all submissions of Round 2 - Task 1.

View post 2

Changelog for Round 1

Hi Everyone,

We’ve launched Round of the challenge, as you know, this comes with an updated the dataset and leaderboard. Along with this, we introduced some new major updates, as mentioned below.

1. Prompt Engineering Track - We have introduced the option to use OpenAI API for prompt engineering based submissions. This will significantly lower the barrier to entry to make submissions. Check out latest documentation starter kit to find out how to make submissions with access to OpenAI API.

2. Changes in function format - The generate_responses function in the agent class has an updated interface to support the prompt engineering track and gpu track simultaneously. Please make the changes to the agent class according to the new format. Check this starter kit commit for the updates to the documentation. It’s recommended that you pull the latest updates made on the starter kit to your submission repos.

3. Removal of persona A information - The organizers have asked for the removal of persona A information from the input data. This brings the benchmark closer to a real world setting where the persona of the conversation partner may not be known. If your agent used persona A information, please update it accordingly.

View post 0

How to know my submission was successful and what is the right way to make a successful submission?

Hi @kcy4 ,

Unfortunately your submission failed, you can see the logs here

Apologies for the confusion in the documentation, will fix them in the starter kit. You should change the model name in models/user_config.py and the function name should be generate_design, please ignore any other conflicting names in the starter kit.

View post 0

Can't open baseline starter kit

Thanks for informing @Camaro , it’s public now.

View post 1

Request for More Comprehensive Error Reports for Recent Submission Failures Task1

Hi @nazlicanto , unfortunately most these seem to be caused by intermittent issue in the OpenAI API while handling large number of API calls. I’ve made changes to try and fix this, have resubmitted your submission 248051 and will keep on eye on it so that it passes.

View post 0

Prize of Task1

Hello, the tracks are meant to share the prizes. In fact, the leaderboard and dataset for both tracks are also shared, the only difference is the trade off between using a single onboard GPU and a smaller finetuned model vs a larger model via API and only retrieval and prompt engineering.

So to answer your question, no, there will be only a single set of winners who will receive the prize.

View post 0

Submissions getting failed

Okay I’ll create it in a few days

View post 0

Submissions getting failed

Hi @saidinesh_pola I’ve replied to got gitlab issue.

View post 0

Evaluation Score Update

TL;DR: We are re-calculating scores using the correct normalization. We don’t expect any significant changes in the ranking, only on the numerical values.

Dear Participants - We are making an amendment to the evaluation score shown on the leaderboard. Unfortunately we noticed that score computation had a normalisation discrepancy with that of Melting Pot 2.0. We don’t expect the ranking of any of the submissions to change, only the numerical score. The Melting Pot 2.0 tech report defines a normalization of return for a scenario to be zero for the baseline with the lowest focal per-capita return, and one for the highest one. The normalization currently reported used the exploiter return from Melting Point 2.0 as the upper bound and random performance as lower bound. For most scenarios these are the same, but there are some for which this is not the case.

Going forward we are updating the leaderboard score to use the normalization from Melting Pot 2.0. We will use this updated normalization for the generalization score to be displayed from Nov 1 too.

Please note: This change will only reflect as a scale factor on overall score where in your performance on some scenarios may slightly decrease by the same scale for all teams and hence we do not expect any significant change in the current leaderboard ranking due to this. We will be using the new score moving forward for both development and generalization phase and also final evaluation.

View post 0

Select submissions for final leaderboard

Hi Everyone,

All teams need to select upto 3 submissions for their final leaderboard entries using the form on this page.

View post 0

Git Clone Runs infinitely

Hi Everyone,

We’re looking into what’s wrong with the submission script. In the meantime, please use the advice from @hca97. The same is also described in the starter kit. Please note the you need to push to the correct remote repo. I think for most participants, this would be set to aicrowd instead of origin, please check the name of your remote repo using git remote -v before trying to push the tag.

View post 0

Submission failed during Inference

@saidinesh_pola @gavinwangshuo , both of these failed due to timeout.

View post 0

Submission failed during Inference

@hca97 No it is not the first prediction, I see many print statements before this with varying time values.

View post 1

Submission failed during Inference

@fkemeth Can you give me the submission ids where you get the failure and I’ll check if it’s due to timeout or something else.

View post 0

Support for central_agent=False

Hi Everyone,

We now support setting central_agent=False in the Citylearn Environment. Since the creation of the environment is not directly controlled by participants. You need to provide a separate key in the aicrowd.json file in your starter kits, to let us know to use non central agents.

Things to keep in mind:

• The actions you provide will be passed to the environment as is, so make sure to use the correct action format when using non central agent.
• Make sure the aicrowd.json is formatted correctly as a valid json file.
• Make sure to use “False” in quotes when setting central agent to False in the json file.

All the best for the challenge.

View post 0

How to load a trained model during AIcrowd evaluation?

@skywuuuu You an use git-lfs using the instructions provided in the starter kit.

View post 0