‼️ ✍️ Important: Submission Documentation Guidelines

Hello,

Thank you for actively participating in the challenge! As part of the challenge, we request all participants to document their submissions and methods and provide a formal write-up. Please note that winners are required to submit this documentation to The Web Conference. However, we strongly encourage all participants to share their approaches with the community.

Please use the provided template to create your submission documentation to ensure consistency. Upload the completed PDF file to the Google Form , including your name and AIcrowd username, no later than 3rd February 2025, 23:05 AoE. A 24-hour grace period will be provided for participants to make updates and incorporate last-minute changes before finalising their submissions.

Note: If you are part of a team, only one documentation should be submitted by the team leader, who will be responsible for filling out the form.

Key Dates and Deadlines

• Submission Deadline: 3rd February 2025, 23:05 AoE (aligned with the competition end time). Follow the template provided.

• Grace Period: An additional 24 hours will be provided to allow participants to make updates and final adjustments.

How to Submit

Please submit your documentation via the following Google Form: https://forms.gle/uqFg5zWdSEcpt7Ht5.

Thank you for your active participation, and we look forward to seeing your creative solutions!

All the best,
Team AIcrowd

View post 0

💬 Feedback & Suggestions

Hi Thomas,
Thank you for pointing that out. You are correct—the metric used is Macro F1.
The typo has now been corrected.

View post 0

🧑‍🏫 Live Townhall 11th January 2025 | Saturday 10AM CET

Hi you can find it here: https://youtu.be/kYyIguY2Kso

View post 1

📹 Brick by Brick Challenge Townhall #1 Recording & Community Contribution Baseline 📕

Hello all,

Last Sunday, we hosted the Brick by Brick Challenge Townhall, featuring insightful presentations from our organisers:

• Matt Amos: Shared the mission statement, the purpose of the challenge, details on data collection, and the processes behind the data, including background on BMS and Brick.

• Arian Prabowo: Discussed the dataset (statistics and splits), problem formulation, and benchmarking methods.

Missed the session? Watch the recording here: https://youtu.be/kYyIguY2Kso

Slide Deck: Download it here: 📹 Brick by Brick Challenge Townhall #1 Slides – Google Drive

Community Contribution Baseline: To help you ace Round 2, we’re excited to share a new baseline created by @saidinesh_pola. This baseline is built on tslib, an open-source toolkit that helps preprocess data following recommended methods, using a consistent 4-hour sampling approach. It includes a user-friendly Colab Notebook, making it easy to clone the repository, set up dependencies, and explore its tools. While the baseline is already effective, future updates could add new models, improve loss functions, and refine data preprocessing, though improving accuracy through preprocessing remains a challenge.

Have follow-up questions? Feel free to drop them in the comments below.

Screenshot 2025-01-11 at 15.35.231920×1141 121 KB

View post 0

💬 Feedback & Suggestions

Yes, the team freeze has been in effect since 10th January. It typically begins three weeks before the challenge deadline to prevent individuals from submitting duplicate entries through multiple teams.

View post 0

💬 Feedback & Suggestions

Update: the issue is now fixed.

View post 1

💬 Feedback & Suggestions

Response from the organisers:

That’s correct—the 20% and 45% values do not refer to the number of “rows” or chunks. Instead, they refer to the proportion of time series allocated to the private, public, and secret sets during dataset preparation, as described here: AIcrowd | Brick by Brick 2024 | Challenges

Specifically, during the preparation stage:

All data from the three buildings are combined into a single dataset and then segmented into distinct sets for training, leaderboard testing, and secret competition testing.

In this step, approximately 20%, 45%, and 35% of the time series are assigned to the private, public, and secret sets, respectively. However, note that the length of each time series can vary. Later, the dataset undergoes further processing which yield the “rows” and “chuncks” available to the participants:

Time Series Chunking: The dataset is further divided into shorter segments or chunks with durations ranging from 2 to 8 weeks.

I hope this clarifies your concern.

View post 0

🧑‍🏫 Live Townhall 11th January 2025 | Saturday 10AM CET

Hi, The townhall will be recorded and I will relay your question to the organisers.

View post 0

💬 Feedback & Suggestions

Response from the organisers:

This is indeed not the typical machine learning setup; however, it reflects practical realities. Unlike text or image data, publicly available datasets for buildings are extremely rare, and this is unlikely to change due to privacy concerns. Additionally, the distribution shifts between buildings are significant, driven by differences in size, design, use, legal restrictions, and occupant behaviors.

The goal of this challenge is to test generalisation capabilities. To achieve this, we intentionally moved a significant portion of the data from the training set to the testing set. This allows us to evaluate how well algorithms perform under different distributions. For context, you can think of this as analogous to a weak supervision or semi-supervised learning setup, where extensive time series data is available, but only a subset is labeled.

View post 0

💬 Feedback & Suggestions

Hi @chan_jun_hao,

There is an additional test set that is not currently avaliable. The final score will be how the model performs on the current public leaderboard test set and this holdout test set which only becomes avaliable after 3rd Feb?

We have provided all of the features from the test set. However, the scores on the current public leaderboard is only based on a part of the test set. The final score will be based on the full set.

If the above is true, is it possible to know the distribution of the seperation. For eg, (training 20%, public test set 20%, private test set 60%)

Approximately:

• 20% training set
• 45% public test set
• 35% private test set

What is final submission deliverables for round 2 on 3rd Feb? Is it the current format of just the predicted csv, or the entire model pipeline (to be run on the unseen test set)?

The deliverables are:

• The current format of the predicted CSV
• The code to the model (for validation purposes)
• Solution Documentation

View post 1

🧑‍🏫 Live Townhall 11th January 2025 | Saturday 10AM CET

Hello all,

As Round 1 concludes, we invite you to join the Brick by Brick Challenge Townhall! This session offers a unique opportunity to engage with the organisers, gain valuable insights into the challenge, and get your questions answered. Prepare to boost your Round 2 submissions and refine your strategies!

Saturday, 11th January, 2024, 10:00 AM CET
Join the Townhall on Zoom

For those unable to attend, a recording will be made available. Feel free to drop your questions on this post, and the organisers will address them during the session.

Townhall Highlights:

• Direct engagement with the organisers
• Overview of the data collection process & background on BMS and the “Brick” dataset
• Problem formulation for the challenge & methods used in the benchmarks
• Interactions with fellow participants
• Live Q&A session

Can’t attend? No problem! Leave your questions in the comments, and they will be addressed during the session.

Mark your calendars, prepare questions, and join the live Townhall.

Looking forward to seeing you there!

Team AIcrowd

View post 0

💬 Feedback & Suggestions

Hi Patrick,

There is no significant difference between the two rounds. We aim to incorporate any feedback and suggestions shared by participants during Round 1 into Round 2, but there are no changes to the dataset or structure. The final winners will be determined based on the Round 2 leaderboard.

I hope this clarifies. Thank you!

View post 1

💬 Feedback & Suggestions

Hi Maghnie,

You can create a new post by going to the Discussion tab and clicking the “New Topic” button. Attached is an image for your reference:

Screenshot 2024-12-16 at 18.26.401920×1143 99 KB

1. The challenge and rules mention that participants can make ten submissions per day. There are no other tracks.

Participants can upload up to ten submissions per day in CSV format. Each submission must adhere strictly to the prescribed format to ensure accurate leaderboard evaluations, reflecting the test set’s real-time performance.

2. Up to 5 failed submissions are allowed every day.

I hope this helps!

View post 1

📹 Welcome to Brick by Brick Challenge 2024

View post 0

👥 Looking for teammates?

Competing is more fun with a team!

Introduce yourself here, and find others who are looking to team up!

Format:

• A short introduction about you and your background.
• What brings you to this challenge?
• Some ideas you wish to explore as a part of this challenge?

All The Best!

View post 0

💬 Feedback & Suggestions

We are constantly trying to improve this challenge for you and would appreciate any feedback you might have!

Please reply to this thread with your suggestions and feedback on making the challenge better for you!

• What have been your major pain points so far?
• What would you like to see improved?

All The Best!

View post 0

📹 Welcome to Brick by Brick Challenge 2024

View post 0

⏰ Challenge Breakdown and Q&A: Live with the Organisers | 23rd January, 10 AM CET

Hello everyone,

As Round 2 continues, we’re excited to invite you to the first Sounding Video Generation Challenge Townhall! This session offers an opportunity to engage with the organisers, gain valuable insights into the challenge, and get your questions answered. Prepare to boost your Round 2 submissions and refine your strategies!

Date: 23rd January 2025 (Thursday)
Time: 6:00 PM JST / 10:00 AM CET
Zoom Link: Join the Townhall

Can’t make it? Don’t worry! A recording will be available after the event. You can also drop your questions in the comments for the organisers, and they’ll be addressed during the town hall.

What to Expect:

• Overview of the challenge and problem statement
• Explanation of the Temporal Alignment track
• Explanation of the Spatial Alignment track
• Live Q&A session

Panellists
This townhall features Sony AI’s leading experts:

• Takashi Shibuya, Senior Manager at Sony AI: Generative AI for content creation, including audio and 3D-aware image generation.
• Masato Ishii, Senior Research Scientist at Sony AI: Generative modeling for multimodal data.
• Christian Simon, Research Scientist at Sony Group Corporation: Audio-visual generative model development.
• Kazuki Shimada, AI Engineer at Sony AI: Audio understanding and generation.

Mark your calendars, prepare questions, and join us live for this event.

Looking forward to seeing you there!
Team AIcrowd

View post 0

💬 Feedback & Suggestions

Hi, The issue is now fixed. You should be able to see your submission on the leaderboard now.

View post 0

Winner’s Solution Overview: KDD Cup 2024 - Team NVIDIA

Team NVIDIA, a group of data scientists and technologists, brought diverse skills to the KDD Cup 2024. Key members include Gilberto, a former #1 ranked Kaggle competitor with a background in Electrical Engineering; Chris, a Ph.D. holder in computational science and mathematics with experience across various professions; Benedikt Schifferer, a manager of Applied Research with expertise in recommender systems and LLMs; Ivan Sorokin, a Senior LLM Technologist; Ahmet Erdem, a Kaggle Grandmaster and open-source contributor; and Simon, a senior LLM technologist specializing in deep learning applications in computer vision and NLP.

Winning Strategy:

Team NVIDIA’s strategy for the KDD Cup 2024 involved the deployment of five fine-tuned Qwen2-72B LLM models, one for each of the competition’s tracks, leveraging cutting-edge techniques and substantial computational resources:

1. Data Transformation and Model Training:

• The team transformed data from six public datasets, including Amazon-M2 and MMLU, into 500k question-answer pairs across 40 tasks and 5 task types.

• They fine-tuned multiple Qwen2-72B models using QLoRA on NVIDIA’s powerful 8xA100 80GB GPUs, employing techniques like DeepSpeed and Axolotl for efficiency.

2. Model Optimization and Fine-Tuning:

• Fine-tuning involved adjusting LoRA parameters and experimenting with different weights for model adapters to optimize performance across various tasks.

• The models were trained with specific prompts tailored to simulate an online shopping assistant, enhancing task-specific performance.

3. Quantization and Inference Optimization:

• To meet the competition’s stringent hardware limitations and inference time constraints, Team NVIDIA employed 4-bit AWQ quantization and batch inference strategies using software vLLM, significantly reducing the model’s memory footprint.

• During inference, logits processors were added to the model’s predictions to ensure output accuracy, particularly in handling structured responses like numbers and commas.

4. Ensemble Techniques:

• The final submissions for each track involved sophisticated ensembles of base models and multiple LoRA adapters, fine-tuned to enhance the accuracy and robustness of the solutions.

Impact and Contributions:

Team NVIDIA’s comprehensive approach showcased their technical prowess and ability to innovate within constraints, leading to their first-place victory in all five competition tracks. Their work demonstrates the powerful capabilities of LLMs in handling diverse and complex real-world NLP tasks, particularly in a competitive setting with limited hardware resources.

View post 0