Week 2 leaderboard is out! Comments and upcoming feedback

Thanks for the clarifications, much appreciated!

View post 1

Week 2 leaderboard is out! Comments and upcoming feedback

Hi @alfarzan,
Would it be possible to get some further details about the reason that resulted in a submission to be excluded in the from the Average Profit Week 2 leaderboard?

I noticed that PASSED_PARTICIPATION_THRESHOLD_ON indicates 0.0 (https://www.aicrowd.com/challenges/insurance-pricing-game/submissions/111795), though I’m scratching my head as to why it wouldn’t have made the cut given the underlying RMSE from the claims was in top 10 the the pricing was a slightly inflated version. Considering that a flat average baseline seems to make the cut, I’m a bit puzzled.

Thanks!

View post 0

Tips on creating a successful submission

Thanks @jyotish. Following the above steps, the zip file submission now works.
Something I noticed from downloading my Colab submission was that there seems to be a reversal between claims and prices at the end of the predict.R file:

  claims = predict_premium(trained_model, Xraw)

Not sure if this would have been a problem I introduced or if it related to the template linked to the submit.sh utility.

On another note, leaderboard seems to be based on my worst submission (id 110383) instead of the best or most recent ( 110384, 110389)

View post 1

Tips on creating a successful submission

For the zip submission, I went for the shotgun approach and likely loaded the libraries at too many places, put at least the following does work. I added require() / library() right after the beginning of each of the functions preprocess_X_data, predict_expected_claim, predict_premium and fit_model, for example:

preprocess_X_data <- function (x_raw){
  require("data.table")
  require("xgboost")
  ...

Note that for the zip submission, no need to include the fit_model.R, it really seems like all that is necessary are the files invoked by the predict.R (so the model is directly loaded from trained_model.RData.

View post 0

Tips on creating a successful submission

Good hint!
I finally managed to get a R zip submission to work.

I’m unsure which moving part was critical, having all of the following does seem to work:

config.json

{"language": "r"}

Install.R

install_packages <- function() {
  install.packages("data.table")
  install.packages("xgboost")
}

install_packages()

And then have the the various functions split into seperate files as listed in the source call in the predict.R file:

source("fit_model.R")  # Load your code.
source("load_model.R")
source("predict_expected_claim.R")
source("predict_premium.R")
source("preprocess_X_data.R")

# This script expects sys.args arguments for (1) the dataset and (2) the output file.
output_dir = Sys.getenv('OUTPUTS_DIR', '.')
input_dataset = Sys.getenv('DATASET_PATH', 'training_data.csv')  # The default value.
output_claims_file = paste(output_dir, 'claims.csv', sep = '/')  # The file where the expected claims should be saved.
output_prices_file = paste(output_dir, 'prices.csv', sep = '/')  # The file where the prices should be saved.
model_output_path = 'trained_model.RData'

args = commandArgs(trailingOnly=TRUE)

if(length(args) >= 1){
  input_dataset = args[1]
}
if(length(args) >= 2){
  output_claims_file = args[2]
}
if(length(args) >= 3){
  output_prices_file = args[3]
}

# Load the dataset.
# Remove the claim_amount column if it is in the dataset.
Xraw = read.csv(input_dataset)

if('claim_amount' %in% colnames(input_dataset)){
  Xraw = within(Xraw, rm('claim_amount'))
}


# Load the saved model, and run it.
trained_model = load_model(model_output_path)

if(Sys.getenv('WEEKLY_EVALUATION', 'false') == 'true') {
  claims = predict_premium(trained_model, Xraw)
  write.table(x = claims, file = output_claims_file, row.names = FALSE, col.names=FALSE, sep = ",")
} else {
  prices = predict_expected_claim(trained_model, Xraw)
  write.table(x = prices, file = output_prices_file, row.names = FALSE, col.names=FALSE, sep = ",")
}

View post 4

Tips on creating a successful submission

@jyotish I’ve been unsuccessful giving another shot at the R zip submission following the update at:

It resulted in the same error message, although running the test.bat works fine.
The only thing I could note was that I had to add the line :
set WEEKLY_EVALUATION=false prior to the first call to predict in order to generate the claims.csv file.

Otherwise, I can’t see what could be wrong. Packages are installed in install.R and loaded where indicated in model.R (even tried to load them within the fit functions which was needed in the Colab submissions - still no success). Have you performed a test on the template to validate it works when using packages? It would be useful to get a working example of scripts that use packages.

View post 0

Tips on creating a successful submission

Regarding the zip file submission with R, the following error messages were returned in the submission page:

"DockerBuildError: The command '/bin/sh -c r -e \"`cat /tmp/install.R`\"' returned a non-zero code: 1"

and

" [bt] (6) /usr"

Although the install.R file in the .zip file only contains:

install.packages("data.table")
install.packages("xgboost")

The zip files had those files:

And running the following works locally:

Rscript predict.R training_data.csv output_predicted_claims.csv output_prices.csv

Any insight on some further considerations for zip file submission?

View post 0

🚀 R Starter Kit

Would it be possible to provide some further details on how to proceed to a proper submission?
Despite having passed the tests and and getting a messing acknowledging a proper submission, the leaderboard evaluation returns errors with little indications on where the error is coming from.

For example, I first wanted to use data.table library, despite having the following to work properly in the Colab notebook, it resulted in the error message "Error in data.table(x_raw) : could not find function \"data.table\"" on the submission evaluation section.

install_packages <- function() {
  install.packages("data.table")
    require("data.table")
}

Looking at the submissions, it seems like others also faced issues package installation/usage, so I think some further examples would be welcome.

Also, when opting for the zip submission instead of the Colab, it resulted in "DockerPushError: An image does not exist locally with the tag: aicrowd/imperial-pricing-game". The submission experience has been a little rough so far on the R side!

View post 2