Log in

Classroom An Introduction

Overview

AIcrowd Classroom is a toolkit which can enables you to create challenge graders on AIcrowd which can be used as Assignments in just a few clicks!
We have a library of problems sets to choose from that you can fork and be ready to go. You also have grader templates ranging from simple regression to a custom grader that you can tailor according to your requirements. All that is needed from your side is just a few attributes and some data files to create an end to end challenge in minutes!



Creating a challenge




  • -> Click on 'Create' on AIcrowd Classroom home page
  • -> Pick a curated problem from Problem Sets or tailor a template as per your requirements
  • -> Choose your organization (create a new one if needed) and fill in the challenge and evaluator details carefully. Upload the datafiles as needed. Look out for tooltips to know more about each attribute. Once you are done click on 'CYOG' to proceed.
  • -> You will be redirected to My Challenges page where you can find your newly created challenge and its graders! :)
  • -> Activate the grader once the setup is complete.
  • -> Click on 'View Challenge' to see your challenge on AIcrowd
  • -> Click on 'Organizer Actions' -> 'Edit Challenge' to customize the challenge according to your needs. Upload relevant datasets in the 'Resources' section for your participants.
  • -> Once you are satisfied with your challenge, change its status to 'Running' to make it live!


Creating a Grader for existing Challenge


  • -> Choose a template for your grader on the Templates page
  • -> Choose your challenge from the dropdown and evaluator details carefully. Upload the datafiles as needed. Look out for tooltips to know more about each attribute. Once you are done click on 'CYOG' to proceed.
  • -> You will be redirected to My Graders page where you can find your newly created graders under the specified challenge.
  • -> Activate the grader once the setup is complete.


Activating a Grader


  • -> All your graders can be seen on the My Challenges page
  • -> Once the setup and the test for the grader is complete, you can activate it for the challenge by pressing 'Activate Graders'
  • -> In the activate form, choose the round and the grader version you want to activate.
  • * Once activated, all the other graders for the same challenge round will be deactivated. Atleast one grader should be active for your challenge to work.






Editing a Grader


  • -> All your graders can be seen on the My Challenges page
  • -> Start editing a grader by pressing 'edit'. You will be taken to the corresponding template form where you change the attributes and upload new files.
  • * Post editing, a new version of the grader will be created and the older one will be retianed.





FAQs


Which grader template suits the best for my challenge?

Graders for challenges involving linear/polynomial/multivariable regression can be made with 'Simple Regression' and binary/multiclass classification can be made with 'Simple Classification' templates. If you have a test-cases based setup or a competitive coding environment, Online Judge template is suited best for you. For a slightly more custom use case, you can create your grader using 'Custom Template'. Lastly for an extremely complex and flexible setup, feel free to make your own grader repository and link it using the 'Existing Repository' template


Where can the participants view the training and testing datasets or other resources?

Any sort of dataset or resources required by the participants can be uploaded and viewed on the 'Resources' section of the official AIcrowd Challenge page. The resource would only be available to the participants of the challenge once they accept the rules and conditions of the challenge.


Which of the evaluation metrics should I choose as the primary and secondary scoring metrics?

Evaluation metrics help us evaluate and score the predictions created by the participant's model. There are variety of sklearn metric available, You can also write your custom ones! :). The primary metric is used to determine the leaderboard and in the case of a tiebreak, the secondary scoring metric is used. These scoring metrics can also be used to give users insights in their model.


What should the ground truth file contain?

The ground truth file should be a single columned csv file containing the correct answers to the cases in the test set (in order). It acts as a source of truth for the evaluation of the participant's predictions. Sample files can be found as tooltips on the respective challenge pages.


Why did my grader setup fail?

This could most possibly be due to incorrect format of the file uploads or form details. Please recheck the format of the files. For debugging purposes, logs are available as well.


Setup of a grader is successful, but the test failed

The grader is tested once with the sample submission uploaded by you. If the test fails, there could be an error in the grader script or the sample submission. Both grader and test logs are available for debugging. If the problem persists, feel free to reach out! :)


Setup and Tests are successful, but the challenge is not deployed

For the grader to be deployed on the AIcrowd challenge page, it has to be 'Activated'. Please refer to the section on activating the grader for more details.



For any other queries, drop us a mail at hello@aicrowd.com