Composite Scoring, FAQs, and One-Day Registration Extension

Composite Scoring

Overview
Your overall score for the 2019 World Championship will be a composite that is made up of four components, each described below.

Weight (Points) Component Description
35 Plan Quality, component 1 (PQ1)

This component is based on your plan quality score as calculated with your RT structure set (RTSS), RT plan (RTP), and RT dose (RTD).

This component will be based on the plan score that you will see on the ProKnow DS scorecard during the event.

Your plan score will be normalized to the sum of all possible metric points (150), and then given a total weight of 35 (out of 100) for this component of the composite score.

35 Plan Quality, component 2 (PQ2)

This component is based on your plan quality score as calculated with the “gold” RTSS, your RTP, and your RTD.

You will not see this score on ProKnow DS during the event, but it will be added as a custom metric to analyze during the review phase.

This metric will inherently incorporate your contouring accuracy for critical organs-at-risk (OAR) that have metrics on the scorecard but are not provided in the partial RTSS that you will download when you start.

This plan score will be normalized to the sum of all possible metric points (150), and then given a total weight of 35 (out of 100) for this component of the composite score.

20 Efficiency (EFF)

This component is based on the elapsed planning time, starting from when you press “Start” on the dashboard (which will make your uniquely tagged patient dataset available for download) to when you upload your final DICOM object (e.g., final RT plan and RT dose).

You will not see this component score on ProKnow DS during the event, but you can easily estimate it based on the elapsed planning time that the dashboard shows you. The elapsed planning time will be added as a custom metric to analyze during the review phase.

The efficiency component will be calculated as a function of your planning time with total weight of 20 (out of 100), as described here:

  • If your time is 4.0 hours or less, you get 100% of the points for this component (i.e., 20 points).
  • If your time is 5.0 hours, you get 90% of the point (i.e., 18 points out of 20).
  • If your time is between 4.0 and 5.0 hours, your score will be linearly interpolated between 20 points (at 4 hours) and 18 points (at 5 hours).
  • If your time is above 5.0 hours, your score will be linearly interpolated from 18 points (at 5 hours) down to zero points (at 7 hours).
10 Contouring Accuracy (CA)

Two of the OARs that you will need to contour will be used to calculate the DICE coefficients between your structures and the gold structures. You will not know which two in advance, but the same two OARs will be used for all participants.

You will not see the resulting DICE coefficients during the event, but they will be added as custom metrics to analyze during the review phase.

The component for the composite score will be based on the average DICE coefficient for the two selected OARs, then multiplied by a weight of 10 (out of 100).

100 TOTAL

Example

Let’s say you participate with the following results:

  • The plan score you see on ProKnow DS is 115.5 (out of 150).
  • The plan score calculated with your plan and dose but with the gold RTSS is 112.1 (out of 150)
  • Your elapsed planning time is 6.0 hours.
  • Your DICE coefficient for selected OAR1 is 0.980 and for OAR2 is 0.625.

With those inputs, your composite score would be calculated the following way:

  • PQ1 = 35 x (115.5/150) = 26.950
  • PQ2 = 35 x (112.1/150) = 26.157
  • EFF = Linear interpolation of 6.0 hrs between 5 hrs (18 points) and 7 hours (0 points) = 9.000
  • CA = 10 x [(0.980 + 0.625)/2] = 8.025
  • TOTAL SCORE = PQ1 + PQ2 + EFF + CA = 70.132 (out of 100)

FAQs

I did not register by the deadline of August 31. Can I register late?

Late registration is now closed. There will be no more opportunities to register for this year’s competition. Good luck to everyone that made it in … for those that couldn’t, we hope to see you on the podium next year!

Which browsers do you support?

ProKnow DS officially supports Chrome and Firefox. Older browsers such as Internet Explorer 11 are not supported.

Should I add the treatment table in the structure set I create and include it in the dose calculation?

Yes! You must add the treatment table/couch that is appropriate for the delivery machine that your plan will use! The partial RTSS we provide will have the patient external contour, but obviously not the treatment table, as those vary. If there is an extension to the couch that is radio-transparent, then you do not need to add a couch. The plan and dose you submit must include the effects of the treatment table. As in any plan study, you must ensure that the plan you design is clinically viable, i.e., a plan you would use for a real patient in your clinic.

Note that the gold RTSS will not include your treatment table, but that is fine, as your RT Dose does include the effects, and the gold RTSS is only used to calculate new DVHs from your dose and the gold structures.

Is there any type of “handicap” or scoring adjustment if I am competing as an individual instead of a team of two?

The allowance for teams of two people is meant to accommodate those who are treatment planning specialists but perhaps not trained in anatomical contouring (e.g., medical physicists). We wanted to give such people a way to participate and compete at a high level rather than exclude them. Thus, we allow for teams of two, e.g., a medical physicist might partner with a physician or dosimetrist with contouring experience.

If you are competing as an individual because you have both contouring and planning skills, then you are not really at a disadvantage since the process is mostly serial (i.e., download images and partial RTSS, contouring missing OARs, then start optimizing). Feel free to partner with somebody if you would like, even if that person (e.g., a physician) will only be used to review your contours.

Can team members complete on multiple teams?

No, team members are not allowed to compete on multiple teams.

Which contours am I supposed to add on my own?

When you see the scorecard, you will see the list of required structures. Many of these will be provided already (make sure that do not edit, rename or delete any of those). The structures you need to contour are those that are required for the scorecard but that are not in your partial RTSS that we will provide at the start. When you create them, be sure to use the correct name (i.e., the exact, case-sensitive name as shown in the scorecard).

Are there any restrictions on the plan I create (e.g., the number of fields, the use of non-coplanar beams, the calculation grid size, the position of the isocenter)?

Yes. Plan practicality details will be announced prior to the event. Please stay tuned for more updates.

Must we create plans that avoid passing through the patient’s shoulder?

There will be scorecard metrics in place to ensure that your plan does not have beams that pass through the shoulder.

What dose calculation algorithm should I use?

You must use the valid dose algorithm and beam model that you use clinically! Using special models is considered cheating and may be flagged during dose audits where 3rd party dose calculators might be used.

You must use heterogeneity corrections, and you must use a modern, clinically-acceptable algorithm (i.e., pencil beam is not allowed).

Can we use dose normalization? Can we normalize dose at something other than 100%?

RT Dose files must be in absolute units of Gy. You should do your dose normalization prior to exporting as an RT Dose file.

If I use a 3rd party contouring software system in conjunction with my TPS, is that okay? Will that slow me down?

Use whatever contouring system you are comfortable with, even if that means contouring in that system first before transferring to your TPS for planning. If that process slows you down significantly for whatever reason, then you should weigh that into your strategy and consider the efficiency component of the score, as described earlier.

What treatment planning systems (TPS) can be used?

Any clinically-commissioned and commercially-available TPS can be used. You cannot use non-clinical or non-commercial TPS. If you want to use a version of an existing TPS that is being validated but is not yet released, we may make an exception if you make a direct request, but you will not be eligible for the cash prizes.

ProKnow allows me to upload data in two ways. Which do I use?

You must activate the patient that was automatically created for you when you began, then use the Actions menu (upper right of patient module) and either “Upload Files” or “Upload Directory” option.

Do not use the Uploads button on the far left, as that might create a new patient if you have somehow altered your patient identifying information (e.g., during import/export from your TPS).

Note: You will only need to upload three types of files: RT structure set, RT plan, and RT dose. You do not need to upload the CT images as those will already be there upon initialization.

Can multiple Eclipse planners using the same Eclipse installation plan at the same time?

Yes! We have generated a unique dataset for each user, which should allow each planner to import their assigned datasets into Eclipse without overlap.

Can I register twice and submit two different plans?

No! This method is sometimes allowed for the conventional plan studies if a user wants to try two modalities, but the World Championship events are different. For these events, we require that each competitor only submit one plan and dose.

If you registered twice (i.e., with two different emails), then please pick one account to use for the World Championship and ignore the other account. If you compete with two accounts, you will be disqualified from all overall awards.

If I upload a new plan iteration but ultimately delete it and keep a prior one, will that affect my planning time?

If you upload another plan attempt but decide to ultimately delete it, that time will not be counted against you. Your planning time as recorded on the dashboard will increase when you upload the new plan iteration, but then it will decrease again (revert back to the last upload) if you delete the new plan.

We suggest that you always keep your “current best” plan while you work on the next plan iteration, and only delete your “current best” if you decide that your new plan iteration is the new “current best.” If you delete patient objects and then re-upload them, you will have a new completion time. The practice phase is a good time to experiment with uploading multiple plans and doses and seeing how they affect your submission time.

Do people try to cheat?

They better not! We do have audits for cheating, and if caught those people will be disqualified and banned from this and all future ProKnow plan study competitions/challenges.

Examples of cheating include, but are not limited to, the following: getting help from anyone other than your teammate, using an invalid or non-clinical dose algorithm or beam model, not using heterogeneity corrections or using invalid CT-to-electron-density conversions, looking at someone else’s dataset before you have started your time, using a non-commercial TPS or version that is not yet released, creating a plan that is unrealistic and/or overly complex, etc.

We do these events for for science, learning, and fun. Cheating means sullying the data and certainly removing the fun. Please, do not attempt to bend the rules just to maximize your score. It’s not worth it, ever.