Advertisement

Development of a multicentre automated model to reduce planning variability in radiotherapy of prostate cancer

Open AccessPublished:August 08, 2019DOI:https://doi.org/10.1016/j.phro.2019.07.005

      Abstract

      Background and purpose

      Inter-institutional studies highlighted correlation between consistent radiotherapy quality and improved overall patient survival. In treatment planning automation has the potential to address differences due to user-experience and training, promoting standardisation. The aim of this study was to evaluate implementation and clinical effect of a multicentre collaboratively-developed automated planning model for Intensity-Modulated Radiation Therapy/Volumetric-Modulated Arc Therapy of prostate. The model was built using a variety of public institutions’ clinical plans, incorporating different contouring and dose protocols, aiming at minimising their variation.

      Methods and materials

      A model using 110 clinically approved and treated prostate plans provided by different radiotherapy centres was built with RapidPlan (RP), for use on intact and post-prostatectomy prostate cases. The model was validated, distributed and introduced into clinical practice in all institutions. To investigate its impact a total of 126 patients, originally manually inverse planned (OP), were replanned using RP without additional planner manual intervention. Target and organ-at-risk (OAR) metrics were statistically compared between original and automated plans.

      Results

      For all centres combined and individually, RP provided plans comparable or superior to OP for all dose metrics. Statistically significant reductions with RP were found in bladder (V40Gy) and rectal (V50Gy) low doses (within 2.3% and 3.4% for combined and 4% and 10% individually). No clinically significant changes were seen for the PTV, independently of seminal vesicle inclusion.

      Conclusion

      This project showed it is feasible to develop, share and implement RP models created with plans from different institutions treated with a variety of techniques and dose protocols, with the potential of improving treatment planning results and/or efficiency despite the original variability.

      1. Introduction

      In recent years automated approaches have become popular in radiotherapy with the aim of reducing variability in tasks which are still heavily dependent on user-experience and training. With the widespread use of inverse planning techniques, these tools have become employed in Intensity-Modulated Radiation Therapy (IMRT) and Volumetric-Modulated Arc Therapy (VMAT) treatment planning, due to the development of dedicated algorithms [
      • Wu B.
      • Ricchetti F.
      • Sanguineti G.
      • Kazhdan M.
      • Simari P.
      • Chuang M.
      • et al.
      Patient geometry-driven information retrieval for IMRT treatment plan quality control.
      ,
      • Wu B.
      • Ricchetti F.
      • Sanguineti G.
      • Kazhdan M.
      • Simari P.
      • Jacques R.
      • et al.
      Data-driven approach to generating achievable dose-volume histogram objectives in intensity-modulated radiotherapy planning.
      ,
      • Chanyavanich V.
      • Das S.
      • Lee W.
      • Lo W.
      Knowledge based IMRT treatment planning for prostate cancer.
      ,
      • Moore K.L.
      • Brame S.R.
      • Low D.A.
      • Mutic S.
      Experience based quality control of clinical intensity modulated radiotherapy planning.
      ,
      • Appenzoller L.
      • Michalski J.
      • Thorstad W.
      • Mutic S.
      • Moore K.
      Predicting dose volume histograms for organs-at-risk in IMRT planning.
      ], some of which have been included in commercial treatment planning systems (TPS) [
      • Fogliata A.
      • Wang P.
      • Belosi F.
      • Clivio A.
      • Nicolini G.
      • Vanetti E.
      • et al.
      Assessment of a model based optimization engine for volumetric modulated arc therapy for patients with advanced hepatocellular cancer.
      ,
      • Fogliata A.
      • Belosi F.
      • Clivio A.
      • Navarria P.
      • Nicolini G.
      • Scorsetti M.
      • et al.
      On the pre-clinical validation of a commercial model-based optimization engine: application to volumetric modulated arc therapy for patients with lung or prostate cancer.
      ,
      • Tol J.P.
      • Delaney A.R.
      • Dahele M.
      • Slotman B.J.
      • Verbakel W.F.A.R.
      Evaluation of a knowledge based planning solution for head and neck cancer.
      ,
      • Fogliata A.
      • Nicolini G.
      • Clivio A.
      • Vanetti E.
      • Laksar S.
      • Tozzi A.
      • et al.
      A broad scope knowledge based model for optimisation of VMAT in esophageal cancer: validation and assessment of plan quality among different treatment centres.
      ,
      • Delaney A.R.
      • Dahele M.
      • Tol J.P.
      • Slotman B.J.
      • Verbakel W.F.A.R.
      Knowledge-based planning for stereotactic radiotherapy of peripheral early stage lung cancer.
      ,
      • Hussein M.
      • South C.P.
      • Barry M.A.
      • Adams E.J.
      • Jordan T.J.
      • Stewart A.J.
      • et al.
      Clinical validation and benchmarking of knowledge-based IMRT and VMAT treatment planning in pelvic anatomy.
      ,
      • Powis R.
      • Bird A.
      • Brennan M.
      • Hinks S.
      • Newman H.
      • Reed K.
      • et al.
      Clinical implementation of a knowledge based planning tool for prostate VMAT.
      ]. These algorithms are based on different methods [
      • Wu B.
      • Ricchetti F.
      • Sanguineti G.
      • Kazhdan M.
      • Simari P.
      • Chuang M.
      • et al.
      Patient geometry-driven information retrieval for IMRT treatment plan quality control.
      ,
      • Wu B.
      • Ricchetti F.
      • Sanguineti G.
      • Kazhdan M.
      • Simari P.
      • Jacques R.
      • et al.
      Data-driven approach to generating achievable dose-volume histogram objectives in intensity-modulated radiotherapy planning.
      ,
      • Chanyavanich V.
      • Das S.
      • Lee W.
      • Lo W.
      Knowledge based IMRT treatment planning for prostate cancer.
      ,
      • Moore K.L.
      • Brame S.R.
      • Low D.A.
      • Mutic S.
      Experience based quality control of clinical intensity modulated radiotherapy planning.
      ], however they all aim to learn from previous site-specific treatment plans in order to predict the optimal dose-volume constraints for organs-at-risk (OAR) for a given prescribed dose in future patients treated with the same anatomical target. There are a range of automated solutions available for treatment planning, already shown by several groups to improve efficiency and provide plans equivalent to those obtained by experienced planners [
      • Fogliata A.
      • Wang P.
      • Belosi F.
      • Clivio A.
      • Nicolini G.
      • Vanetti E.
      • et al.
      Assessment of a model based optimization engine for volumetric modulated arc therapy for patients with advanced hepatocellular cancer.
      ,
      • Fogliata A.
      • Belosi F.
      • Clivio A.
      • Navarria P.
      • Nicolini G.
      • Scorsetti M.
      • et al.
      On the pre-clinical validation of a commercial model-based optimization engine: application to volumetric modulated arc therapy for patients with lung or prostate cancer.
      ,
      • Tol J.P.
      • Delaney A.R.
      • Dahele M.
      • Slotman B.J.
      • Verbakel W.F.A.R.
      Evaluation of a knowledge based planning solution for head and neck cancer.
      ,
      • Fogliata A.
      • Nicolini G.
      • Clivio A.
      • Vanetti E.
      • Laksar S.
      • Tozzi A.
      • et al.
      A broad scope knowledge based model for optimisation of VMAT in esophageal cancer: validation and assessment of plan quality among different treatment centres.
      ,
      • Hussein M.
      • South C.P.
      • Barry M.A.
      • Adams E.J.
      • Jordan T.J.
      • Stewart A.J.
      • et al.
      Clinical validation and benchmarking of knowledge-based IMRT and VMAT treatment planning in pelvic anatomy.
      ,
      • Kubo K.
      • Monzen H.
      • Ishii K.
      • Tamura M.
      • Kawamorita R.
      • Sumida I.
      • et al.
      Dosimetric comparison of RapidPlan and manually optimized plans in volumetric arc therapy for prostate cancer.
      ].
      Institutions in Australia have focused on minimising variations in radiotherapy quality, in line with the repeated finding that consistent radiotherapy is significantly and substantially tied to improved overall patient survival [
      • Moore K.L.
      • Brame R.S.
      • Low D.A.
      • Mutic S.
      Experience-based quality control of clinical intensity-modulated radiotherapy planning.
      ,
      • Nelms B.E.
      • Robinson G.
      • Markham J.
      • Velasco K.
      • Boyd S.
      • Narayan S.
      • et al.
      Variation in external beam treatment plan quality: an inter-institutional study of planners and planning systems.
      ,
      • Peters L.J.
      • O’Sullivan B.
      • Giralt J.
      • Fitzgerald T.J.
      • Trotti A.
      • Bernier J.
      • et al.
      Critical impact of radiotherapy protocol compliance and quality in the treatment of advanced head and neck cancer: results from TROG 0202.
      ,
      • Fitzgerald T.J.
      What we have learned: the impact of quality from a clinical trials perspective.
      ,
      • Fairchild A.
      • Straube W.
      • Laurie F.
      • Followill D.
      Does quality of radiation therapy predict outcomes of multicenter cooperative group trials? A literature review.
      ]. As a critical step towards this objective, an effort was made to reduce variability in treatment planning and improve overall plan quality by introducing automation. These institutions provided clinically approved treatment plans for a range of tumour sites (prostate, head and neck and anal canal [
      • Duchesne G.
      • Grand M.
      • Kron T.
      • Haworth A.
      • Corry J.
      • Jackson M.
      • et al.
      The TransTasman Radiation Oncology Group (TROG) ‘ANROTAT’* Framework – development of a model for the early evaluation of new technologies and techniques in Radiation Oncology (*Assessment of new radiation oncology technology and treatments).
      ]), in order to create heterogeneous training datasets, and models, with potential benefits, as previously highlighted [
      • Cagne E.
      • Botti A.
      • Micera R.
      • Galeandro M.
      • Sghedoni R.
      • Orlandi M.
      • et al.
      Knowledge-based treatment planning: an inter-technique and inter-system feasibility study for prostate cancer.
      ]. A similar initiative was previously promoted by Schubert et al. and Roach et al. [
      • Schubert C.
      • Waletzko O.
      • Weiss C.
      • Voelzke D.
      • Toperim S.
      • Roeser A.
      • et al.
      Intercenter validation of a knowledge based model for automated planning of volumetric modulated arc therapy for prostate cancer. The experience of the German RapidPlan Consortium.
      ,
      • Roach D.
      • Wortel G.
      • Ochoa C.
      • Jensen H.R.
      • Damen E.
      • Vial P.
      • et al.
      Adapting automated treatment planning configurations across international centres for prostate radiotherapy.
      ], however their models were created using data from a single institution, and then distributed to other centres. Ueda et al. [
      • Ueda Y.
      • Fukunaga T.
      • Kamima T.
      • Adachi Y.
      • Nakamatsu K.
      • Monzen H.
      Evaluation of multiple institutions’ models for knowledge-based planning of volumetric arc therapy (VMAT) for prostate cancer.
      ] instead looked at the performance of a series of automated models created in different institutions using their local treatment plans, on the same two datasets used by each centre independently.
      This work presents the development and implementation of a multicentre automated prostate cancer model across the range of participating centres with different treatment approaches. Several series of results for the use of automated models for prostate planning are available [
      • Fogliata A.
      • Belosi F.
      • Clivio A.
      • Navarria P.
      • Nicolini G.
      • Scorsetti M.
      • et al.
      On the pre-clinical validation of a commercial model-based optimization engine: application to volumetric modulated arc therapy for patients with lung or prostate cancer.
      ,
      • Hussein M.
      • South C.P.
      • Barry M.A.
      • Adams E.J.
      • Jordan T.J.
      • Stewart A.J.
      • et al.
      Clinical validation and benchmarking of knowledge-based IMRT and VMAT treatment planning in pelvic anatomy.
      ,
      • Kubo K.
      • Monzen H.
      • Ishii K.
      • Tamura M.
      • Kawamorita R.
      • Sumida I.
      • et al.
      Dosimetric comparison of RapidPlan and manually optimized plans in volumetric arc therapy for prostate cancer.
      ,
      • Cagne E.
      • Botti A.
      • Micera R.
      • Galeandro M.
      • Sghedoni R.
      • Orlandi M.
      • et al.
      Knowledge-based treatment planning: an inter-technique and inter-system feasibility study for prostate cancer.
      ,
      • Schubert C.
      • Waletzko O.
      • Weiss C.
      • Voelzke D.
      • Toperim S.
      • Roeser A.
      • et al.
      Intercenter validation of a knowledge based model for automated planning of volumetric modulated arc therapy for prostate cancer. The experience of the German RapidPlan Consortium.
      ,
      • Ueda Y.
      • Fukunaga T.
      • Kamima T.
      • Adachi Y.
      • Nakamatsu K.
      • Monzen H.
      Evaluation of multiple institutions’ models for knowledge-based planning of volumetric arc therapy (VMAT) for prostate cancer.
      ,
      • Scaggion A.
      • Fusella M.
      • Roggio A.
      • Bacco S.
      • Pivato N.
      • Rossato M.A.
      • et al.
      Reducing inter- and intra-planner variability in radiotherapy plan output with a commercial knowledge-based planning solution.
      ], however, this project is unique as the model and its validation combines contouring, planning experience, protocols and fractionation regimens from different institutions. To evaluate its performance, the results of the analysis conducted on the dose distributions collected in the first months after its clinical implementation are presented, looking at inter-centre and intra-centre variability, along with the overall effect of introducing the same multicentre model across all participating centres.

      2. Methods and materials

      Eight radiotherapy centres, treating over 10,000 patients per year, were included in the project. These were all public radiotherapy centres in the state of Victoria, the second most populous state of Australia, that use the Eclipse treatment planning system (Varian Medical Systems, Palo Alto). To facilitate this process, the Victorian Public Sector RapidPlan Group (VPSRG) was formed. This group comprised oncologist, therapist and physicist representatives from participating institutions.
      Two out of the eight centres were involved in development, training and validation of the prostate model, specifically:
      • both centres developed an individual model using the same patient data, which was then cross-validated by the other;
      • the most reliable and consistent of the models to achieve the lowest OAR doses without compromising target coverage (Table 1) with a single automated optimisation was selected as the final clinical model;
        Table 1Optimisation Objectives used in the RP2 model. The far right column summarises the agreed OAR DVH constraints for the VPSRG prostate model.
        Model structureOptimisation

        Objective
        Volume (%)Dose (% or Gy)PriorityVPSRG

        DVH constraints
        PTV-highUpper

        Lower
        0%

        100%
        101.5%

        100%
        120

        140
        Clinically acceptable parameters for each centre
        CTVUpper

        Lower
        0%

        100%
        104%

        103%
        120

        120
        Clinically acceptable parameters for each centre
        BladderUpper

        Line (DVH as lines obtained in the objective generation phase
        • Fogliata A.
        • Belosi F.
        • Clivio A.
        • Navarria P.
        • Nicolini G.
        • Scorsetti M.
        • et al.
        On the pre-clinical validation of a commercial model-based optimization engine: application to volumetric modulated arc therapy for patients with lung or prostate cancer.
        )
        0%

        Model generated
        100%

        Model generated
        Model generated

        Model generated
        V50Gy ≤ 50%

        V60Gy ≤ 40%

        V65Gy ≤ 35%

        V70Gy ≤ 25%
        Fem Heads L-RLineModel generatedModel generatedModel generatedV35Gy ≤ 100%

        V45Gy ≤ 60%

        V50Gy ≤ 50%

        V60Gy ≤ 30%
        Penile BulbLineModel generatedModel generated25Mean ≤ 52.5 Gy
        RectumUpper

        Line
        0%

        Model generated
        100%

        Model generated
        85

        85
        V40Gy ≤ 65%

        V50Gy ≤ 50%

        V60Gy ≤ 30%

        V70Gy ≤ 20%

        V75Gy ≤ 5%
      • once validated and endorsed by the remaining six participating centres, that model was approved by the VPSRG, published in Eclipse, and introduced into clinical practice at all centres.
      In addition to developing models, the VPSRG met monthly to oversee the implementation process. The group was in charge of standardising dose-volume-histograms constraints for all centres (VPSRG-DVH constraints Table 1), establishing structure naming conventions, and discussing model validation and potential improvements.

      2.1 Model development

      The RapidPlan automated optimisation engine implemented in Eclipse was used for the project. A description of the engine is presented elsewhere [
      • Fogliata A.
      • Belosi F.
      • Clivio A.
      • Navarria P.
      • Nicolini G.
      • Scorsetti M.
      • et al.
      On the pre-clinical validation of a commercial model-based optimization engine: application to volumetric modulated arc therapy for patients with lung or prostate cancer.
      ,
      • Hussein M.
      • South C.P.
      • Barry M.A.
      • Adams E.J.
      • Jordan T.J.
      • Stewart A.J.
      • et al.
      Clinical validation and benchmarking of knowledge-based IMRT and VMAT treatment planning in pelvic anatomy.
      ]. The system works by using a statistical model which mines a library of clinically approved plans using principal-component-analysis to establish the optimal correlation between the geometrical and dosimetric characteristics of the plans [
      • Hussein M.
      • South C.P.
      • Barry M.A.
      • Adams E.J.
      • Jordan T.J.
      • Stewart A.J.
      • et al.
      Clinical validation and benchmarking of knowledge-based IMRT and VMAT treatment planning in pelvic anatomy.
      ]. To build the VPSRG prostate model, a total of 135 approved and treated training prostate plans were submitted from five of the eight participating centres. Each treatment plan was originally created according to the local centre’s protocol, with either IMRT or VMAT using 6MV X-rays, and fulfilling the VPSRG DVH constraints (Table 1). These plans were created for a variety of fractionation regimes and were a combination of intact and post-prostatectomy treatments (PPP) (Table A1-Appendix). These diverse inclusion criteria were used with the goal of developing a single versatile model applicable to a wide range of prostate patients.
      All training plans were anonymised at the centre of origin then re-imported into the model developers’ TPS. The Eclipse Model Configuration module was then used to build and train the model. The statistical parameters estimated in Model Configuration, and the Model Analytics tool, in addition to treatment plan review were used to exclude plans indicated as outliers for one or more structures, as previously described [
      • Delaney A.R.
      • Dahele M.
      • Tol J.P.
      • Slotman B.J.
      • Verbakel W.F.A.R.
      Knowledge-based planning for stereotactic radiotherapy of peripheral early stage lung cancer.
      ,

      Eclipse Photon and Electron reference guide v.13.7, Varian Medical System, June 2015, https://www.myvarian.com.

      ,
      • Delaney A.R.
      • Tol J.P.
      • Dahele M.
      • Cuijpers J.
      • Slotman B.J.
      • Verbakel W.F.A.R.
      Effect of dosimetric outlier on the performance of a commercial knowledge-based planning solution.
      ,
      • Wu H.
      • Jiang F.
      • Yue H.
      • Zhang H.
      • Wang K.
      • Zhang Y.
      Applying a RapidPlan model trained on a technique and orientation to another: a feasibility and dosimetric evaluation.
      ]. A total of 110 training plans were retained for the trained model: 77 IMRT and 33 VMAT. Due to the differences in planning techniques between centres and plan availability, both treatment techniques were used in the model to represent each centre’s practice.
      To understand the capability of automated planning, three separate models were initially created by two developer centres: one for the intact prostate, one for post-prostatectomy cases, and a combined one with both sites. Local validation was performed on all three models in the six remaining centres not involved in model building using previously approved treatment plans (>15 per site), and comparing OAR and Planning-Target-Volume (PTV) dose distributions. This process resulted in the selection of a single combined model, which was distributed to participating centres for clinical use. After planning 30 new patients, the model was modified to increase dose objectives on the rectum, priority on PTV and smoothing parameters. The revised VPSRG prostate model (called RP2) was subsequently disseminated as the standard prostate model for all clinical cases treated at the VPSRG centres. Optimisation objectives used in the model and VPSRG standardised constraints are shown in Table 1.

      2.2 Plan evaluation

      To investigate the effect of introducing RP2 into participating centres, analysis was performed assessing variations in the entire cohort of patients, and between centres. The methodology was as follows: all centres (designated A to H) submitted validation treatment plan data, not included in the training set, representing a sample of their prostate patients (Table 2). These original plans (OP) were all approved by radiation oncologists and originally created using “manual” inverse optimisation following each centre’s protocol with 6MV X-ray beams. The validation plans were then re-optimised using the RP2 model. No planner intervention was used on the RP2 plan and no additional dose control structures were added for optimisation. Identical field setup as in the OP was used in the RP2 plan. Overall plan metrics from 126 patients were accrued for analysis. For all centres the calculation was performed with the Analytical Anisotropic Algorithm, except for one site which used the AcurosXB algorithm (dose-to-water).
      Table 2Characteristics of the validation patient plans analysed. The number of intact or post-prostatectomy (PPP) patients and those with PTV seminal vesicles involvement (SV) are shown.
      CentreNo of patientsPrescription Dose

      FractionsTechniqueintactPPPSV
      A2046, 64, 66 and 74 Gy23, 32, 33 and 37IMRT13718
      B2070 and 78 Gy35 and 39IMRT10100
      C3067.5, 70, 75.6, 76 and 77 Gy27, 28, 36, 38 and 35VMAT30028
      D1150, 68, 74 and 78 Gy25, 34, 37 and 39VMAT7411
      E666, 70 and 78 Gy33, 35 and 39IMRT/VMAT156
      F1370 and 78 Gy35 and 39IMRT1030
      G1670 and 78 Gy35 and 39IMRT1061
      H1060, 70 and 78 Gy30, 35 and 39VMAT828
      Total126893772
      All centres OAR metrics for bladder and rectum were studied (Bladder: V70Gy, V65Gy, V60Gy and V50Gy, Rectum: V70Gy, V60Gy, V50Gy, V40Gy). For the PTV D98%, D2%, V95%, conformity (CI [
      • Lomax N.J.
      • Scheib S.G.
      Quantifying the degree of conformity in radiosurgery treatment planning.
      ]) and heterogeneity (HI) indices (as (D2%–D98%)/D50%) were evaluated. Information on monitor units (MU) and PTV seminal vesicle (SV) involvement was also obtained. Seminal vesicles inclusion was based on each centre’s institutional prostate protocol. Left and right femoral heads, and penile bulb parameters (Table 1), for the centres that routinely include it in the contouring (A, D) were also compared.
      Firstly, results were studied for the combined cohort of patients from all centres. Due to the variety of prescription doses used, metrics were grouped into three cohorts according to prescription levels with a similar biological-effective-dose: 66–70 Gy, 74–76 Gy and 77–78 Gy in approximately 2 Gy per fraction. For this part of the analysis 37 (34 PPP and 3 intact prostate, 17 of which had seminal vesicle involvement (SV)), 40 (all intact prostate, 36 SV) and 39 patients (all intact prostate, 11 SV) were evaluated, respectively. A group with lower doses (46–60 Gy in 2 Gy fractions) was excluded as patients were treated with a combined brachytherapy boost. Secondly, the results were studied for each centre separately, to identify any individual trends and inter-centre differences, to assess if introducing automation could reduce inter-centre variability.

      2.3 Statistics

      Descriptive statistics and sample numbers informed from the literature were used. Paired-samples t-test (α = 0.05-two-tail-test) was used to highlight the statistical significance of the differences between metrics obtained with the original and RP2 plans. Normality of the data was tested prior to the analysis with the Anderson-Darling test. Seminal vesicle involvement in the PTV and OAR doses was evaluated using a point-biserial-correlation of the difference between the OP and RP2 analysed dose metrics (for example DeltaV70Gy for the rectum), and the presence or not of seminal vesicles (binary variable) for each patient for the dose groups used in the combined analysis (Minitab 17.2.1).

      3. Results

      During the validation stage, it became evident that the combined model produced plans that were either dosimetrically equivalent or superior to those obtained with the intact and post-prostatectomy models. One developer centre found average decreases in rectum, and bladder metrics of up to 13% and 1% respectively for the combined model compared to the post-prostatectomy model, and found equivalent doses (within 1%) between the combined and intact model. Similar findings were reported by the other validation sites so the combined model was selected to maximise flexibility of the final model.

      3.1 Plan evaluation: structures and dose levels

      For all patients combined, all dose parameters tended to be either equivalent or lower when RP2 was used. In general, there was a larger statistically significant improvement in bladder metrics for the lower dose parameters, such as V50Gy, which decreased on average by 2.3% (20.5%-OP vs. 18.2%-RP2) and 1.4% (33.2%-OP vs. 31.8%-RP2) respectively for the 77–78 Gy and the 66–70 Gy dose group. A similar trend was observed for rectal doses, except for the highest dose group (77–78 Gy), which showed a dose increase in the V40Gy metric of 1.2% (30.3%-OP vs. 31.5%-RP2), as shown in Table 3 and Fig. 1.
      Table 3All centres combined data showing the absolute difference in bladder, rectal and PTV mean doses of the original and RP2 plans (OP-RP2) and associated standard errors. Positive values show reductions obtained using RP2, while negative values represent an increase in dose using RP2. Values in bold represent statistically significant differences.
      Abs Difference (%)BLADDERAbs Difference (%)RECTUMAbs Difference (%)PTV
      77–78 Gy74–76 Gy66–70 Gy77–78 Gy74–76 Gy66–70 Gy77–78 Gy74–76 Gy66–70 Gy
      V50Gy2.3 ± 0.960.7 ± 0.441.4 ± 0.61V40Gy−1.2 ± 1.003.4 ± 1.510.9 ± 0.77V95%−0.2 ± 0.24−0.8 ± 0.200.1 ± 0.18
      V60Gy1.4 ± 0.330.4 ± 0.320.5 ± 0.43V50Gy0.1 ± 0.902.3 ± 1.150.9 ± 0.63D2%0.4 ± 0.28−0.1 ± 0.200.5 ± 0.15
      V65Gy1.1 ± 0.280.2 ± 0.22−0.2 ± 0.41V60Gy0.0 ± 0.490.8 ± 0.661.3 ± 0.71D98%0.1 ± 0.340.0 ± 0.120.1 ± 0.17
      V70Gy0.5 ± 0.25−0.1 ± 0.170.1 ± 0.44V70Gy−0.2 ± 0.34−0.1 ± 0.36−0.9 ± 0.34
      Figure thumbnail gr1
      Fig. 1Bladder and rectal mean doses for the original (full symbols) and RP2 plans (RP-empty symbols) and associated standard errors for all centres combined data, grouped according to the three different dose groups (66–70 Gy (diamonds), 74–76 Gy (squares) and 77–78 Gy (circles)).
      For the PTV statistically non-significant differences were observed for D2% and D98% for the 77–78 Gy and the 74–76 Gy groups while a small statistically significant average dose decrease of 0.5% (102.8%-OP vs. 102.3%-RP2) was found for D2% for the lower dose group (Table 3), which was not found for D98%. For V95% statistically significant improvement in PTV coverage of 0.8% (98.9% vs. 99.6%) (p < 0.001) was found for the 74–76 Gy group when RP2 was used (Fig. 2).
      Figure thumbnail gr2
      Fig. 2PTV D2% (a), D98% (b), V95% (c) differences between the OP and RP2 plan for all centres divided by prescription doses. The column represents the mean dose and the error bars show the standard deviation.
      CI and HI analysis showed that plans obtained with RP2 were as conformal as the original plans (1.1 vs. 1.08, 1.09 vs. 1.12 and 1.04 vs. 1.05 for the 66–70 Gy, 74–76 Gy, 77–78 Gy respectively, OP vs. RP2, p > 0.05). Dose distributions generated with RP2 tended to be either equally or more homogenous than those for the OP plans (77–78 Gy group p = 0.001, 0.06 vs. 0.05, and p > 0.05 for all the other groups). For the MUs there was a small statistically non-significant average increase in MUs when RP2 was used (928 vs. 935, 688 vs. 696 and 850 vs. 870 from the smaller to the larger dose group for OP vs. RP2). Looking at the seminal vesicle involvement, for all dose metrics studied for OARs and for the PTV V95%, the point-biserial-correlation coefficient (r) was, in absolute value, lower than 0.22 (Supplementary Table S1) and the two-tailed test of significance (p) was always larger than 0.05, indicating that the inclusion of seminal vesicles in the PTV made no significant difference to the RP2 model’s ability to achieve PTV coverage and OAR sparing.

      3.2 Plan evaluation: variation across centres

      Similarly to the combined analysis for the bladder metrics, for each centre, the use of RP2 either reduced, or at least provided dose parameters equivalent to the original plan for all but V65Gy for centre D for which the average dose increased by 0.6% (11.9% vs. 12.5%) (Table 4 and Supplementary Table S2). Statistically significant dose reductions were seen for many institutions and dose metrics; most notably, for centre B there was a consistent reduction on average doses for all metrics, between 3.6% (24.6% vs. 21%) for V50Gy and 0.7% (15.2% vs. 14.5%) for V65Gy.
      Table 4Absolute difference in bladder, rectal and PTV mean doses calculated with the original and RP2 plans (OP-RP2) with associated standard errors for each centre’s cohort. Positive values show reductions obtained using RP2, while negative values represent an increase in dose using RP2. Values in bold represent statistically significant differences.
      BLADDERRECTUMPTV
      CentreV50Gy (%)V60Gy (%)V65Gy (%)V70Gy (%)V40Gy (%)V50Gy (%)V60Gy (%)V70Gy (%)V95% (%)D2% (%)D98% (%)
      A1.9 ± 1.00.9 ± 0.6−0.9 ± 0.9−0.1 ± 0.29.9 ± 2.18.1 ± 1.84.0 ± 1.00.1 ± 0.40.5 ± 0.20.6 ± 0.30.3 ± 0.2
      B3.6 ± 1.71.5 ± 0.30.7 ± 0.31.5 ± 0.40.4 ± 0.80.7 ± 0.51.4 ± 1.1−0.2 ± 0.40.1 ± 0.21.4 ± 0.20.1 ± 0.1
      C−0.2 ± 0.40.0 ± 0.20.0 ± 0.20.1 ± 0.20.2 ± 0.9−0.2 ± 0.6−0.1 ± 0.40.2 ± 0.3−1.3 ± 0.3−0.5 ± 0.2−0.1 ± 0.1
      D1.8 ± 2.0−0.3 ± 0.4−0.6 ± 0.2−0.2 ± 0.2−3.0 ± 1.9−1.5 ± 1.7−1.3 ± 4.1−1.1 ± 0.8−0.4 ± 0.10.8 ± 0.20.1 ± 0.3
      E4.0 ± 2.71.3 ± 2.10.9 ± 1.5−1.1 ± 1.2−2.2 ± 3.4−1.2 ± 2.8−0.9 ± 2.1−2.5 ± 1.10.4 ± 0.8−0.4 ± 0.30.1 ± 1.0
      F2.5 ± 0.91.6 ± 0.70.9 ± 0.70.1 ± 0.8−0.8 ± 1.11.8 ± 1.91.2 ± 0.7−0.4 ± 0.60.3 ± 0.11.5 ± 0.2−0.6 ± 0.1
      G2.2 ± 1.01.4 ± 0.71.3 ± 0.7−0.2 ± 0.6−1.2 ± 1.5−0.7 ± 0.9−0.3 ± 0.5−0.7 ± 0.3−1.1 ± 0.30.8 ± 0.3−0.9 ± 0.3
      H0.5 ± 0.50.3 ± 0.50.5 ± 0.30.2 ± 0.5−0.7 ± 3.1−0.2 ± 1.8−0.6 ± 1.3−0.4 ± 0.81.1 ± 0.30.1 ± 0.9−0.2 ± 1.0
      Similar results were obtained for the rectum. For centre A, there was consistent and statistically significant reduction in average dose parameters between 4% for V60Gy (19% vs. 15%) and 9.9% V40Gy (41% vs. 31.1%). For each centre, results did not show any standard deviation reduction of average OAR metrics obtained by using the RP2 model (Table 4 and Supplementary Table S2). Additionally, evaluating the OAR metrics minimum and maximum values when comparing all centres there were still differences in the range of doses obtained by each centre with RP2. This indicates that introducing an automated model in this case did not promote planning DVH uniformity between centres (for example for the bladder V50Gy centre A still ranged between 0% and 73.2%, while centre B between 3.3% and 40%) (Supplementary Table S2).
      For the left and right femoral heads the OP plan already provided on average dose parameters lower than 3% for all centres, which were equivalent or lower when RP2 was used, and well within the VPSRG constraints (Table 1). For the two centres contouring the penile bulb (A, D) there were statistically non-significant differences when RP2 was used (A: 19.9 Gy vs. 18.6 Gy and D: 48.7 Gy vs. 51.4 Gy) (Supplementary Table S2). PTV results confirmed the trend found in the combined groups with average changes within 1.5% (Table 4). A consistent change for all centres was not found, for example for centre C coverage was statistically improved, with V95% increasing on average by 1.3% (98.4%-OP vs.99.7%-RP2), while for H it decreased by 1.1% (95.8%-OP vs. 94.7%-RP2).

      4. Discussion

      This study demonstrated a feasible and reasonable workflow to develop and implement a multicentre automated planning model. This was particularly notable because of the heterogeneity of the institutional planning processes, and patient populations provided by eight institutions with different contouring and dose protocols. Follow-up and refinement of the final version of the model was obtained utilising a consortium of experienced staff from each centre, who achieved standardising the data included in the model such as naming convention and OAR dose-constraints over all centres. This group validated model performance and provided feedback to the final clinical version. Our experience showed that a collaboration of this type has the potential to develop models faster since, as previously been highlighted, development can be time-consuming [
      • Wu H.
      • Jiang F.
      • Yue H.
      • Zhang H.
      • Wang K.
      • Zhang Y.
      Applying a RapidPlan model trained on a technique and orientation to another: a feasibility and dosimetric evaluation.
      ], and more robustly due to the prompt availability and diversity of the plans included in the model training [
      • Cagne E.
      • Botti A.
      • Micera R.
      • Galeandro M.
      • Sghedoni R.
      • Orlandi M.
      • et al.
      Knowledge-based treatment planning: an inter-technique and inter-system feasibility study for prostate cancer.
      ]. It was not previously shown that such a heterogeneous input population can be used to create uniform models in a diverse plan environment.
      The initial model testing confirmed that it was not necessary to separate the intact and post-prostatectomy cases, as the results of the model composed of both cases provided superior plans to a post-prostatectomy only model, and were equal to the intact one. Similarly, the model performed equally well regardless of whether or not seminal vesicles were included in the PTV.
      Overall the data analysis showed that plans calculated with RP2 achieved OAR doses that were, on average, either equivalent or reduced with respect to the original manually optimised plans. For individual centres OAR reductions were up to 10% for the rectum, and importantly these results were obtained using the RP2 model generated plan without any additional planner intervention, representing a considerable saving in planning time. Dose reductions were consistent for some centres in all parameters, such as centre B for the bladder and A for the rectum (Table 4), providing consistent overall improvement of their dose distributions. The variable improvement amongst centres highlights the differences in local practice prior to the introduction of RapidPlan. In the combined analysis it was noteworthy that, for the bladder (Table 3), the improvement in OAR sparing was generally larger for lower dose parameters (such as V50Gy), indicating that in the original plans, each centre was focusing mainly on directing the optimiser results toward limiting the higher doses. However, using RapidPlan it appeared there were no additional constraints to limiting the lower doses being applied. This represents a further potential benefit of automated models which impose optimisation constraints to all organ volume dose levels by using line objectives [
      • Fogliata A.
      • Belosi F.
      • Clivio A.
      • Navarria P.
      • Nicolini G.
      • Scorsetti M.
      • et al.
      On the pre-clinical validation of a commercial model-based optimization engine: application to volumetric modulated arc therapy for patients with lung or prostate cancer.
      ], without a planning time penalty.
      While the OAR dose parameters fulfilled the initial constraints established by the VPSRG consortium, there was no uniform reduction in the OAR variation between centres or within each centre itself, as hoped when introducing identical models and objectives (Tables 4 and Supplementary Table S2). This finding is similar to other published work [
      • Hussein M.
      • South C.P.
      • Barry M.A.
      • Adams E.J.
      • Jordan T.J.
      • Stewart A.J.
      • et al.
      Clinical validation and benchmarking of knowledge-based IMRT and VMAT treatment planning in pelvic anatomy.
      ,
      • Kubo K.
      • Monzen H.
      • Ishii K.
      • Tamura M.
      • Kawamorita R.
      • Sumida I.
      • et al.
      Dosimetric comparison of RapidPlan and manually optimized plans in volumetric arc therapy for prostate cancer.
      ,
      • Schubert C.
      • Waletzko O.
      • Weiss C.
      • Voelzke D.
      • Toperim S.
      • Roeser A.
      • et al.
      Intercenter validation of a knowledge based model for automated planning of volumetric modulated arc therapy for prostate cancer. The experience of the German RapidPlan Consortium.
      ], and it could be explained by the fact that each centre applied its own contouring protocol, fractionation regime and beam arrangement. Different dose schedules should be easily handled by RapidPlan due to the availability in the dose evaluation module of rescaling the optimisation objectives, and the range of doses used in our analysis showed this to be so. However, contouring and beam arrangement are still dependent on the experience and training of the person performing it, and could affect the results of the optimisation, due to the way RapidPlan handles the overlap between OAR and target volume [
      • Moore K.L.
      • Brame S.R.
      • Low D.A.
      • Mutic S.
      Experience based quality control of clinical intensity modulated radiotherapy planning.
      ,
      • Schubert C.
      • Waletzko O.
      • Weiss C.
      • Voelzke D.
      • Toperim S.
      • Roeser A.
      • et al.
      Intercenter validation of a knowledge based model for automated planning of volumetric modulated arc therapy for prostate cancer. The experience of the German RapidPlan Consortium.
      ]. Our results suggest that to further reduce plan variability across different centres, or within a centre, it may be necessary to include strict contouring and/or beam arrangement guidelines along with the model description. The VPSRG consortium aims to analyse the results of other anatomical sites in order to understand if these findings are specific to prostate cases, or more generalisable, as shown in previous work [
      • Moore K.L.
      • Brame S.R.
      • Low D.A.
      • Mutic S.
      Experience based quality control of clinical intensity modulated radiotherapy planning.
      ], particularly for head and neck cases. Improvement is also expected by updating the models with the newly generated high quality RapidPlan plans produced by all centres.
      Overall no significant changes were seen for the PTV metrics (Fig. 2). The use of RP2 appeared to achieve more homogeneous PTV doses, however close inspection of the 3D dose-distributions highlighted a tendency for RP2 plan to produce under-coverage of the PTV near the anterior edge of the rectum, as previously reported [
      • Hussein M.
      • South C.P.
      • Barry M.A.
      • Adams E.J.
      • Jordan T.J.
      • Stewart A.J.
      • et al.
      Clinical validation and benchmarking of knowledge-based IMRT and VMAT treatment planning in pelvic anatomy.
      ].
      The VPSRG model development and validation included different techniques (IMRT and VMAT) due to the variety of centres participating in the VPSRG project. The VPSRG committee had extensive discussion on the effectiveness of using a model covering both techniques, or if separate models should be developed. Validation results confirmed that, for the prostate case, a universal model could provide clinically acceptable treatment plans, independent of the treatment modality or centre.
      This project has facilitated an on-going collaboration to implement an automated approach with the aim of reducing inter and intra-centre planning variability. The review of the first months of clinical data has shown that it was possible to successfully develop and clinically implement multicentre models at unrelated centres utilising patients treated in different institutions with a variety of techniques and dose protocols. Data from the first model developed for prostate showed that automated plans were generally of equivalent or higher quality to the manually optimised ones after a single optimisation and no further planner intervention. Following these results, automation is now used by all centres in clinical practice with more models being collaboratively developed highlighting the importance and benefit of shared knowledge.

      Funding

      No specific funding has been provided for this work.

      Declaration of Competing Interest

      The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

      Acknowledgements

      The authors would like to acknowledge the Department of Health and Human Services of the State of Victoria, Australia, for the support provided in this project.

      Appendix A

      Table A1Characteristics of the 135 patient plans used in the training set. The number of intact or post-prostatectomy (PPP) patients and those with PTV seminal vesicles involvement (SV) are shown.
      CentreNo of patientsPrescription DoseFractionsTechniqueintactPPPSV
      A3266, 68, 70 and 74 Gy33, 34, 35 and 37IMRT

      (5, 7 Fields)
      211118
      B4070 and 78 Gy35 and 39IMRT

      (7 Fields)
      20200
      C1575.6, 76 Gy36, 38 and 42VMAT

      (1 or 2 Arcs)
      15013
      D2046, 74 and 78 Gy23, 37, and 39VMAT

      (1 or 2 Arcs)
      20019
      G2870 and 78 Gy35 and 39IMRT

      (7 Fields)
      1991
      Total135954051

      Appendix B. Supplementary data

      The following are the Supplementary data to this article:

      References

        • Wu B.
        • Ricchetti F.
        • Sanguineti G.
        • Kazhdan M.
        • Simari P.
        • Chuang M.
        • et al.
        Patient geometry-driven information retrieval for IMRT treatment plan quality control.
        Med Phys. 2009; 36: 5497-5505https://doi.org/10.1118/1.3253464
        • Wu B.
        • Ricchetti F.
        • Sanguineti G.
        • Kazhdan M.
        • Simari P.
        • Jacques R.
        • et al.
        Data-driven approach to generating achievable dose-volume histogram objectives in intensity-modulated radiotherapy planning.
        Int J Radiat Oncol Biol Phys. 2011; 79: 1241-1247https://doi.org/10.1016/j.ijrobp.2010.05.026
        • Chanyavanich V.
        • Das S.
        • Lee W.
        • Lo W.
        Knowledge based IMRT treatment planning for prostate cancer.
        Med Phys. 2011; 38: 2515-2522https://doi.org/10.1118/1.3574874
        • Moore K.L.
        • Brame S.R.
        • Low D.A.
        • Mutic S.
        Experience based quality control of clinical intensity modulated radiotherapy planning.
        Int J Radiat Oncol Biol Phys. 2011; 81: 545-551https://doi.org/10.1016/j.ijrobp.2010.11.030
        • Appenzoller L.
        • Michalski J.
        • Thorstad W.
        • Mutic S.
        • Moore K.
        Predicting dose volume histograms for organs-at-risk in IMRT planning.
        Med Phys. 2012; 39: 7446-7461https://doi.org/10.1118/1.4761864
        • Fogliata A.
        • Wang P.
        • Belosi F.
        • Clivio A.
        • Nicolini G.
        • Vanetti E.
        • et al.
        Assessment of a model based optimization engine for volumetric modulated arc therapy for patients with advanced hepatocellular cancer.
        Radiat Oncol. 2014; 9: 236https://doi.org/10.1186/s13014-014-0236-0
        • Fogliata A.
        • Belosi F.
        • Clivio A.
        • Navarria P.
        • Nicolini G.
        • Scorsetti M.
        • et al.
        On the pre-clinical validation of a commercial model-based optimization engine: application to volumetric modulated arc therapy for patients with lung or prostate cancer.
        Radiother Oncol. 2014; 113: 385-391https://doi.org/10.1016/j.radonc.2014.11.009
        • Tol J.P.
        • Delaney A.R.
        • Dahele M.
        • Slotman B.J.
        • Verbakel W.F.A.R.
        Evaluation of a knowledge based planning solution for head and neck cancer.
        Int J Radiat Oncol Biol Phys. 2015; 91: 612-620https://doi.org/10.1016/j.ijrobp.2014.11.014
        • Fogliata A.
        • Nicolini G.
        • Clivio A.
        • Vanetti E.
        • Laksar S.
        • Tozzi A.
        • et al.
        A broad scope knowledge based model for optimisation of VMAT in esophageal cancer: validation and assessment of plan quality among different treatment centres.
        Radiat Oncol. 2015; 10: 220https://doi.org/10.1186/s13014-015-0530-5
        • Delaney A.R.
        • Dahele M.
        • Tol J.P.
        • Slotman B.J.
        • Verbakel W.F.A.R.
        Knowledge-based planning for stereotactic radiotherapy of peripheral early stage lung cancer.
        Acta Oncol. 2017; 56: 490-495https://doi.org/10.1080/0284186X.2016.1273544
        • Hussein M.
        • South C.P.
        • Barry M.A.
        • Adams E.J.
        • Jordan T.J.
        • Stewart A.J.
        • et al.
        Clinical validation and benchmarking of knowledge-based IMRT and VMAT treatment planning in pelvic anatomy.
        Radiother Oncol. 2016; 120: 473-479https://doi.org/10.1016/j.radonc.2016.06.022
        • Powis R.
        • Bird A.
        • Brennan M.
        • Hinks S.
        • Newman H.
        • Reed K.
        • et al.
        Clinical implementation of a knowledge based planning tool for prostate VMAT.
        Radiat Oncol. 2017; 12: 81https://doi.org/10.1186/s13014-017-0814-z
        • Kubo K.
        • Monzen H.
        • Ishii K.
        • Tamura M.
        • Kawamorita R.
        • Sumida I.
        • et al.
        Dosimetric comparison of RapidPlan and manually optimized plans in volumetric arc therapy for prostate cancer.
        Phys Med. 2017; 44: 199-204https://doi.org/10.1016/j.ejmp.2017.06.026
        • Moore K.L.
        • Brame R.S.
        • Low D.A.
        • Mutic S.
        Experience-based quality control of clinical intensity-modulated radiotherapy planning.
        Int J Radiat Oncol Biol Phys. 2011; 81: 545-551https://doi.org/10.1016/j.ijrobp.2010.11.030
        • Nelms B.E.
        • Robinson G.
        • Markham J.
        • Velasco K.
        • Boyd S.
        • Narayan S.
        • et al.
        Variation in external beam treatment plan quality: an inter-institutional study of planners and planning systems.
        Pract Radiat Oncol. 2012; 2: 296-305https://doi.org/10.1016/j.prro.2011.11.012
        • Peters L.J.
        • O’Sullivan B.
        • Giralt J.
        • Fitzgerald T.J.
        • Trotti A.
        • Bernier J.
        • et al.
        Critical impact of radiotherapy protocol compliance and quality in the treatment of advanced head and neck cancer: results from TROG 0202.
        J Clin Oncol. 2010; 28: 2996-3001https://doi.org/10.1200/JCO.2009.27.4498
        • Fitzgerald T.J.
        What we have learned: the impact of quality from a clinical trials perspective.
        Semin Radiat Oncol. 2012; 22: 18-28https://doi.org/10.1016/j.semradonc.2011.09.004
        • Fairchild A.
        • Straube W.
        • Laurie F.
        • Followill D.
        Does quality of radiation therapy predict outcomes of multicenter cooperative group trials? A literature review.
        Int J Radiat Oncol Biol Phys. 2013; 87: 246-260https://doi.org/10.1016/j.ijrobp.2013.03.036
        • Duchesne G.
        • Grand M.
        • Kron T.
        • Haworth A.
        • Corry J.
        • Jackson M.
        • et al.
        The TransTasman Radiation Oncology Group (TROG) ‘ANROTAT’* Framework – development of a model for the early evaluation of new technologies and techniques in Radiation Oncology (*Assessment of new radiation oncology technology and treatments).
        J Med Imag Radiat Oncol. 2015; 59: 363-370https://doi.org/10.1111/1754-9485.12255
        • Cagne E.
        • Botti A.
        • Micera R.
        • Galeandro M.
        • Sghedoni R.
        • Orlandi M.
        • et al.
        Knowledge-based treatment planning: an inter-technique and inter-system feasibility study for prostate cancer.
        Phys Med. 2017; 36: 38-44https://doi.org/10.1016/j.ejmp.2017.03.002
        • Schubert C.
        • Waletzko O.
        • Weiss C.
        • Voelzke D.
        • Toperim S.
        • Roeser A.
        • et al.
        Intercenter validation of a knowledge based model for automated planning of volumetric modulated arc therapy for prostate cancer. The experience of the German RapidPlan Consortium.
        PLoS ONE. 2017; 12e0178034https://doi.org/10.1371/journal.pone.0178034
        • Roach D.
        • Wortel G.
        • Ochoa C.
        • Jensen H.R.
        • Damen E.
        • Vial P.
        • et al.
        Adapting automated treatment planning configurations across international centres for prostate radiotherapy.
        Phys Imag Radiat Oncol. 2019; 10: 7-13https://doi.org/10.1016/j.phro.2019.04.007
        • Ueda Y.
        • Fukunaga T.
        • Kamima T.
        • Adachi Y.
        • Nakamatsu K.
        • Monzen H.
        Evaluation of multiple institutions’ models for knowledge-based planning of volumetric arc therapy (VMAT) for prostate cancer.
        Radiat Oncol. 2018; 13: 46https://doi.org/10.1186/s13014-018-0994-1
        • Scaggion A.
        • Fusella M.
        • Roggio A.
        • Bacco S.
        • Pivato N.
        • Rossato M.A.
        • et al.
        Reducing inter- and intra-planner variability in radiotherapy plan output with a commercial knowledge-based planning solution.
        Phys Med. 2018; 53: 86-93https://doi.org/10.1016/j.ejmp.2018.08.016
      1. Eclipse Photon and Electron reference guide v.13.7, Varian Medical System, June 2015, https://www.myvarian.com.

        • Delaney A.R.
        • Tol J.P.
        • Dahele M.
        • Cuijpers J.
        • Slotman B.J.
        • Verbakel W.F.A.R.
        Effect of dosimetric outlier on the performance of a commercial knowledge-based planning solution.
        Int J Radiat Oncol Biol Phys. 2016; 94: 469-477https://doi.org/10.1016/j.ijrobp.2015.11.011
        • Wu H.
        • Jiang F.
        • Yue H.
        • Zhang H.
        • Wang K.
        • Zhang Y.
        Applying a RapidPlan model trained on a technique and orientation to another: a feasibility and dosimetric evaluation.
        Radiat Oncol. 2016; 11: 108https://doi.org/10.1186/s13014-016-0684-9
        • Lomax N.J.
        • Scheib S.G.
        Quantifying the degree of conformity in radiosurgery treatment planning.
        Int J Radiat Oncol Biol Phys. 2003; 55: 1409-1419https://doi.org/10.1016/S0360-3016(02)04599-6

      Linked Article