Advertisement

Combining inter-observer variability, range and setup uncertainty in a variance-based sensitivity analysis for proton therapy

Open AccessPublished:December 02, 2021DOI:https://doi.org/10.1016/j.phro.2021.11.005

      Abstract

      Margin concepts in proton therapy aim to ensure full dose coverage of the clinical target volume (CTV) in presence of setup and range uncertainty. Due to inter-observer variability (IOV), the CTV itself is uncertain. We present a framework to evaluate the combined impact of IOV, setup and range uncertainty in a variance-based sensitivity analysis (SA). For ten patients with skull base meningioma, the mean calculation time to perform the SA including 1.6 × 104 dose recalculations was 59 min. For two patients in this dataset, IOV had a relevant impact on the estimated CTV D95% uncertainty.

      Keywords

      1. Introduction

      Treatment plans in proton therapy are affected by range and setup uncertainties. These are typically compensated through margin concepts or robust planning approaches. Margin concepts aim at covering the clinical target volume (CTV) in presence of range and setup uncertainty [
      • Unkelbach J.
      • Alber M.
      • Bangert M.
      • Bokrantz R.
      • Chan T.C.Y.
      • Deasy J.O.
      • et al.
      Robust radiotherapy planning.
      ]. However, due to inter-observer variability (IOV), the CTV itself is uncertain. While there are many studies assessing IOV, only few studies have investigated dosimetric consequences of IOV [
      • Vinod S.K.
      • Jameson M.G.
      • Min M.
      • Holloway L.C.
      Uncertainties in volume delineation in radiation oncology: A systematic review and recommendations for future studies.
      ], e.g Lobefalo et al. [
      • Lobefalo F.
      • Bignardi M.
      • Reggiori G.
      • Tozzi A.
      • Tomatis S.
      • Alongi F.
      • et al.
      Dosimetric impact of inter-observer variability for 3D conformal radiotherapy and volumetric modulated arc therapy: The rectal tumor target definition case.
      ] who investigated the dosimetric impact of IOV in three-dimensional conformal radiotherapy (3D-CRT) and volumetric modulated arc therapy for rectal tumours, Hellebust et. al. [
      • Hellebust T.P.
      • Tanderup K.
      • Lervåg C.
      • Fidarova E.
      • Berger D.
      • Malinen E.
      • et al.
      Dosimetric impact of interobserver variability in MRI-based delineation for cervical cancer brachytherapy.
      ] who assessed the dosimetric impact of IOV in brachytherapy for cervical cancer and Eminowicz et al. [
      • Eminowicz G.
      • Rompokos V.
      • Stacey C.
      • McCormack M.
      The dosimetric impact of target volume delineation variation for cervical cancer radiotherapy.
      ], who studied the dosimetric impact of IOV in VMAT for cervical cancer. To the best of our knowledge, there is no study assessing the combined and relative impact of range, setup uncertainty and IOV in proton therapy in a quantitative way. The statistical method of variance-based sensitivity analysis (SA) is suited for this, since it can be used to assess the impact of uncertainty of multiple input parameters on the output of a quantitative model [
      • Saltelli A.
      • Annoni P.
      • Azzini I.
      • Campolongo F.
      • Ratto M.
      • Tarantola S.
      Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index.
      ]. In the context of patient dose calculation in medical physics, the technique has been previously applied to relative biological effectiveness (RBE) uncertainties in carbon ion therapy [
      • Kamp F.
      • Brüningk S.
      • Cabal G.
      • Mairani A.
      • Parodi K.
      • Wilkens J.J.
      Variance-based sensitivity analysis of biological uncertainties in carbon ion therapy.
      ,
      • Kamp F.
      • Wilkens J.J.
      Application of variance-based uncertainty and sensitivity analysis to biological modeling in carbon ion treatment plans.
      ] and to estimate the impact of interpatient variability on organ dose estimates in nuclear medicine [
      • Zvereva A.
      • Kamp F.
      • Schlattl H.
      • Zankl M.
      • Parodi K.
      Impact of interpatient variability on organ dose estimates according to MIRD schema: Uncertainty and variance-based sensitivity analysis.
      ]. Recently, a framework to evaluate the combined impact of range, setup and RBE uncertainty in a variance-based SA has been presented by our group [
      • Hofmaier J.
      • Dedes G.
      • Carlson D.J.
      • Parodi K.
      • Belka C.
      • Kamp F.
      Variance-based sensitivity analysis for uncertainties in proton therapy: A framework to assess the effect of simultaneous uncertainties in range, positioning and RBE model predictions on RBE-weighted dose distributions.
      ]. In this technical note, an extension of the framework to include IOV is shown. The feasibility of the approach was demonstrated by using it to investigate the relative impact of IOV, range and setup uncertainty on proton plans for a dataset with ten patients with skull base meningioma.

      2. Materials and methods

      2.1 Variance-based sensitivity analysis

      In the Monte Carlo method of global variance-based SA, the output of a model Y=f(X) with k input factors X=(x1,x2,,xk) which are subject to uncertainty is recalculated many times while simultaneously and randomly varying the input factors within their assumed distributions. In our particular case, the model f(X) corresponded to a dose calculation followed by a dose volume histogram (DVH) calculation. The output Y corresponded to DVH parameters of interest. The input factors (x1,x2xk) included patient shifts in three spatial dimensions, absolute and relative range shifts as well as IOV, resulting in k=6 input factors. The resulting variance V(Y) is decomposed as [
      • Saltelli A.
      • Annoni P.
      • Azzini I.
      • Campolongo F.
      • Ratto M.
      • Tarantola S.
      Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index.
      ]:
      V(Y)=l=1kVl+l=1km>lkVlm+l=1km>lkn>mkVlmn++V1k
      (1)


      resulting in (2k-1) terms. The first order terms are
      Vl=V[E(Y|Xl)]
      (2)


      The expectation value E(Y|Xl) is hereby calculated over all possible values of all input factors except for Xl, which is kept fixed. The second order terms, which are representing the interaction between the inputs Xl and Xm, are
      Vlm=V[E(Y|Xl,Xm)]-Vl-Vm
      (3)


      Higher order terms are defined in an analoguous fashion. Sensitivity indices are defined by normalising to the overall variance
      Sl=VlV(Y)
      (4)


      Slm=VlmV(Y)
      (5)


      and so on. Total effect indices are defined by summing all terms of any order containing l:
      STl=Sl+mlkSlm++S1k
      (6)


      Like in a previous study from our group [
      • Hofmaier J.
      • Dedes G.
      • Carlson D.J.
      • Parodi K.
      • Belka C.
      • Kamp F.
      Variance-based sensitivity analysis for uncertainties in proton therapy: A framework to assess the effect of simultaneous uncertainties in range, positioning and RBE model predictions on RBE-weighted dose distributions.
      ], the efficient Monte Carlo method proposed by Saltelli [
      • Saltelli A.
      • Annoni P.
      • Azzini I.
      • Campolongo F.
      • Ratto M.
      • Tarantola S.
      Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index.
      ] was used for direct calculation of Sl and STl, and sampling from low-discrepancy quasi-random sequences was employed to improve convergence. This method requires N(k+2) model evaluations, where N is typically of the order of 103. In our study, as described above, we had k=6 input factors. We set N=2048, which resulted in approximately 1.6·104 model evaluations. The sensitivity analysis framework was extended to include IOV. Additionally to the fast, graphics processing unit (GPU) based pencil beam algorithm capable of modeling setup and range variations described in the previous publication from our group [
      • Hofmaier J.
      • Dedes G.
      • Carlson D.J.
      • Parodi K.
      • Belka C.
      • Kamp F.
      Variance-based sensitivity analysis for uncertainties in proton therapy: A framework to assess the effect of simultaneous uncertainties in range, positioning and RBE model predictions on RBE-weighted dose distributions.
      ], the possibility to include multiple treatment plans and to switch randomly between them was added.

      2.2 Clinical dataset

      Datasets of ten patients with benign (WHO grade I) meningioma of the skull base were included in this study. For all patients, contrast enhanced magnetic resonance imaging (MRI) and DOTATATE positron emission tomography (PET) images were available in addition to a planning computed tomography (CT).

      2.3 Target delineation and treatment planning

      A rigid image registration of MRI, PET and planning CT images was performed. For each patient, four clinicians independently delineated the gross tumor volume (GTV) taking into account all imaging modalities (GTVobserver). A consensus GTV (GTVSTAPLE) was created using the simultaneous truth and performance level estimation (STAPLE) algorithm [
      • Warfield S.K.
      • Zou K.H.
      • Wells W.M.
      Simultaneous truth and performance level estimation (STAPLE): An algorithm for the validation of image segmentation.
      ] in the research treatment planning system computational environment for radiological research (CERR) [
      • Deasy J.O.
      • Blanco A.I.
      • Clark V.H.
      CERR: A computational environment for radiotherapy research.
      ]. This implementation of an expectation-maximization algorithm generates a probabilistic estimate of the true volume based on the volumes delineated by multiple observers. The GTVSTAPLE was used as the ”ground truth” GTV. As an example, the four GTVobserver and the GTVSTAPLE contours for patient number 1 are shown in the supplementary material. The CTVobserver and the CTVSTAPLE were defined as the respective GTV without any margins applied (i.e. GTV = CTV), as suggested in a current guideline [
      • Combs S.E.
      • Baumert B.G.
      • Bendszus M.
      • Bozzao A.
      • Brada M.
      • Fariselli L.
      • et al.
      ESTRO ACROP guideline for target volume delineation of skull base tumors.
      ]. To obtain the planning target volumes (PTVs), gantry-angle specific margins were applied. To compensate for proton range uncertainty, larger margins were applied in beam direction than laterally. The applied margins were 6, 5 and 3 mm in distal, proximal and lateral directions, respectively. For a typical margin receipe of 3.5% + 3 mm, the distal margin of 6 mm would correspond to a radiological target depth of approximately 9 cm. Since all tumours were at the skull base and therefore at similar depths, the same absolute margins were applied to all patients for simplicity. For each CTVobserver a PTVobserver was created. For each PTVobserver of each patient a spot scanning proton treatment plan with one beam was generated using non-robust optimization, resulting in a total number of 40 treatment plans (four treatment plans for each of the ten patients). The gantry angle was chosen individually for each patient. The proton plans were optimized to deliver 1.8 Gy(RBE) per fraction to the PTVobserver. A spatially constant RBE of 1.1 was assumed.

      2.4 Application of the SA framework

      Like in the previous study from our group [
      • Hofmaier J.
      • Dedes G.
      • Carlson D.J.
      • Parodi K.
      • Belka C.
      • Kamp F.
      Variance-based sensitivity analysis for uncertainties in proton therapy: A framework to assess the effect of simultaneous uncertainties in range, positioning and RBE model predictions on RBE-weighted dose distributions.
      ], the variance-based SA was performed assuming the following uncertainty distributions for the input factors mentioned in Section 2.1: For patient shifts in X,Y and Z directions, a normal distribution with standard deviation σX,Y,Z = 1 mm truncated to 2σX,Y,Z was assumed. For relative range shifts the probability density was set to a normal distribution with standard deviation σr,rel = 3 % truncated to 2σr,rel. Additionally, absolute range shifts following a normal distribution with standard deviation σr,abs = 1 mm truncated to 2σr,abs were assumed. For IOV, an equal probability of p  = 0.25 for each of the four observer treatment plans was chosen. To perform the SA, the dose distribution was re-calculated approximately 1.6·104 times (corresponding to N=2048 and k=6 in the Saltelli formalism, as described in Section 2.1) while simultaneously sampling from the above uncertainty distributions. An Nvidia Quadro RTX 8000 GPU with 48 gigabytes of memory was used. For the resulting dose distributions, DVHs were calculated for the CTVSTAPLE. Confidence intervals (CIs) and sensitivity indices for the dose level enclosing 95% of the CTVSTAPLE (D95%) were calculated. Convergence plots of the sensitivity indices were created. The obtained total effect indices ST were converted to SIIOV, the sum of all interaction terms with involvement of IOV and SIother, the sum of all interaction terms without involvement of IOV. By definition is
      SIIOV=STIOV-SIOV
      (7)


      and due to normalization
      SIother=1-Ssetup-Srange-STIOV
      (8)


      3. Results

      The mean calculation time to perform the 1.6·104 dose calculations was 59 min. Large differences were observed for the calculation times for different patients, which ranged from 11 min to 195 min. Convergence plots for Sl and STl for an exemplary patient are shown in panels A and B of Fig. 1. By visual inspection of the convergence plots it becomes evident that a sufficient convergence was achieved well below N=2048.
      Figure thumbnail gr1
      Fig. 1Convergence plots for Sl and STl for one patient (panels A and B) and DVHs for CTVSTAPLE for the four patients with the largest overall D95% uncertainties (panels C to F). The variability of the DVH in presence of setup uncertainty, range uncertainty and IOV is visualized by the shaded areas (68% and 95% CIs). The solid line indicates the mean value over all simulated error scenarios.
      Results for the D95% are presented in Table 1. For six patients, the width of the CI95% for the D95% was below 0.18 Gy (10% of the prescribed dose of 1.8 Gy). Uncertainties of more than 10 % were observed for patients 2, 3, 7 and 9. Here the widths of the CI95% for the D95% were 0.57, 0.24, 0.28 and 0.48 Gy, respectively. Plots of the DVHs for the CTVSTAPLE for these four patients with their corresponding 95 % and 68 % CIs are shown in panels C to F of Fig. 1. For two of these patients, the overall influence of IOV was negligible (SIOV + SIIOV < 0.05 for patients 7 and 9). In both cases, range uncertainty was the most important contribution to overall uncertainty (Srange was 0.53 and 0.70 for patients 7 and 9, respectively). For patients 2 and 3, however, IOV played a major role for overall uncertainty (SIOV + SIIOV was 0.43 and 0.63 for patients 2 and 3, respectively).
      Table 1Uncertainty and sensitivity analysis results for the D95% for (CTV)STAPLE. For each patient, the mean value and 95% and the 68% CIs have been calculated. The relative contribution to the overall uncertainty is broken down to first order indices Ssetup, Srange and SIOV, higher order indices with involvement of IOV (SIIOV) and higher order indices without involvement of IOV (SIother).
      pat.mean [Gy]CI95% [Gy]CI68% [Gy]SsetupSrangeSIOVSIIOVSIother
      11.711.62–1.741.69–1.730.210.290.110.100.29
      21.561.16–1.731.35–1.720.120.350.340.090.10
      31.661.49–1.731.59–1.710.140.110.620.010.12
      41.701.59–1.731.68–1.720.190.140.100.120.45
      51.731.71–1.741.72–1.740.370.310.000.140.18
      61.731.70–1.741.72–1.730.200.290.030.040.44
      71.671.45–1.731.61–1.730.250.530.000.020.20
      81.691.58–1.741.65–1.730.230.490.040.050.19
      91.621.25–1.731.48–1.710.130.700.010.000.16
      101.731.70–1.741.72–1.740.260.240.020.060.42

      4. Discussion

      A framework for the variance-based SA of setup, range and IOV has been presented. To the best of our knowledge, this study is the first to assess the relative dosimetric impact of setup uncertainty, range uncertainty and IOV in a variance-based SA. In a first analysis of ten patients, calculation times were of the order of a few minutes to a few hours. These calculation times are fast enough for offline plan evaluation. Although this was not investigated in this study, it can be assumed that the differences in calculation time were caused by differences in the sizes and depths of the target volumes. The convergence plots in Fig. 1 suggest that actually less than N=2048 would have been sufficient to achieve convergence, therefore the calculation times could be reduced by stopping the calculation after reaching a predefined convergence criterion. While for the majority of patients, the overall uncertainties in CTV coverage were small, in some cases the coverage was deteriorated. The dominating contributions to overall uncertainty were either range uncertainty or IOV. This suggests that IOV might have a relevant effect on target coverage in some patients.
      In this work, the analysis was restricted to skull base meningioma, since the framework does not support organ motion at the moment. Furthermore, a pencil beam algorithm was used, whose accuracy is known to decrease in regions of high heterogeneity. The framework would be applicable without modification to other tumour sites for which these limitations are acceptable. The possibility to model motion could be included by extending the framework to use multiple CT geometries (e.g. phases of a 4D-CT to model breathing motion), at the cost of an increased memory usage and longer calculation times. In the previous publication from our group [
      • Hofmaier J.
      • Dedes G.
      • Carlson D.J.
      • Parodi K.
      • Belka C.
      • Kamp F.
      Variance-based sensitivity analysis for uncertainties in proton therapy: A framework to assess the effect of simultaneous uncertainties in range, positioning and RBE model predictions on RBE-weighted dose distributions.
      ], uncertainties in variable RBE models were evaluated in combination with setup and range uncertainty. In this study, since the focus was on IOV, RBE uncertainty was not taken into account and a constant RBE of 1.1 was assumed. However, the combined evaluation of all four types of uncertainty could in principle also be included in the framework. This could be used in future studies to assess the combined impact of range, setup and RBE uncertainty and IOV. The evaluation of the CTV D95% in presence of IOV required a ”ground truth” CTV. Unfortunately, this volume is not known. In this work, the consensus target volume created with the STAPLE algorithm was used to define a ”ground truth” target volume, as has been done previously [
      • Kristensen I.
      • Nilsson K.
      • Agrup M.
      • Belfrage K.
      • Embring A.
      • Haugen H.
      • et al.
      A dose based approach for evaluation of inter-observer variations in target delineation.
      ]. Since this algorithm provides a maximum likelyhood estimate for the actual CTV based on the observer CTVs themselves, this approach is well suited to capture the variability within a group of observers. However, it cannot correct systematic deviations from the ground truth CTV within the observer group. Furthermore, in this study only data from four observers was available, which was considered sufficient to show the feasibility of the approach. However, outlier contours could have considerable effect on the evaluation. For this reason, both the number of patients and the number of observers needs to be increased for future systematic evaluations of the impact of IOV in combination with setup and range uncertainties. Another limitation is that in our study simple proton plans with only one beam direction were used. More clinically realistic plans with multiple beam directions are supported by the framework without modifications, but have higher memory requirements and will lead to longer calculation times.
      In this technical note, no metrics of contour similarity such as Dice coefficients or Hausdorff distances were evaluated. The presented framework might be used in future studies to investigate the correlation of these metrics with dosimetric parameters. It could also have potential applications in the investigation of the implications of uncertainty reduction. If technical advances such as dual energy computed tomography (DECT), proton CT and improved image guidance reduce range and setup uncertainty, the relative impact of IOV on overall uncertainty becomes larger. The SA framework could complement studies such as [
      • van de Water S.
      • van Dam I.
      • Schaart D.R.
      • Al-Mamgani A.
      • Heijmen B.J.M.
      • Hoogeman M.S.
      The price of robustness; impact of worst-case optimization on organ-at-risk dose and complication probability in intensity-modulated proton therapy for oropharyngeal cancer patients.
      ,
      • Tattenberg S.
      • Madden T.M.
      • Gorissen B.L.
      • Bortfeld T.
      • Parodi K.
      • Verburg J.
      Proton range uncertainty reduction benefits for skull base tumors in terms of normal tissue complication probability (NTCP) and healthy tissue doses.
      ,
      • Wagenaar D.
      • Kierkels R.G.J.
      • van der Schaaf A.
      • Meijers A.
      • Scandurra D.
      • Sijtsema N.M.
      • et al.
      Head and neck IMPT probabilistic dose accumulation: Feasibility of a 2 mm setup uncertainty setting.
      ], who have investigated the impact of range and setup margin reduction. By also including IOV into the analysis, questions such as how far the overall uncertainty can be reduced by reduction of setup and range uncertainty before IOV becomes the limiting factor could be comprehensively investigated in future studies. Similarly, the following question could be assessed: Although not explicitly accounted for in the PTV concept, it can be assumed that IOV is compensated by the margins (or, in an analogous manner in the case of robust optimization, the plan robustness settings) to a certain extent. The SA framework could help to investigate whether a CTV-to-PTV margin reduction (or reduction of plan robustness settings) justified by reduced range and setup uncertainties would lead to an unexpected increase in uncertainty of CTV coverage caused by IOV.
      In conclusion, a previously presented framework for variance-based sensitivity analysis has been extended to include IOV. The approach is feasible and enables the evaluation of the combined impact of setup and range uncertainty and IOV. In a first analysis of ten patients, IOV had a relevant impact on the CTV D95% for two of these patients. This suggests that IOV could have a deteriorating effect on CTV coverage in some cases.

      Declaration of Competing Interest

      The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: CB holds research grants from Elekta, Brainlab, Viewray and C-Rad without any relation to the research described here. The remaining authors declare that they have no competing interest.

      Acknowledgements

      This project was supported by the DFG grant KA 4346/1-1 and the DFG Cluster of Excellence Munich Center for Advanced Photonics (MAP).

      Supplementary data

      The following are the Supplementary data to this article:

      References

        • Unkelbach J.
        • Alber M.
        • Bangert M.
        • Bokrantz R.
        • Chan T.C.Y.
        • Deasy J.O.
        • et al.
        Robust radiotherapy planning.
        Phys Med Biol. 2018; 63: 22TR02https://doi.org/10.1088/1361-6560/aae659
        • Vinod S.K.
        • Jameson M.G.
        • Min M.
        • Holloway L.C.
        Uncertainties in volume delineation in radiation oncology: A systematic review and recommendations for future studies.
        Radiother Oncol. 2016; 121: 169-179https://doi.org/10.1016/j.radonc.2016.09.009
        • Lobefalo F.
        • Bignardi M.
        • Reggiori G.
        • Tozzi A.
        • Tomatis S.
        • Alongi F.
        • et al.
        Dosimetric impact of inter-observer variability for 3D conformal radiotherapy and volumetric modulated arc therapy: The rectal tumor target definition case.
        Radiat Oncol. 2013; 8: 176https://doi.org/10.1186/1748-717X-8-176
        • Hellebust T.P.
        • Tanderup K.
        • Lervåg C.
        • Fidarova E.
        • Berger D.
        • Malinen E.
        • et al.
        Dosimetric impact of interobserver variability in MRI-based delineation for cervical cancer brachytherapy.
        Radiother Oncol. 2013; 107: 13-19https://doi.org/10.1016/j.radonc.2012.12.017
        • Eminowicz G.
        • Rompokos V.
        • Stacey C.
        • McCormack M.
        The dosimetric impact of target volume delineation variation for cervical cancer radiotherapy.
        Radiother Oncol. 2016; 120: 493-499https://doi.org/10.1016/j.radonc.2016.04.028
        • Saltelli A.
        • Annoni P.
        • Azzini I.
        • Campolongo F.
        • Ratto M.
        • Tarantola S.
        Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index.
        Comput Phys Commun. 2010; 181: 259-270https://doi.org/10.1016/j.cpc.2009.09.018
        • Kamp F.
        • Brüningk S.
        • Cabal G.
        • Mairani A.
        • Parodi K.
        • Wilkens J.J.
        Variance-based sensitivity analysis of biological uncertainties in carbon ion therapy.
        Phys Med. 2014; 30: 583-587https://doi.org/10.1016/j.ejmp.2014.04.008
        • Kamp F.
        • Wilkens J.J.
        Application of variance-based uncertainty and sensitivity analysis to biological modeling in carbon ion treatment plans.
        Med Phys. 2019; 46: 437-447https://doi.org/10.1002/mp.13306
        • Zvereva A.
        • Kamp F.
        • Schlattl H.
        • Zankl M.
        • Parodi K.
        Impact of interpatient variability on organ dose estimates according to MIRD schema: Uncertainty and variance-based sensitivity analysis.
        Med Phys. 2018; 45: 3391-3403https://doi.org/10.1002/mp.12984
        • Hofmaier J.
        • Dedes G.
        • Carlson D.J.
        • Parodi K.
        • Belka C.
        • Kamp F.
        Variance-based sensitivity analysis for uncertainties in proton therapy: A framework to assess the effect of simultaneous uncertainties in range, positioning and RBE model predictions on RBE-weighted dose distributions.
        Med Phys. 2021; 48: 805-818https://doi.org/10.1002/mp.14596
        • Warfield S.K.
        • Zou K.H.
        • Wells W.M.
        Simultaneous truth and performance level estimation (STAPLE): An algorithm for the validation of image segmentation.
        IEEE Trans Med Imaging. 2004; 23: 903-921https://doi.org/10.1109/TMI.2004.828354
        • Deasy J.O.
        • Blanco A.I.
        • Clark V.H.
        CERR: A computational environment for radiotherapy research.
        Med Phys. 2003; 30: 979-985https://doi.org/10.1118/1.1568978
        • Combs S.E.
        • Baumert B.G.
        • Bendszus M.
        • Bozzao A.
        • Brada M.
        • Fariselli L.
        • et al.
        ESTRO ACROP guideline for target volume delineation of skull base tumors.
        Radiother Oncol. 2021; 156: 80-94https://doi.org/10.1016/j.radonc.2020.11.014
        • Kristensen I.
        • Nilsson K.
        • Agrup M.
        • Belfrage K.
        • Embring A.
        • Haugen H.
        • et al.
        A dose based approach for evaluation of inter-observer variations in target delineation.
        Tech Innov Patient Support Radiat Oncol. 2017; 3-4: 41-47https://doi.org/10.1016/j.tipsro.2017.10.002
        • van de Water S.
        • van Dam I.
        • Schaart D.R.
        • Al-Mamgani A.
        • Heijmen B.J.M.
        • Hoogeman M.S.
        The price of robustness; impact of worst-case optimization on organ-at-risk dose and complication probability in intensity-modulated proton therapy for oropharyngeal cancer patients.
        Radiother Oncol. 2016; 120: 56-62https://doi.org/10.1016/j.radonc.2016.04.038
        • Tattenberg S.
        • Madden T.M.
        • Gorissen B.L.
        • Bortfeld T.
        • Parodi K.
        • Verburg J.
        Proton range uncertainty reduction benefits for skull base tumors in terms of normal tissue complication probability (NTCP) and healthy tissue doses.
        Med Phys. 2021; 48: 5356-5366https://doi.org/10.1002/mp.15097
        • Wagenaar D.
        • Kierkels R.G.J.
        • van der Schaaf A.
        • Meijers A.
        • Scandurra D.
        • Sijtsema N.M.
        • et al.
        Head and neck IMPT probabilistic dose accumulation: Feasibility of a 2 mm setup uncertainty setting.
        Radiother Oncol. 2021; 154: 45-52https://doi.org/10.1016/j.radonc.2020.09.001