Reliability, replicability and reproducibility in PET imaging

Positron Emission Tomography (PET) is a non-invasive biomedical imaging method which can quantify biochemical markers as well as functional and metabolic activity in vivo. Following image analysis, quantification can be performed using various pharmacokinetic models, or using simplified semi-quantitative methods. There are numerous methods by which PET data can be analysed, and outcomes which can be reported, which differ in their accuracy, stability and specificity. The quantification of PET data, as well as the statistical procedures used to test clinical hypotheses, leads to conclusions which may differ in their degree of correctness. This thesis explores themes of reliability, replicability and reproducibility for PET research. Reliability concerns the consistency and accuracy of an outcome for distinguishing between individuals. Replicability concerns the accuracy of research conclusions, and whether they can be obtained again using the same procedures in new studies. Reproducibility concerns steps towards increasing the transparency of data analysis, by recording and sharing the exact procedures used to arrive at the conclusions. Across the studies in this thesis, these themes are detailed, expanded upon, and used in a series of methodological and applied clinical PET studies.

In Study I, the performance of surface-based methods for normalisation and smoothing of PET data were compared with volumetric methods for exploratory parametric analysis using PET test-retest data measuring [11C]SCH23390 BPND in cortical regions. We replicated previous results of decreased spread, and showed that these methods also show improved test-retest repeatability. In Study II, using the same data we evaluated the performance of post-reconstruction movement correction as well as automatic and manual methods for delineation of regions of interest. We showed that motion correction improves the reliability and repeatability of binding estimates, and that automatic methods for delineation do not perform less well than manual methods, and appear to be more consistent. Study III evaluated the test-retest performance of simplified ratio-based outcome measures for quantification of translocator protein (TSPO) binding using [11C]PBR28. We showed that these methods exhibit poor reliability, and little to no association with the gold-standard outcome measure VT, suggesting that caution is warranted for interpretation of studies making use of these measures.

In Study IV, diurnal and seasonal changes in the availability of the serotonin 1A receptor and the serotonin transporter were measured across the day and year in a large sample of healthy controls. We replicated previous findings of seasonal changes in the availability of the serotonin 1A receptor, failed to replicate findings of seasonal changes in the availability of the serotonin transporter, and additionally showed diurnal changes in both targets. In Study V, the importance of reliability is discussed with reference to study design, and a new method is presented for making approximations of the reliability for new samples. This approach allows researchers to more effectively gauge the feasibility of new between-individual studies before collection of any data, and to focus their efforts on research questions which can be expected to yield more interpretable outcomes. In Study VI, we perform a direction replication of a previous finding of a strong association between the Self-Transcendence scale of the Temperament and Character Inventory using a much larger sample to assess the veracity of the original findings. We showed moderate to strong evidence for no effect relative to the the previous results, suggesting that the original results were more likely to be either a false positive or greatly overestimated.

In Study VII, we carried out an individual-participant data meta-analysis of TSPO binding measured using second-generation tracers in healthy controls compared with schizophrenia and psychotic disorder patients. Contrary to the original hypothesis of increases in TSPO binding, we showed strong evidence for decreases in TSPO in patients compared to controls in both cortical and subcortical regions. In Studies VIII and IX, we hypothesised that D1 receptor binding would be higher with increasing proneness to develop psychosis and in the early stages of the disease prior to medication exposure, respectively. In Study VIII, we showed convergent evidence of no association between D1 receptor availability and delusional ideation in healthy controls. In Study IX, contrary to our hypotheses, we found moderate evidence in favour of lower levels of D1 receptor availability in the dorsolateral prefrontal cortex of first-episode drug-naive psychosis patients compared to healthy controls.

While reliability and replicablity of previous findings were directly assessed, the theme of reproducibility concerned our sharing the analysis code, and the data where possible, such that all analysis steps including those which could not be adequately described in the papers were recorded to ensure transparency, and demonstrate the correctness of our conclusions. All of the three themes of the thesis concern efforts to improve the quality, robustness, and utility of scientific research. For a field such as PET imaging, which is not only resource-intensive, but also requires exposing participants to harmful radiation, it is especially important both from a scientific as well as an ethical perspective that data are processed and analysed in a manner which is transparent, generalisable and optimal such that they are made use of to their full potential.

List of scientific papers

I. Matheson GJ, Stenkrona P, Cselényi Z, Plavén-Sigray P, Halldin C, Farde L & Cervenka S. (2017). Reliability of volumetric and surface-based normalisation and smoothing techniques for PET analysis of the cortex: A test-retest analysis using [11C]SCH-23390. NeuroImage. 155, 344-353.
https://doi.org/10.1016/j.neuroimage.2017.04.031

II. Stenkrona P, Matheson GJ, Cervenka S, Plavén-Sigray P, Halldin C & Farde L. (2018). [11C] SCH23390 binding to the D1-dopamine receptor in the human brain-a comparison of manual and automated methods for image analysis. EJNMMI research. 8, 74.
https://doi.org/10.1186/s13550-018-0416-2

III. Matheson GJ, Plavén-Sigray P, Forsberg A, Varrone A, Farde L & Cervenka S. (2017). Assessment of simplified ratio-based approaches for quantification of PET [11C] PBR28 data. EJNMMI research. 7, 58.
https://doi.org/10.1186/s13550-017-0304-1

IV. Matheson GJ, Schain M, Almeida R, Lundberg J, Cselényi Z, Borg J, Varrone A, Farde L & Cervenka S. (2015). Diurnal and seasonal variation of the brain serotonin system in healthy male subjects. NeuroImage. 112, 225-231.
https://doi.org/10.1016/j.neuroimage.2015.03.007

V. Matheson GJ. (2018). We need to talk about reliability: Making better use of test retest studies for study design and interpretation. bioRxiv. [Submitted]
https://doi.org/10.1101/274894

VI. Griffioen G, Matheson GJ, Cervenka S, Farde L & Borg J. (2017). Serotonin 5-HT1A receptor binding and self-transcendence in healthy control subjects-a replication study using Bayesian hypothesis testing. bioRxiv. [Submitted]
https://doi.org/10.1101/226092

VII. Plavén-Sigray P, Matheson GJ, Collste K, Ashok AH, Coughlin JM, Howes OD, Mizrahi R, Pomper MG, Rusjan P, Veronese M, Wang Y, & Cervenka S. (2018). Positron Emission Tomography Studies of the Glial Cell Marker Translocator Protein in Patients With Psychosis: A Meta-analysis Using Individual Participant Data. Biological psychiatry. 84, 433-442.
https://doi.org/10.1016/j.biopsych.2018.02.1171

VIII. Matheson GJ, Plavén-Sigray P, Louzolo A, Borg J, Farde L, Petrovic P & Cervenka S. (2018). Dopamine D1 receptor availability is not associated with delusional ideation measures of psychosis proneness. bioRxiv. [Submitted]
https://doi.org/10.1101/321646

IX. Stenkrona P, Matheson GJ, Halldin C, Cervenka S & Farde L. Reduced frontal dopamine D1-receptor binding in first episode drug naive patients with schizophreniform psychosis - a PET-study using [11C]SCH23390. [Submitted]