Predictive modeling of psychiatric symptom outcomes in psychological treatment : a methodological evaluation

Predicting treatment outcome at an early stage of psychological treatment can benefit patients by informing clinicians and allowing them to make adaptations. Yet, the predictive performance of these models is unclear despite a renewed interest during the last decade. The methodological issues regarding the development of predictive models for clinical use remain pervasive, making it hard to draw conclusions about their useability or performance for treatment outcomes after psychological treatment.

The aim of this thesis was to assess and increase performance of predictive models for symptom outcome after psychological treatment, using data from internet-delivered cognitive behavioral therapy in routine psychiatric care. This was done by exploring different: methodologies of validation, predictors, and predictive models.

In Study I, we investigated a multitude of ways to develop predictive models. Predictive performance: increased linearly up to the final prediction timepoint of day 28, was maximised by pooling all treatment conditions, was increased by imputing data, was best with symptom variable predictors, and showed minimal differences across models. The linear regression with the benchmark set of predictors had a balanced accuracy of 77.8% and an R2 of 0.538. This study had a high risk of overfitting due to the imputation and validation procedure.

In Study II, we investigated if predictive models which account for the time- dependency in symptom-ratings are superior to methods that do not. There was no differential effect depending on model type. The predictive performance for the last timepoint, which was treatment outcome, had a RMSE across models of 0.121, and a balanced accuracy of 73.46%, with a low risk of overfitting due to nested cross-validation and multiple imputation.

In Study III, we investigated whether the use of natural language processing methods for the patient-therapist text interactions could improve predictions, compared to a simple symptom model. For each tested text-model, an additional was trained which modified the model to also incorporate the symptom variables. Linear regression using only symptom variables had the best predictive performance for both an RMSE of 0.14, and a balanced accuracy of 70%. Due to large variation across imputation datasets, for the RMSE metric, only the text model BERT, with an RMSE of 0.17, was significantly better than the dummy regression with 0.18. Adding symptom variables to the text models only significantly improved the BERT model from 60% to 68% balanced accuracy, and only for this metric.

In Study IV, we investigated whether a psychometric improvement of symptom variables using Rasch measurement theory affected predictive performance, given the strong influence of the quality of a measurement for predictive models. A dataset for each datatype was created, 'Base' using the original scales, and 'Rasch' with the psychometrically improved version. For linear regression the Base dataset had an RMSE of 0.132, and Rasch had an RMSE of 0.139. It is inconclusive if Rasch improved precision and reduced overfitting, or if the Base dataset retained useful variability.

There are several important methodological challenges for valid predictive models for treatment outcome after psychological treatment, including a lack of evidence for their clinical utility and generalizability. While models can predict with a potentially useful performance, the more important question is what they are predicting and if this performance results in a clinically favorable outcome. Once psychology is rooted in stronger foundations, the potential benefits of predictive modeling will be much easier to realise.

List of scientific papers

I. Hentati Isacsson, N., Ben Abdesslem, F., Forsell, E., Boman, M., & Kaldo, V. (2024). Methodological choices and clinical usefulness for machine learning predictions of outcome in Internet-based cognitive behavioural therapy. Communications Medicine. 4(1), 196. https://doi.org/10.1038/s43856-024-00626-4

II. Hentati Isacsson, N., Zantvoort, K., Forsell, E., Boman, M., & Kaldo, V. (2024). Making the most out of timeseries symptom data: A machine learning study on symptom predictions of internet-based CBT. Internet Interventions. 38, 100773. https://doi.org/10.1016/j.invent.2024.100773

III. Hentati Isacsson, N., Gómez-Zaragozá, L., Ben Abdesslem, F., Boman, M., & Kaldo, V. (2025). Natural Language Processing Models for Predicting Treatment Outcomes in Internet-Based Cognitive Behavioural Therapy. [Manuscript]

IV. Hentati Isacsson, N., Johansson, M., & Kaldo, V. (2025). Latent Trait or Sum Score: Addressing Measurement Challenges in the Prediction of Self-Rated Symptom Outcomes in Psychological Treatment. [Manuscript]