Adaptive treatment strategies in internet-delivered cognitive behavior therapy : predicting and avoiding treatment failures

Background: Internet-delivered Cognitive Behavior Therapy (ICBT) is efficacious for a number of psychiatric disorders and can be successfully implemented in routine psychiatric care. Still, only about half of patients experience a good enough treatment outcome. Using data from the early part of treatment to identify patients with high risk of not benefitting from it, and target them with additional resources to prevent the predicted failure is a potential way forward. We call this an Adaptive Treatment Strategy, and a very important part of it is the ability to predict the outcome for a specific patient.

Aims: To establish a proof of concept for an Adaptive Treatment Strategy in ICBT, and explore outcome prediction further by evaluating the accuracy of an empirically supported classification algorithm, the time point in treatment when acceptable accuracy can be reached, and the accuracy of ICBT-therapists’ own predictions. Preliminary benchmarks regarding the clinical usefulness of prediction will be established.

Studies: Four studies were performed: Study I was a randomized controlled trial (RCT; n=251) where patients’ risk of treatment Failure (Red=high risk of failure, Green=low risk) was predicted during week 4 out of 9 in ICBT for Insomnia. Red patients (n=102) were then randomized to either continuing with standard treatment (n=51) or having their treatment individually adapted (n=51). In Study II, the classification algorithm from Study I was evaluated in terms of classification accuracy and the contribution of the different predictors used. In Study III, data from 4310 regular care ICBT-patients having received treatment for either Depression, Social anxiety disorder or Panic disorder were analyzed in a series of multiple regression models using weekly observations of the primary symptom measure as predictors to classify risk of Failure. As a contrast, Study IV examines ICBT therapists’ own predictions on both categorical and continuous treatment outcomes, as they made predictions for each of their patients (n=897) during week 4 in the same three treatments as in Study III.

Results: The RCT was successful in that Red patients receiving Adapted treatment improved significantly more than Red patients receiving standard treatment, and their odds of failure were nearly cut in half. Green patients did better than Red patients, indicating that the accuracy of the classification algorithm was clinically useful. Study II showed that the balanced accuracy of the classifier was 67% and that only 11 of 21 predictors correlated significantly with Failure. Notable predictors were symptom levels as well as different markers of treatment engagement. Study III and IV showed that acceptable predictions could be made halfway through treatment using only symptom scores and basic statistics, and that ICBT-therapists predicted outcomes better than chance but on average 9.5 % less accurate than the statistical models. Therapist predictions reached the clinical acceptance benchmark only for remission in Social anxiety disorder. At treatment week four, therapist could predict on average 16% of the variance in continuous outcomes, compared to a statistical model explaining 39%.

Conclusions: We find support for the clinical usefulness of an Adaptive Treatment Strategy in ICBT for insomnia, and establish a preliminary benchmark that a classification algorithm with at least 67% balanced accuracy should be sufficient for clinical purposes. Simple statistical models using only symptom scores can reach clinically acceptable levels of accuracy halfway through 12-week ICBT-programs. Previous findings that therapists’ predictions are less accurate than statistical models seem to hold also for therapists providing ICBT. However, it was also indicated that clinicians’ ratings of adherence and activity do add unique information to prediction algorithms. In line with previous findings, the vast majority of useful prediction variables were found during early treatment, rather than before treatment start.

List of scientific papers

I. Forsell, E., Jernelöv, S., Blom, K., Kraepelien, M., Svanborg, C., Andersson, G., Lindefors, N. & Kaldo, V. (2019). Proof of Concept for an Adaptive Treatment Strategy to Prevent Failures in Internet-Delivered CBT: A Single-Blind Randomized Clinical Trial With Insomnia Patients. American Journal of Psychiatry. 176(4), 315-323.
https://doi.org/10.1176/appi.ajp.2018.18060699

II. Forsell, E., Jernelöv, S., Blom, K. & Kaldo, V. Clinically sufficient classification accuracy and key predictors of treatment failure in a randomized controlled trial of Internet-delivered Cognitive Behavior Therapy for Insomnia. [Submitted]

III. Forsell, E., Isacsson, N., Blom, K., Jernelöv, S., Ben Abdesslem, F., Lindefors, N., Boman, M. & Kaldo, V. (2019). Predicting treatment failure in regular care Internet-Delivered Cognitive Behavior Therapy for depression and anxiety using only weekly symptom measures. Journal of Consulting and Clinical Psychology. 88(4):311-321.
https://doi.org/10.1037/ccp0000462

IV. Forsell, E., Mattsson, S. & Kaldo, V. Accuracy of therapists’ predictions of outcome in Internet delivered Cognitive Behavior Therapy for depression and anxiety in routine psychiatric care. [Submitted]