The utilization of natural language processing (NLP) to gauge healthcare decision-maker perceptions of cell and gene therapies (CGTs)
Breyanne Bannister, PharmD; Joseph Washington,
PharmD, MPH;
Claire Gorey, MA; Andrew Gaiser, PharmD, MBA,
MS; Justin
Merritt; Melissa McCart, PharmD, MS
Xcenda, L.L.C., Carrollton, TX, USA
Background
- Cell and gene therapies (CGTs) represent a rapidly evolving market with the potential to provide significant clinical and economic benefits to patients, caregivers, and society at large.
- However, high costs and clinical uncertainties have created varied healthcare decision-maker (HCDM) perceptions and barriers to coverage, reimbursement, and patient access.
- Methods to readily assess HCDM perceptions of CGTs are needed to facilitate relevant and timely discussions that enable biopharma companies to tailor evidence generation and market access strategies to meet the needs of HCDMs.
- Artificial intelligence (AI), such as natural language processing (NLP), is increasingly being utilized in healthcare to improve outcomes and care delivery.1,2 This includes evaluating large, unstructured datasets to inform real-time decision-making.
- FormularyDecisions® is an online platform connecting biopharma companies to their HCDM customers. HCDMs have access to a variety of resources to inform formulary decision making and the ability to provide valuable insights to biopharma companies through product-specific surveys.
- Sentiment analysis is an NLP technique that can be used to automatically determine whether qualitative data is negative, neutral, positive, or mixed.2
Objective
- To assess qualitative HCDM survey responses regarding CGTs using a Microsoft (MS) NLP sentiment analysis.
Methods
- Open-ended survey responses and accompanying numerical ratings from November 15, 2018 to May 31, 2022 were collected from FormularyDecisions to gather insights on the clinical efficacy and economic value of 3 FDA-approved CGTs with active subscriptions.
- Survey respondents included HCDMs from managed care organizations (MCOs), pharmacy benefit managers (PBMs), academic institutions, provider organizations, consultant agencies, government, and other organizations who were verified to have a role in the formulary decision-making process.
- The AI Builder capability from
the MS Power Platform was used to generate an NLP sentiment
analysis and automatically process the open-ended survey
responses.
- Responses were categorized into the following sentiment valence types: positive, negative, neutral, mixed.
- Descriptive statistics were used to evaluate categorical trends. Results are reported in aggregate.
- To validate the AI's categorization of sentiments into valence categories, numerical ratings for clinical efficacy and economic value by sentiment valence type were assessed through a one-way analysis of variance (ANOVA).
Results
Respondent demographics
- There were 563 total survey responses; 280 for clinical efficacy and 283 for economic value. Respondents were primarily from managed care (247; 43.9%), provider (149; 26.4%), and PBM (120; 21.3%) organizations (Figure 1).
Sentiment analysis
- There was no substantial difference among positive, neutral, or negative sentiment across the 3 products. The distribution of overall sentiment was slightly more negative (40.3%) and neutral (28.6%) (Figure 2).
- Sentiment for clinical efficacy was 37.1% neutral (Figure 3), while 55.5% of responses for economic value were classified as negative (Figure 4).
- Overall sentiment was most positive across MCOs (28.7%) and provider (22.8%) organizations, and most negative across other (50.5%) and PBM organizations (44.2%) (Figure 5).
- In general, sentiment
was more positive for clinical efficacy and more
negative for economic value across all organization
types.
- For clinical efficacy, positive sentiment was highest among MCOs (38.5%) and PBMs (28.3%), while negative sentiment was highest among other (38.6%) and PBM organizations (33.3%) (Figure 6).
- Positive sentiment for economic value was highest among provider organizations (23.0%) followed by MCOs (19.2%). All organization types had a largely negative sentiment of at least 40.0% (Figure 7).
Figure 1: Count of survey responses by organization type
N=563.
aIncludes respondents from organizations such as academia, government, and consultants.
Key: MCO – managed care organization; PBM – pharmacy benefit manager.
aIncludes respondents from organizations such as academia, government, and consultants.
Key: MCO – managed care organization; PBM – pharmacy benefit manager.
Sentiment analysis
- There was no substantial difference among positive, neutral, or negative sentiment across the 3 products. The distribution of overall sentiment was slightly more negative (40.3%) and neutral (28.6%) (Figure 2).
- Sentiment for clinical efficacy was 37.1% neutral (Figure 3), while 55.5% of responses for economic value were classified as negative (Figure 4).
- Overall sentiment was most positive across MCOs (28.7%) and provider (22.8%) organizations, and most negative across other (50.5%) and PBM organizations (44.2%) (Figure 5).
- In general, sentiment
was more positive for clinical efficacy and more
negative for economic value across all organization
types.
- For clinical efficacy, positive sentiment was highest among MCOs (38.5%) and PBMs (28.3%), while negative sentiment was highest among other (38.6%) and PBM organizations (33.3%) (Figure 6).
- Positive sentiment for economic value was highest among provider organizations (23.0%) followed by MCOs (19.2%). All organization types had a largely negative sentiment of at least 40.0% (Figure 7).
Use the filter above to view
results by overall sentiment, sentiment for clinical
efficacy, or sentiment for economic value.
Figure 5: Distribution of overall sentiment by organization typea
N=563. Note: The sum of percentages for each
organization type may not equal 100% due to
rounding. The percentages for other organization
types reflect an average across multiple
organizations; therefore, the sum of percentages
may be greater than 100%.
aFigure reflects aggregate sentiment data for clinical efficacy and economic value. Other organization types include respondents from organizations such as academia, government, and consultants.
Key: MCO – managed care organization; PBM – pharmacy benefit manager.
aFigure reflects aggregate sentiment data for clinical efficacy and economic value. Other organization types include respondents from organizations such as academia, government, and consultants.
Key: MCO – managed care organization; PBM – pharmacy benefit manager.
Figure 6: Distribution of sentiment for clinical efficacy by organization type
n=280. Note: The sum of percentages for each
organization type may not equal 100% due to
rounding. The percentages for other organization
types reflect an average across multiple
organizations; therefore, the sum of percentages
may be greater than 100%.
aIncludes respondents from organizations such as academia, government, and consultants.
Key: MCO – managed care organization; PBM – pharmacy benefit manager.
aIncludes respondents from organizations such as academia, government, and consultants.
Key: MCO – managed care organization; PBM – pharmacy benefit manager.
| Sample open-ended responses by sentiment typea | |||
|---|---|---|---|
| Negative | Neutral | Positive | Mixed |
| "Not enough real-world
or long-term data to
speak to
[Product X’s] effectiveness." – PBM organization |
"Although there are likely some
treatment
differences, both products are
ranked
equivalently
in [clinical guidelines]. I
therefore
ranked
them
the same." – PBM organization |
"Clinical efficacy from the initial
studies
are
positive and, fortunately, there’s
been
some
good
long-term studies to show continuing
benefit.
Specifically, data presented at [X
annual
meeting]
has shown that nearly X% were still
alive at
X
years
which is a great finding considering
the
severity of
this population." – MCO |
"Efficacy looks good from initial
trials
and
then
some of the longer-term data at 2 to
3
years
out.
However, safety is a notable
concern." – MCO |
| "[Product X] use entails ongoing
treatment
administered through [X route] (>30
over
a
10-year period) resulting in
substantial
lifetime healthcare costs and
considerable
burden to patients, caregivers, and
the
healthcare system, as well as a risk
of
procedure-related
complications." – Other organization |
"Data on long-term durability of
effect
is still pending. Efficacy is based
on a
similar magnitude of effect and
endpoints as [Product X] and
[Product
Y]." – MCO |
"Clinical studies have shown
significant
efficacy, and the field of oncology
is
very
excited for the impact [CGT] will
have
on
the future of care." – Provider organization |
"The clinical trial for this therapy
has
shown outstanding results. However,
there is
still limited information available
on
overall clinical efficacy for
patients
with
[disease X] and the overall
durability
of
the therapy." – Other organization |
aOpen-ended responses have been
blinded to product mentions or product-specific
characteristics. Other organization types
include respondents from organizations such as
academia, government, and consultants.
Figure 7: Distribution of sentiment for economic value by organization type
n=283. Note: The sum of percentages for each
organization type may not equal 100% due to
rounding. The percentages for other organization
types reflect an average across multiple
organizations; therefore, the sum of percentages
may be greater than 100%.
aIncludes respondents from organizations such as academia, government, and consultants.
Key: MCO – managed care organization; PBM – pharmacy benefit manager.
aIncludes respondents from organizations such as academia, government, and consultants.
Key: MCO – managed care organization; PBM – pharmacy benefit manager.
| Sample open-ended responses by sentiment typea | |||
|---|---|---|---|
| Negative | Neutral | Positive | Mixed |
| "The cost is a significant challenge
for
payers. Not just the dollar amount,
but
also the [administration] creates a
situation where payer assumes all
financial risk without guarantee of
future premiums." – MCO |
"As I think the clinical aspects of
the
products are similar and the cost is
the
same, I view the total economic
value as
equal." – Provider organization |
"While the data shows pretty similar
effects to other therapies approved,
my
opinion is that [Product X] is just
a
notch more valuable than the others
because it is able to be delivered
to
patients [aged X]. The younger these
patients, the fewer the
complications,
and the more they stand to
benefit." – Provider organization |
"Cost of [Product X] is incredibly
high
up-front. Although in the long term
you
could likely make a good argument
over
this product being cost-effective
the
up-front hit is a big one." – MCO |
| "All are expensive, but [Product X]
is
the most costly for us." – Other organization |
Specific economic data is not
readily
available; however, the expectation
is
that the drug could reach upwards of
[X
price] per patient. – PBM organization |
Compared to Product X, [Product Y]
does
not require regular [X route]
administration. [Product Y] is a [X
therapy] which is favorable for
patient
quality of life. Additionally,
roughly 4
years after receiving [Product Y],
overall drug costs begin to be lower
than that of [Product X]. This
product
has a real potential to decrease
medical
costs and complications compared to
other comparators. – PBM organization |
"Based on the estimated price tag of
[Product X], I cannot give it more
economic value than [Product Y].
[Product Y] is highly effective
across
all [disease X] and has a plethora
of
real-world data. Although highly
priced,
it would take years of [Product Y]
to
come close to the estimated cost of
[Product X]. Patients may not even
have
the same insurance [in the] long
term." – Other organization |
aOpen-ended responses have been
blinded to product mentions or other
product-specific characteristics. Other
organization types include respondents from
organizations such as academia, government, and
consultants.
One-way ANOVAs suggested there was no statistically
significant difference in numerical clinical efficacy
ratings across sentiment valence types; however, there was a
significant difference for economic value ratings.
Post hoc analyses were conducted to further evaluate the
significant difference in economic value ratings. Pairwise
comparisons suggested that economic value ratings were
significantly higher for neutral and positive sentiment
valence types compared to negative types (p < .05; p <
.001).
Figure 8: Clinical efficacy ratings by sentiment type
n=280. Note: Standard error bars reflect how reliable
the sample means are for each sentiment type; smaller
standard errors suggest that the sample mean is a more
accurate reflection of the true population mean.
Standard error of the mean was calculated by dividing
the standard deviation for each sentiment type by the
root of the sample size for that sentiment type.
Figure 9: Economic value ratings by sentiment type
n=283. Note: Standard error bars reflect how reliable
the sample means are for each sentiment type; smaller
standard errors suggest that the sample mean is a more
accurate reflection of the true population mean.
Standard error of the mean was calculated by dividing
the standard deviation for each sentiment type by the
root of the sample size for that sentiment type.
Limitations
- Results were reported in aggregate and may not fully represent HCDM sentiment regarding any particular product. As this research only evaluated survey responses for 3 CGT products, caution should be used in generalizing the results to all CGTs.
- Inherent limitations of the sentiment analysis technology, including inaccuracies due to insufficiently labeled data or complex sentences, should be considered when reviewing the results.
- This research reflects the perspectives of HCDMs identified from users of FormularyDecisions; other user types (eg, patients, manufacturers) were not represented in this survey.
- The respondent sample had greater representation from MCOs, PBMs, and provider organizations, which could affect generalizability of the results across all types of organizations and HCDMs.
- Because all respondents voluntarily completed the survey, voluntary response bias may exist, and survey results may overrepresent respondents with stronger interest in payer-manufacturer partnerships.
Conclusions
- The results suggest that NLP sentiment analyses have utility in rapidly evaluating qualitative HCDM data.
- Sentiment for the 3 CGTs supports that there are varied perceptions across HCDMs regarding clinical efficacy and economic value and that these insights may vary by organization type.
- While overall sentiment varied across the evaluated CGTs, sentiment was slightly more negative and neutral compared to positive. When evaluating clinical efficacy, providers and MCOs appear to have a relatively more positive sentiment, while MCOs, PBMs, and provider perceptions of economic value appear to be largely negative.
- As HCDMs continue to provide feedback for CGTs, manufacturers should consider innovative methods for timely and targeted assessments to identify and address barriers that may impact successful commercialization and patient access.
References
- Bohr A, Memarzadeh K. The rise of artificial intelligence in healthcare applications. In: Bohr A, Memarzadeh K, eds. Artificial Intelligence in Healthcare. Academic Press; 2020;25-60.
- Zunic A, Corcoran P, Spasic I. Sentiment analysis in health and well-being: systematic review. JMIR Med Inform. 2020;8(1):e16023.
Presented at: AMCP 2023 Annual Meeting, March 21-24, 2023;
San
Antonio, Texas.
Direct questions to Breyanne Bannister at Breyanne.Bannister@xcenda.com
This research was funded by Xcenda.
Direct questions to Breyanne Bannister at Breyanne.Bannister@xcenda.com
This research was funded by Xcenda.