Files
Download Full Text (502 KB)
DOI
https://doi.org/10.63853/PLRG5419
Description
Purpose: Organizational Health Literacy (OHL) refers to the ability of healthcare institutions to support patients and families to access, understand, and use health information. Nurses play a key role in OHL using evidence-based health literacy practices (i.e. plain language, teach-back, open-ended questions, chunking information, and three to five key points) when educating patient families. Yet, implementation is challenging. Audit and Feedback (A&F) is an implementation strategy for promoting fidelity to health literacy practices. Conventionally requiring expert human evaluations that are costly, labor-intensive, and difficult to scale.
Methods: HealthLit was trained using a dataset of simulated nurse-patient education encounters and applies natural language processing, machine learning, and retrieval-augmented generation to evaluate fidelity to health literacy practices. The team refined the model through HLEs’ feedback and prompt engineering, enhancing contextual understanding and precision. HealthLit’s performance was validated using an adapted AHRQ Communication Observation Form to rate the presence of plain language, teach-back, and open-ended questions on a 0-2 scale (0 = absent, 1 = partially present, 2 = present). Feedback Quality Evaluation assessed specificity, actionability, constructiveness, accuracy on a 0-2 scale, with higher scores indicated better feedback quality. A focus group of 11 clinical nurses with training in health literacy provided feedback on the perceived usefulness and acceptability of outputs for further refinement.
Results: A dataset of 33 nurse-patient transcripts was used for the model demonstrating moderate overall quality scores with a mean feedback quality score of 6.56/8 suggesting that while its feedback is somewhat helpful, it occasionally lacks details, specificity, and accuracy. The model demonstrated high specificity (m = 1.78), actionability (m = 1.81), and constructiveness (m = 1.89), but required improvement in accuracy over health literacy practices (m = 1.11). It was most accurate for open-ended questions (m = 1.22). It aligned closely with the health literacy expert evaluators across all health literacy practices in terms of actionability (m= 1.78 to 2) and constructiveness (m = 1.78 to 2). Focus group findings revealed key themes: positive response to HealthLit outputs and a desire for enhanced learning and analytics. Nurses recognized the usefulness in tracking health literacy fidelity and wished for case-based learning and aggregate clinic-level analytics.
Publication Date
11-24-2025
Disciplines
Pediatric Nursing
Recommended Citation
Howe, Carol J.; Feng, Yunhe; Fu, Song; Chen, Donger; and Patton, Lindsey, "HealthLitPro™- Development of an AI driven Audit & Feedback Tool for Organizational Health Literacy" (2025). 2025. 24.
https://scholarlycollection.childrens.com/nursing-anf2025/24
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.
