Clinical documents play a pivotal role in patient care, research, and Health Information Management (HIM). At Intermountain Health, we present a comprehensive approach to unlocking valuable insights from these documents through Natural Language Processing (NLP) on the Databricks Lakehouse Platform. Our strategy involves leveraging advanced Electronic Health Records (EHRs) management, processing, and analysis to transform hundreds of millions of unstructured clinical documents into precise, accessible, and collaborative resources. This not only aids in guiding medical decisions but also enhances patient care and accelerates research endeavors.

Our workflow begins with decompressing raw files and storing them as delta tables, followed by the application of John Snow Labs’ medical NLP libraries to deidentify Personally Identifiable Information (PII). The resulting data is then made available for research purposes and shared with Intermountain’s research subsidiaries. Next, we employ medical text summarization models to create synopses of the notes.

The process reduces review time from 10 minutes to 3 minutes per document. Finally, we utilize Generative AI applications for seamless querying of the database using natural language, facilitating a more intuitive and user-friendly interaction with the wealth of information stored within the clinical documents. This integrated approach not only enhances the efficiency of research initiatives but also contributes to the overall improvement of healthcare outcomes.