Automated Care Pathway Modeling Using Agentic and Knowledge-Aware LLMs

Abstract

Clinical pathways (CPWs) translate evidence-based guidance into stepwise care but are often disseminated as free text, obscuring control-flow semantics needed for clarity and computability. Formalizing CPWs as process models - e.g., in the Business Process Model and Notation (BPMN) - improves comprehensibility and enables downstream automation.

This thesis designs, implements, and evaluates LLM4CPW, a pipeline for automatic guideline-to-BPMN modeling using large language models (LLMs). We compare two contemporary frameworks - MAO (agentic, multi-role orchestration) and ProMoAI (single-agent with self-refinement loop) - under controlled execution with standardized evaluation. Automated metrics combine node-level and structural similarity (after Dijkman et al.) with graph-edit distance; a clinician provides fidelity ratings with qualitative annotations. We then investigate knowledge-aware modeling via a categorization of recurrent errors and curated clinical statements, testing prompt-only injections versus a dedicated Knowledge Advisor agent placed at different stages of the workflow.

Across four Ontario stroke Quality-Based Procedures (QBPs; n = 15 runs per framework), MAO attains higher node similarity (≈ 0.782 vs. ≈ 0.696) and structural similarity (≈ 0.630 vs. ≈ 0.585) than ProMoAI with large effects and p < 0.001, and exhibits markedly lower run-to-run variability; expert ratings also favor MAO. Knowledge-aware variants of MAO yield measurable gains: introducing a Knowledge Advisor after semantic review phase improves node similarity by > 4 points and reduces Graph Edit Distance by ∼ 13 (to∼ 97), with statistically comparable outcomes when placed before review; expert deltas likewise favor the advisor-based designs. Refining statement wording improves medians and interpretability without shifting means.

Contributions. This thesis contributes (i) an auditable LLM4CPW pipeline and evaluation protocol; (ii) empirical evidence that agentic orchestration improves fidelity and stability; and (iii) a principled, deployable strategy for knowledge-enhanced modeling via a specialized advisory phase. Collectively, the findings demonstrate the feasibility of reliable, automatically extracted BPMN models from concise clinical guidelines and chart a path toward broader, clinically grounded automation.