Evidence-Grounded Multimodal Knowledge Graph Construction for Multi-Lecture Educational Reasoning

# Evidence-Grounded Multimodal Knowledge Graph Construction for Multi-Lecture Educational Reasoning
Source: [https://arxiv.org/html/2608.03161](https://arxiv.org/html/2608.03161)
###### Abstract

Lecture videos distribute knowledge across speech, slide text, diagrams, equations, and presentation order, which transcript\-only retrieval does not fully preserve\. This paper presents an evidence\-grounded multimodal pipeline that transcribes lectures, selects semantic anchors, applies optical character recognition \(OCR\), and uses a vision\-language model to extract only concepts and typed relationships supported by transcript, OCR, or visual evidence\. Mentions are validated and canonicalized into a provenance\-rich knowledge graph\. On three neural\-network lectures, the pipeline processed 3,118 frames, 756 transcript segments, and 559 anchors\. It retained 1,022 concept and 312 relationship mentions, yielding 172 canonical concepts and 282 relationships with 90\.38% endpoint coverage\. A preliminary three\-question retrieval test achieved 100% top\-1 and top\-3 accuracy and 100% mean top\-5 recall\. The contribution is an auditable construction method rather than a state\-of\-the\-art performance claim\.

## IIntroduction

Educational concepts may be spoken, written on slides, drawn in diagrams, represented in equations, and revisited across lectures\. Students integrate these signals to connect definitions, examples, and prerequisites\. Automated lecture question answering therefore needs a representation that retains concepts, relationships, evidence, and temporal context rather than only flat transcript chunks\.

Retrieval\-augmented generation \(RAG\) grounds language models in external context, but conventional transcript RAG is weakest on explicit dependencies, concept evolution, and visually communicated information\. A knowledge graph \(KG\) can instead represent concepts as nodes and typed relations as edges while retaining the lecture, timestamp, frame, and evidence quotation that justify each extraction\. Such auditability is important in education because unsupported claims can mislead learners and instructors need to inspect or correct extracted knowledge\.

We present an end\-to\-end pipeline combining timestamped automatic speech recognition \(ASR\), high\-recall anchor selection, OCR, grounded vision\-language extraction, validation, canonicalization, KG construction, and GraphRAG\-style retrieval\. The design constraint is simple: a concept or relationship is promoted only when tied to lecture evidence\. The study uses three conceptually connected 3Blue1Brown lectures on neural networks, gradient descent, and backpropagation\.

The contributions are: \(1\) a multimodal lecture\-to\-KG workflow; \(2\) an extraction schema retaining evidence, confidence, modality, lecture, timestamp, frame, and anchor identifiers; \(3\) a provenance\-rich graph with 172 canonical concepts and 282 relationships; \(4\) hybrid semantic, lexical, fuzzy, and evidence\-aware retrieval; and \(5\) reproducible figures and tables derived from the final run\.

## IIRelated Work

### II\-ARetrieval\-Augmented and Graph\-Structured Generation

Lewis*et al\.*introduced RAG for knowledge\-intensive natural\-language processing and showed the value of conditioning generation on retrieved documents\[[1](https://arxiv.org/html/2608.03161#bib.bib1)\]\. Subsequent surveys have organized the fast\-growing RAG literature around indexing, retrieval, and generation stages and have consistently identified hallucination and outdated internal knowledge as central failure modes that external retrieval is meant to address\[[12](https://arxiv.org/html/2608.03161#bib.bib12)\]\. GraphRAG extends this motivation with entity, relation, and community indexes for corpus\-level questions that flat chunk retrieval can miss\[[2](https://arxiv.org/html/2608.03161#bib.bib2)\]\. More recent work generalizes GraphRAG\-style retrieval to non\-textual evidence: multimodal RAG surveys report that grounding generation in images, video, and other modalities alongside text further reduces hallucination relative to text\-only retrieval, particularly when visual and textual cues are jointly necessary to answer a question\[[11](https://arxiv.org/html/2608.03161#bib.bib11)\]\. Our work applies graph\-structured retrieval to multimodal lectures and, unlike most RAG systems that retrieve opaque passages, retains source evidence for every extracted item so that each answer can be traced to a specific transcript span, OCR string, or frame\.

### II\-BMultimodal Knowledge Graph Construction

Multimodal knowledge graphs \(MMKGs\) integrate text, images, and other modalities into a single structured representation to support cross\-modal reasoning\. Zhu*et al\.*survey MMKG construction and completion built primarily from text–image pairs and highlight symbol grounding—linking a textual entity to its supporting visual evidence—as a central open problem\[[6](https://arxiv.org/html/2608.03161#bib.bib6)\]\. Large language models have since been used to automate parts of this pipeline\. A recent survey of LLM\-empowered KG construction describes how vision\-language models can be cascaded to translate visual features into text before extraction and catalogs LLM\-driven entity\-fusion and canonicalization methods that merge duplicate mentions using embedding similarity rather than hand\-written rules\[[7](https://arxiv.org/html/2608.03161#bib.bib7)\]\. This strategy is conceptually similar to the alias, fuzzy\-matching, and embedding\-based merging used in our canonicalization stage\.

Video\-native MMKGs such as Kuaipedia link entities to short\-video evidence at scale but are built for open\-domain, single\-clip content rather than multi\-lecture instructional material with typed pedagogical relations such asprerequisite\_oforcomputed\_by\[[15](https://arxiv.org/html/2608.03161#bib.bib15)\]\. Our pipeline differs by constraining extraction to a fixed, education\-oriented relation vocabulary and requiring every mention to cite the transcript, OCR, or visual evidence that supports it before entering the graph\.

### II\-CEducational Knowledge Graphs

Educational knowledge graphs \(EduKGs\) structure course concepts and their relationships to support curriculum design, prerequisite discovery, and personalized learning\. A systematic review of KG construction and application in education finds that most existing EduKGs are built from static, already\-structured materials such as textbooks, syllabi, or course descriptions, and identifies automatic construction directly from raw instructional content as an open challenge\[[10](https://arxiv.org/html/2608.03161#bib.bib10)\]\. Work on the CourseMapper platform compares top\-down \(ontology\-first\) and bottom\-up \(extraction\-first\) strategies for automatically building EduKGs from lecture slides and later reports an optimized pipeline that improves concept\-extraction accuracy and processing efficiency over the initial approach\[[8](https://arxiv.org/html/2608.03161#bib.bib8),[9](https://arxiv.org/html/2608.03161#bib.bib9)\]\. These efforts operate on slide text or transcripts alone\. In contrast, our pipeline treats video frames, OCR output, and transcript segments as three simultaneous evidence sources for the same anchor, allowing it to capture diagram\- and equation\-only content that slide\- or transcript\-only EduKG pipelines cannot see\.

### II\-DSpeech, Vision, and Text Recognition Components

Lecture processing depends on robust ASR\. Whisper demonstrated strong generalization from large\-scale weak supervision\[[3](https://arxiv.org/html/2608.03161#bib.bib3)\]; we use Faster\-Whisperlarge\-v3and align each timestamped segment with a primary frame\. For joint interpretation of transcript, slide text, diagrams, and equations, we use Qwen2\.5\-VL\[[4](https://arxiv.org/html/2608.03161#bib.bib4)\]under a constrained JSON extraction prompt rather than as an unconstrained generator\. Because slide and diagram text is useful only when read correctly, OCR acts as a second visual\-evidence channel\. Surveys of deep\-learning\-based scene\-text recognition describe how cluttered backgrounds, varied fonts, and non\-frontal capture make this setting harder than scanned\-document OCR\[[13](https://arxiv.org/html/2608.03161#bib.bib13)\], matching the noisy, camera\-captured slide frames encountered in lecture video\. Canonical graph text is embedded with BGE\-large English\[[5](https://arxiv.org/html/2608.03161#bib.bib5)\]\.

### II\-EVideo Question Answering

Because the graph is intended to answer lecture questions, our retrieval stage relates to video question answering \(VideoQA\), which pairs video featurization, question featurization, and joint embedding to produce an answer directly from raw video\[[14](https://arxiv.org/html/2608.03161#bib.bib14)\]\. VideoQA systems typically reason over a fixed clip without an explicit intermediate knowledge representation, so they cannot easily aggregate evidence for a single concept across multiple, separately recorded lectures or expose why a particular answer was produced\. Our graph\-grounded retrieval first resolves a question to canonical concepts and their one\-hop neighborhood, then generates an answer only from retrieved definitions, relationships, and evidence\. This keeps reasoning inspectable and reusable across lectures rather than tied to a single video pass\. Unlike educational KGs manually curated from textbooks or structured materials, the proposed graph is built from raw videos while preserving inspectable multimodal provenance\.

## IIIMethod

### III\-APipeline and Alignment

Fig\.[1](https://arxiv.org/html/2608.03161#S3.F1)summarizes the workflow\. Videos are downloaded, frames are sampled at 1 frame/s, and audio is converted to a 16\-kHz mono waveform\. Faster\-Whisper produces segments with start time, end time, and text; the segment midpoint identifies its primary frame\. This synchronization lets transcript and visual evidence share a temporal neighborhood\.

![Refer to caption](https://arxiv.org/html/2608.03161v1/figures/Im1.png)Figure 1:Evidence\-grounded multimodal KG construction pipeline\.
### III\-BSemantic Anchors and OCR

Running a vision\-language model \(VLM\) on every frame is expensive and redundant\. A recall\-oriented selector combines visual\-change, transcript\-keyword, relationship\-cue, and first\-mention scores\. It targets approximately 18% of frames with temporal spacing to suppress near\-duplicates\. Each anchor receives transcript context from 15 s before through 22 s after its timestamp\. EasyOCR processes the anchor frame and contributes slide labels, diagram annotations, symbols, and equations that may not be spoken\.

### III\-CGrounded Extraction and Validation

For every anchor, Qwen2\.5\-VL receives the frame, transcript window, OCR text, and locally derived candidate terms\. It returns strict JSON arrays for concepts and relationships; empty arrays are permitted\. A concept records name, definition, evidence quotation, source modality, and confidence\. A relationship records source concept, target concept, one of eight permitted types \(prerequisite\_of,component\_of,uses,optimizes,computed\_by,example\_of,contrasts\_with, orrelated\_to\), evidence quotation, source modality, and confidence\.

Validation removes missing or low\-information fields, unsupported evidence, invalid relation types, and confidence below 0\.55\. Transcript\- and OCR\-sourced claims must occur in the evidence pool; visual\-only concepts require stronger confidence\. Validation improves auditability but does not by itself guarantee factual correctness\.

### III\-DCanonicalization and Graph Construction

Validated mentions are merged using aliases, normalized and fuzzy string matching, token overlap, and embedding similarity\. Each canonical node retains its identifier, display name, aliases, definitions, mention list, lecture coverage, evidence count, and average confidence\. Relationship endpoints are mapped only after concept deduplication; this order prevents otherwise valid edges from being lost when raw endpoint names differ from canonical names\.

The result is a NetworkXMultiDiGraph\. Nodes are canonical concepts and edges are typed relationship instances\. Edge metadata includes relation type, lecture, timestamp, anchor, evidence quotation, and confidence\. Multiple edge instances between the same nodes are allowed when supported by different evidence\.

### III\-EGraph\-Grounded Retrieval and QA

Canonical concept text combines name, definition, aliases, and evidence snippets and is embedded using BGE\-large English\. The retrieval score combines semantic similarity, exact\-name and alias matches, fuzzy similarity, and an evidence\-count prior\. The top six concepts are expanded to a one\-hop subgraph\. The answer prompt receives only formatted definitions, relationships, and evidence and requests lecture identifiers and timestamps when available\.

## IVExperimental Setup

### IV\-ADataset and Configuration

The evaluation covers three lectures on neural networks, gradient descent, and backpropagation\. Table[I](https://arxiv.org/html/2608.03161#S4.T1)reports processing and OCR coverage; Fig\.[2](https://arxiv.org/html/2608.03161#S4.F2)visualizes per\-lecture volume\. The run used Faster\-Whisperlarge\-v3, EasyOCR, Qwen2\.5\-VL\-7B\-Instruct, BGE\-large\-en\-v1\.5 embeddings, 1\-frame/s sampling, an 18% anchor target, and 0\.55 minimum concept and relation confidence\.

TABLE I:Dataset, Anchor, and OCR Summary![Refer to caption](https://arxiv.org/html/2608.03161v1/figures/Im2.png)Figure 2:Per\-lecture frame, transcript\-segment, anchor, and OCR counts\.TABLE II:Models and Processing ConfigurationThe overall anchor and non\-empty OCR rates were 17\.93% and 94\.63%, respectively\. Reported metrics cover frames, transcripts, anchors, OCR, raw and validated mentions, canonical nodes, graph edges, endpoint coverage, evidence density, and retrieval\. Retrieval uses only three seed questions and is a sanity check, not a final benchmark\.

## VResults

### V\-AExtraction Yield

The VLM returned 1,155 raw concept and 400 raw relationship mentions\. Evidence and confidence checks retained 1,022 concepts \(88\.48%\) and 312 relationships \(78\.00%\)\. Canonicalization produced 172 concepts, and endpoint mapping retained 282 edges \(90\.38% of validated relationships\), as summarized in Table[III](https://arxiv.org/html/2608.03161#S5.T3)and Fig\.[3](https://arxiv.org/html/2608.03161#S5.F3)\. The reduction from 1,022 mentions to 172 nodes is expected because course concepts recur across anchors and lectures\.

TABLE III:Extraction and Graph\-Construction Yield![Refer to caption](https://arxiv.org/html/2608.03161v1/figures/Im3.png)Figure 3:Yield from raw mentions to validated and canonical graph items\.
### V\-BKnowledge Graph Statistics

The graph contains 172 canonical concepts and 282 relationships, with 5\.94 evidence mentions per concept on average\. Fig\.[4](https://arxiv.org/html/2608.03161#S5.F4)shows the constructed graph\. Canonical concepts are represented as nodes and extracted typed relationships as edges\. Dense clusters indicate neural\-network concepts recurring across lectures and anchors, whereas peripheral nodes represent lower\-frequency or localized evidence\.

TABLE IV:Graph and Starter\-Retrieval Summary![Refer to caption](https://arxiv.org/html/2608.03161v1/figures/Im4.png)Figure 4:Constructed educational knowledge graph\.
### V\-CConcept Evidence and Retrieval

Central training concepts dominate the evidence distribution\. Table[V](https://arxiv.org/html/2608.03161#S5.T5)lists the 15 most frequent nodes, and Fig\.[5](https://arxiv.org/html/2608.03161#S5.F5)provides the corresponding visualization\. Remaining singular/plural variants reveal conservative but incomplete entity resolution\.

TABLE V:Canonical Concepts with the Most Evidence![Refer to caption](https://arxiv.org/html/2608.03161v1/figures/Im5.png)Figure 5:Concepts with the most evidence\-backed mentions\.All three seed questions placed their target concept first and within the top three, with mean top\-5 recall of 1\.00 \(Table[VI](https://arxiv.org/html/2608.03161#S5.T6)\)\. Fig\.[7](https://arxiv.org/html/2608.03161#S5.F7)contrasts these preliminary scores with endpoint coverage\. Definition, relation, prerequisite, and cross\-lecture trials retrieved relevant concepts and produced grounded descriptions, but generated answers sometimes added correct background knowledge not explicitly supported by retrieved evidence\.

Fig\.[6](https://arxiv.org/html/2608.03161#S5.F6)shows representative retrieval and answer outputs for definition, relation, prerequisite, and cross\-lecture questions\. The examples illustrate how the system ranks relevant graph concepts before generating an answer, while the accompanying evaluation records whether the gold concepts occur among the highest\-ranked results\.

TABLE VI:Starter Retrieval EvaluationTABLE VII:Qualitative Grounded\-QA Behavior![Refer to caption](https://arxiv.org/html/2608.03161v1/figures/Im6.jpg)Figure 6:Representative questions, retrieved concepts with relevance scores, generated answers, and retrieval\-evaluation records\.![Refer to caption](https://arxiv.org/html/2608.03161v1/figures/Im7.png)Figure 7:Starter retrieval metrics and relationship endpoint coverage\.

## VIDiscussion

Evidence grounding makes the graph inspectable: each retained mention records lecture, anchor, frame, timestamp, modality, quotation, and confidence; each edge records endpoints, type, and supporting evidence\. This reduces the risk of a structurally plausible but unsupported VLM\-generated graph\. The 94\.63% non\-empty OCR rate also indicates that recall\-oriented anchors frequently capture visible educational content that transcript chunks omit\.

Canonicalization merged repeated mentions effectively but remains incomplete\. Variants such as*weight/weights*,*bias/biases*, and*cost/loss function*demonstrate the tension between over\-merging distinct ideas and fragmenting aliases\. Endpoint coverage is therefore a critical diagnostic: 90\.38% shows that most validated relations matched the final node inventory after deduplication\.

Perfect retrieval on three seeds only confirms that obvious concepts are reachable\. A credible benchmark must include substantially more definition, relation, prerequisite, example, first\-mention, temporal\-evolution, visually grounded, and cross\-lecture questions\.

## VIILimitations and Future Work

The dataset contains only three lectures from one series, and extraction lacks a manually annotated concept\-and\-relation gold standard\. The three\-query retrieval set cannot support statistical claims\. Canonicalization leaves aliases and singular/plural duplicates; isolated or noisy nodes may encode generic terms, numeric labels, or visual artifacts\. OCR and VLM accuracy depend on frame resolution, handwriting, transitions, and diagram complexity\. Finally, answer generation may add unsupported background knowledge even when it is correct\.

Future work will \(1\) annotate concept mentions, canonical entities, relations, evidence validity, and QA; \(2\) add domain\-aware lemmatization, alias dictionaries, and merge blocklists; \(3\) prune low\-evidence isolated nodes and enrich reliable central relations; \(4\) compare transcript\-only RAG, transcript\-plus\-OCR RAG, ungrounded extraction, grounded extraction without images, and the full pipeline; \(5\) evaluate answer faithfulness, citation validity, and unsupported\-answer rate; and \(6\) scale across courses and domains with potentially different relation ontologies\.

## VIIIConclusion

This paper presented an evidence\-grounded multimodal pipeline for turning lecture videos into a provenance\-rich KG for multi\-lecture reasoning\. Across three neural\-network lectures it processed 3,118 frames, 756 transcript segments, and 559 anchors; retained 1,022 concept and 312 relation mentions; and constructed 172 canonical nodes and 282 edges with 90\.38% endpoint coverage\. Preliminary retrieval succeeded on three seed questions, but larger annotations, controlled ablations, stronger canonicalization, and rigorous faithfulness evaluation are required\. The principal result is an auditable method that connects graph structure to explicit lecture evidence\.

## Reproducibility

## References

- \[1\]P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Kuttler, M\. Lewis, W\.\-t\. Yih, T\. Rocktäschel, S\. Riedel, and D\. Kiela, “Retrieval\-augmented generation for knowledge\-intensive NLP tasks,” in*Advances in Neural Information Processing Systems*, vol\. 33, 2020, pp\. 9459–9474, doi: 10\.48550/arXiv\.2005\.11401\.
- \[2\]D\. Edge, H\. Trinh, N\. Cheng, J\. Bradley, A\. Chao, A\. Mody, S\. Truitt, and J\. Larson, “From local to global: A graph RAG approach to query\-focused summarization,”*arXiv preprint arXiv:2404\.16130*, 2024, doi: 10\.48550/arXiv\.2404\.16130\.
- \[3\]A\. Radford, J\. W\. Kim, T\. Xu, G\. Brockman, C\. McLeavey, and I\. Sutskever, “Robust speech recognition via large\-scale weak supervision,”*arXiv preprint arXiv:2212\.04356*, 2022, doi: 10\.48550/arXiv\.2212\.04356\.
- \[4\]S\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, S\. Song, K\. Dang, P\. Wang, S\. Wang, J\. Tang,*et al\.*, “Qwen2\.5\-VL technical report,”*arXiv preprint arXiv:2502\.13923*, 2025, doi: 10\.48550/arXiv\.2502\.13923\.
- \[5\]Beijing Academy of Artificial Intelligence, “BGE large English embedding model: BAAI/bge\-large\-en\-v1\.5,” Hugging Face model card\. \[Online\]\. Available:[https://huggingface\.co/BAAI/bge\-large\-en\-v1\.5](https://huggingface.co/BAAI/bge-large-en-v1.5)\. Accessed: Aug\. 3, 2026\.
- \[6\]X\. Zhu, Z\. Li, X\. Wang, X\. Jiang, P\. Sun, X\. Wang, Y\. Xiao, and N\. J\. Yuan, “Multi\-modal knowledge graph construction and application: A survey,”*arXiv preprint arXiv:2202\.05786*, 2022, doi: 10\.48550/arXiv\.2202\.05786\.
- \[7\]H\. Bian, “LLM\-empowered knowledge graph construction: A survey,”*arXiv preprint arXiv:2510\.20345*, 2025, doi: 10\.48550/arXiv\.2510\.20345\.
- \[8\]Q\. U\. Ain, M\. A\. Chatti, A\. Shakhshir, J\. Qussa, R\. Alatrash, and S\. Joarder, “Top\-down vs\. bottom\-up approaches for automatic educational knowledge graph construction in CourseMapper,” in*Digital Education: Shaping Sustainable Lifelong Learning for All in the Era of AI \(EMOOCs 2025\)*, 2025, pp\. 119–129, doi: 10\.1007/978\-3\-032\-00056\-9\_11\.
- \[9\]Q\. U\. Ain, M\. A\. Chatti, J\. Qussa, A\. Shakhshir, R\. Alatrash, and S\. A\. Joarder, “An optimized pipeline for automatic educational knowledge graph construction,”*arXiv preprint arXiv:2509\.05392*, 2025, doi: 10\.48550/arXiv\.2509\.05392\.
- \[10\]B\. Abu\-Salih and S\. Alotaibi, “A systematic literature review of knowledge graph construction and application in education,”*Heliyon*, vol\. 10, no\. 3, Art\. no\. e25383, 2024, doi: 10\.1016/j\.heliyon\.2024\.e25383\.
- \[11\]L\. Mei, S\. Mo, Z\. Yang, and C\. Chen, “A survey of multimodal retrieval\-augmented generation,”*arXiv preprint arXiv:2504\.08748*, 2025, doi: 10\.48550/arXiv\.2504\.08748\.
- \[12\]S\. Wu, Y\. Xiong, Y\. Cui, H\. Wu, C\. Chen, Y\. Yuan, L\. Huang, X\. Liu, T\.\-W\. Kuo, N\. Guan, and C\. J\. Xue, “Retrieval\-augmented generation for natural language processing: A survey,”*arXiv preprint arXiv:2407\.13193*, 2024, doi: 10\.48550/arXiv\.2407\.13193\.
- \[13\]X\. Chen, L\. Jin, Y\. Zhu, C\. Luo, and T\. Wang, “Text recognition in the wild: A survey,”*ACM Computing Surveys*, vol\. 54, no\. 2, pp\. 1–35, 2021, doi: 10\.1145/3440756\.
- \[14\]P\. J\. Jeshmol and B\. C\. Kovoor, “Video question answering: A survey of the state\-of\-the\-art,”*Journal of Visual Communication and Image Representation*, vol\. 105, Art\. no\. 104320, 2024, doi: 10\.1016/j\.jvcir\.2024\.104320\.
- \[15\]H\. Pan, Z\. Zhai, Y\. Zhang, R\. Fu, M\. Liu, Y\. Song, Z\. Wang, and B\. Qin, “Kuaipedia: A large\-scale multi\-modal short\-video encyclopedia,”*arXiv preprint arXiv:2211\.00732*, 2022, doi: 10\.48550/arXiv\.2211\.00732\.