On-Premises Multi-Course RAG Tutoring for Business Education: Hardware-Software Trade-offs in a Campus AI Tutor

# On-Premises Multi-Course RAG Tutoring for Business Education:Hardware–Software Trade-offs in a Campus AI Tutor
Source: [https://arxiv.org/html/2610.02510](https://arxiv.org/html/2610.02510)
Joshua LindemannAffiliation:Dhillon School of Business, University of Lethbridge

###### Abstract

Campus AI tutors based on retrieval\-augmented generation \(RAG\) must ground answers in assigned course materials while keeping textbooks and student dialogue on institutional infrastructure\. We present CourseChat, an on\-premises, multi\-course RAG tutor for undergraduate business education, deployed behind a campus web gateway and intended for Moodle\-embedded use\. Six isolated course offerings—each keyed by its own course reference number \(CRN\)—share twin edge AI hosts running a FastAPI service, a local vector database, and a local large language model \(LLM\) served by Ollama\. We report two generation\-model bake\-off rounds, a separate fixed\-evidence source\-fidelity comparison, and conversation and quiz audits\. Several larger models failed a classroom speed gate, but a 12B model and a 7B alternative passed it\. A separate mixture\-of\-experts candidate improved some corrections while introducing new factual and continuity errors\. We therefore retain the 8B production model pending a demonstrated overall improvement, rather than claiming that 8B is universally optimal\. Software changes improved follow\-up topic resolution while preserving course scope; 435 prebuilt questions across 65 modules decouple practice from live generation\. The results support treating model choice, evidence selection, serving compatibility, and product design as a joint engineering decision\. They do not establish learning gains: faculty ratings, peak\-load capacity, and complete public\-gateway acceptance remain separate evaluation needs\.

Keywords:retrieval\-augmented generation \(RAG\); on\-premises large language models; campus AI tutoring; business education; Moodle; course isolation; source fidelity; hardware–software co\-design; educational technology

## 1Introduction

Campus AI tutors promise grounded help on assigned materials without sending copyrighted textbooks or student dialogue to external APIs\. Realizing that promise under institutional constraints is a systems problem: limited GPU memory, concurrent classroom use, LMS embeds, and a hard requirement that CourseAAnever answer from CourseBB’s corpus\.

This paper documents CourseChat, a multi\-course RAG tutor deployed for a business school\. Students interact through a Moodle\-embeddable chat that supports \(i\) conversation grounded in the selected offering’s indexed materials and \(ii\) module\-scoped practice quizzes with answer keys, hints, and source excerpts\. Six offerings \(anonymized here as Courses 1–6\) share one application binary while remaining isolated at the data and routing layers\.

Our central claim is that building such a tutor requires an explicit hardware–software trade\-off strategy: choose models and pipelines that fit the host memory and concurrency envelope; push quality into retrieval, scoping, and evaluation rather than into ever\-larger generators; and keep product surfaces \(conversation vs\. quiz\) aligned with what the stack can guarantee\.

The practical question is which component should change when a tutor gives a poor answer\. An unhelpful follow\-up can originate in lost conversational context, a missing passage, a relevance filter that rejects a useful passage, or a generator that misstates correctly retrieved evidence\. A larger model addresses only some of these failures, and its benefits must be weighed against response time and serving compatibility\. We use this distinction to organize both the architecture and the evaluation\.

The paper makes three contributions\. First, it describes a replicated local RAG architecture and its course\-scoping and artifact\-management boundaries\. Second, it reports a generation\-model comparison with fixed retrieval and token limits, alongside a deliberately separate model\-and\-response\-contract experiment\. Third, it connects observed failures to concrete design choices: retrieval\-topic resolution, bounded evidence selection, and prebuilt practice banks\. Together these provide an evidence\-based account of why the current release retains an 8B model despite having enough memory to trial larger ones\.

CourseChat began as a pilot project at the Dhillon School of Business to explore how a locally developed AI tutor could add value to students’ learning\. The university already provides students with access to Copilot and Gemini, and students also use ChatGPT and other generative AI tools\. The pilot’s purpose was therefore to complement these existing options with support tailored to local courses, teaching materials, and learning expectations\. Rather than replicate general\-purpose AI assistance, CourseChat was designed to help students engage with their assigned material and develop the confidence to work independently\. Course\-grounded explanations, practice questions, hints, and feedback were intended to encourage students to attempt problems themselves, examine their reasoning, and return to the course sources\. The intended educational value lay in supporting students’ own effort and understanding, with AI serving as a guide within that process\.

## 2Related work and positioning

CourseChat brings together three research concerns: how tutoring supports learning, how generated explanations remain faithful to course evidence, and how an institution can serve those interactions reliably\. The relevant literature therefore extends beyond comparisons of language\-model capability\. It also provides reasons to distinguish successful task completion, useful instructional interaction, and learning that persists after assistance is removed\.

### 2\.1Tutoring, practice, and self\-regulated learning

Research on intelligent tutoring systems predates conversational language models and emphasizes the organization of instructional support\. VanLehn’s review distinguishes systems by the granularity of their interaction, including whether feedback addresses a final answer or intermediate problem\-solving steps\[[2](https://arxiv.org/html/2610.02510#bib.bib2)\]\. Kulik and Fletcher’s meta\-analysis of fifty controlled evaluations found generally positive learning effects for intelligent tutoring systems, while also showing that outcomes depend on the comparison condition, assessment, and implementation\[[3](https://arxiv.org/html/2610.02510#bib.bib3)\]\. These findings support studying the instructional process around a response rather than treating fluent dialogue as sufficient evidence of tutoring effectiveness\. They do not imply that a language\-model chatbot inherits the benefits of a validated tutor\.

Practice and feedback provide a more specific rationale for the pilot’s student controls\. In experiments using prose passages, Roediger and Karpicke found that retrieval practice improved delayed retention relative to repeated study, even when restudy produced better immediate performance\[[4](https://arxiv.org/html/2610.02510#bib.bib4)\]\. The result motivates opportunities to attempt a question before viewing its answer, although it does not establish the effectiveness of CourseChat’s multiple\-choice banks\. Nicol and Macfarlane\-Dick’s account of formative feedback links useful feedback to students’ ability to judge their work and regulate subsequent learning\[[5](https://arxiv.org/html/2610.02510#bib.bib5)\]\. In this framing, a hint or explanation should help a learner identify a gap and decide what to do next\. CourseChat adopts practice attempts, optional hints, and explanatory feedback as design choices informed by this literature; whether they improve study habits or retention in this setting remains an empirical question\.

### 2\.2Generative AI assistance and independent learning

Recent experiments show why the manner of providing AI assistance matters\. Bastani et al\. studied generative AI support in high\-school mathematics, comparing a relatively unrestricted assistant with a tutor configured to provide teacher\-informed instructional support\[[6](https://arxiv.org/html/2610.02510#bib.bib6)\]\. Access to AI improved performance during assisted practice, but students using the relatively unrestricted version subsequently performed worse than the no\-AI control group on an unassisted exam\. The tutoring safeguards mitigated that reduction; this should not be interpreted as evidence that the guarded condition produced a positive unassisted learning gain\. The study distinguishes help with the current exercise from developing the ability to solve another exercise independently\. That distinction is directly relevant to a pilot intended to encourage students’ own work rather than merely make answers easier to obtain\.

Kestin et al\. report a complementary result from a randomized crossover study in undergraduate physics: a deliberately designed AI tutor produced higher immediate learning gains than the comparison active\-learning class sessions\[[7](https://arxiv.org/html/2610.02510#bib.bib7)\]\. The intervention embedded pedagogical guidance and structured content, so it was more specific than access to a general chat interface\. Its short instructional scope also limits conclusions about long\-term retention, other disciplines, or semester\-long use\. Read together, these studies suggest that the instructional design and help policy are part of the intervention being evaluated\. They do not support either a universal claim that AI tutoring improves learning or a universal claim that it undermines it\.

For the Dhillon pilot, this evidence shifts the design question toward the additional educational value of a local service\. Students already have other ways to obtain generated text\. CourseChat’s intended contribution is to connect that help to the selected course and give students a practical route between attempting a question, requesting limited assistance, checking an explanation, and returning to the assigned material\. This is a rationale for the design, not a claim that the pilot has reproduced either study’s learning outcomes\.

### 2\.3Course grounding and source fidelity

Retrieval\-augmented generation provides a technical basis for connecting that help to local materials\. Lewis et al\. combined a generative model with an external document index, demonstrating improvements on knowledge\-intensive tasks and a mechanism for updating accessible evidence separately from model parameters\[[1](https://arxiv.org/html/2610.02510#bib.bib1)\]\. CourseChat uses the broader retrieval\-conditioned generation pattern rather than reproducing that paper’s training procedure\. Its course collections and module filters determine which teaching materials are eligible to support a response\. The resulting benefit is control over the evidence supplied to generation; it does not make the generated answer automatically correct\.

Gao et al\.’s ALCE benchmark evaluates generated answers along separate dimensions of correctness, fluency, and citation quality\[[8](https://arxiv.org/html/2610.02510#bib.bib8)\]\. Their results show that even citation\-producing systems can leave claims without adequate support\. This distinction matters in education because a valid link to a textbook passage can coexist with an incorrect interpretation of its conditions\. CourseChat therefore treats source selection, citation resolution, and fidelity of the explanation as distinct checks\. Source cards make evidence inspectable, while answer review must still establish whether the cited passage supports the claim\.

Increasing the context window is also an incomplete response to grounding failures\. Liu et al\. found that the position of relevant information affected performance on document question answering and key\-value retrieval, with information in the middle of long contexts often used less effectively\[[9](https://arxiv.org/html/2610.02510#bib.bib9)\]\. Their experiments concern particular models and tasks, not the present classroom deployment\. Nevertheless, they motivate testing evidence selection and organization instead of assuming that supplying more textbook content will improve an answer\. Bounded context, explicit scope, and a fixed retrieval configuration make CourseChat’s model comparisons more interpretable\.

### 2\.4Evaluation and serving constraints

RAG evaluation frameworks reinforce the need to assess components separately\. RAGAs proposes reference\-free measures for retrieved context, faithfulness, and answer quality\[[10](https://arxiv.org/html/2610.02510#bib.bib10)\]\. ARES uses trained judges to evaluate context relevance, answer faithfulness, and answer relevance, together with human annotations to account for judge error\[[11](https://arxiv.org/html/2610.02510#bib.bib11)\]\. These approaches provide useful models for diagnosis, but their scores are not measures of student understanding\. CourseChat’s reported evaluation does not claim to implement either framework\. It follows the same separation of questions: whether the intended evidence was found, whether the response was supported, and whether the interaction was useful\. Faculty assessment and unassisted learning measures address additional questions that automated retrieval or citation checks cannot settle\.

Serving research also shows why the suitability of a model cannot be inferred from parameter count alone\. Kwon et al\.’s work on PagedAttention demonstrates that managing the memory used by active generation requests affects batching and throughput\[[12](https://arxiv.org/html/2610.02510#bib.bib12)\]\. Their vLLM results do not predict the performance of CourseChat’s Ollama stack or establish that this deployment would benefit from a particular serving migration\. They do establish a relevant systems concern: model weights are only part of the resource demand during concurrent use\. CourseChat accordingly treats latency, request overlap, context limits, and client compatibility as evaluation criteria alongside answer quality, while acknowledging that its serial trials do not measure classroom capacity\.

### 2\.5Position of the CourseChat pilot

Related course\-specific tutors combine retrieval with explicit instructional control\. CourseAssist uses course\-grounded retrieval, user intent classification, and question decomposition to support computer science students\[[13](https://arxiv.org/html/2610.02510#bib.bib13)\]\. KITE combines multimodal retrieval from course materials with an intent\-aware Socratic strategy that provides hints, guiding questions, and progressive scaffolding for algorithmic problem\-solving\[[14](https://arxiv.org/html/2610.02510#bib.bib14)\]\. CourseChat shares their emphasis on course\-aligned assistance and develops the operational side of that design: replicated on\-premises serving across segregated offerings, model\-selection trials under a fixed latency allowance, and prebuilt quiz banks with source bindings and item\-review records\. Its contribution is evidence about these deployment choices and their observed failure modes in business education\.

The pilot’s contribution is the integration and evaluation of these concerns within a local business\-school setting\. It combines course\-specific evidence, conversation continuity, prebuilt practice, and a shared local generator across six separately scoped offerings\. The paper contributes an account of the resulting tradeoffs and failure modes, including why a working 8B configuration was retained despite credible alternatives\. It does not propose a new retrieval algorithm, demonstrate superiority over institutionally available AI services, or report a controlled learning\-outcome comparison\.

This position also shapes the next evaluation\. A useful local tutor should be assessed for its ability to support appropriate independent effort as well as produce accurate explanations\. Measures of unaided problem solving, retention, source use, and patterns of hint requests would connect the engineering results reported here to the educational aims established in the tutoring literature\. The present study supplies the implementation and diagnostic evidence needed to design that evaluation\.

## 3System overview

### 3\.1Student experience

The student experience is organized around the material assigned in a particular course offering\. In the intended Moodle integration, students enter through an instructor\-provided link embedded in the LMS\. CourseChat is meant to sit inside that Moodle course context rather than as a public textbook or tutoring destination: the embed is the intended entry path, and the course slug is not advertised to students as a standalone bookmark, although slug secrecy and enrollment checks are not claimed as proven access controls\. That link establishes the full\-course, module, or named\-activity scope; the header displays the course, instructor, course reference number \(CRN\), and selected material\. This visible context helps students understand which materials the tutor can use and connects the interaction to their immediate coursework\. Conversation and quiz functions are implemented in the course pages, while complete acceptance of the public gateway remains a separate deployment task\. The design seeks local value through this connection to assigned materials and practice, complementing students’ access to general\-purpose AI services\.

In Conversation mode, students ask questions in ordinary language and receive explanations accompanied by citations to passages from the selected offering\. Inline citations open source cards containing excerpts and available chapter, section, or page information\. Keeping these cards collapsed initially gives the explanation room while preserving a route back to the original material\. Students can request a shorter answer, more detail, or another explanation, and can adjust the explanation level\. These controls are intended to support checking and revisiting a concept rather than treating the first generated answer as the endpoint of study\. The interface also distinguishes source relevance from answer correctness: its relevance score describes the match between a passage and a query, not the probability that an explanation is correct\. Course\-configured relevance floors set a trusted lower bound; students may raise but not lower that floor\.

Conversational continuity matters because students need not restate a complete question at every turn\. After an explanation of net present value, for example, “Can you show me that with numbers?” should preserve that topic and the selected module\. A brief social aside should not erase it\. The router uses recent dialogue to resolve the reference, then retrieves current course evidence; earlier assistant text does not become an authoritative source\. Where several referents are plausible, it asks a short clarification question\. Social replies do not acquire artificial course citations\. When retrieval finds no qualifying passage within the selected module, a reported fallback may broaden the explanation using the model’s current conversational context and material from the same course offering\. It does not retrieve material or conversation history from another segregated course\. If qualifying course evidence remains unavailable, the system reports the evidence gap\. The presence of a qualifying passage cannot by itself guarantee a faithful explanation\.

Quiz mode provides a different form of participation\. Selecting Quiz starts a practice session from a prepared bank and presents one multiple\-choice item at a time\. The student can attempt the item, request a stored hint, or skip it\. A hint leaves the current question and progress unchanged and does not reveal the answer key or supporting excerpt\. After an answer, the system reports whether it is correct, supplies an explanation and source evidence, and presents the next item; skipping also reveals the explanation\. This sequence is intended to make independent attempts and reflection convenient, although it does not require an attempt before disclosure\. The 435 prepared items cover 65 configured modules, and practice uses stored keys and feedback without live generation\. The banks are assistant\-curated and await instructor review; their technical audits establish reproducible behavior rather than educational validity\.

Students can move from practice to discussion while retaining the conversation and instructor\-selected scope, allowing a difficult item to prompt a broader explanation of the concept\. Selecting Quiz again starts a new session\. Clearing the conversation retains the scope and mode, while history is separated by course and scope\. This supports a sequence of asking, attempting, checking, and revisiting within a familiar course context\. Its value for motivation, independent effort, or learning remains an intended outcome to be evaluated with students, rather than a measured result\.

The interface also provides progress messages and incremental answer text through server\-sent events\. Visible generation can signal that a request is being processed before the complete explanation is available\. This perceived responsiveness depends on the application and gateway forwarding events incrementally; server\-local complete\-answer timings do not establish the student’s time to first visible text through Moodle\.

### 3\.2Logical modules

Figure[1](https://arxiv.org/html/2610.02510#S3.F1)shows how request handling, retrieval, generation, and course artifacts compose\.

Student UIcourse \+ modemodule \+ historyFastAPIresolve modelock course \+ moduleCourse settingscollection \+ mapstutoring promptPrebuilt quiz bankkeys, hints, evidenceprepared offlineConversation routerrules first; classify ambiguous turns;preserve the latest course topicQuiz sessionmodule\-scoped itemsdeterministic gradingCourse routesresolve topic \+ embedscoped Qdrant searchhybrid retrieval \+ rerankSocial / clarifysocial generation orone clarifying questionPractice responsestored hint or feedbackcited explanationAnswer \+ citationsOllama 8B \+ evidencebounded dialogue historyUncited responseno course retrievalor course citationsquizconversationFigure 1:Application request paths\. Quiz practice uses stored items and deterministic grading\. Course, mixed, and dependent follow\-up routes retrieve scoped evidence before generation; social and clarification routes do not retrieve course content\. Responses return through the same API and UI\. The dashed arrow supplies course configuration\.
### 3\.3Deployment topology

Figure[2](https://arxiv.org/html/2610.02510#S3.F2)summarizes the campus deployment architecture\. The IIS gateway is assigned TLS termination, course\-slug rewrites, and routing to two AI nodes \(Node A and Node B\)\. Each node runs the full stack locally: FastAPI, Qdrant, sentence\-transformer embeddings, a cross\-encoder reranker, and Ollama\. Backend health and server\-local audits establish the state of the two nodes; they do not establish that every public slug or proxy route has passed acceptance\.

Browser / Moodle embedcourse\-specific public URLCampus IIS gatewayTLS termination, course\-slug rewrites,ARR load balancing and health checksNode A: serve \+ ingestFastAPI \+ static course UILocal embeddings and rerankerQdrant: one collection per offeringOllama:llama3\.1:8bCanonical corpus ingestionNode B: serve \+ replicateFastAPI \+ static course UILocal embeddings and rerankerQdrant: one collection per offeringOllama:llama3\.1:8bRestored course indexesHTTPSinternal HTTP / responsesCollection snapshots: Node A→\\rightarrowNode BQuiz banks, source ledgers, review records, and application assetsare deployed separately to both nodes\.Figure 2:Campus access architecture and deployed twin local stacks\. Solid arrows indicate request/response traffic; the dashed path transfers index snapshots\. Quiz and application artifacts are deployed separately\. The node audits reported here bypass the gateway; complete public course\-slug acceptance remained open in the deployment snapshot\.Hardware envelope \(classroom hosts\)\.Each node is a DGX Spark\-class appliance with a Grace Blackwell GB10 and on the order of 128 GB unified memory shared by the API workers, vector database, embedding/rerank models, and Ollama\. Two uvicorn workers allow overlapping application requests, although actual generation overlap depends on the inference server’s scheduling and memory use\. These are independent local serving stacks, not a pooled GPU model spread across both machines\. This envelope frames the model trials in Section[6](https://arxiv.org/html/2610.02510#S6); worker count alone does not establish classroom capacity\.

## 4Hardware–software trade\-off strategy

We treat model selection as a constrained operating decision\. The desired outcome is a useful explanation supported by the selected offering’s materials, delivered within a tolerable classroom wait\. The bake\-off makes one performance constraint explicit: median complete\-answer latency must be no more than 30% above the production 8B baseline measured in the same round\. Other promotion criteria require usable overlapping requests, preservation of course scope, and a meaningful improvement in reviewed answers\. These are promotion requirements; a listed requirement is not evidence that the corresponding experiment was completed\.

Several design choices act on different parts of this objective\. Model family and size affect answer generation, whereas retrieval depth, context limits, and reranking affect which evidence reaches the model and how soon generation can begin\. Server\-wide application profiles package these performance settings; they do not select a course or change the collection boundary\. A separate product decision moves quiz authoring out of the request path\. A separate deployment decision moves document extraction and review to a workstation, leaving the serving hosts responsible for indexing and inference\.

Changing these components together would obscure why a candidate succeeded or failed\. We therefore keep the serving retrieval stack fixed during the model bake\-off, and identify the separate structured\-answer experiment as a different comparison\. Course content remains local, quiz questions are prepared before practice, and reasoning\-capable models must deliver student\-visible output within the configured budget before they are considered deployable\. These choices trade some flexibility for traceable failures and a bounded release process\.

Table[1](https://arxiv.org/html/2610.02510#S4.T1)brings together the main interventions, their observed outcomes, and the resulting design decisions\. It distinguishes generator comparisons from changes to retrieval and practice delivery; Section[6](https://arxiv.org/html/2610.02510#S6)gives the model\-by\-model measurements\. This overview makes explicit why the project retained some components, changed others, and deferred alternatives that had not established an overall advantage\.

Table 1:Decision summary across model trials and system\-design evaluations\. Results come from distinct protocols and are not a common accuracy ranking\. Model latency ratios use the 8B baseline from the corresponding bake\-off round; all reported timings are server\-local\.
## 5Design choices

### 5\.1Isolation without six fine\-tuned LLMs

Isolation is enforced by*separate course ids*,*separate on\-disk trees*,*separate Qdrant collections*, and?course=routing\. A shared Ollama generator receives passages from the routed collection\. Optional per\-course embedding fine\-tunes are a potential retrieval\-quality intervention; they are not required for isolation and were not changed in the model trials\. The collection and request boundaries determine which indexed materials can be selected, while source\-faithful generation remains a separate requirement\.

Each uploaded offering is keyed by its own course reference number \(CRN\)\. Instructors can therefore maintain different versions of the same catalog course—for example, distinct sections or successive terms—and control which materials enter that offering’s retrieval collection\. Uploading documents updates the evidence available to that CRN; it does not retrain the shared language model or merge corpora across offerings\. Two offerings may share a catalog code \(e\.g\., two sections of the same strategic management course\) yet must keep distinct corpora when instructors and materials differ\. The same principle applies to conversation history and quiz state: carrying a previous course’s session into a newly selected offering would defeat collection\-level separation\. Scope therefore belongs to request handling and session validation as well as storage\. Passing the reported scope checks supports these specific paths, rather than proving that a shared pretrained model can never produce an unsupported statement\.

### 5\.2RAG for content; prompts for pedagogy

Course knowledge is indexed, not baked into LLM weights\. Tutoring tone and Socratic instructions come from system prompts and optional bot\-profile YAML\. Chapter scope is applied through request validation and retrieval metadata filters; any reported fallback remains within the selected course offering\. Content updates are re\-ingest \+ replicate, not a training job\. This allows revisions to textbooks and decks to be associated with explicit source artifacts and ingestion receipts\. A weight update would not by itself provide the same passage\-level provenance, and would add another artifact to version across the two serving nodes\.

### 5\.3Prebuilt quizzes

Early demos showed that generating a multiple\-choice item at chat time is insufficient for a reliable practice session\. The platform therefore prebuilds banks containing source\-backed items, keys, explanations, hints, and evidence ledgers\. Automatic generation is gated by index and answer evaluation\. A separate publication path accepts independently reviewed items bound to the exact current source and question hashes; that path requires the author and reviewer identities to differ, and that item\-level review does not clear a failed free\-form chat audit\. Serving quiz turns is deterministic grading against the bank \(warm medians on the order of milliseconds server\-local\), while conversation retains the generative path\. Answer keys remain server\-side until an attempt or reveal, so the bank is not exposed as a downloadable key file in the student UI\.

Across Courses 1–6 the deployed banks total 435 questions over 65 modules\. Structural audits and live API grading checks pass on both nodes; instructor pedagogical review remains an explicit separate gate \(instructor\_reviewed: falseuntil faculty sign\-off\)\.

This choice makes practice behavior reproducible: the hint, key, explanation, and supporting excerpt can all be inspected before use\. It also makes content maintenance explicit\. When source material changes, a bank must be checked against the new corpus rather than silently continuing to cite stale evidence\. The cost is a finite pool of questions and less spontaneous adaptation to a student’s wording, which is why open\-ended explanation remains a separate conversation mode\.

### 5\.4Conversational turns over shared course evidence

Conversation gives students a way to work through the meaning of course material, including questions that cannot be anticipated in a prepared quiz\. The design combines three resources with different roles: a pretrained model provides language generation, indexed passages provide current teaching evidence, and prebuilt quiz items provide stable questions, hints, and grading keys\. Ingesting a professor’s materials updates the evidence collection; it does not retrain the language model\. Conversation can therefore extend the uses of the same corpus while preserving the distinction between an open\-ended explanation and a stored practice item\.

A conversational turn carries more than a new search phrase\. If a student asks whether a different assumption would change a worked example, the tutor must retain the example’s setting, identify the changed assumption, and find the course evidence needed to explain its effect\. An exchange about net present value might move from a definition to a numerical illustration and then to the student’s own account of why a higher discount rate changes the result\. This illustrates the intended interaction: the student can test an interpretation and request a targeted explanation\. The model’s hypothetical numbers must remain clearly distinguished from facts attributed to course sources, and a change of topic must still respect the selected module\.

The implemented path resolves dependent turns against recent dialogue while passing the student’s actual wording to generation\. Recent dialogue is held in the browser and submitted with each request rather than stored as a durable server session: history must match the selected course, is bounded in length, and does not require sticky affinity between the twin nodes\. The shared resolver raised intended\-topic follow\-up rates to 90–100% across the five\-course replay, without changing the generator \(Section[8](https://arxiv.org/html/2610.02510#S8)\)\. The conversation router handles social, course, mixed, follow\-up, and clarification turns, using deterministic rules first and a short classification call for ambiguity\. A casual aside can preserve the course topic; subsequent content questions retrieve fresh evidence\. Earlier assistant text helps interpret a reference but is not promoted into authoritative teaching material\. Only designated meta or policy FAQ entries short\-circuit retrieval; content FAQs remain in the course vector index and compete with other evidence\.

LangChain components already provide model, prompt, document, and vector\-store interfaces in the retrieval application\. They offer a basis for extending the conversation workflow around the existing corpus: assemble the learning instructions and bounded dialogue, retrieve material for the resolved turn, and generate an explanation with source references\. LangChain’s memory facilities support thread\-level history and strategies for limiting its size; durable conversation state would additionally require shared persistence and course isolation\[[18](https://arxiv.org/html/2610.02510#bib.bib18)\]\. The current router uses explicit Python control flow and browser\-supplied history\. A more extensive LangChain workflow, or LangGraph orchestration for resumable multi\-step activities, would be a future implementation choice, evaluated against this simpler baseline\.

This separation creates room for additional instructional modes without building another knowledge base for each one\. A future case\-discussion mode could ask students to justify an interpretation using a professor’s supplementary case; a guided\-practice mode could focus on the next reasoning step; and a revision mode could revisit a misconception exposed by a quiz\. Each would use an explicit interaction policy over the same scoped evidence and keep factual support checks in place\. Their local value would come from the instructor’s selected material and teaching expectations, combined with opportunities for student reasoning\. These are proposed extensions to the deployed conversation and quiz modes, to be tested for usefulness and appropriate independent effort\.

### 5\.5Streaming vs\. raw latency

Non\-streaming live eval reports full\-answer wall times \(often∼\\sim6–10 s on 8B\)\. Perceived speed is dominated by time\-to\-first\-token through the gateway\. The UI requests SSE, and gateway configuration must preserve incremental delivery\. The first metadata event, the first visible text, and completed answer time are different measurements\. The bake\-off below measures the last of these on the node itself; it does not quantify browser\-perceived waiting time through IIS\. Streaming can expose an answer earlier without reducing total inference work, so it complements rather than replaces model\-latency evaluation\.

### 5\.6Workstation / Node A / Node B pipeline

Scanned textbook OCR and page review run on a Mac workstation \(Docling, Apple Vision, optional on\-device formula assistance\)\. Reviewed corpora are prepared offline, ingested on Node A with the same encoder the API uses, then snapshot\-replicated to Node B\. This split keeps OCR engines off the classroom hosts and keeps A/B parity enforceable \(parity\-checkon git SHA and health\)\.

## 6Model bake\-off

### 6\.1Protocol

On Node A we compared Ollama instruct tags while holding fixed:APP\_PROFILE, context length 8192, output cap 768 tokens, retrieval knobs, and the same live\-eval probe set\. Node B remained on the production 8B model as failover\. The embedding encoder and reranker were held fixed to avoid confounding the generator comparison; changing the encoder would also require rebuilding stored vectors\. Thinking\-mode defaults were treated as disqualifying when they emptied student\-visible content or exhausted the token budget\.

Promotion rule: a candidate replaces 8B only if it is clearly better at grounded tutoring*and*clears the classroom speed gate \(p50 total latency no worse than\+30%\+30\\%vs\. 8B\) with usable two\-request overlap\.

Requests were sent to the loopback API on Node A, bypassing IIS\. The twelve live\-evaluation probes exercise keyword presence and citation behavior, not a complete pedagogical rubric\. We report each candidate’s median complete\-answer wall time divided by the 8B median from the same round\. A fresh 8B baseline was measured in each round, so ratios use the corresponding round rather than a single pooled reference\. Context and output caps were fixed at 8,192 and 768 tokens, respectively; fixed caps do not imply identical realized output lengths\.

The protocol calls for reviewing directness, groundedness, synthesis, and course\-appropriate tone, alongside scope and overlap checks\. The documented quality target includes at least 95% supported factual claims and a clear paired\-review improvement\. These targets were not established by the twelve automated passes\. The reported tables contain point estimates without repeated run uncertainty intervals or a completed concurrency study, so a speed pass is only one prerequisite for promotion\.

### 6\.2Results

Tables[2](https://arxiv.org/html/2610.02510#S6.T2)–[3](https://arxiv.org/html/2610.02510#S6.T3)summarize two rounds on Node A \(18–19 September 2026\)\. Live eval scores are keyword/citation smoke checks \(necessary but not sufficient\)\.

Table 2:Bake\-off round 1 \(includes thinking\-capable tags\)\.†Thinking filled the reply; empty student\-facing content under the live client\.

Table 3:Bake\-off round 2 \(non\-thinking / think\-disabled candidates\)\.‡Integration mismatch with the chat client \(emptycontent\), not a claim the model cannot answer via raw Ollama\.

#### Deployment decision\.

Production remainsllama3\.1:8bon both nodes\. No round\-1 larger model cleared the classroom speed gate\. Round 2 identifiedqwen2\.5:7b\-instructas a same\-speed side\-grade with a small completeness bump, andmistral\-nemo:12bas a larger option still inside\+30%\+30\\%\. Promotion waits on faculty side\-by\-side review of answer quality\.

Round 1 illustrates the difference between memory feasibility and acceptable response time: Qwen2\.5 14B and Gemma 12B returned all twelve expected smoke\-test responses but required approximately 1\.75 and 2\.25 times the baseline median\. Round 2 provides a counterexample to rejecting larger models categorically: Nemo 12B completed at 1\.24 times its fresh baseline\. Qwen 7B completed at 1\.01 times the baseline and had a modest reported completeness\-proxy increase \(0\.93 versus 0\.91\)\. That proxy is not a supported\-claim rate or an instructor rating\. The result is a shortlist for further review, rather than evidence that parameter count alone determines suitability\.

The zero\-pass rows require a different interpretation\. The tested Qwen MoE configuration spent its budget in a thinking channel and supplied no student\-visible answer through the live client\. Granite also returned empty content through the application despite answering through raw Ollama\. These are failures of the evaluated serving configuration\. They matter to deployment, but they do not establish that the models cannot answer the questions\.

### 6\.3Separate fixed\-evidence fidelity comparison

A separate diagnostic compared the production 8B path with aqwen3\.6:35b\-a3b\-q4\_K\_Mcandidate on twelve frozen cases\. The fixture preserved the current question, earlier dialogue, selected source passages, and source checksums\. It included corrections after an incorrect assistant answer, numerical illustrations, missing facts, topic switching, format constraints, and references to a previous example\. Unlike the generator\-only bake\-off, the final candidate changed the model, prompt, structured response contract, validation, and transport together\. Its results therefore evaluate that complete configuration, not an isolated model\-size effect\.

The candidate returned an evidence list of source identifiers and exact quotations followed by an answer\. Validation checked structure, current source identity, quotation text, and cited source numbers; rejected drafts were replaced by a fixed conversational gap response\. Thinking was explicitly disabled\. Direct requests to the generation endpoint produced twelve valid JSON objects, resolving a format\-handling problem observed with the installed chat route\. Nine answers passed structural checks, including one materially wrong claim\. Three were withheld: two appropriately lacked evidence, whereas one was an answerable follow\-up\.

The independent comparison recorded three substantial improvements, two modest improvements, three ties, one mixed result, and three regressions\. The larger candidate corrected earlier accounting and pricing mistakes, but also turned a qualified benefit of holding cash into a guarantee and lost continuity on a reference to a prior example\. Warm serial medians were 3\.469 s for the production path and 2\.997 s for the candidate\. A separate cold candidate request took 25\.417 s, including 23\.269 s loading\. With only twelve warm observations, the reported nearest\-rank p95 is the maximum, not a reliable estimate of classroom tail latency\. The candidate occupied approximately 26\.4 GB on the GPU with the configured context, demonstrating that memory fit was not the decisive obstacle in this trial\.

The configuration was not activated\. Its warm speed and correction gains were promising, but new factual and continuity regressions prevented an overall quality win\. Production streaming, dynamic instructor instructions, and the broader interaction settings were also outside this diagnostic’s validated contract\. Section[9](https://arxiv.org/html/2610.02510#S9)uses these results to explain why retaining 8B is a conditional release decision\.

## 7Content and RAG pipeline

Course materialstextbooks, slides, notesand transcriptsExtract \+ reviewworkstation OCRpage \+ numeric checksPrepare corpusJSONL \+ module mapssource hashesAutomatic quiz buildonly after all corpusquality gates passEvaluate on Node Aretrieval \+ answersfailed checks retainedIngest on Node AAPI encoder \+ upsertverify sources; receiptReviewed additionsindependent item reviewsource \+ content hashesValidate artifactsevidence \+ coverageruntime gradingRestore on Node BQdrant collectionsnapshot from Node ACopy quizzes to A/Bbank \+ source ledgeritem\-review historyFigure 3:Knowledge preparation and deployment\. Automatic quiz building requires passing corpus quality gates\. Independently reviewed additions provide a separate path, bound to current primary evidence, without clearing failed chat evaluations\. Both paths require artifact validation\. Dashed arrows denote deployment transfers; instructor pedagogical review remains separate and pending for the published banks\.Production chat uses a Fast RAG path: embed query, hybrid search over the course collection, metadata filters \(module/chapter\), cross\-encoder rerank, optional cache, then Ollama generation with bounded history and citations\. Relevance floors are course\-configured; students may raise but not lower the server floor\. The displayed relevance score summarizes passage–query match from the reranker; it is not a confidence that the generated explanation is correct\. Retrieval also applies evidence tiers: original lecture and reading passages are preferred over generated study aids such as cards, and navigation or placeholder entries are excluded from grounding\. When an instructor locks a chapter or module and the filtered set is empty, a counted fallback may widen scope using the model’s current conversational context and material from the same course offering, and report that it did so\. This fallback never retrieves materials or dialogue from another segregated course; the course collection boundary remains unchanged\. Lecture slides enter the index through explicit lecture\-to\-module maps rather than filename heuristics; in the reported preparation, eighty lecture files contributed 2,242 passages under those maps\.

The four newly completed textbook\-course corpora contain 15,730 prepared passages\. This is a prepared\-corpus count, not the total number of Qdrant points across all six offerings; live collections can also contain legacy material\. Course metadata distinguishes a teaching module from a textbook chapter, so a shared chapter number does not implicitly determine the scope of a teaching activity\.

The pipeline preserves original\-file hashes, reviewed page metadata, stable chunk identifiers, and ingestion receipts\. These artifacts make it possible to check that a stored quotation still belongs to the current source before publishing a question\. Scanned tables and formulas need additional numerical review: extracting a page does not certify its arithmetic or layout\. Generated study aids are not treated as primary evidence for the reviewed quiz items\. This provenance work adds preparation effort, but allows content defects to be distinguished from retrieval and generation defects\.

The split between automatic and independently reviewed quiz publication is material to interpreting the deployment\. None of the four textbook courses passed the complete automatic regeneration gate in the reported ingestion run\. Their 45 added items followed the separate primary\-source authoring and independent item\-review path, with distinct author and reviewer identities\. That path verified current evidence and item integrity without clearing the failed chat evaluations or implying faculty approval\. Snapshot replication moves the index, while banks, source ledgers, and cumulative review histories require separate deployment to both nodes\.

## 8Evaluation

### 8\.1Multi\-turn conversation

Frozen fixtures contain 20 dialogues and 80 turns per course, including 60 follow\-ups\. They cover numerical examples, misconceptions, topic changes, brevity requests, locked\-module boundaries, and transitions between quiz and conversation\. The harness records the resolved retrieval query, scope, selected passages, model\-visible context, reply, and latency\. Topic\-hit scores measure whether a dependent question retains its intended subject; they do not measure whether every resulting explanation is correct\.

Course 1 was evaluated separately, and Courses 2–6 were replayed serially on Node B with the production 8B generator\. Table[4](https://arxiv.org/html/2610.02510#S8.T4)reports the documented before/after topic checks\. Course 1 has two live replays; the Courses 2–6 baseline scores come from a resolver\-only preflight with the prior production resolver, followed by live evaluation of the update\. In the five\-course run, all 400 scope checks passed with zero API errors, and each course reached the 90% follow\-up target\. Median per\-turn latency ranged from 3\.24 to 6\.19 s\. The separate Course 1 replay improved from 31 to 60 successful follow\-ups, while its median increased from 4\.92 to 5\.81 s\. This is consistent with treating quality and speed as distinct outcomes rather than assuming that every resolver improvement also reduces latency\.

Table 4:Intended\-topic follow\-up checks\. Course 1 has a live baseline; Courses 2–6 have resolver\-only baselines\. Updated results are live for all courses with the unchanged 8B generator\. Scope checks also include initial turns\.The five\-course run still contained 93no\_matchturns\. Twelve were intentional locked\-module refusals, fifteen were quiz follow\-ups whose wording did not retrieve a qualifying passage, and 66 involved other requests spanning 24 topics\. Cleaner concept probes found evidence for nineteen of those topics; five still required content or index investigation\. A bounded facet expansion recovered eleven of the 24 original questions without lowering the relevance floor\. These diagnostics identify recoverable retrieval failures; they are not an overall answer\-accuracy score\. Faculty ratings remain pending, so neither the proposed 4/5 usefulness target nor the 95% supported\-claim target is claimed as achieved\.

### 8\.2Source fidelity and routing checks

A targeted, agent\-reviewed ten\-request live textbook audit gives a complementary view of quality\. It found three supported replies, four retrieval misses despite available evidence, two source contradictions, and one partly grounded answer\. Every returned citation resolved to its source\. Thus, citation integrity coexisted with substantive error: locating a real passage did not ensure that the answer preserved its meaning\. The requests were selected diagnostics, not a representative sample from which to estimate a population error rate\.

The 20 September router extension has a different evaluation scope\. Its frozen contract contains 32 crossover cases, supplemented by focused checks and live probes for social, mixed, course, follow\-up, and ambiguous turns\. Both nodes passed 306 focused Python tests and the documented live probes before activation\. This supports the routing implementation; it does not retroactively turn the earlier conversation runs into tests of that extension, or establish faculty\-rated usefulness\.

### 8\.3Quizzes

Both nodes passed full artifact and grading audits for all 65 modules\. The artifact audit bound each question to its current source excerpt and original file hashes, then exercised all 435 questions, including deliberately wrong answers\. It checked that hints do not reveal keys or advance progress, and that requests cannot carry another course’s quiz state into the selected offering\. These checks establish reproducible practice behavior\.

A separate 166\-request live API audit included all 65 module starts on each node, plus warm repeats, hints, and graded answers\. Warm medians were 4\.8 ms on Node A and 5\.8 ms on Node B, measured server\-local\. They describe stored quiz operations and must not be compared as if they were generative tutoring latencies or browser response times\. Instructor review of answer correctness, distractors, difficulty, and instructional coverage remains incomplete\.

## 9Discussion

### 9\.1Why retain the 8B classroom generator?

Retainingllama3\.1:8bis a deployment decision under incomplete quality evidence, rather than a claim that eight billion parameters is an optimal model size\. The first bake\-off established that several plausible upgrades increased full\-answer latency substantially under the same retrieval and output budget\. The second round qualified that result: Qwen 7B ran at almost the baseline speed, and Mistral Nemo 12B stayed inside the prescribed 30% latency allowance\. Consequently, “larger models are too slow” does not adequately explain the final choice\. The production policy requires a clear grounded\-tutoring benefit in addition to acceptable speed\. Passing twelve keyword\-and\-citation checks, or producing slightly more complete answers on that set, does not establish that benefit\. The documented decision is to keep the existing generator while faculty compare the alternatives on the quality of their actual explanations\.

This choice preserves a known serving configuration while acknowledging its material weaknesses\. The fidelity diagnostic reproduced incorrect accounting claims, failure to correct an earlier pricing error, and attribution of an equation to an exercise that did not state it\. Those failures preclude treating continued deployment as evidence that the 8B model has passed the proposed claim\-support target\. The defensible rationale is narrower: no evaluated replacement has yet demonstrated the complete combination of quality, continuity, integration, and operational readiness required for promotion\. A reviewed Qwen 7B or Nemo 12B improvement could change that decision without changing the architecture\.

### 9\.2A model is evaluated through its serving contract

The separate structured\-response experiment illustrates why parameter count alone is a poor predictor of either speed or usefulness\. With thinking disabled, the 35B\-A3B mixture\-of\-experts candidate had a lower warm median than the 8B baseline on the twelve fixed\-source cases\. It corrected important misconceptions that persisted in the baseline, but also overstated the protection provided by holding cash and refused an answerable reference to a previous example\. These are consequential tutoring regressions: a student needs an explanation to preserve both the source’s conditions and the setting established in the conversation\.

That comparison held the evidence and dialogue fixed but changed the model, prompt, output schema, validation, and transport together\. Its gains therefore belong to the complete candidate configuration; they cannot be attributed solely to model scale\. Conversely, an empty response in the original production\-client bake\-off does not show that the underlying model lacks knowledge\. The installed serving stack did not consistently apply the structured format to thinking\-capable chat requests when thinking was disabled\. A direct generation request resolved the observed JSON\-formatting problem, while factual and continuity failures remained\. Runtime compatibility thus belongs in the experimental specification alongside model tag, quantization, context length, and sampling options\. The experimental answer contract also left production streaming and instructor\-specific instructions uncertified, providing another reason to withhold activation\.

### 9\.3Grounding requires useful answers and appropriate abstention

The evidence suggests that stricter output validation trades some unsupported answers for refusals, without eliminating semantic error\. In the final candidate, all twelve outputs were valid JSON and nine passed the quotation and citation checks\. An accepted answer nevertheless strengthened a conditional financial benefit into a guarantee\. Of the three withheld answers, two lacked sufficient evidence and one was an answerable follow\-up\. Exact quotation verifies a textual relationship with a source; it does not verify that the explanation preserves the source’s meaning\.

The separate entailment experiment reinforces this distinction\. Its proposed contradiction threshold missed both original production errors, even though the classifier detected simplified versions of some errors\. A hypothetical teaching example also received a neutral label, showing why rejecting all non\-entailed prose would suppress legitimate instruction\. A useful tutor must distinguish course\-supported concepts, explicitly invented example values, unavailable facts, and unjustified real\-world guarantees\. Evaluation should therefore report unnecessary abstention and loss of explanatory context alongside unsupported claims\. The proposed bounded revision experiment addresses these failure modes, but it remains future work; neither a runtime factual guard nor a reliable semantic repair mechanism has been demonstrated\.

### 9\.4Retrieval and conversation control are complementary to generation

The multi\-course resolver results identify a substantial source of improvement outside the generator\. Resolving a short follow\-up into its intended retrieval topic allows the existing model to receive relevant teaching material while still answering the student’s actual wording\. Course and module boundaries remain enforceable in this step\. However, a correct topic does not ensure that an adequate passage will be retrieved, and an adequate passage does not ensure that the model will explain it faithfully\. The five\-course audit’s remaining retrieval misses, and the fixed\-evidence fidelity failures, demonstrate these distinct failure layers\.

This decomposition helps allocate engineering effort\. Clean\-concept probes recovered evidence for nineteen of twenty\-four affected topics, indicating that many misses arose from query formulation\. Five topics still failed under that favorable probe and required corpus, index, or threshold investigation\. A bounded expansion using only words in the student question recovered eleven of the original queries without lowering the relevance floor\. These findings support targeted retrieval work; they do not imply that query processing can repair missing material or generation errors\. Preserving each turn’s resolved query, selected passages, and displayed answer makes that distinction auditable and reduces the temptation to attribute every poor answer to model size\.

The subsequent conversation router, activated on both nodes on 20 September, extends this separation to social, course, mixed, follow\-up, and clarification turns\. Obvious cases use deterministic rules; ambiguous cases use a short structured classification request to the same 8B model\. Sharing the resident model avoids introducing a second model and a possible model\-loading penalty while routing is evaluated\. Its validated intent schema is a deployed control mechanism, distinct from the experimental evidence\-first answer contract discussed above\. The implementation also illustrates a deliberate complexity trade\-off: bounded browser\-supplied history and a small Python router meet the current requirements, while durable cross\-device conversations would require additional persistence\. LangGraph remains an option for that future requirement, rather than a dependency needed to obtain the reported improvements\.

### 9\.5On\-premises control carries operational obligations

Local inference and separate offering collections give the institution control over where course materials and student dialogue are processed and which evidence enters an answer\. Dialogue is handled on the campus hosts rather than sent to an external generation API\. Each offering is keyed by its own CRN, so instructors can maintain different versions of the same catalog course and decide which materials that version may retrieve\. Sharing one generator across offerings avoids maintaining six content\-specific language models, while separate collections, data trees, browser\-held histories, and request routing keep those CRN boundaries in place\. Uploaded materials update the retrieval collection for the selected offering; they are not used to fine\-tune a shared model on faculty or student content\. Access is organized around the LMS course context and course routing rather than an application\- level enrollment check in CourseChat itself\. These mechanisms support reproducible content updates and instructor\-controlled corpora\. Their evaluation demonstrates scoped behavior on the recorded probes, rather than a general proof of security, privacy compliance, or enrollment enforcement\.

The two complete local stacks provide an operational alternative when one node is under maintenance, but duplicate state introduces explicit maintenance work\. Updating the primary index is insufficient: the replica must receive the corresponding snapshots, and quiz banks and evidence ledgers must be copied separately\. Code, model configuration, and course artifacts also need consistent versions\. This is the cost of retaining local ownership and independently serviceable hosts\. The reported work does not measure capital amortization, electricity, administration time, or cost per student; it therefore supports no claim that this deployment is cheaper than a cloud service\. Its demonstrated motivation is control under the institution’s chosen deployment constraints\.

### 9\.6Prebuilt practice moves uncertainty into a reviewable build stage

Prebuilding quizzes changes both the computational and educational trade\-off\. Serving a stored question and grading against its key avoids a generation request for each practice turn, while making hints and explanations stable across sessions\. It also makes evidence and arithmetic review possible before an item reaches a student\. In exchange, the bank offers finite variation and requires rebuilding or re\-review when course material changes\. Six or seven questions per module establish availability across the module map, rather than exhaustive coverage of each learning objective\.

The distinction between artifact integrity and teaching quality remains essential\. Successful grading and exact source bindings demonstrate that practice behaves consistently, but they cannot establish suitable difficulty, plausible distractors, or a single pedagogically defensible answer\. All banks still record instructor review as incomplete\. Similarly, a module with limited source material can offer bounded, supported practice without claiming comprehensive coverage\. This explicit limitation is preferable to letting live generation silently supply missing instructional content\.

### 9\.7Faculty review, student uptake, and educational value

The next pilot stage should connect content maintenance to teaching practice\. Instructors could identify recurring misconceptions, contribute supplementary cases or worked examples, and review the questions generated from those materials\. A useful approval workflow would present each question alongside its answer, distractors, hint, explanation, and supporting excerpts, allowing the instructor to edit, approve, or return it for revision\. The existing source bindings and item\-review records provide an artifact foundation for this workflow; a streamlined faculty interface and completed instructor approval remain future work\. Preserving approved edits and requesting renewed review when their supporting material changes would make this feedback loop part of maintaining a course, rather than a one\-time inspection of generated content\.

Student uptake should be evaluated over a teaching period\. Initial visits would show reach, while return visits and continued use across later modules would help distinguish sustained use from novelty\. Reporting use separately for explanation, conversational follow\-up, and quiz practice would reveal which activities students find valuable\. Course coverage also matters: a service used intensively in one offering may fit that instructor’s materials without transferring to other courses\. These observations should be paired with student feedback on usefulness, ease of use, trust in source\-supported answers, and reasons for abandoning an interaction\. Acceptance cannot be inferred from usage counts alone, particularly where an instructor explicitly encourages participation\.

Educational evaluation should examine what students can do after assistance\. A future study could combine interaction measures, such as attempts before revealing answers and responses to hints, with delayed assessments and new problems completed without the tutor\. Appropriate comparisons include the existing course resources and the general\-purpose AI tools students already use\. Comparing equivalent learning tasks would help establish whether access to instructor\-selected evidence and reviewed practice adds value beyond convenient text generation\. Perceived usefulness, assisted task performance, and independent learning should remain separate outcomes\.

Faculty acceptance is equally consequential\. The pilot should record the time needed to prepare materials, review questions, resolve reported errors, and update a course between offerings\. A proposed feedback mechanism could let students flag an unclear explanation or disputed answer for instructor review, with confirmed issues feeding into corpus corrections and regression cases\. This would connect observed difficulties to concrete revisions while making the teaching workload visible\. Together, student and faculty evidence would guide which interaction modes to extend and which content gaps deserve attention first\.

### 9\.8Limits of the evidence and next evaluation priorities

The reported latency measurements are predominantly serial and server\-local\. Two API workers permit overlapping requests, but worker count and available memory do not establish usable classroom throughput\. Warm model timings also exclude the substantial cold\-load delay observed for the larger candidate\. The two\-request overlap gate remains a requirement for promotion; the presented results do not constitute a completed load study\. Gateway time\-to\-first\-token requires separate measurement because proxy buffering can delay visible output without changing generation time\. SSE support is implemented, whereas public gateway readiness and verification for every course slug remained open in the documented deployment snapshot\.

The evaluation sets are engineering regressions rather than a representative sample of student use\. Repeated development on frozen fixtures can improve those cases without transferring to unseen misconceptions or question forms\. Faculty ratings for usefulness and factual support remain pending, and no learning\-outcome or controlled classroom study is reported\. The next evidence needed is therefore paired faculty review on held\-out dialogues, explicit measurement of false refusals and source\-condition errors, and concurrent browser\-to\-gateway timing under representative load\. These checks would make a future model change defensible while preserving the current distinction between an implemented system, measured engineering behavior, and demonstrated educational benefit\.

## 10Conclusion

CourseChat began as a Dhillon School of Business pilot to provide useful local learning support alongside the generative AI services already available to students\. Its contribution rests on the design of the whole teaching system: separate CRN\-scoped collections preserve instructional context and let instructors control which materials each offering may retrieve, a traceable ingestion pipeline connects responses to supplied materials, conversation controls support follow\-up questions, and prebuilt quizzes make practice and feedback inspectable\. These choices allow one shared generator to serve different offerings while giving course content and instructional interaction their own development and review processes\. They also expose practical trade\-offs, including preparation effort, finite question\-bank variety, and the need to keep replicated course artifacts consistent\.

The model trials inform one part of this design\. We retainllama3\.1:8bbecause the evaluated alternatives have not yet demonstrated an overall advantage across grounded explanation, conversational continuity, latency, and serving compatibility\. Credible 7B and 12B candidates remain available for further assessment\. The choice is therefore conditional on evidence and can change without rebuilding the course architecture\. The observed retrieval and fidelity failures also show why improving a tutor requires attention to evidence preparation and dialogue handling alongside model capability\.

The next development stage should extend the course ingestion pipeline with relevant, reviewed material supplied by instructors: supplementary cases, worked examples, lecture explanations, and contextual material beyond the core assigned text\. Better coverage can address missing evidence, but adding documents alone does not ensure better answers; extraction quality, module mapping, retrieval, and explanation fidelity must be evaluated together\. A shared, versioned corpus can support several interaction modes, including course discussion, guided examples, concept comparison, and quiz practice, without maintaining a separate knowledge base for each mode\. These modes would complement general\-purpose assistants such as ChatGPT by centering the instructor’s selected context, expectations, and review of teaching materials\. The local service’s purpose is to help students reason within that context and apply it themselves\.

Faculty participation should shape that development\. A review workflow can allow professors to inspect, revise, and approve quiz items and their supporting evidence, while student reports of unclear or incorrect explanations can inform subsequent content and evaluation updates\. Existing review artifacts provide a starting point; instructor approval of the current banks is still pending\. The next pilot should assess sustained student uptake, acceptance of different interaction modes, and the workload this process creates for faculty\. Delayed and unassisted assessments, alongside comparisons with existing resources and general\-purpose assistants, would establish whether the system supports independent learning\. These findings should guide its expansion toward an instructor\-informed learning environment whose value comes from local context, useful feedback, and students’ own effort\.

## Acknowledgments

We thank the University of Lethbridge Information Technology \(IT\) team for its support of the pilot and its infrastructure, and colleagues at the Dhillon School of Business for their contributions to the project\.

## References

- \[1\]P\. Lewis et al\.Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.*Advances in Neural Information Processing Systems*, 33:9459–9474, 2020\.[https://proceedings\.neurips\.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5\-Abstract\.html](https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html)\.
- \[2\]K\. VanLehn\.The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems\.*Educational Psychologist*, 46\(4\):197–221, 2011\.[https://doi\.org/10\.1080/00461520\.2011\.611369](https://doi.org/10.1080/00461520.2011.611369)\.
- \[3\]J\. A\. Kulik and J\. D\. Fletcher\.Effectiveness of intelligent tutoring systems: A meta\-analytic review\.*Review of Educational Research*, 86\(1\):42–78, 2016\.[https://doi\.org/10\.3102/0034654315581420](https://doi.org/10.3102/0034654315581420)\.
- \[4\]H\. L\. Roediger, III and J\. D\. Karpicke\.Test\-enhanced learning: Taking memory tests improves long\-term retention\.*Psychological Science*, 17\(3\):249–255, 2006\.[https://doi\.org/10\.1111/j\.1467\-9280\.2006\.01693\.x](https://doi.org/10.1111/j.1467-9280.2006.01693.x)\.
- \[5\]D\. J\. Nicol and D\. Macfarlane\-Dick\.Formative assessment and self\-regulated learning: A model and seven principles of good feedback practice\.*Studies in Higher Education*, 31\(2\):199–218, 2006\.[https://doi\.org/10\.1080/03075070600572090](https://doi.org/10.1080/03075070600572090)\.
- \[6\]H\. Bastani et al\.Generative AI without guardrails can harm learning: Evidence from high school mathematics\.*Proceedings of the National Academy of Sciences*, 122\(26\):e2422633122, 2025\.[https://doi\.org/10\.1073/pnas\.2422633122](https://doi.org/10.1073/pnas.2422633122)\.
- \[7\]G\. Kestin, K\. Miller, A\. Klales, T\. Milbourne, and G\. Ponti\.AI tutoring outperforms in\-class active learning: An RCT introducing a novel research\-based design in an authentic educational setting\.*Scientific Reports*, 15:17458, 2025\.[https://doi\.org/10\.1038/s41598\-025\-97652\-6](https://doi.org/10.1038/s41598-025-97652-6)\.
- \[8\]T\. Gao, H\. Yen, J\. Yu, and D\. Chen\.Enabling large language models to generate text with citations\.In*Proceedings of EMNLP*, pages 6465–6488, 2023\.[https://doi\.org/10\.18653/v1/2023\.emnlp\-main\.398](https://doi.org/10.18653/v1/2023.emnlp-main.398)\.
- \[9\]N\. F\. Liu et al\.Lost in the middle: How language models use long contexts\.*Transactions of the Association for Computational Linguistics*, 12:157–173, 2024\.[https://doi\.org/10\.1162/tacl\_a\_00638](https://doi.org/10.1162/tacl_a_00638)\.
- \[10\]S\. Es, J\. James, L\. Espinosa Anke, and S\. Schockaert\.RAGAs: Automated evaluation of retrieval augmented generation\.In*Proceedings of EACL: System Demonstrations*, pages 150–158, 2024\.[https://doi\.org/10\.18653/v1/2024\.eacl\-demo\.16](https://doi.org/10.18653/v1/2024.eacl-demo.16)\.
- \[11\]J\. Saad\-Falcon, O\. Khattab, C\. Potts, and M\. Zaharia\.ARES: An automated evaluation framework for retrieval\-augmented generation systems\.In*Proceedings of NAACL: Human Language Technologies*, volume 1, pages 338–354, 2024\.[https://doi\.org/10\.18653/v1/2024\.naacl\-long\.20](https://doi.org/10.18653/v1/2024.naacl-long.20)\.
- \[12\]W\. Kwon et al\.Efficient memory management for large language model serving with PagedAttention\.In*Proceedings of the 29th Symposium on Operating Systems Principles*, pages 611–626, 2023\.[https://doi\.org/10\.1145/3600006\.3613165](https://doi.org/10.1145/3600006.3613165)\.
- \[13\]T\. Feng, S\. Liu, and D\. Ghosal\.CourseAssist: Pedagogically appropriate AI tutor for computer science education\.arXiv:2407\.10246, 2024\.[https://arxiv\.org/abs/2407\.10246](https://arxiv.org/abs/2407.10246)\.
- \[14\]M\. Jain et al\.Retrieval\-augmented tutoring for algorithm tracing and problem\-solving in AI education\.In*Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications \(BEA 2026\)*, pages 841–850, 2026\.[https://doi\.org/10\.18653/v1/2026\.bea\-1\.57](https://doi.org/10.18653/v1/2026.bea-1.57)\.
- \[15\]Ollama\.[https://ollama\.com/](https://ollama.com/)
- \[16\]Qdrant\.[https://qdrant\.tech/](https://qdrant.tech/)
- \[17\]FastAPI\.[https://fastapi\.tiangolo\.com/](https://fastapi.tiangolo.com/)
- \[18\]LangChain\.Short\-term memory\.Official software documentation, accessed September 2026\.[https://docs\.langchain\.com/oss/python/langchain/short\-term\-memory](https://docs.langchain.com/oss/python/langchain/short-term-memory)\.

## Appendix ACampus request path \(deployment contract\)

1. 1\.Student openshttps://tutor\.example\.edu/<slug\>through an LMS embed\. The embed is the intended entry path; the slug is not advertised to students as a standalone bookmark, and enrollment enforcement is not claimed here\.
2. 2\.IIS rewrites the slug to/?course=<id\>and proxies to Node A or Node B without requiring sticky session affinity\.
3. 3\.UI loads; student submits a turn with mode, optional chapter lock, and bounded browser\-held history for the selected course\.
4. 4\.API resolves course config→\\rightarrowcollection\. Quiz mode reads the bank; the conversation router selects scoped retrieval, social response, instant FAQ, or clarification according to the turn\.
5. 5\.The response returns through the gateway, optionally streamed\. Course answers include citations confined to the selected course, with any module\-scope fallback reported; social and clarification responses do not invent course citations\.