Multi-sourced AI Tutor for Non-Native English Speaker Students

What it claims

A GPT-4 tutor for an intermediate Data Science programming course. It takes questions in Mandarin, answers in both English and Mandarin, and appends a glossary of technical terms. The paper attributes most of its value to two pre-retrieval steps:

• Machine translation into English before retrieval. On non-English posts, this makes retrieved context much closer to what the English post would have retrieved.
• A small-LLM "question extraction" (QE) step. It resolves bare references like "P2 Q10" or "FA22 Exam 2 Q10" into the actual question text. About 35% of the course's Piazza posts were of that kind, and the authors argue neither embedding nor keyword retrieval can resolve them.

How it works

flowchart TD
  A[Student post] --> B{Mandarin?}
  B -- yes --> C[Azure Translator to English]
  B -- no --> D[Question extraction]
  C --> D
  D --> E[Rule lookup in question bank]
  A --> F[FAISS retrieval, top 10]
  E --> F
  A --> G[GPT-4 bilingual answer + glossary]
  E --> G
  F --> G

• QE: Llama 3 8B runs under Ollama on one RTX 4090. A few-shot prompt has it emit a standardized question number. A rule-based lookup then fetches the question from a bank built two ways: past exam PDFs parsed with fitz, and Canvas quizzes pulled via the Canvas API.
• RAG: Built with LangChain and FAISS, using text-embedding-ada-002 embeddings over every course material type.
• Parsing: notebooks go through nbconvert to Python; slides and PDFs go to Markdown via pptx2md and fitz.
• Chunking: LangChain Markdown/HTML header splitters (a modified one for notebooks), then RecursiveCharacterTextSplitter.
• Each chunk carries its source name, so answers can cite it.
• Past exams and quizzes are deliberately excluded from similarity search so they don't crowd out lecture material.
• Generation: Azure OpenAI GPT-4. The prompt forbids revealing solutions or writing code snippets, requires source citations, and asks for answers in both languages plus a glossary of hard terms and error messages.

Evidence

Data. The authors collected 620 Piazza posts from a 400-student course.

• They dropped 238 posts containing images, because GPT-4 vision was too expensive before GPT-4o.
• That left 382 text posts: 90 logistics, 54 conceptual, 41 environment setup, and 197 programming.
• The Mandarin set was made by Google-translating these posts, with review by a native-speaker author.
• Judging was done by two native-Mandarin authors, with the course instructor (also an author) breaking ties.

QE accuracy on English posts (logistics / conceptual / env / programming): 97.78% / 90.74% / 100% / 97.46%.

Agreement of QE results with the English-post results:

RAG results. On English posts, 6.1 / 7.7 / 7.3 / 6.6 of the top 10 retrieved chunks were relevant. Overlap with the English retrievals, out of 10:

• Raw Mandarin input: 7.0 / 5.7 / 7.5 / 6.8.
• Translated input: 8.5 / 7.0 / 8.9 / 8.2.

Translating first therefore improves retrieval in every category, while making little difference to QE.

How to use it

Code and course materials are promised "upon publication" and are not available yet. The few-shot QE prompt and the generation prompts are linked via bit.ly URLs (https://bit.ly/3Lv3Eiw and https://bit.ly/3SeXvLg). The paper describes both translation APIs as swappable.

Limits

• Mandarin inputs are not real student questions. They are machine translations of English posts, so real student phrasing is untested.
• Text only. Image posts (38% of the corpus) were excluded.
• Conceptual questions are the weak spot for non-English input: about 72–74% QE agreement and the lowest retrieval overlap.
• Narrow, author-graded evaluation. It covers one course and one language, and the authors themselves did the grading, including the course instructor.
• IRB details are pending.
• The fetched text is truncated partway through the results, so the full answer-quality breakdown behind the 91-post figures was not visible.