Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering

Standard RAG treats tickets as flat text split into fixed-size chunks. That throws away structure (tickets are "related to", "cloned from" or "caused by" other tickets) and can split a problem from its fix, which leads to incomplete answers.

• Parses each ticket into a tree of sections (summary, description, priority, steps to reproduce, root cause). Fixed fields use rules, and an LLM guided by a YAML template handles the free text.
• Connects tickets into a graph using explicit links from the issue tracker plus implicit links when ticket titles are similar enough (cosine similarity of embeddings above a threshold).
• Embeds text-heavy sections with E5/BERT into a vector database. Because each section is embedded on its own, chunking never splits one section from another.
• At query time, an LLM extracts entities and intent from the question, embedding search finds the most relevant tickets, then the LLM writes a Cypher query against Neo4j to pull exactly the sub-graph it needs and answers from it. It falls back to plain text retrieval if the graph query fails.

• Same LLM (GPT-4) and embedding model (E5) as the baseline, but retrieval MRR rose from 0.522 to 0.927.
• Answer quality improved on every text metric: BLEU 0.057 → 0.377, METEOR 0.279 → 0.613, ROUGE 0.183 → 0.546.
• In a roughly six-month randomized rollout to LinkedIn's support team, the group using the tool resolved issues faster: mean 40h → 15h, median 7h → 5h, P90 87h → 47h.

• The graph template is written by hand for customer-support tickets. Extracting templates automatically is listed as future work.
• The graph isn't updated live as new questions come in, and the paper only tests the customer-service domain.
• Offline results come from a curated "golden" dataset; the paper is a short five-page industry write-up.