Ankit Singh Chauhan

Ankit Singh Chauhan

AI Researcher — Large Language Models (agents, RAG, PEFT/QLoRA), Causal Inference and Probes, Knowledge Graphs

LLM-Driven Community Asset Knowledge Graph (Mistral-7B + Bedrock)

Published:

Problem

Community-research transcripts — interviews, surveys, focus-group notes — describe rich local ecosystems of people, organizations, services, and places. To be useful for analysis, that text has to become a queryable graph of assets and the relationships between them. Hand-coding tens of thousands of documents is not viable, and off-the-shelf LLM extraction on this kind of long, messy text gives shallow, inconsistent results.

Approach

Extraction pipeline

  • Built a two-pass LLM extraction pipeline that ingests raw documents and writes assets and their relationships into a Neo4j property graph — 8.3k human-reviewed records to date.
  • Fine-tuned Mistral-7B with LoRA, combined with retrieval-augmented prompting over the project ontology, to push structured-extraction accuracy from 0.32 → 0.78 Triple-F1.

Platform on Amazon Bedrock

  • Designed the extraction platform on Amazon Bedrock Agents (AWS Lambda, DynamoDB).
  • A Claude Haiku classifier routes each document between Claude Sonnet and the fine-tuned Mistral-7B based on extraction difficulty — giving roughly 20× lower per-token cost than running Sonnet across the board, with no drop in graph quality on the easy cases.

Human-in-the-loop review

  • Developed a React + Cytoscape.js front-end backed by FastAPI for graph review and correction.
  • Every researcher correction is captured as an audit trail and as training data for the next fine-tune.
  • Annotation quality validated at Cohen’s κ = 0.82 on a double-annotated sample.

Results

  • Triple-F1: 0.32 → 0.78 on structured graph extraction after LoRA + RAG.
  • 0.82 Cohen’s κ inter-annotator agreement on the review-UI–generated labels.
  • ~20× lower per-token cost from Haiku-routed dispatching versus an all-Sonnet baseline.
  • Platform is in active use by the partnering research group for mapping community assets across multiple geographies.

Tech stack

Python · PyTorch · HuggingFace Transformers · PEFT/LoRA · Mistral-7B · Amazon Bedrock · AWS Lambda · DynamoDB · FastAPI · React · Cytoscape.js · Neo4j · Cypher