Event-Driven Batch Processing Pipeline

• We designed and implemented a fully serverless, event-driven batch processing pipeline on AWS.
• The system is not a chatbot or RAG architecture — it processes entire cohorts of courses once and generates comprehensive, structured curriculum maps that persist in Aurora PostgreSQL and S3 for downstream search and reporting.
• IMSCC files uploaded to S3 automatically trigger the pipeline after a configurable 10-minute debounce period, ensuring all files in a cohort are fully uploaded before processing begins.

Content Extraction and Hierarchical Chunking

• Unpacked and parsed IMSCC files, extracting imsmanifest.xml to build full resource graphs and course hierarchies
• Implemented file-type-specific extraction: HTML via BeautifulSoup, PDF via PyMuPDF, PPTX via python-pptx, DOCX via python-docx, QTI/XML for assessments, and Claude vision OCR for image-only pages
• Applied hierarchical chunking at 800 tokens with 100-token overlap using tiktoken, structured across three levels: Course → Module/Unit → Resource/Item
• Expanded large PDFs (over 25 pages) into 25-page parts to fit within batch time budgets; compressed images to max 1024px JPEG quality 70, reducing Bedrock token usage by 60-70%
• Generated Titan Embed Text v2 embeddings (1536 dimensions) for every chunk and stored them in Aurora PostgreSQL with pgvector

AAMC Competency Mapping and Course Summarization

• During chunking, Claude extracted learning objectives from each resource and mapped them to the four AAMC competency domains
• GenerateCourseSummary Lambda computed AAMC domain coverage percentages per course, with direct strategy (up to 120k tokens) and map_reduce strategy (batched summarization) for larger courses
• Structured JSON course summaries captured: title, description, key topics, prerequisite topics, AAMC domain coverage, and detailed module breakdowns

Two-Pass Curriculum Map Generation

• Pass 1 — Prerequisite Dependency Map: Claude analyzed each course's prerequisite topics against other courses' key topics to build an explicit dependency graph
• Pass 2 — Blueprint Generation: The dependency map was injected as ordering constraints; Claude generated a module sequence respecting all prerequisite relationships
• Pass 3 — Core and Supplementary Maps: Two full curriculum maps generated from the approved blueprint — one for required AAMC competencies, one for supplementary learning
• Outputs delivered as Excel workbooks (openpyxl) and JSON stored in S3, with structured records written to Aurora for search

Hybrid Semantic Search API

• Implemented hybrid search combining Titan v2 vector similarity (cosine) against chunk embeddings and PostgreSQL ILIKE keyword matching across chunk text and module titles
• Chunks appearing in both vector and keyword results ranked highest; Claude generates a natural language narrative summarizing top results with source attribution
• Exposed via API Gateway HTTP v2 GET /search endpoint with cohort-level filtering by input folder

Step Functions Orchestration for Scale

• AWS Step Functions orchestrated the complete pipeline with parallel Map states — up to 5 courses processed concurrently, with nested Map states for up to 3 batches per course
• Each Lambda invocation used ThreadPoolExecutor (3-5 workers) for concurrent Bedrock API calls, maximizing throughput within Lambda's timeout ceiling
• Built-in retry logic, checkpoint recovery, and idempotent reprocessing (force_reprocess flag) ensured resilience across long-running cohort runs