This project is a fully functional, end-to-end Retrieval-Augmented Generation (RAG) system built entirely on the Google Cloud Platform. It features a conversational AI agent, powered by Dialogflow CX and Google's Gemini models, that can answer user questions about public health topics (specifically, the flu) based on real, up-to-date information scraped from the Centers for Disease Control and Prevention (CDC) website.
The entire system is serverless, event-driven, and designed to be a low-cost, scalable solution for providing factual, AI-driven answers from a trusted knowledge base.
Features
- Conversational AI Interface: Users can interact with the assistant in natural language through a simple web chat widget.
- Automated Data Ingestion: A Cloud Scheduler job runs daily to trigger a web scraping function, ensuring the knowledge base stays current.
- Sophisticated RAG Pipeline:
- Retrieval: Uses Vertex AI Vector Search to find the most semantically relevant text chunks from the knowledge base.
- Augmentation: Retrieves the full text of relevant chunks from a Firestore database.
- Generation: Uses the powerful Gemini 2.5 Flash model to synthesize a helpful, conversational answer based only on the retrieved factual context.
- De-duplication Logic: The data pipeline intelligently overwrites existing data, preventing the accumulation of duplicate information from daily scrapes.
- Production-Ready Publishing: The Dialogflow agent is configured with separate Draft and Production environments, following best practices for safe deployment and testing.
Architecture
The project is composed of two main asynchronous workflows: a daily data ingestion pipeline and a real-time query pipeline.
Technology Stack
Google Cloud Platform Services
- Compute: Google Cloud Functions (Gen 2)
- AI and ML: Vertex AI (Vector Search, Embedding API, Gemini 2.5 Flash)
- Conversational AI: Dialogflow CX
- Storage: Google Cloud Storage, Firestore
- Automation and Ops: Cloud Scheduler, Cloud Build, Cloud Run, IAM, Cloud Logging
Key Python Libraries
functions-frameworkgoogle-cloud-firestoregoogle-cloud-storagegoogle-authrequestsbeautifulsoup4
The Journey: A Note on Real-World Implementation
Built and stabilized a modern cloud stack on a bleeding-edge platform by systematically uncovering and fixing deep, often undocumented issues, for example, stubborn container start-up failures, non-intuitive console navigation for setting up workflows, persistent agent misbehaviors and API endpoints failures. The work highlights a persistent, methodical, and deeply technical approach to cloud engineering and debugging.
Future Improvements
- Incorporate more data sources (for example, from the WHO) to enrich the knowledge base.
- Implement a content-hashing strategy in the scraper to only trigger the processing pipeline when source documents have actually changed.
- Add conversation history to allow for follow-up questions.
- Stream responses from Gemini for a more interactive user experience.
Demo
A short walkthrough of the assistant experience and how the RAG pipeline produces grounded answers from the knowledge base.