Student: Jiayi Weng, Master of Urban Science, at the Center of Urban Science + Progress at the Tandon School of Engineering.
Advisors: Dr. Zhaoxi Zhang, Faculty Fellow in the Center of Urban Science + Progress at the Tandon School of Engineering and Dr. Tamir Mendel, Postdoctoral Researcher in the Department of Technology of Management and Innovation, NYU Tandon School of Engineering, New York University.
Project Abstract: This research explores the impact of different embedding models on the accuracy and effectiveness of AI chatbots in responding to urban-related queries. While AI chatbots have been increasingly applied in various urban domains, they also face the problem of misinformation. For example, NYC’s ” MyCity” chatbot once provided incorrect information to business owners, which could lead to illegal actions. Such issues, along with the increasing need for citizen participation, underscore the need for more reliable AI chatbots in urban governance. This study leverages Retrieval-Augmented Generation (RAG) technique, which enhances large language models (LLMs) by integrating external knowledge bases. The study uses LangChain to build a pipeline, focusing specifically on the embedding models used in it, to evaluate their performance in retrieving text from urban documents. At the current stage, we compared 3 popular open-source embedding models on HuggingFace and the Llama embedding model, and evaluated key metrics such as answer relevance and context accuracy using the RAGAS tool. Preliminary results show that all-MiniLM-L6 model outperforms other models in terms of accuracy and retrieval ability. This research highlights the importance of fine-tuning LLM pipelines to optimize AI chatbots in specific domains such as urban governance, and lays the foundation for future work including using larger document datasets, more complex models, and applying multimodal language models.
Pictures from the demo survey:
- Zhaoxi Zhang: Zhaoxi.Zhang@nyu.edu
- Jiayi Weng: jw8608@nyu.edu