Break the Vector Search Dependency for Robust RAG Systems