The Full Flow of RAG Data Preparation — Selection, Cleansing, Chunking, Metadata, and Evaluation Sets
While working on the FEMS project, I got curious about 'what and how should go into a vector DB.' Key insight — data should be selected backward from 'questions that need to be retrieved,' not from the 'domain' criterion. Also covers cleansing / the effect of metadata (document title, section path) before chunks / OCR preprocessing for analog data / building an evaluation set — the full flow of the RAG data pipeline.
