Portable Vector Index: Exporting & Sharing Compressed Knowledge

A cornerstone of the project’s technical infrastructure is the use of vector indices to enable semantic search and analysis of the seed collection. After gathering the textual and visual data for each species, we convert these rich descriptions into high-dimensional vectors (embeddings) using machine learning models. For instance, the semantic copy of each species can be transformed into a vector in an embedding space where similar species end up near each other (based on textual similarity of their descriptions). Likewise, images can be embedded in a vector space using pre-trained CNN encoders so that visually similar seeds or fruits are clustered. These embeddings allow rapid similarity search – a user’s query or an unknown sample can be turned into a vector and nearest neighbors in the index reveal related species.

To support collaboration and reproducibility, our project emphasizes the ability to export and import full vector index files. Whether using Facebook’s FAISS library for similarity search, Spotify’s Annoy, HNSWlib, or a cloud service like Pinecone, we ensure that our indices (the data structures containing all the vectors and their arrangement for fast searching) can be saved to disk and shared. This portability has several key benefits:

  • Seamless Data Transport: Researchers at different institutions can work with the exact same vector index without needing to rebuild it. For example, we can generate a FAISS index of all seed description embeddings and share the index file (perhaps a few hundred MB) with a collaborator. They can load it on their machine and immediately have the same semantic search capability as we do, without re-running the embedding model on every record. This is crucial in cases where computing resources or access to raw data might be limited.

  • Compressed Knowledge, Minimal Hallucination: The vector index is essentially a compressed representation of the entire dataset’s knowledge. By sharing the index rather than just a summary or, say, an AI model, we avoid the risk of hallucination in downstream use. For instance, if someone were to rely solely on a language model to answer questions about the seeds, the model might fabricate answers not in the data. Instead, with a vector index, any query can be answered by retrieving the nearest actual entries from the seed data, ensuring the response is grounded in real, original information. In other words, the index doesn’t “guess” – it pulls from factual neighbors – thereby preserving fidelity to the source data.

  • Efficient Analysis: Querying a vector index is extremely fast (milliseconds to find nearest neighbors among thousands of vectors). This speed, combined with portability, means researchers can integrate the index into interactive tools or AI systems for real-time analysis. A botanist could, for example, input a paragraph about an unknown plant and get instant suggestions of species with similar descriptions from our index.

  • Minimal Information Loss: Unlike heavy compression or summary techniques, using a vector embedding (especially with a high dimensional space and a suitable model) retains subtle nuances of the data. Two species with slightly different seed morphology might end up with different embeddings. By sharing the full index, we ensure that such subtle differences are not averaged away. The nearest-neighbor relationships remain exact as computed, so if a species is unique, it will remain an outlier in the vector space rather than being lost in a summary.

Below is an example of how you might create a FAISS index for text embeddings and then save and load it for sharing:

				
					import faiss
import numpy as np

# Assume we have a 2D NumPy array `embeddings` of shape (N_species, D) 
# containing the semantic vector for each species (e.g., from a language model).
N, D = embeddings.shape

# 1. Build a FAISS index (Flat L2 index in this example)
index = faiss.IndexFlatL2(D)  # L2 distance for similarity
index.add(embeddings)         # Add all N vectors to the index

print(f"Built FAISS index with {index.ntotal} vectors of dimension {D}.")

# 2. Save the index to a file for portability
faiss.write_index(index, "seeds_vector_index.faiss")
print("FAISS index saved to seeds_vector_index.faiss")

# ... Later or on another machine, the index can be loaded:
loaded_index = faiss.read_index("seeds_vector_index.faiss")
print(f"Loaded FAISS index with {loaded_index.ntotal} vectors.")

# 3. Use the loaded index to perform a sample search (e.g., find 5 nearest neighbors of the first species)
query_vector = embeddings[0:1]  # take the first species vector as a query
distances, indices = loaded_index.search(query_vector, k=5)
print("Nearest neighbors of species[0]:", indices[0], "at distances", distances[0])

You might (using something like the above script) generate a flat L2 FAISS index from an array of embedding vectors. The index is then written to disk (write_index), producing a file that encapsulates the entire vector database.

Later, you can test the reading of this file (read_index) to reconstruct the index. A test search should show that the loaded index returns nearest neighbors identical to the original – confirming no loss of information. In practice, we might use more complex indices (like IVF for large datasets or HNSW for efficient approximate search), but all these index types in FAISS (and similarly in Annoy or HNSWlib) support serialization to files.

We have similarly experimented with Annoy (Approximate Nearest Neighbors) and HNSWlib, both of which allow saving indexes (e.g., Annoy indexes are saved as .ann files). If using a cloud-based vector database like Pinecone, the concept of export is a bit different – one can typically use their API to fetch all vectors or rely on their persistent storage – but we prioritize solutions that guarantee our ability to extract and share our data in full.

This ensures longevity and flexibility: even if the original website funding is in question, the knowledge encoded in our vector indices can be preserved and passed on to other systems or researchers, avoiding any single point of failure.