Vector Database Examples - Seeds Of Hawaii

A brief Tutorial On Creating Local Vector Databases

Imagine being able to instantly search through complex datasets of text, images, or other high-dimensional data with precision and speed—all right on your local machine. With local tools like FAISS (Facebook AI Similarity Search), Annoy (Approximate Nearest Neighbors Oh Yeah), HNSWlib (Hierarchical Navigable Small World), and NMSLIB (Non-Metric Space Library), as well as emerging technologies like Chroma, PGVector, ScaNN, and Milvus, you can build custom vector databases that power intelligent search, recommendation engines, and data discovery applications. These libraries let you create highly efficient indices that drastically reduce query times, enabling real-time results for applications ranging from semantic text matching to image retrieval, even when handling millions of data points.

Overview: FAISS is one of the most mature libraries for similarity search. It’s ideal for high-dimensional vectors and offers both CPU and GPU modes.

Installation: Make sure you have Python installed, then:

				
					pip install faiss-cpu  # or faiss-gpu if using GPU mode

Sample Code Example:

				
					import numpy as np
import faiss

# Generate a sample dataset: 1000 vectors of dimension 128
d = 128  # dimension
nb = 1000  # number of database vectors
np.random.seed(1234)
data = np.random.random((nb, d)).astype('float32')

# Build a flat (brute-force) index
index = faiss.IndexFlatL2(d)  # L2 distance metric
index.add(data)  # add vectors to index

# Create a query vector
query = np.random.random((1, d)).astype('float32')

# Perform a search (find 5 nearest neighbors)
k = 5
distances, indices = index.search(query, k)

print("Nearest neighbor indices:", indices)
print("Distances:", distances)

Explanation: We generate random vectors. We construct a FAISS index (here, a basic flat index for simplicity). A query is run against the index, returning the top 5 nearest neighbors.

Overview: Annoy is developed by Spotify and is optimized for approximate nearest neighbor searches. It’s lightweight and works well with large datasets by memory-mapping.

Installation: Make sure you have Python installed, then:

				
					pip install annoy

Sample Code Example:

				
					from annoy import AnnoyIndex
import random

# Define parameters
d = 128  # dimensionality
n_trees = 10  # number of trees to build
nb = 1000

# Create an Annoy index using angular distance (other metrics like Euclidean are also available)
index = AnnoyIndex(d, 'angular')

# Add vectors
for i in range(nb):
    vector = [random.random() for _ in range(d)]
    index.add_item(i, vector)

# Build the index with the specified number of trees
index.build(n_trees)

# Query the index for the top 5 nearest neighbors for a random query vector
query_vector = [random.random() for _ in range(d)]
result = index.get_nns_by_vector(query_vector, 5)

print("Top 5 nearest neighbor indices:", result)

Explanation: Annoy builds a forest of trees to allow fast approximate queries. You add items one-by-one before building the trees. The query returns IDs of the closest items.

Overview: HNSWlib implements Hierarchical Navigable Small World graphs. It tends to be very efficient with a trade-off between speed and accuracy.

Installation: Make sure you have Python installed, then:

				
					pip install hnswlib

Sample Code Example:

				
					import hnswlib
import numpy as np

# Settings
d = 128  # dimension
nb = 1000  # number of elements

# Generate random data
np.random.seed(42)
data = np.random.random((nb, d)).astype('float32')

# Initialize index
p = hnswlib.Index(space='l2', dim=d)
p.init_index(max_elements=nb, ef_construction=200, M=16)
p.add_items(data)

# Set ef parameter for search (trade-off between speed and accuracy)
p.set_ef(50)

# Query: top 5 neighbors for a random vector
query = np.random.random((1, d)).astype('float32')
labels, distances = p.knn_query(query, k=5)

print("Nearest neighbor indices:", labels)
print("Distances:", distances)

Explanation: The index is built with parameters like M (maximum number of links) and ef_construction (construction time parameter) that affect performance. Adjusting ef during query time controls accuracy and speed.

Overview: NMSLIB provides a variety of index types optimized for non-metric spaces as well, making it versatile for different kinds of data.

Installation: Make sure you have Python installed, then:

				
					pip install nmslib

Sample Code Example:

				
					import nmslib
import numpy as np

# Data parameters
d = 128  # dimension
nb = 1000  # number of vectors

# Generate random dataset
np.random.seed(101)
data = np.random.random((nb, d)).astype('float32')

# Create an NMSLIB index with the 'hnsw' method
index = nmslib.init(method='hnsw', space='l2')
index.addDataPointBatch(data)
index.createIndex({'post': 2}, print_progress=True)

# Query for the 5 nearest neighbors using a random vector
query = np.random.random(d).astype('float32')
ids, distances = index.knnQuery(query, k=5)

print("Nearest neighbor indices:", ids)
print("Distances:", distances)

Explanation: NMSLIB is flexible in terms of the methods and spaces it supports. Here, we use it with the HNSW method to get a balance of speed and accuracy.

Additional Approaches

Chroma (for local in-memory vector stores)

What It Is: Chroma is designed as a lightweight vector store that’s easy to set up and integrate.
Usage: Typically used in conjunction with language model applications.
Embedding: Code examples would be similar in spirit to the above libraries, focusing on how vectors are added, indexed, and queried.

PGVector with PostgreSQL

What It Is: PGVector extends PostgreSQL to allow you to store and index vector data.
Usage: This is great if your project already runs on Postgres and you need to integrate vector similarity search into your SQL queries.
Embedding:
- You would provide SQL commands to install the extension and create an index.
- For example:

				
					-- Install PGVector (if not already installed)
CREATE EXTENSION IF NOT EXISTS vector;

-- Create a table with a vector column (dimension 128)
CREATE TABLE items (
    id serial primary key,
    embedding vector(128)
);

-- Insert sample data
INSERT INTO items (embedding) VALUES ('[0.1,0.2,...]');

-- Create an index for fast similarity search
CREATE INDEX ON items USING ivfflat (embedding vector_l2_ops);

Then, you can run similarity queries using SQL.

ScaNN or Milvus

ScaNN: Google’s library for efficient nearest neighbor search.
Milvus: A full-fledged vector database that can be run locally (often via Docker).