Maintaining Metadata Fidelity for Ecological Data
In digitizing conservation data from SeedsOfHawaii.org, metadata fidelity is paramount. Each species page is not merely a static article but a structured record of scientific, morphological, ecological, and regional attributes. The scraping pipeline we’ve developed captures every discrete data point—not just text—ensuring that no detail is lost from field to database.
Every species page contains the following critical data, which is extracted into structured JSON fields:
Family
Hawaiian Name
Seed Dimensions (min/max length, width, mass in mm and mg)
2D and 3D Shape
Texture
Geographic Dispersal
Federal Listing Status
PEPP Status
Native Status
Storage Behavior
Each field is parsed individually and preserved with exact labeling, allowing for fine-grained filtering, sorting, and modeling.
Rather than treating images as detached media, each image URL is scraped and stored in a field like:
These links serve both for display and for machine learning applications, where correct image-to-species mapping is vital. Storing URLs instead of detached binaries means images can be traced, updated, or re-used across systems.
Each species page often includes a paragraph or two of semantic description—covering morphology, ecology, conservation status, and cultural context. This is extracted cleanly into a single description field, stripped of HTML but retaining full prose:
With the above data preserved:
Users and researchers can query:
“All PEPP species under 3mm with pubescent texture”
“All endemic plants in the Northwestern Hawaiian Islands with Orthodox storage behavior”Front-end search facets (e.g., dropdown filters for island or conservation status) map directly to these JSON fields, ensuring accurate and consistent results.
Clustering algorithms can meaningfully group species based on trait vectors or island distribution without relying on inference.
Standards and Interoperability
To facilitate interoperability with global biodiversity databases, we use consistent field names and export schemas that can map to standards like Darwin Core, GBIF, or the IUCN Red List. For instance:
scientific_namemaps todwc:scientificNamenative_statuscan map todwc:establishmentMeansfederal_listingaligns withiucn:threatStatus
All output is UTF-8 encoded to preserve:
Hawaiian diacritics (e.g., ʻokina, kahakō)
Unicode botanical names
Cultural terminology important for ethical and scientific integrity
A Platform for Scalable Collaboration
By retaining every field in its original, uncompressed, unaltered form, we allow future researchers to:
Extend the dataset without re-scraping
Build portable vector databases (e.g., FAISS, HNSWLib, Annoy)
Train local image classifiers even with minimal labeled examples
Share self-contained index files across systems with minimal data degradation