Schema Creation: Suggested Web Scraping Pipeline for Comprehensive Data Collection
This basic process can be adopted by any institution with web-facing information on their collections OR offline material that can be fed into a python script. In our case, each page follows a consistent format that includes essential taxonomic, morphological, and conservation-related data. We recommend adopting this sort of precision for a myriad number of reasons, but chief among them is information integrity when it comes to vectorizing your knowledge base.
Scraping Pipeline Breakdown
As a working example, these are the key features of each web-facing species profiles on SeedsOfHawaii.org. The particulars aren’t as important (insofar as they’re contingent upon the organization’s mission) as the organizational consistency and adherence to whichever metrics are most salient.
In our case, each plant species entry contains:
Scientific name and family
Hawaiian name (if available)
Detailed seed metrics (length, width, mass)
Morphological descriptors (2D shape, 3D shape, texture)
Island-specific distribution
Federal and PEPP conservation status
Native/endemic classification
Seed storage behavior (e.g., Orthodox)
Descriptive paragraph and citation
One or more reference images
We recommend first generating a list of species URLs either from your site’s sitemap or an index page that links to every specimen or species profile:
For each species URL, the scraper performs an HTTP GET request to obtain the page’s HTML content.
Parsing and Field Extraction
Using an HTML parser such as BeautifulSoup (Python), the scraper extracts the following structured fields:
title: Scientific name (e.g., Abutilon eremitopetalum)family: Botanical family (e.g., Malvaceae)hawaiian_name: If available; often marked as “N/A”length_mm:[min, max]in millimeterswidth_mm:[min, max]in millimetersmass_mg:[min, max]in milligramsshape_2d,shape_3d: Morphological traitstexture: Surface description (e.g., Pubescent)distribution: List of Hawaiian islands where the species is foundfederal_listing: Endangered/threatened/etc.pepp_status: PEPP program statusnative_status: Native or Endemicstorage_behavior: Seed storage classificationdescription: Paragraph summary about the speciescitation: Citation text if providedimages: List of image URLs (seeds, plant forms)
All extracted content is stored in structured JSON format for each species, e.g.:
Your scraper should loop through all known species pages, writing each JSON entry to a local or cloud-based repository. This process results in a complete, structured, and versionable dataset — decoupled from the website itself and suitable for downstream research, ML training, or conservation planning.
This pipeline can be implemented in Python for reliability and ease of use with web data. Below is a simplified example snippet illustrating how a single page can be scraped and saved as JSON:
import requests
from bs4 import BeautifulSoup
import json
# Example: Scrape a single plant page
url = "https://seedsofhawaii.org/plants/ExamplePlantSpecies"
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
# Extract title (e.g., species name)
title = soup.find('h1').get_text(strip=True)
# Extract main text content (assuming it lies in a <div class="content">)
content_div = soup.find('div', class_='content')
text_content = content_div.get_text(separator="\n", strip=True)
# Extract all image links on the page
images = []
for img in soup.find_all('img'):
img_url = img.get('src')
if img_url:
images.append(img_url)
# Extract PDF links (if any)
pdf_links = []
for link in soup.find_all('a', href=True):
href = link['href']
if href.lower().endswith('.pdf'):
pdf_links.append(href)
# Extract video links (if any embedded videos, e.g., YouTube iframes)
video_links = []
for iframe in soup.find_all('iframe', src=True):
video_links.append(iframe['src'])
# Compile the data into a structured dictionary
plant_data = {
"title": title,
"url": url,
"text": text_content,
"images": images,
"pdf_links": pdf_links,
"video_links": video_links
}
# Save the data to a JSON file
with open("ExamplePlantSpecies.json", "w") as f:
json.dump(plant_data, f, indent=4)
print(f"Saved data for {title} to JSON.")
Code explanation: Fetch the HTML of a specimen page, parse it to extract relevant elements, and build a dictionary of fields. This structured data can then be saved as a JSON or txt file. In practice, the actual scraper would repeat this for all species URLs and handle nuances of the site’s HTML structure (such as different sections or additional metadata fields).
By capturing all content elements, this pipeline goes beyond basic text scraping – it preserves images, documents, and links so that nothing is lost. The resulting JSON dataset forms the backbone of the Seeds of Hawaiʻi digital collection, enabling downstream processing such as search indexing, data analysis, and integration with other databases.