Few-Shot Image Classification with Pre-Trained Models
Many species in the Seeds of Hawaiʻi collection have only a handful of images available – often five or fewer images per species – which poses a challenge for traditional image classification algorithms. To address this, the project leverages transfer learning with state-of-the-art pre-trained models (such as ResNet, YOLOv5, and EfficientNet) to perform image classification in a few-shot learning context. Our goal is to build a system that can suggest the identity of a seed or plant from a photograph, despite the scarcity of training examples for each species, by capitalizing on the rich features learned from large datasets like ImageNet.
We suggest experimenting with multiple CNN architectures:
ResNet-50: A deep residual network that has become a workhorse for image recognition tasks. We’ve had some early success using ResNet as a baseline feature extractor; its convolutional layers (pre-trained on millions of images) can detect generic patterns like shapes, edges, and textures present in seed photos. We fine-tune the final classification layer (and in some cases a few last residual blocks) on our specific set of species. Given only a few images per class, we typically freeze most of the early layers to avoid overfitting, and apply heavy data augmentation (random rotations, flips, color jitter, etc.) to make the most of each example.
EfficientNet-B0/B1: EfficientNet models are known for achieving high accuracy with fewer parameters by scaling depth, width, and resolution in a balanced way. These models, pre-trained on ImageNet, offer an excellent starting point for our task. We found that EfficientNet can sometimes outperform ResNet on our seed images due to its efficiency in learning fine-grained details, especially when combined with aggressive augmentation. The smaller size of EfficientNet-B0 also means we can train and deploy it with limited compute resources – an important consideration if the model will be used in the field on mobile devices for identification.
YOLOv5: While YOLO (You Only Look Once) is primarily an object detection model, we leverage YOLOv5 for two purposes. First, in cases where images contain multiple objects (e.g., a photo of several seeds or a seed within a fruit), YOLOv5’s detection capability helps localize and crop individual seed instances before classification. Second, we can configure YOLOv5 in a classification mode or use its backbone network for feature extraction. YOLOv5’s advantage is its speed and its prowess in learning from smaller datasets with appropriate transfer learning. By detecting the region of interest (the seed), we reduce background noise and feed a focused image to the classifier, which improves accuracy when data is scarce.
With as few as 1–5 images per species, overfitting is the biggest concern. Our approach to training includes:
Transfer Learning: As mentioned, we never train from scratch. The pre-trained weights give the model a rich starting point. For example, the first layers of ResNet might already recognize “texture” or “shape” features that are relevant to seed coats or leaf venation.
Freezing Layers: We often freeze the majority of layers (all convolutional blocks) and only train the final fully connected layer (which maps to our species classes). This drastically reduces the number of parameters that need tuning (<1% of the total model parameters in many cases), which is feasible with few samples. In some cases, we unfreeze one or two of the last convolutional layers to fine-tune higher-level feature representations if we have slightly more data to allow it.
Data Augmentation: Each original image is transformed in dozens of ways to simulate a larger training set. For instance, a single seed photo can be rotated to various angles (since orientation shouldn’t matter for identification), cropped or zoomed to different scales, given slight color shifts (to mimic different lighting conditions), or have noise added. These augmented images are treated as additional training samples. This technique helps the model generalize better, so it’s not just “memorizing” the exact appearance of the one or two seed photos it saw.
Cross-Validation: With so little data, we use leave-one-out or k-fold cross-validation where feasible to evaluate the model. This means repeatedly training on all but one image and testing on the held-out image, cycling through. While this isn’t robust in a traditional sense, it ensures the model isn’t just echoing back the training examples and gives a sense of how it might perform on a truly unseen image of the species.
Because species-level classification can be uncertain with so few examples, the system is designed to interpret the model’s confidence percentages and provide higher-level taxonomic suggestions when appropriate:
The output of our image classifier is a probability distribution across all species in the dataset (via a softmax layer in the network). In an ideal scenario, the correct species would have an overwhelmingly highest probability. However, with limited training data, the model might be unsure – for example, it might give 30% to species A, 25% to species B, and smaller percentages to others.
We set a confidence threshold (for instance, 50% for top prediction confidence). If the top predicted species has less than 50% confidence, the system will refrain from making a definitive species ID. Instead, it looks at the higher-level groupings of the top predictions:
Genus suggestion: We check if the top few predicted species (say 3-5 species with highest probabilities) share the same genus. If, for example, Species A and Species B in the top guesses are both from genus Metrosideros, their combined probability might be 55%. This gives a clue that the image is likely of genus Metrosideros, even if the model isn’t certain which exact species. In such a case, the system would output a suggestion like “Genus likely: Metrosideros (with ~55% confidence)” rather than naming a species outright.
Family suggestion: Similarly, we aggregate probabilities by family. It might be that the top predictions are from different genera but the same family. For instance, if an unknown seed is either a Cyanea (genus in Campanulaceae) or a Clermontia (another genus in Campanulaceae) with combined confidence, the model can confidently say “this is in the Bellflower family (Campanulaceae)” even though at species level it’s torn between two possibilities. Family-level suggestions are broader, but they provide useful information for conservationists—knowing the family can inform traits like germination methods or pollinator type.
The hierarchical approach means the AI embraces uncertainty gracefully. Rather than giving a possibly incorrect species name (which could be misleading, especially for critical decisions like identifying an endangered plant in the wild), it provides a sensible fallback that is more likely to be correct. This mirrors how human experts behave as well: a botanist might say, “I’m not sure which species, but it looks like an aster (family Asteraceae).”
We also include the raw confidence scores in the output for transparency. A researcher using the system can see that, for example, the model was 45% confident it’s Species X, 30% for Species Y, which informs how much trust to place in the automated suggestion. Over time, if more images are added to the training set (e.g., new photographs taken in the field), these confidence metrics can be retrained and improved.