ECCV Marine Vision Workshop 2026 · Oral Presentation

WildFin: An In-the-Wild Dataset for Fish Behavioral Recognition

Abigail G. Grassick^1,*iD, Jerome Tze-Hou Hsu^1,*iD, Ethan Lin^1,*iD, Ziang Liu¹iD, Max Whitton¹, Madelyn Hair²iD, Liam Gutierrez², Haozheng Yu¹iD, Kristin Branson³iD, Vivek Jayaraman³iD, Michael A. Gil²iD, Andrew M. Hein¹iD, Jennifer J. Sun¹iD

¹ Cornell University ² University of Colorado Boulder ³ HHMI Janelia Research Campus

arXiv Code Dataset

Abstract

Recent advances in field technology have led to a massive influx of in-the-wild video data for ecological science. The primary bottleneck in leveraging this data is the high cost of expert annotation. While computer vision offers a potential solution, current models frequently fail when deployed in complex marine environments. To characterize these failures, we introduce WildFin, a novel benchmark for fish behavior recognition collected and annotated by ecologists. WildFin spans two critical real-world paradigms: stationary cameras monitoring groups of fish and dynamic divers following individual subjects. The dataset represents a massive curation effort, involving 1,350 hours of fieldwork and 600 hours of expert annotation to produce 9 hours of behavioral data with over 2 million frame-by-frame labels. We benchmark modern vision foundation models and quantify tradeoffs between static and spatiotemporal architectures, revealing the substantial gap that remains between current model capabilities and the demands of real-world underwater behavioral analysis.

Motivation

Marine ecology pipeline from raw footage through manual annotation or computer-vision behavior classification to a behavior distribution

Many in-situ or “in the wild” marine ecology pipelines require the annotation of fish behavior from raw video. Historically, domain experts spend many hours manually annotating this behavior data (including in our work), and thus it is extremely expensive to gather enough data to generate any downstream ecological insights.

With recent advances in computer vision, the automation of labeling ecological behavior data is becoming increasingly possible. WildFin is a new benchmark to evaluate the performance of computer vision models on fish behavior classification tasks in real-world ecological data --- the data that ecologists actually use for science.

In addition to benchmarking vision model capabilities, we are excited to see how WildFin can bridge the gap between computer vision research and the practical challenges of ecological applications.

Dataset Details

WildFin is composed of two datasets, CoralCam and FishFollow, that benchmark fish behavior recognition across two different real-world data-collection paradigms. We provide short Colab notebooks showing how to load and work with each dataset (CoralCam / FishFollow). We recommend using our codebase VisionBench when training models on the dataset. It is a modular and scalable PyTorch framework for video classification tasks and handles many of the challenges of working with large amounts of video data.

Stationary · Multi-agent

CoralCam

Open in Colab

Fixed cameras observe schools of brown chromis across 12 wild reef sites in Curaçao, capturing dense social behavior amid occlusion, small fish, and complex coral backgrounds.

Footage: 1.2 hours · 258,998 frames
Capture: 12 sites · 1080p at 60 fps
Behavior labels: 213 verified tracks · 3 behaviors
Detection labels: 21,116 boxes · 3 species
Quality control: Inter-annotator F1 0.74

CoralCam Ethogram

CoralCam Detection Species Distribution

Stacked bar showing the overall species distribution across all CoralCam detection boxes — Overall species composition across 22,811 CoralCam detection boxes: 3,263 bi-color damselfish, 2,627 bluehead wrasse, and 16,921 brown chromis boxes. Select to enlarge.

How it was constructed

Expert bounding boxes trained a YOLOv8 detector (AP@0.5 74.5), detections were linked with BoTSort, and ecologists then verified selected tracks and labeled behavior frame by frame. Mouth visibility identified frames where behavior could not be judged reliably.

Benchmark task Classify behavior on tracked individuals in crowded, stationary reef scenes.

Mobile · Focal-follow

FishFollow

Open in Colab

Divers follow individual parrotfish and surgeonfish through wild reefs in Curaçao, preserving the camera motion and viewpoint shifts of real behavioral field observations.

Footage: 8 hours · 1,799,894 frames
Capture: 81 videos · 4K at 60 fps
Behavior labels: 20 behaviors · frame-level
Quality control: 35% independently double-annotated

FishFollow Ethogram

How it was constructed

Because each recording follows one focal fish, trained annotators labeled behavior directly on every frame without an upstream tracking stage. Agreement reached F1 0.78 at 0.25 seconds and F1 0.93 at 0.5 seconds.

Benchmark task Classify behavior through continuous camera motion, habitat transitions, and scale changes.

Benchmark

Modern vision foundation models learn reusable representations from large-scale image and video data, making them promising for ecological tasks where expert labels are scarce and expensive. We use WildFin to test whether these general features transfer to real-world fish behavior recognition and when explicit temporal modeling is necessary.

What We Evaluate

We compare image backbones (DINOv3 and ResNet50) with video backbones (VideoMAE and V-JEPA 2). Image models process individual frames, while video models process 16-frame clips to capture motion and interaction cues.

Adaptation Protocol

Pre-trained backbones remain frozen while lightweight mean- or cross-attention-pooling and two-layer classification heads are trained. Performance is measured with macro F1, precision, and recall using a ±7-frame tolerance for uncertain behavior boundaries.

WildFin benchmarking pipeline comparing frozen image and video foundation models with a trainable classification head — Our benchmarking pipeline for WildFin. We evaluate frozen image and video foundation models with different classification head tuning on the behavior classification task.

Main Results

CoralCam

Temporal cues unlock dynamic behaviors

VideoMAE-L achieves the strongest overall macro F1 (0.450). Image models remain competitive for visually defined biting behavior, but nearly fail on aggression, where VideoMAE-L reaches 0.332. This gap shows that interaction-driven behaviors require temporal context.

FishFollow

Appearance helps, but rare behaviors remain difficult

Video models lead overall and on movement, while DINOv3-L performs best on habitat labels that rely heavily on static appearance. Across architectures, performance remains low for long-tailed bite and social-interaction classes.

Other Takeaways

Class imbalance matters

Naive random sampling produces near-zero performance on rare behaviors. Balanced sampling is essential, while focal loss generally trades modest precision for improved recall.

Pooling is model-dependent

Cross-attention pooling improves most backbones by emphasizing informative tokens, but mean pooling works substantially better for V-JEPA 2's distributed representations.

Full fine-tuning can overfit

A fully tuned ResNet50 is competitive on the smaller CoralCam task, but its advantage disappears on FishFollow, where robust frozen DINOv3 features transfer better.

Comprehensive results and full ablation tables are provided in the paper and appendix.

If you find WildFin useful, please consider citing our paper:

@misc{grassick2026wildfininthewilddatasetfish,
  title         = {WildFin: An In-the-Wild Dataset for Fish Behavioral Recognition},
  author        = {Abigail G. Grassick and Jerome Tze-Hou Hsu and Ethan Lin and Ziang Liu and Max Whitton and Madelyn Hair and Liam Gutierrez and Haozheng Yu and Kristin Branson and Vivek Jayaraman and Michael A. Gil and Andrew M. Hein and Jennifer J. Sun},
  year          = {2026},
  eprint        = {2608.21281},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2608.21281}
}