GPR1200

A challenging benchmark for general-purpose content-based image retrieval

Similar to most vision related tasks, deep learning models have taken over in the field of content-based image retrieval (CBIR) over the course of the last decade. However, most publications that aim to optimize neural networks for CBIR, train and test their models on domain specific datasets. It is therefore unclear if those networks can be used as a general-purpose image feature extractor. After analyzing popular image retrieval test sets we decided to manually curate GPR1200, an easy to use and accessible but challenging benchmark dataset with 1200 categories and 10 class examples. Classes and images were manually selected from six publicly available datasets of different image areas, ensuring high class diversity and clean class boundaries.

Number of Images: 12000

Number of Classes: 1200

Number of Subdomains: 6

Key Features

Domain-Rich Image Dataset: GPR1200 encompasses a diverse array of real-world domains, capturing a wide spectrum of content commonly encountered in photography. This includes landmarks, distinctive objects, diverse flora and fauna, commercial products, artistic sketches, and human faces. The dataset consists of a total of 1200 classes with 10 images for each class

Manual Image Curation: Although the GPR1200 dataset is derived from subsets of pre-existing datasets, meticulous manual selection was employed to prioritize solvability. Each class and image was carefully chosen to eliminate class overlap, ensuring that visually similar images are not grouped under different classes. Furthermore, all 10 images within each class were selected to consistently share distinct visual characteristics, minimizing the presence of intra-class outliers and enhancing the dataset's reliability.

Robust and Intuitive Evaluation Protocol: The GPR1200 dataset simplifies and strengthens the evaluation process with a clear and robust protocol. Unlike traditional evaluation datasets that require a query and database split, GPR1200 treats every image as a query, calculating the mean-average-precision (mAP) metric across all queries to derive the final result. This approach ensures consistent comparability across related research, while reducing the potential for confusion and complexity, making it easier for researchers to assess and benchmark performance.

How to Use GPR1200?

Download Instructions: The images are available under this link. Unzipping the content will result in an "images" folder, which contains all 12 000 images. Each filename consists of a combination of the GPR1200 category ID and the original name: "{category ID}_{original name}.jpg"

Evaluation Examples: This github repository shows how to evaluate the GPR1200 performance of your model.

License Information

This dataset is available for non-commercial research and educational purposes only and the copyright belongs to the original owners. If any of the images belong to you and you would like it removed, please kindly inform us, we will remove it from our dataset immediately. Since all images were curated from other publicly available datasets, please visit the respective dataset websites for additional license information:

Publications

List of relevant publications

Improving Image Encoders for General-Purpose Nearest Neighbor Search and Classification

Konstantin Schall, Kai Uwe Barthel, Nico Hezel, Klaus Jung
2023 International Conference on Multimedia Retrieval (ICMR '23)

This paper evaluates vision foundation models for content-based image-to-image retrieval, focusing on zero-shot retrieval and k-NN classification. By benchmarking and fine-tuning these models with diverse datasets, the study shows improved generalization, making them effective as general-purpose embedding models for image retrieval.

GPR1200: A Benchmark for General-Purpose Content-Based Image Retrieval

Konstantin Schall, Kai Uwe Barthel, Nico Hezel, Klaus Jung
28th International Conference on Multimedia Modeling (MMM 2022)

We introduce GPR1200, a new benchmark dataset designed to evaluate the generalization quality of image retrieval models across diverse categories. Our experiments show that large-scale pretraining significantly enhances retrieval performance.

List all publications