Publications

Welcome to the publications section, where we present our research papers along with accompanying code and presentations, where available, to provide a deeper insight to our contributions to the field.

If you would like to receive a publication that is not freely available, please feel free to contact us.

Creating Sorted Grid Layouts with Gradient-based Optimization

Kai Uwe Barthel, Florian Tim Barthel, Peter Eisert, Nico Hezel, Konstantin Schall
2024 International Conference on Multimedia Retrieval (ICMR '24)

This paper introduces a gradient-based optimization method for efficiently arranging high-dimensional vectors on 2D grids, using a novel loss function to balance valid permutations and similarity-based arrangement for improved sorting quality.

Libro - Lifelog Search Browser

Nico Hezel, Konstantin Schall, Bruno Schilling, Klaus Jung, Kai Uwe Barthel
7th Annual ACM Workshop on the Lifelog Search Challenge (LSC '24)

This paper introduces Libro, a powerful tool for interactive image retrieval and browsing, which won the Runner-up and the Best New Entrant awards at the Lifelog Search Challenge 2024.

An Exploration Graph with Continuous Refinement for Efficient Multimedia Retrieval

Nico Hezel, Kai Uwe Barthel, Konstantin Schall, Klaus Jung
2024 International Conference on Multimedia Retrieval (ICMR '24)

As datasets get larger, searching within them becomes slower. This paper presents a continuous refining Exploration Graph (crEG) for Approximate Nearest Neighbor Search in large multimedia databases. The new graph provides a best balance between search speed and accuracy.

Optimizing the Interactive Video Retrieval Tool Vibro for the Video Browser Showdown 2024

Konstantin Schall, Nico Hezel, Kai Uwe Barthel, Klaus Jung
30th International Conference on MultiMedia Modeling (MMM 2024)

Vibro is an interactive video retrieval system that won the Video Browser Showdown in 2022 and 2023. This paper outlines its core concepts and recent updates for the 2024 competition. It also proposes an evaluation method used to optimize the system's embedding model for image- and text-to-image retrieval.

Fast Approximate Nearest Neighbor Search with a Dynamic Exploration Graph using Continuous Refinement

Nico Hezel, Kai Uwe Barthel, Konstantin Schall, Klaus Jung
arXiv (2023)

We introduce the Dynamic Exploration Graph (DEG), a graph-based algorithm that improves search and exploration efficiency in approximate nearest neighbor search. Our method outperforms existing algorithms by continuously optimizing the graph structure, leading to increased search efficiency and better handling of both indexed and unindexed queries.

navigu.net: NAvigation in Visual Image Graphs gets User-friendly

Kai Uwe Barthel, Nico Hezel, Konstantin Schall, Klaus Jung
2023 International Conference on Multimedia Retrieval (ICMR '23)

We propose a solution that significantly reduces search times and increases efficiency in visual search systems for large image collections. By combining two separate image graphs, our method enables fast approximate nearest neighbor search and real-time visual exploration through a web browser.

Improving Image Encoders for General-Purpose Nearest Neighbor Search and Classification

Konstantin Schall, Kai Uwe Barthel, Nico Hezel, Klaus Jung
2023 International Conference on Multimedia Retrieval (ICMR '23)

This paper evaluates vision foundation models for content-based image-to-image retrieval, focusing on zero-shot retrieval and k-NN classification. By benchmarking and fine-tuning these models with diverse datasets, the study shows improved generalization, making them effective as general-purpose embedding models for image retrieval.

Improved Evaluation and Generation Of Grid Layouts Using Distance Preservation Quality and Linear Assignment Sorting

Kai Uwe Barthel, Nico Hezel, Klaus Jung, Konstantin Schall
Computer Graphics Forum, Volume 42, Issue 1 (2023)

We introduce distance preservation quality (DPQ) as a new metric for evaluating visually sorted grid layouts, which shows a stronger correlation with user-perceived quality and retrieval performance compared to existing metrics. Additionally, we present Fast Linear Assignment Sorting (FLAS), an efficient algorithm for creating high-quality grid layouts while optimizing runtime and computational resources.

Vibro: Video Browsing with Semantic and Visual Image Embeddings

Konstantin Schall, Nico Hezel, Klaus Jung, Kai Uwe Barthel
29th International Conference on MultiMedia Modeling (MMM 2023)

Vibro represents a powerful tool for interactive video retrieval and browsing and is the winner of the Video Browser Showdown 2022. In this iteration the text-to-image & image-to-image search quality got improved by using general-purpose finetuneing regime.

Combining Semantic and Visual Image Graphs for Efficient Search and Exploration of Large Dynamic Image

Kai Uwe Barthel, Nico Hezel, Konstantin Schall, Klaus Jung
2nd International Workshop on Interactive Multimedia Retrieval (IMuR '22)

We demonstrate a method to significantly reduce search times for high-dimensional image feature vectors by combining two image graphs, achieving faster approximate nearest neighbor search and improved search efficiency. Our approach also allows real-time visual exploration of large image collections in a standard web browser and adapts dynamically to changes in the image dataset.

PicArrange - Visually Sort, Search, and Explore Private Images on a Mac Computer

Klaus Jung, Kai Uwe Barthel, Nico Hezel, Konstantin Schall
28th International Conference on Multimedia Modeling (MMM 2022)

PicArrange integrates state-of-the-art image sorting and similarity search to enable users to get a better overview of their images on a Mac computer.

GPR1200: A Benchmark for General-Purpose Content-Based Image Retrieval

Konstantin Schall, Kai Uwe Barthel, Nico Hezel, Klaus Jung
28th International Conference on Multimedia Modeling (MMM 2022)

We introduce GPR1200, a new benchmark dataset designed to evaluate the generalization quality of image retrieval models across diverse categories. Our experiments show that large-scale pretraining significantly enhances retrieval performance.

Efficient Search and Browsing of Large-Scale Video Collections with Vibro

Nico Hezel, Konstantin Schall, Klaus Jung, Kai Uwe Barthel
28th International Conference on Multimedia Modeling (MMM 2022)

We present the latest version of our interactive video browser tool, Vibro, featuring an enhanced user interface for easier temporal search, an upgraded shot-detection algorithm, and improved query efficiency through graph-based approximate nearest neighbor search.

Video Search with Sub-Image Keyword Transfer Using Existing Image Archives

Nico Hezel, Konstantin Schall, Klaus Jung, Kai Uwe Barthel
27th International Conference on Multimedia Modeling (MMM 2021)

This paper details our frame-based Ad-hoc video search system for the Video Browser Showdown 2021 (VBS2021), featuring enhanced automatic keywording, fine-tuned visual feature vectors, and improved search result presentation.

Real-Time Visual Navigation in Huge Image Sets Using Similarity Graphs

Kai Uwe Barthel, Nico Hezel, Konstantin Schall, Klaus Jung
27th ACM International Conference on Multimedia (MM '19)

We demonstrate a system for real-time visual exploration of millions of images using a standard web browser, addressing the challenge of inspecting large collections. Our approach leverages graph-based image navigation and a hierarchical similarity graph to present images in a dynamically updated, zoomable 2D map.

Visual Navigation of Large Image Graphs

Nico Hezel, Kai Uwe Barthel, Konstantin Schall, Klaus Jung
2019 IEEE 21st International Workshop on Multimedia Signal Processing (MMSP '19)

We present a graph-based system for visually exploring and navigating millions of images in a web browser. This demo retrieves subsets of images from a similarity graph and displays them as a zoomable, draggable 2D map.

Deep Metric Learning using Similarities from Nonlinear Rank Approximations

Konstantin Schall, Kai Uwe Barthel, Nico Hezel, Klaus Jung
2019 IEEE 21st International Workshop on Multimedia Signal Processing (MMSP '19)

We introduce a metric learning algorithm that enhances image similarity search by focusing on feature vectors that most impact retrieval quality. By computing normalized approximated ranks and using a nonlinear transfer function in a new loss function, our approach significantly improves deep feature embeddings across multiple datasets.

Deep Aggregation of Regional Convolutional Activations for Content Based Image Retrieval

Konstantin Schall, Kai Uwe Barthel, Nico Hezel, Klaus Jung
2019 IEEE 21st International Workshop on Multimedia Signal Processing (MMSP '19)

We present a supervised aggregation method for deep learning-based image retrieval that combines regional pooling with weighted activation averages to create a highly representative feature vector. Our approach, which includes fine-tuning with a new NRA loss function, achieves state-of-the-art results on the INRIA Holidays dataset and competitive results on the Oxford Buildings and Paris datasets, while significantly reducing training time.

Visually Exploring Millions of Images using Image Maps and Graphs

Kai Uwe Barthel, Nico Hezel
Big Data Analytics for Large‐Scale Multimedia Search, Chapter 11, John Wiley & Sons, Ltd (2019)

This work introduces various image sorting algorithms and a new measure for evaluating 2D image arrangements. It presents an enhanced self-sorting maps (SSM) algorithm for efficiently sorting millions of images, discusses a graph-based approach for browsing large collections, and explains how high-quality image features are generated using convolutional neural networks.

Dynamic Construction and Manipulation of Hierarchical Quartic Image Graphs

Nico Hezel, Kai Uwe Barthel
2018 International Conference on Multimedia Retrieval (ICMR '18)

We present new techniques for dynamically constructing and manipulating image graphs, which connect similar images and perform well in retrieval tasks. Using an enhanced fast self-sorting map algorithm, we enable efficient exploration of large image collections.

ImageX - Explore and Search Local/Private Images

Nico Hezel, Kai Uwe Barthel, Klaus Jung
24th International Conference on Multimedia Modeling (MMM 2018)

We present a system for exploring and searching large sets of untagged images using standard hardware and operating systems, leveraging compact 64-byte feature vectors derived from convolutional neural network activations. Our approach enables fast search-by-example queries and keyword search by generating reference features from clustered web images.

Fusing Keyword Search and Visual Exploration for Untagged Videos

Kai Uwe Barthel, Nico Hezel, Klaus Jung
24th International Conference on Multimedia Modeling (MMM 2018)

We present a system for searching untagged videos using sketches, example images, and keywords. By analyzing frequent search terms and using multiple image features, our system retrieves thousands of relevant video scenes, displayed in a visually sorted hierarchical map that allows users to quickly explore and find images of interest through zooming and dragging.

Visually Browsing Millions of Images Using Image Graphs

Kai Uwe Barthel, Nico Hezel, Klaus Jung
2017 International Conference on Multimedia Retrieval (ICMR '17)

We present a new method for visually browsing large sets of untagged images using high-quality features from convolutional neural network activations to build a hierarchical image graph. Our approach offers an efficient and user-friendly way to explore the image collection interactively.

Graph Navigation for Exploring Very Large Image Collections

Kai Uwe Barthel, Nico Hezel
12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISAPP 2017)

We describe a method for generating high-quality image descriptors from convolutional neural network activations, which are used to model image similarities and build an efficient hierarchical image graph. Projecting sub-graphs onto a 2D image map provides the best user experience and ease of use.

Navigating a Graph of Scenes for Exploring Large Video Collections

Kai Uwe Barthel, Nico Hezel, Radek Mackowiak
22nd International Conference on Multimedia Modeling (MMM 2016)

We present an enhanced method for browsing large video scene collections using a hierarchical graph and 2D image maps. Users can start searches with various tools, and the system efficiently projects images onto a 2D plane to facilitate navigation and display of relevant video frames.

ImageMap - Visually Browsing Millions of Images

Kai Uwe Barthel, Nico Hezel, Radek Mackowiak
21st International Conference on Multimedia Modeling (MMM 2015)

We present ImageMap, an image browsing system for exploring and searching millions of images using a map-like interface. The system employs an image pyramid built from sorting and clustering techniques, enabling efficient navigation and targeted search results.

Graph-Based Browsing for Large Video Collections

Kai Uwe Barthel, Nico Hezel, Radek Mackowiak
21st International Conference on Multimedia Modeling (MMM 2015)

We introduce a graph-based browsing system that extends ImageMap for visually searching large video collections, using a hierarchical graph to preserve complex inter-image relationships. This system supports various visualization and navigation modes, allowing users to apply filters and search tools to efficiently explore and locate relevant video frames.

Perceived Color Difference - ein spielerisches Experiment zur Erfassung empfundener Farbunterschiede

Kai Uwe Barthel, Nico Hezel, Moritz Klack, Andy Lindemann, Christopher Möller, Christine Wiederer
19. Workshop Farbbildverarbeitung, Gesellschaft zur Förderung angewandter Informatik e. V. (FWS 2013)

Determining perceived color differences accurately is challenging, as traditional distance metrics based on color coordinates often don't align with human perception. This article describes a method to systematically capture these differences through a game where participants unknowingly provide data on perceived color differences.

Automatic Generation of Volumetric Transfer Functions

Kai Uwe Barthel
4th ImageJ User & Developer Conference, 2012

This paper introduces new features in the ImageJ Volume Viewer plugin, including automatic suggestions for 1D and 2D transfer functions to improve 3D volume renderings. These functions can be adjusted by the user to enhance visualization quality.

Image Retrieval using Collaborative Filtering and Visual Navigation

Kai Uwe Barthel, Sebastian Müller, David Backstein, Dirk Neumann, Klaus Jung
ACM Special Interest Group on Computer Graphics and Interactive Techniques Conference (SIGGRAPH '10)

We propose a new image search system that combines keywords, low-level visual features, and collaborative filtering to create a network of semantic relationships between images. Unlike other methods, our system models image similarity through a network of linked images, with link weights learned solely from user interactions.

Improved Image Retrieval using Visual Sorting and Semi-Automatic Semantic Categorization of Images

Kai Uwe Barthel, Sebastian Richter, Anuj Goyal, Andreas Follmann
Metadata Mining for Image Understanding (VISIGRAPP 2008)

The growing volume of digital images presents challenges in organizing them for effective search and retrieval, as current keyword-based and content-based systems struggle with high-level human semantics. We propose a new image search system that integrates keyword annotations, low-level visual metadata, and semantic relationships learned from user interactions.

3D-Data Representation with ImageJ

Kai Uwe Barthel
First ImageJ User and Developer Conference, Luxembourg, 2006

The paper discusses three ImageJ plugins – 3D Color Inspector, 3D Surface Plot, and Volume Viewer – that use 3D display techniques for image data. It highlights the shared algorithms among these plugins and introduces a new Java framework designed to simplify the creation of other plugins that require 3D data visualization.

Verlustlose Bildkompression

Kai Uwe Barthel
Lossy image compression can achieve high compression factors, but lossless compression is more challenging, typically achieving only 2-3x reduction. This article explains standard lossless image compression techniques and explores newer algorithm ideas.

Festbildcodierung bei niedrigen Bitraten unter Verwendung fraktaler Methoden im Orts- und Frequenzbereich

Kai Uwe Barthel
This dissertation advances the field of fractal image coding by analyzing and optimizing block-based methods, leading to the development of new coders that outperform existing techniques, including JPEG, at extremely high compression ratios. It also reveals that fractal coding is closely related to other coding methods and can be effectively integrated with them.

Awards

We achieved the Runner-up position and won the Best New Entrant Award at the Life Search Challenge 2024, competing against 21 teams.

Winner of the 2024 VBS Textual Known Item Search challenge

Consecutive Winner of the 2023 Video Browser Showdown

Overall winner of the 2022 Video Browser Showdown

See all our awards