Fast Approximate Nearest Neighbor Search with a Dynamic Exploration Graph using Continuous Refinement
TL;DR
Abstract
For approximate nearest neighbor search, graph-based algorithms have shown to offer the best trade-off between recall and search time. We propose the Dynamic Exploration Graph (DEG), which is superior to existing algorithms in terms of search and exploration efficiency by combining two new ideas: First, a single undirected even-regular graph is incrementally built by partially replacing existing edges to integrate new vertices and to update old neighborhoods at the same time. Secondly, an edge optimization algorithm is used to continuously improve the quality of the graph. Combining this ongoing refinement with the graph construction process leads to a well-organized graph structure at all times, resulting in: (1) increased search efficiency, (2) predictable index size, (3) guaranteed connectivity and therefore reachability of all vertices, and (4) a dynamic graph structure. In addition we investigate how well existing graph-based search systems can handle indexed queries where the seed vertex of a search is the query itself. Such exploration tasks, despite their good starting point, are not necessarily easy. High efficiency in approximate nearest neighbor search (ANNS) does not automatically imply good performance in exploratory search. Extensive experiments show that our new Dynamic Exploration Graph significantly outperforms existing algorithms for indexed and unindexed queries.
BibTeX
If you use our work in your research, please cite our publication:
@misc{hezel2023fastapproximatenearestneighbor,
title={Fast Approximate Nearest Neighbor Search with a Dynamic Exploration Graph using Continuous Refinement},
author={Nico Hezel and Kai Uwe Barthel and Konstantin Schall and Klaus Jung},
year={2023},
eprint={2307.10479},
archivePrefix={arXiv},
primaryClass={cs.IR},
url={https://arxiv.org/abs/2307.10479},
}