ReorderBench : A Benchmark for Matrix Reordering
Matrix reordering permutes the rows and columns of a matrix to reveal meaningful visual patterns, such as blocks that represent clusters. A comprehensive collection of matrices, along with a scoring method for measuring the quality of visual patterns in these matrices, contributes to building a benchmark. This benchmark is essential for selecting or designing suitable reordering algorithms for revealing specific patterns. In this paper, we build a matrix-reordering benchmark, ReorderBench, with the goal of evaluating and improving matrix-reordering techniques. This is achieved by generating a large set of representative and diverse matrices and scoring these matrices with a convolution- and entropy-based method. Our benchmark contains 2,835,000 binary matrices and 5,670,000 continuous matrices, each generated to exhibit one of four visual patterns: block, off-diagonal block, star, or band, along with 450 real-world matrices featuring hybrid visual patterns. We demonstrate the usefulness of ReorderBench through three main applications in matrix reordering: 1) evaluating different reordering algorithms, 2) creating a unified scoring model to measure the visual patterns in any matrix, and 3) developing a deep learning model for matrix reordering.
What is ReorderBench?
ReorderBench is a large-scale matrix benchmark built for matrix reordering. ReorderBench has the following features:
- Visual pattern recognition
- 2,835,000 binary matrices, 5,670,000 continuous matrices, and 450 real-world matrices with expert-annotated visual patterns
- 4 visual patterns: block, off-diagonal block, star, and band
- 4 matrix sizes: 100x100, 200x200, 300x300, and 400x400
- Scores to measure the quality of visual patterns
Benchmark
We provide the ReorderBench test set with 1,701,000 matrices and the annotated real-world collection of 450 matrices at Hugging Face.
Click HERE to explore more examples.
Benchmark generation pipiline
To ensure the representativeness and diversity of the ReorderBench, we first generate a set of representative matrix templates for each visual pattern. Then, based on these matrix templates, a large number of matrix variations with diverse degrees of degeneration are generated. The diversity is achieved by combining different variation methods, including index swapping and two types of typical anti-patterns: noise anti-patterns and noise-cluster anti-patterns. To accurately evaluate the quality of visual patterns in a matrix, we develop a scoring method by combining the matching capability of convolutional kernels and the disorder detection capability of entropy.
The code for generating the dataset is available at Github.
Unified scoring model
We build a unified scoring model based on the ReorderBench. This model aligns with the convolution- and entropy-based scoring method across all four visual patterns in both binary and continuous matrices and can also measure matrices of varying sizes.
The unified scoring model is available at Hugging Face. The related code is available at Github.
Deep reordering model
The extensive and diverse matrices in ReorderBench offer valuable supervision for training a deep reordering model. By treating the matrices with index swaps as negative samples and their ground-truth matrices as positive samples, we build a deep model for matrix reordering.
The models are available at Hugging Face. The related code is available at Github. You can also try the models with docker. The Docker image is available at Docker Hub.
