publications
preprints
2026
- PreprintA note on the finite-exceedance dependence principle for sum–maximum asymptotic independence2026
Asymptotic independence between sum-type and maximum-type statistics arises repeatedly across many areas of probability and statistics, particularly in high-dimensional testing and random matrix theory. Motivated by the shared structure underlying existing proofs, which is obscured by their model-specific formulations, this note isolates a common intermediate principle. We introduce a finite-exceedance dependence principle: if the finite-exceedance dependence error vanishes at every fixed order and high-order exceedances are mildly controlled, then a finite-order Bonferroni argument yields asymptotic independence between the sum and the maximum. We further give an independent-proxy criterion for verifying this: for each finite set of exceedances, it suffices to construct an exactly independent proxy for the sum-type statistic whose approximation error is negligible in aggregate. Local deletion under dependency graphs and Gaussian regression and residualization are two examples. The framework gives modular derivations of existing sum–max independence results.
@article{shi2026FiniteExceedance, title = {A note on the finite-exceedance dependence principle for sum--maximum asymptotic independence}, author = {Shi, Hongjian}, year = {2026}, arxiv = {2611.03351},} - PreprintCombinatorial arbitrage in prediction markets2026
We develop a framework to identify and quantify arbitrage across logically related prediction-market contracts. We prove that prices outside the convex hull of admissible payoff vectors admit arbitrage and identify optimal arbitrage portfolios through integer programming. Using venue metadata and semantic analysis of resolution criteria, we recover 1.9 million logical relations and compile them into constraint systems covering more than 255,000 contracts. We replay full-depth Polymarket order books to simulate an arbitrageur executing these portfolios against resting orders. Among settled portfolios, an ideal arbitrageur that takes the full displayed depth of every opportunity and reuses its capital at resolution earns a risk-free return of 3.5% a week after spreads and fees. We find that arbitrage opportunities persist longer and attract less observed participation where relation discovery and execution are more costly.
@article{shi2026ArbitragePredMarket, title = {Combinatorial arbitrage in prediction markets}, author = {Shi, Hongjian and Zhai, Daojing}, year = {2026}, arxiv = {2610.23517}, ssrn = {7587019},}
2025
- PreprintExistence of direct density ratio estimatorsErika Banzato, Mathias Drton, Kian Saraf-Poor, and Hongjian Shi2025
Many two-sample problems call for a comparison of two distributions from an exponential family. Density ratio estimation methods provide ways to solve such problems through direct estimation of the differences in natural parameters. The term direct indicates that one avoids estimating both marginal distributions. In this context, we consider the Kullback–Leibler Importance Estimation Procedure (KLIEP), which has been the subject of recent work on differential networks. Our main result shows that the existence of the KLIEP estimator is characterized by whether the average sufficient statistic for one sample belongs to the convex hull of the set of all sufficient statistics for data points in the second sample. For high-dimensional problems it is customary to regularize the KLIEP loss by adding the product of a tuning parameter and a norm of the vector of parameter differences. We show that the existence of the regularized KLIEP estimator requires the tuning parameter to be no less than the dual norm-based distance between the average sufficient statistic and the convex hull. The implications of these existence issues are explored in applications to differential network analysis.
@article{shi2025existence, title = {Existence of direct density ratio estimators}, author = {Banzato, Erika and Drton, Mathias and Saraf-Poor, Kian and Shi, Hongjian}, year = {2025}, arxiv = {2502.12738},}
2024
- PreprintOn universal inference in Gaussian mixture models2024
Recent work on game-theoretic statistics and safe anytime-valid inference (SAVI) provides new tools for statistical inference without assuming any regularity conditions. In particular, the framework of universal inference proposed by Wasserman, Ramdas, and Balakrishnan (2020) offers new solutions by modifying the likelihood ratio test in a data-splitting scheme. In this paper, we study the performance of the resulting split likelihood ratio test under Gaussian mixture models, which are canonical examples for models in which classical regularity conditions fail to hold. We first establish that under the null hypothesis, the split likelihood ratio statistic is asymptotically normal with increasing mean and variance. Moreover, contradicting the usual belief that the flexibility of SAVI and universal methods comes at the price of a significant loss of power, we are able to prove that universal inference surprisingly achieves the same detection rate (n^−1 \log \log n)^1/2 as the classical likelihood ratio test.
@article{shi2024universal, title = {On universal inference in {G}aussian mixture models}, author = {Shi, Hongjian and Drton, Mathias}, year = {2024}, arxiv = {2407.19361},}
journal articles
2025
- Distribution-free tests of multivariate independence based on center-outward quadrant, Spearman, Kendall, and van der Waerden statisticsBernoulli 2025
Due to the lack of a canonical ordering in ℝ^d for d>1, defining multivariate generalizations of the classical univariate ranks has been a long-standing open problem in statistics. Optimal transport has been shown to offer a solution in which multivariate ranks are obtained by transporting data points to a grid that approximates a uniform reference measure (Chernozhukov et al., 2017, Hallin, 2017, Hallin et al., 2021), thereby inducing ranks, signs, and a data-driven ordering of ℝ^d. We take up this new perspective to define and study multivariate analogues of the sign covariance/quadrant statistic, Spearman’s rho, Kendall’s tau, and van der Waerden covariances. The resulting tests of multivariate independence are fully distribution-free, hence uniformly valid irrespective of the actual (absolutely continuous) distribution of the observations. Our results provide the asymptotic distribution theory for these new test statistics, with asymptotic approximations to critical values to be used for testing independence between random vectors, as well as a power analysis of the resulting tests in an extension of the so-called (bivariate) Konijn model. This power analysis includes a multivariate Chernoff–Savage property guaranteeing that, under elliptical generalized Konijn models, the asymptotic relative efficiency of our van der Waerden tests with respect to Wilks’ classical (pseudo-)Gaussian procedure is strictly larger than or equal to one, where equality is achieved under Gaussian distributions only. We similarly provide a lower bound for the asymptotic relative efficiency of our Spearman procedure with respect to Wilks’ test, thus extending the classical result by Hodges and Lehmann on the asymptotic relative efficiency, in univariate location models, of Wilcoxon tests with respect to the Student ones.
@article{MR4815978, author = {Shi, Hongjian and Drton, Mathias and Hallin, Marc and Han, Fang}, title = {Distribution-free tests of multivariate independence based on center-outward quadrant, {S}pearman, {K}endall, and van der {W}aerden statistics}, journal = {Bernoulli}, fjournal = {Bernoulli. Official Journal of the Bernoulli Society for Mathematical Statistics and Probability}, volume = {31}, year = {2025}, number = {1}, pages = {106--129}, issn = {1350-7265,1573-9759}, mrclass = {99-06}, mrnumber = {4815978}, doi = {10.3150/24-bej1721}, url = {https://doi.org/10.3150/24-bej1721}, arxiv = {2111.15567},}
2024
- On Azadkia-Chatterjee’s conditional dependence coefficientBernoulli 2024
In recent work, Azadkia and Chatterjee (2021) laid out an ingenious approach to defining consistent measures of conditional dependence. Their fully nonparametric approach forms statistics based on ranks and nearest neighbor graphs. The appealing nonparametric consistency of the resulting conditional dependence measure and the associated empirical conditional dependence coefficient has quickly prompted follow-up work that seeks to study its statistical efficiency. In this paper, we take up the framework of conditional randomization tests (CRT) for conditional independence and conduct a power analysis that considers two types of local alternatives, namely, parametric quadratic mean differentiable alternatives and nonparametric Hölder smooth alternatives. Our local power analysis shows that conditional independence tests using the Azadkia–Chatterjee coefficient remain inefficient even when aided with the CRT framework, and serves as motivation to develop variants of the approach; cf. Lin and Han (2022+). As a byproduct, we resolve a conjecture of Azadkia and Chatterjee by proving central limit theorems for the considered conditional dependence coefficients, with explicit formulas for the asymptotic variances.
@article{MR4699537, author = {Shi, Hongjian and Drton, Mathias and Han, Fang}, title = {On {A}zadkia-{C}hatterjee's conditional dependence coefficient}, journal = {Bernoulli}, fjournal = {Bernoulli. Official Journal of the Bernoulli Society for Mathematical Statistics and Probability}, volume = {30}, year = {2024}, number = {2}, pages = {851--877}, issn = {1350-7265,1573-9759}, mrclass = {62G10 (60F05 62H20)}, mrnumber = {4699537}, doi = {10.3150/22-bej1529}, url = {https://doi.org/10.3150/22-bej1529}, arxiv = {2108.06827},}
2023
- A discussion of “A note on universal inference” by Tse and DavisonMathias Drton, Hongjian Shi, and David StriederStat 2023
Invited discussion for Stat of "A note on universal inference" by Tse and Davison (2022)
@article{10.1002/sta4.572, author = {Drton, Mathias and Shi, Hongjian and Strieder, David}, title = {A discussion of ``{A} note on universal inference'' by {T}se and {D}avison}, journal = {Stat}, fjournal = {Stat}, volume = {12}, year = {2023}, number = {1}, pages = {Paper No. e572, 2}, doi = {10.1002/sta4.572}, url = {https://doi.org/10.1002/sta4.572}, arxiv = {2303.17689},}
2022
- On the power of Chatterjee’s rank correlationBiometrika 2022
Chatterjee (2021) introduced a simple new rank correlation coefficient that has attracted much attention recently. The coefficient has the unusual appeal that it not only estimates a population quantity first proposed by Dette et al. (2013) that is zero if and only if the underlying pair of random variables is independent, but also is asymptotically normal under independence. This paper compares Chatterjee’s new correlation coefficient with three established rank correlations that also facilitate consistent tests of independence, namely Hoeffding’s D, Blum–Kiefer–Rosenblatt’s R, and Bergsma–Dassios–Yanagimoto’s τ∗. We compare the computational efficiency of these rank correlation coefficients in light of recent advances, and investigate their power against local rotation and mixture alternatives. Our main results show that Chatterjee’s coefficient is unfortunately rate-suboptimal compared to D, R and τ∗. The situation is more subtle for a related earlier estimator of Dette et al. (2013). These results favour D, R and τ∗ over Chatterjee’s new correlation coefficient for the purpose of testing independence.
@article{MR4430960, author = {Shi, Hongjian and Drton, Mathias and Han, Fang}, title = {On the power of {C}hatterjee's rank correlation}, journal = {Biometrika}, fjournal = {Biometrika}, volume = {109}, year = {2022}, number = {2}, pages = {317--333}, issn = {0006-3444}, mrclass = {62C20 (62G10)}, mrnumber = {4430960}, doi = {10.1093/biomet/asab028}, url = {https://doi.org/10.1093/biomet/asab028}, arxiv = {2008.11619},} - On universally consistent and fully distribution-free rank tests of vector independenceAnn. Statist. 2022
Rank correlations have found many innovative applications in the last decade. In particular, suitable rank correlations have been used for consistent tests of independence between pairs of random variables. Using ranks is especially appealing for continuous data as tests become distribution-free. However, the traditional concept of ranks relies on ordering data and is, thus, tied to univariate observations. As a result, it has long remained unclear how one may construct distribution-free yet consistent tests of independence between random vectors. This is the problem addressed in this paper, in which we lay out a general framework for designing dependence measures that give tests of multivariate independence that are not only consistent and distribution-free but which we also prove to be statistically efficient. Our framework leverages the recently introduced concept of center-outward ranks and signs, a multivariate generalization of traditional ranks, and adopts a common standard form for dependence measures that encompasses many popular examples. In a unified study, we derive a general asymptotic representation of center-outward rank-based test statistics under independence, extending to the multivariate setting the classical Hájek asymptotic representation results. This representation permits direct calculation of limiting null distributions and facilitates a local power analysis that provides strong support for the center-outward approach by establishing, for the first time, the nontrivial power of center-outward rank-based tests over root-n neighborhoods within the class of quadratic mean differentiable alternatives.
@article{MR4474478, author = {Shi, Hongjian and Hallin, Marc and Drton, Mathias and Han, Fang}, title = {On universally consistent and fully distribution-free rank tests of vector independence}, journal = {Ann. Statist.}, fjournal = {The Annals of Statistics}, volume = {50}, year = {2022}, number = {4}, pages = {1933--1959}, issn = {0090-5364}, mrclass = {62G10}, mrnumber = {4474478}, doi = {10.1214/21-aos2151}, url = {https://doi.org/10.1214/21-aos2151}, arxiv = {2007.02186},} - Distribution-free consistent independence tests via center-outward ranks and signsJ. Amer. Statist. Assoc. 2022
This article investigates the problem of testing independence of two random vectors of general dimensions. For this, we give for the first time a distribution-free consistent test. Our approach combines distance covariance with the center-outward ranks and signs developed by Marc Hallin and collaborators. In technical terms, the proposed test is consistent and distribution-free in the family of multivariate distributions with nonvanishing (Lebesgue) probability densities. Exploiting the (degenerate) U-statistic structure of the distance covariance and the combinatorial nature of Hallin’s center-outward ranks and signs, we are able to derive the limiting null distribution of our test statistic. The resulting asymptotic approximation is accurate already for moderate sample sizes and makes the test implementable without requiring permutation. The limiting distribution is derived via a more general result that gives a new type of combinatorial noncentral limit theorem for double- and multiple-indexed permutation statistics. Supplementary materials for this article are available online.
@article{MR4399094, author = {Shi, Hongjian and Drton, Mathias and Han, Fang}, title = {Distribution-free consistent independence tests via center-outward ranks and signs}, journal = {J. Amer. Statist. Assoc.}, fjournal = {Journal of the American Statistical Association}, volume = {117}, year = {2022}, number = {537}, pages = {395--410}, issn = {0162-1459}, mrclass = {Prelim}, mrnumber = {4399094}, doi = {10.1080/01621459.2020.1782223}, url = {https://doi.org/10.1080/01621459.2020.1782223}, arxiv = {1909.10024},}
2020
- High-dimensional consistent independence testing with maxima of rank correlationsAnn. Statist. 2020
Testing mutual independence for high-dimensional observations is a fundamental statistical challenge. Popular tests based on linear and simple rank correlations are known to be incapable of detecting nonlinear, nonmonotone relationships, calling for methods that can account for such dependences. To address this challenge, we propose a family of tests that are constructed using maxima of pairwise rank correlations that permit consistent assessment of pairwise independence. Built upon a newly developed Cramér-type moderate deviation theorem for degenerate U-statistics, our results cover a variety of rank correlations including Hoeffding’s D, Blum–Kiefer–Rosenblatt’s R and Bergsma–Dassios–Yanagimoto’s τ∗. The proposed tests are distribution-free in the class of multivariate distributions with continuous margins, implementable without the need for permutation, and are shown to be rate-optimal against sparse alternatives under the Gaussian copula model. As a by-product of the study, we reveal an identity between the aforementioned three rank correlation statistics, and hence make a step towards proving a conjecture of Bergsma and Dassios.
@article{MR4185806, author = {Drton, Mathias and Han, Fang and Shi, Hongjian}, title = {High-dimensional consistent independence testing with maxima of rank correlations}, journal = {Ann. Statist.}, fjournal = {The Annals of Statistics}, volume = {48}, year = {2020}, number = {6}, pages = {3206--3227}, issn = {0090-5364}, mrclass = {62G10}, mrnumber = {4185806}, doi = {10.1214/19-AOS1926}, url = {https://doi.org/10.1214/19-AOS1926}, arxiv = {1812.06189},}
Ph.D. thesis
2021
- Ph.D.Distribution-Free Consistent Tests of Independence via Marginal and Multivariate Ranks2021
Testing independence is a fundamental statistical problem that has received much attention in literature. In this dissertation, we consider testing independence under two different settings. The first is testing mutual independence of many covariates, and the second is testing independence of two random vectors. For both settings, we propose, for the first time, distribution-free and consistent tests of independence via marginal or multivariate ranks. Moreover, we establish the optimal efficiency in the statistical sense of both tests. In addition, we also investigate the power of a simple consistent rank correlation coefficient recently proposed by Chatterjee (2021) against local alternatives. Our results show that Chatterjee’s coefficient is unfortunately statistically inefficient.
@book{MR4351909, author = {Shi, Hongjian}, title = {Distribution-{F}ree {C}onsistent {T}ests of {I}ndependence via {M}arginal and {M}ultivariate {R}anks}, note = {Thesis (Ph.D.)--University of Washington}, publisher = {ProQuest LLC, Ann Arbor, MI}, year = {2021}, pages = {219}, isbn = {979-8492-72215-9}, mrclass = {Thesis}, mrnumber = {4351909}, url = {http://gateway.proquest.com/openurl?url_ver=Z39.88-2004&rft_val_fmt=info:ofi/fmt:kev:mtx:dissertation&res_dat=xri:pqm&rft_dat=xri:pqdiss:28719887},}