Anomaly detection search for new resonances decaying into a Higgs boson and a generic new particle X in hadronic final states using Formula Presented pp collisions with the ATLAS detector

A search is presented for a heavy resonance Formula Presented decaying into a Standard Model Higgs boson Formula Presented and a new particle Formula Presented in a fully hadronic final state. The full Large Hadron Collider run 2 dataset of proton-proton collisions at Formula Presented collected by the ATLAS detector from 2015 to 2018 is used and corresponds to an integrated luminosity of Formula Presented. The search targets the high Formula Presented-mass region, where the Formula Presented and Formula Presented have a significant Lorentz boost in the laboratory frame. A novel application of anomaly detection is used to define a general signal region, where events are selected solely because of their incompatibility with a learned background-only model. It is constructed using a jet-level tagger for signal-model-independent selection of the boosted Formula Presented particle, representing the first application of fully unsupervised machine learning to an ATLAS analysis. Two additional signal regions are implemented to target a benchmark Formula Presented decay into two quarks, covering topologies where the Formula Presented is reconstructed as either a single large-radius jet or two small-radius jets. The analysis selects Higgs boson decays into Formula Presented, and a dedicated neural-network-based tagger provides sensitivity to the boosted heavy-flavor topology. No significant excess of data over the expected background is observed, and the results are presented as upper limits on the production cross section Formula Presented) for signals with Formula Presented between 1.5 and 6 TeV and Formula Presented between 65 and 3000 GeV.