Magnetic properties and spin dynamics in the spin-orbit driven Jeff= 12 triangular lattice antiferromagnet Ba6Yb2Ti4 O17

Frustration-induced strong quantum fluctuations accompanied by spin-orbit coupling and crystal electric field can give rise to rich and diverse magnetic phenomena associated with unconventional low-energy excitations in rare-earth-based quantum magnets. Herein, we present crystal structure, magnetic susceptibility, specific heat, muon spin relaxation (μSR), and electron spin resonance (ESR) studies on polycrystalline samples of Ba6Yb2Ti4O17, in which Yb3+ ions constitute a perfect triangular lattice in the ab plane without detectable antisite disorder between atomic sites. The Curie-Weiss fit of the low-temperature magnetic susceptibility data suggests spin-orbit driven effective pseudospin Jeff=12 degrees of freedom of Yb3+ spin with weak antiferromagnetic exchange interactions in the Kramers doublet ground state. The zero-field specific heat data reveal the presence of long-range magnetic order at Néel temperature TN=77mK which is suppressed in a magnetic field μ0H≥1T. The broad maximum in specific heat is attributed to the Schottky anomaly implying the Zeeman splitting of the Kramers doublet ground state in a magnetic field. The ESR measurements suggest the presence of anisotropic exchange interaction between the moments of Yb3+ spins. The μSR experiments reveal a fluctuating state of Yb3+ spins in the temperature range 0.1K≤T≤10K owing to depopulation of crystal electric field levels, which suggests that the lowest Kramers doublets with Jeff=12 are well separated, and the low-temperature physics of this frustrated magnet is dominated by Jeff=12 moments. In addition to the intraplane nearest-neighbor superexchange interaction, the interplane exchange interaction and anisotropy are expected to stabilize the long-range ordered state in this triangular lattice antiferromagnet.