The Antares maelstrom is some sort of astronomical object in the Antares system.
Khan Noonien Singh swore he would chase Admiral James T. Kirk "round the moons of Nibia and round the Antares maelstrom and round perdition's flames" before he gave him up. (Star Trek II: The Wrath of Khan)
Background
A maelstrom is basically described as "a powerful whirlpool." While Khan's reference is relatively vague, it would appear to describe some sort of large swirling spatial anomoly found in the Antares system.