The Gambling gene, according to a Dominion simulation of Quark, was a gene present in some species making them susceptible to the allure of gambling. This Quark impostor furthermore suspected that the Jem'Hadar possesed this gene. (DS9: "The Search, Part II")
O'Brien seemed unfamiliar with the term, leaving it unclear if the concept existed in the real world. Additionally, it is even unclear if it was a literal gene, or just an expression.