A caste was a familial rank in a social structure. In caste systems, such as those of Bajor and the Earth nation-state of India, one was born into a type of family and lived according to one's caste. Generally, one was prohibited from advancing one's caste in such systems. The caste system of Bajor was called D'jarra.
In the 22nd century the Klingons had a warrior caste. One Klingon, Antaak, was disowned by his father for becomming a healer even though his house belonged to the warrior caste. (ENT: "Affliction", "Divergence")
The Torothan homeworld had a caste system under which the Torothan clan terrorised Zobral's clan for centuries, forcing them to wear a Yrott amonst other things. The caste system had officially been abolished for a number of years by 2152, but little had changed. (ENT: "Desert Crossing")