(covers information from several alternate timelines)
Red matter was a substance capable of forming a black hole when ignited. A small amount was sufficient enough to collapse a star or consume an entire planet.
Red matter was used by Ambassador Spock in 2387 to create a black hole to absorb the energy of a supernova which threatened the entire galaxy, by shooting a small amount of it into the exploding star from his small, one-man vessel. Although his plan succeeded, and the supernova was eliminated, Spock was too late to save Romulus, which was destroyed. Subsequent to the creation of the black hole, Spock was confronted in his ship by the Romulan mining vessel Narada, commanded by Nero, who held Spock responsible for the destruction of his homeworld. During the confrontation, both ships were pulled into the black hole and transported back in time through a tunnel in spacetime, the Narada arriving in the year 2233, resulting in the creation of an alternate reality. Upon emerging from the black hole, the Narada encountered the Starfleet vessel Kelvin and engaged in battle, but was severely damaged when Lieutenant George Kirk rammed the Narada with the Kelvin.
Twenty-five years later, Spock, who had entered the black hole only seconds after the Narada, emerged, and both he and his ship were captured by Nero. Upon obtaining the red matter aboard Spock's ship, Nero mounted an attack on Vulcan, drilling into the planet and launching a small amount of the red matter into the core, forming a singularity within which consumed the entire planet. Nero later attempted to perform the same action on Earth, but was stopped by the intervention of Spock's alternate reality-counterpart and James Kirk. Spock's counterpart was able to appropriate Spock's ship from the Narada, and piloted it on a collision course with the Romulan vessel. When the ship collided, the red matter stored aboard ignited, generating a black hole within the Narada itself, destroying the ship. (Star Trek)
In Star Trek Magazine issue 144, Scott Chambliss, production designer on Star Trek, noted that the appearance of red matter as a "big red ball" was a reference to other productions that he had worked on with J.J. Abrams, including Alias and Mission: Impossible III. 
According to the Star Trek: Countdown prequel comic book, which details the events leading up to the destruction of Romulus, red matter is an artificial substance created by the Vulcan Science Academy. It is manufactured from decalithium, a rare isotope which the Narada was equipped to mine.
The Path to 2409 (β) on the Star Trek Online website states that the Vulcans had created red matter without the consultation or approval of Federation authorities, earning criticism from the board investigating the destruction of Romulus. In response the Vulcans promised to not conduct any more red matter experiments and stated that the entire supply was on Spock's ship when it vanished. Also, the game has a device called a "Red Matter Capacitor". This device is a equipped to a player's ship and acts as an auxiliary power source.