The Genesis Device activated aboard the USS Reliant
The Genesis Device was a sophisticated technological innovation designed to alleviate sociological problems such as overpopulation and limited food supplies. Its development was completed by a team of scientists led by Carol Marcus and her son, David Marcus, in 2285 on the Spacelab Regula I in the Mutara sector.
The device initiated a process of rapid terraforming, by which previously uninhabitable planets could be turned into class M worlds ready for colonization. This was accomplished by launching the Genesis Device, a torpedo-shaped projectile, into a lifeless planet. Upon impact, the device caused a massive explosion, reducing the entire area to subatomic particles. A preprogrammed matrix then reassembled these subatomic particles into the desired configuration, creating an atmosphere and environment habitable for humans within a matter of hours, regardless of the test area's original composition.
Development and testing of the Genesis Device took place during a year-long period on a space station orbiting Regula, a class D planetoid. Tests were carried out by Dr. Carol Marcus and her team in three stages, the first of which involved experiments conducted in a laboratory on the space station. In the second stage, the device was deployed within a lifeless underground cavern deep inside the Regula planetoid. The intention of the third stage of testing was to introduce the device to a lifeless space body, such as a moon or barren planet.
Unfortunately, if the device was detonated where life already existed, it could prove to be a very powerful doomsday weapon, destroying all life in favor of its new matrix. This possibility seemed to catch its creators off guard, and they were unprepared for the consequences when Khan Noonien Singh stole the device and detonated it aboard the USS Reliant in the middle of the Mutara Nebula. The resulting cataclysmic explosion reorganized the matter that constituted the nebula and formed a new planet, Genesis. The ship USS Enterprise was almost destroyed by the shockwave that resulted from the explosion before Spock sacrificed himself and the ship went to high warp just before the Reliant exploded.
Within an extremely short time, fully developed plant life emerged, but this seemingly tranquil planet had a highly unstable core caused by the inclusion of protomatter in its Genesis matrix; this resulted in its aging rapidly. The planet suffered from wind storms, fires, and earthquakes that became more and more violent, until it literally tore itself apart in an explosion almost as tremendous as the one that had created it, destroying the life that had so recently been created.
Ultimately, the Genesis Device was a grand experiment that failed. The use of protomatter in the matrix was mainly responsible for this, but a series of factors, including the theft of the device and its use in an unintended setting, were also to blame. (Star Trek II: The Wrath of Khan; Star Trek III: The Search for Spock)
Background
It is unclear if the technology of Genesis was at one time covered up or classified, as the only mention since Star Trek IV: The Voyage Home was in VOY: "The Omega Directive", in which Kathryn Janeway mentions "Marcus and her Genesis Device" as a comparison to other technologies deemed dangerously powerful.
It is never explained why the Genesis wave created, not only a planet from the nebula, but a yellow sun at just the correct distance. The explaination and demonstration of the device never mentions star creation, only terraforming.