An invasive program is a benign or malevolent computer program not native to a target computer's environment, and forcibly maintaining itself within that target environment, with or without that system's knowledge of the program's existence.
In 2368, an invasive paradoxical puzzle program was developed by Commanders Geordi La Forge and Data as an unconventional weapon in the war against the Borg for use as an exploit of Hugh's subspace transceiver link to the Borg Collective. The program would initially be "nothing more than a minor distraction" to the Borg, but would require exponentially-increasing amounts of processing time to solve this no-solution puzzle, thus eventually occupying the entire Collective's resources. The ultimate goal was to starve the Borg, as their cortical nodes would be so consumed with solving the puzzle they would forget to nourish their organic components.
In 2369, an alien probe from the Gamma Quadrant transmitted the "Pup" non-biological lifeform to Deep Space 9 space station. "Pup" acted like an invasive program, and thwarted all attempts that were made to return it to the probe. (DS9: "The Forsaken")