A nit was the egg of a louse.
The term "picking nits" was commonly used by Humans to describe meticulously searching for and criticizing trivial errors in detail. In 2365, Ensign Wesley Crusher was concerned that he was "picking nits" when he thought that Ensign Davies should have run an ico-spectrogram on the survey results on Drema IV. (TNG: "Pen Pals")
In 2369, after Minister Toran asked a number of pernickety questions relating to the preparations for the Jeraddo energy transfer project, he apologized to Major Kira, asking her to forgive his "bureaucratic nit-picking". (DS9: "Progress")