The Prime Directive has played a central role in Star Trek lore since The Original Series as a fundamental ethical principle of the Federation that is to be maintained at all costs. It started as a plot conceit, allowing ethical debate to play out across a given episode’s storyline. It proved so compelling that it’s been used in almost every subsequent Trek series to date, including Star Trek: Discovery. The Directive itself is a statement of noninterference, maintaining that every planet has a right to self-determination. Starfleet is strictly forbidden from stepping in to help less scientifically advanced cultures with its vast resources, even if it means abandoning them to certain death.
The specifics were spelled out early and rarely wavered. Indeed the “Prime” part of the Prime Directive refers to its in-world status as General Order 1, superseding all others by stating, "No starship may interfere with the normal development of any alien life or society." There are strong reasons for it, which Star Trek has taken great delight in exploring over the years -- not only when the Prime Directive is maintained, but what happens when it’s broken.
The details are straightforward enough. Starfleet believes that every society has the right to natural, unimpeded development. Those cultures who have not yet discovered warp travel must be spared any exposure to superior technology or knowledge of sentient life beyond their own planet, even at the cost of Federation personnel and starships. To break the Directive -- even for noble or humanitarian reasons -- could have catastrophic consequences for the affected culture. The Federation's duty is to protect developing cultures under its care, one of the things that set them apart from other galactic powers like the Romulans and the Borg. Star Trek: Enterprise centered a number of episodes around the Prime Directive's development and the reasons why it was so important.
The concept first appeared in The Original Series Season 1, Episode 21, “The Return of the Archons.” When Kirk proposes dismantling a computer controlling an entire culture, Spock warns him that doing so could violate the Directive. Kirk argues that the world in question is trapped in stagnancy and thus not naturally developing as the Directive stipulates. The episode sets up the Directive and the fact that there is a lot of room for interpretation, which is why the franchise has used it so many times. The costs of upholding it and the price of breaking it make for compelling drama.
It also leaves the specifics a little fuzzy: just enough to let the Star Trek's heroes eel around it when they need to. For instance, the Prime Directive can be breached to repair the damage done by a previous violation, as Kirk did in The Original Series Season 2, Episode 17, “A Piece of the Action.” It can similarly be broken by misfortune or catastrophe, as in Star Trek: The Next Generation Season 3, Episode 4, “Who Watches the Watchers?” in which the accidental reveal of a Federation anthropology team forces Picard to reveal himself to a primitive culture.
Specifics vary on punishment too. In The Original Series Season 2, Episode 23, "The Omega Glory," Kirk specifically states that a captain must sacrifice himself and his crew rather than violate the Prime Directive. However, The Next Generation Season 4, Episode 21, "The Drumhead," specifies that Picard had done so nine times since taking command of the Enterprise-D. Yet, he faced no disciplinary action for it. This suggests that the draconian approach had softened between the 23rd and 24th centuries, or at least when the writers needed a little flexibility to better explore what the concept might mean.
Now, the Prime Directive is as intrinsically tied to Trek as phasers and transporters. The moral quandaries of the Directive are simply too useful to Star Trek’s style of storytelling to let go. And it's a strong yardstick for the Federation's ethical principles by proving how it strives to do better than species only interested in conquest and exploitation.