Remove duplicate words

Keeps the first occurrence of each word and removes the rest, leaving the original order intact. Comparison ignores case, so "The" and "the" count as the same word.

Keeping the first occurrence rather than the last is what preserves the original order, and that is usually what people want — a de-duplicated list that still reads in the sequence it was written. The case-insensitive comparison is a deliberate choice for words and would be wrong for data: if you are de-duplicating identifiers or keys where case is significant, use the line option, which trims whitespace but compares exactly. Note that punctuation travels with the word, so "cat" and "cat," are treated as different tokens; run the text through the cleaner first if that matters.