String Filtering (words, lines, tokens)

Filtering whole strings keeps or drops tokens, lines, or substrings by rules—split, comprehension over words, or regex. Contrast with per-character filtering: here the unit is often a word or record.

Why this shows up in the real world

Log pipelines drop lines that lack an ERROR tag. Search snippets remove stopwords before highlighting.

Core idea (explained for students)

Pattern: [w for w in s.split() if predicate(w)] then ' '.join(...). For line-based files, iterate lines and filter.

Try this in Python

def keep_alpha_words(s: str) -> str:
    return " ".join(w for w in s.split() if w.isalpha())


print(keep_alpha_words("hi 42 there x2"))

Common mistakes

• split() without args collapses all whitespace—sometimes you need split(',').
• Stripping once vs per-token—duplicates or spacing changes meaning.

Key takeaways

• Decide token definition (whitespace vs punctuation) before writing filters.
• Preserve original indices if downstream UI maps back to source text.