The Stream Processing Playbook
Pipes & Redirects — The Core of UNIX
The UNIX philosophy is: "Write programs that do one thing well. Write programs to work together." Pipes and redirects are HOW programs work together. Every DevOps engineer uses these dozens of times daily.
cmd1 | cmd2 → Pipe: stdout of cmd1 → stdin of cmd2cmd > file → Overwrite file with outputcmd >> file → Append output to filecmd 2> err.log → Redirect errors to filecmd > out.log 2>&1 → Capture EVERYTHING to one file
🔍 Search & Filter
grep -i— Case-insensitive searchgrep -r— Recursive directory searchgrep -c— Count matchesgrep -v— Invert (exclude matches)grep -A 3— Show 3 lines after match
📊 Analyze & Count
wc -l— Count linessort | uniq -c— Count occurrencessort -rn— Sort numerically (highest first)awk '{print $N}'— Extract column Nhead -N / tail -N— First/last N lines
⚠️ The Deadly Difference: > vs >>
> OVERWRITES — the file is completely replaced. Old content is gone forever.
>> APPENDS — new content is added at the end. Old content is preserved.
💀 Horror story: echo "new_var=true" > .env deleted 47 production environment variables. The engineer meant to use >>. Always double-check before pressing Enter!
