Scraping job boards without the mess

Every job board looks different. That is the whole point of Crawl-Job: normalize them into one searchable pipeline.

The problem

Job seekers check ten boards and still miss roles. Titles differ, salary formats differ, remote tags differ, and half the listings are duplicates with different spellings.

The approach

• One scraper per board, each Sam pinned to a stable contract.
• A shared normalizer: title, company, location, salary range, remote flag, posted date.
• Deduping by (company, normalized title) fuzzy match.
• Output as structured JSON lines, easy to feed into anything.

What I learned

Board HTML changes without warning. The resilient pattern is defensive parsing: never assume a field exists, and log what you failed to parse so you can fix it the next morning.

The repo is therayyanawaz/Crawl-Job on GitHub.