About our crawler

mooncrier reads public job feeds that companies publish for other sites to use, and job pages that carry public job data. Our crawler identifies itself as DailyWantBot, respects robots.txt, and visits each source only a few times a day.

Ads we find always link back to the original posting and say where they were found. Employers can claim, edit or remove them in one click, and can ask us never to contact them again.

To keep your site out of our paper, disallow DailyWantBot in your robots.txt or tell us.