exclude content to be crawled
I have run other tools in the past that allowed me to set a list of URLs to ignore. I could add the full URL, or if it contains (for example) /blog/, it could ignore those pages or only run those pages.
Discussion
Voters
