exclude content to be crawled

I have run other tools in the past that allowed me to set a list of URLs to ignore. I could add the full URL, or if it contains (for example) /blog/, it could ignore those pages or only run those pages.

Earl DeMuesy posted 4 months ago

Discussion