The URL Crawler project template is used to crawl a set of seed URLs following all links on the retrieved web pages down to the predefined link depth.
The final result of the URL Crawler comprises the HTML content of all crawled web pages or the extracted text of PDF or Office documents, in case crawled
URLs pointed to non-HTML documents.
The content of all crawled URLs can either be exported from the Content Viewer in the Management Console
or can be retrieved by REST call to the Webservice API