Star Date Bot
Star Date Bot is the name of Star Date's web crawler. Our crawler can be identified by the user agent string 'Star Date Bot'.
As a user agent string can be set to any text, anyone can claim to be the 'Star Date Bot' or spoof the Star Date Bot.
How we Access Your Resources
Star Date can access many different resources through the World Wide Web. Before Star Date will crawl a resource, it must have a well-known Uniform Resource Locator (URL).
The World Wide Web is a collection of resources and Star Date will try to crawl all resources including documents, web pages, images, application programming interfaces, metaverses and more.
Star Date is a good World Wide Web citizen and will attempt to read your robots.txt file associated with your web resource and otherwise moderately crawl resources for inclusion in our search engine. Star Date Bot will follow your instructions including how fast to crawl your resource, what resources to crawl and what resources not to crawl.
Star Date will obey instructions in your robots.txt file and instructions provided inline in a html web page.
How Often Star Date Bot Crawls Your Resources
Star Date Bot will change how often it crawls your resource based on the type of resource, how often a resource is updated, who the author is and how helpful it is, amongst other criteria.
Star Date will probably not crawl all of the web pages associated with a web site at once unless a government or education resource, and it may focus on your top pages and resources first. If you cannot see all of your pages or resources in our search index, this is why.
As Star Date has limited resources, we cannot guarantee that we will crawl all resources. Star Date Bot may not crawl your resource if it is repetitive, adds little value or attempts to rank higher that other resources by keyword stuffing, keyword hiding and other schemes. To obtain the best rank, simply create an informative and useful resource.
If you have concerns about us crawling or not crawling your resource you can contact us here.
You can add a web resource for crawling here.
The Speed Star Date Bot Crawls Your Resources
Star Date Bot will crawl your resource as slow as one page in every four seconds and mostly crawls resources sequentially or one at a time. However, if you indicate there is no crawl speed Star Date Bot may crawl your site asynchronously. As Star Date Bot only crawls one page at a time, the crawl delay can often be longer than specified in your instructions, as Star Date Bot will usually finish crawling a page before requesting another page. As a result, it is unlikely Star Date Bot would create resource issues.
If you would like to limit Star Date Bot's crawl rate, you can add an instruction in your resource's robot.txt file as to one of the following commands:
| Instruction | Delay | Example | Description |
|---|---|---|---|
| crawl-delay: | 4 | crawl-delay:4 | Crawl no more than one resource every four seconds. |
| crawl delay: | 1 | crawl delay:1 | Crawl no more than one resource every one second. |
| crawl-rate: | 3 | crawl-rate:3 | Crawl no more than one resource every three seconds. |
| crawl rate: | 0 | crawl rate:0 | No need to have a delay between crawling pages. |
Star Date Bot understands all of the instructions: 'crawl-delay' or 'crawl delay' or 'crawl-rate' or 'crawl rate'.
Where you specify the crawl delay as zero, you are telling Star Date Bot that there is no need for a crawl delay. In this case, Star Date Bot will assume you have no crawl rate speed limitation and may crawl your site asynchronously or multiple resources at once rather than one at a time or sequentially.