To Prevent overloading the servers when search engine crawlers load many items of material concurrently. Over the flipside, In order for you your complete website blocked from that crawler, you will see a “/”. You may also have particular directories or pages outlined here, all of which must be outlined https://seotoolstube.com/