Create and maintain a robots.txt file

A robots.txt file tells crawlers which parts of a site they're allowed to access. A well-maintained file lets search and AI crawlers reach your valuable pages, keeps them away from irrelevant ones such as admin or cart pages, and points them to your XML sitemap.Josh BlyskalTactic credited to Josh Blyskal. See every tactic from them.