This bulk robots.txt checker lets you check in bulk whether URLs may be visited by crawlers.
Use the Bulk robots.txt checker