MyVidster is a social video bookmarking service. MyVidsterPreviewBot is our HTTP metadata fetcher: it retrieves public pages to create previews for video bookmarks, including page titles, descriptions, thumbnail references, and embed information. Bookmarks link to their source pages.
Requests from this crawler use the following user-agent:
MyVidsterPreviewBot/1.0 (+https://www.myvidster.com/bot.html)
The robots.txt product token is MyVidsterPreviewBot. The crawler runs on MyVidster-operated infrastructure and uses IPv4. Its current outbound address is 212.7.203.250.
The machine-readable crawler IP list contains authorized outbound IPv4 addresses on MyVidster-operated scraper infrastructure. It includes the current metadata fetcher and additional scraper servers. Some clients on these addresses use other user-agents; an address alone does not identify MyVidsterPreviewBot. Check both the user-agent and source address when identifying its requests.
The crawler reads each site's robots.txt before requesting page content, honors matching Allow and Disallow rules, and caches readable policies for up to one hour. If robots.txt is challenged, inaccessible, or cannot be read safely, it defers the page request. Missing robots.txt responses with status 404 or 410 permit crawling.
Requests are serialized per origin and spaced at least one second apart. Longer Crawl-delay directives and Retry-After responses delay further requests. Delays longer than one day cause the crawler to decline the page request. Network and server failures trigger a backoff.
Page redirects are checked against the destination site's robots.txt. Cross-origin redirects of robots.txt itself are deferred. The crawler stops on access denials and Cloudflare challenges and does not solve CAPTCHAs. It fetches public HTTP or HTTPS URLs and does not recursively follow page links.
To exclude this crawler, add the following to your robots.txt:
User-agent: MyVidsterPreviewBot Disallow: /
For crawler questions or access issues, email [email protected] or use the MyVidster contact form. Include the affected URL, approximate request time, and crawler user-agent if available.
Updated October 18, 2026. This page documents requests carrying the MyVidsterPreviewBot identity.