# /search and /profile are NOT disallowed here on purpose: both pages
# serve <meta name="robots" content="noindex, nofollow"> at the HTML
# level, and disallowing them in robots.txt prevents Googlebot from
# crawling them — which means Google never sees the noindex meta and
# happily lists the bare URL in the index anyway ("Indexed, though
# blocked by robots.txt"). Letting Google crawl + see the meta is the
# canonical way to keep these out of the index.
#
# /api/ stays disallowed — those endpoints return JSON, not HTML, so
# the noindex meta route doesn't apply. If any /api/ URL ever surfaces
# in GSC, switch to an X-Robots-Tag: noindex response header instead.
# Faceted-filter crawl trap: the /brands filter rail links every
# speciality/sort combination (76 specialities → tens of thousands of
# parameter permutations). These URLs are already noindexed at the HTML
# level, but the problem here is CRAWL VOLUME, not indexing — a July 2026
# Ahrefs crawl fetched 21K of these SSR permutations and hammered the CMS
# into timeouts. Unlike /search & /profile above, nothing external links
# to these combos, so "indexed though blocked" is a non-risk and Disallow
# is the right tool (Google's own faceted-navigation guidance).
User-agent: *
Disallow: /api/
Disallow: /brands?*speciality=
Disallow: /brands?*sort=
Disallow: /brands?*q=

Sitemap: https://www.dtcetc.com/sitemap-index.xml