Case study
Contractor Directory Case Study
A directory with tens of thousands of pages has a different problem from a brochure site. The question is not whether search engines can find the pages — it is whether they can crawl them efficiently, and whether the pages are worth indexing at all. Most large directories fail on index coverage not because pages are missing, but because the site never explains which pages matter.
This page sets out the method used to engineer index coverage on a large contractor directory: how page types are classified, how sitemaps are generated, how crawl budget is protected, and how coverage is watched and corrected after launch. It describes a pattern, not a particular engagement.
The problem
Contractor directories share a predictable set of failure modes, and they compound each other:
- Inventory outgrows attention. Page count grows faster than anyone’s ability to assess whether each page deserves to be indexed.
- Near-duplicate page types. A template with a place name substituted produces hundreds of pages that are technically unique and practically identical. These dilute the site and rarely earn impressions.
- Sitemaps that list everything equally. A single sitemap containing every URL gives a crawler no signal about which pages carry the site’s value.
- Faceted and internal search URLs. Filters, sorts and search result pages create near-infinite crawlable URLs that consume budget without producing value.
- Unreachable pages. Pages linked only from a sitemap, or buried behind pagination, sit outside the internal linking model.
The consequence is a site that is fully crawlable in theory and poorly crawled in practice, with the commercially important pages indexed slowly or not at all.
Approach
The method treats index coverage as a designed system rather than a byproduct of publishing. Four decisions shape it.
Classify pages before generating them. Every page type is assigned a tier: index-worthy (unique, useful, commercially relevant), conditional (index-worthy only where it meets a data threshold), or excluded (near-duplicate, faceted or empty). The tier determines robots directives, sitemap inclusion and internal linking, and it is applied in the template so it cannot drift as the site grows.
Generate sitemaps by page class. Rather than one sitemap listing everything, the site publishes a sitemap index with a child sitemap per page class, each carrying accurate last-modified dates drawn from the underlying data. This tells a crawler what changed and what matters, and it makes coverage problems visible by class rather than as a single opaque number.
Protect crawl budget deliberately. Faceted and search URLs are handled explicitly — excluded from the index and from sitemaps, and not linked in a way that invites crawling. Pagination is treated as a crawl path, not a content destination. Internal linking is built to keep the pages that matter within a few clicks of a hub.
Watch coverage by class after launch, and prune. Index coverage is monitored per page class, not site-wide. A page type that consistently fails to earn impressions is revised, consolidated or removed rather than left standing. Pruning is treated as routine maintenance, not as an admission of failure.
Build detail
- Custom PHP sitemap generation driven by the page inventory, split by page class and published through a sitemap index.
- Canonical, robots and sitemap rules applied at template level so they survive the site growing.
- Explicit handling of faceted, sorted and internal search URLs to prevent crawl traps.
- An internal linking model built around hubs, keeping important pages within a small number of clicks.
- Index coverage monitored by page class, with a defined threshold for revising or removing a page type.
- A refresh path so a new data release regenerates sitemaps and last-modified dates without hand editing.
What the pattern gets right, and what to watch
The strength of the approach is that it makes the site’s own priorities legible — to crawlers and to the people running it. When coverage drops, the cause is visible by page class rather than hidden in an aggregate figure.
The thing to watch is over-generation. It is easy to produce page types at scale before there is evidence anyone searches for them, and easier still to keep them once generated. The discipline that matters is refusing to publish a page class until the data justifies it, and being willing to remove one that never earns impressions.
The boundary, stated plainly
This page describes a technical method and the reasoning behind it. It does not identify a client, assert an engagement or report a result achieved for anyone. Where a directory engagement is agreed, the specific scope, responsibilities and measurement are set out in writing.
Discuss a directory or index-coverage project
If you operate a directory, marketplace or location-based site and the coverage numbers do not match the page inventory, the first conversation is about which page classes are actually earning their place.