Choosing the Right Wayback Snapshot: Era, Scope, and URL Preservation

A practitioner guide to the decision that turns a Wayback scrape into a usable site

If you have ever pulled the Wayback Machine CDX index for a URL that lived for years, you have seen the problem: hundreds — sometimes thousands — of timestamps. If you grab them all, you end up with a Frankenstein site where one page is from 2014, another is from 2019, and a third has the Wayback toolbar baked into the body.

After running this job for SMBs and developers for two years, the part that fails is almost never "did we get the HTML?" It is "which snapshots did we pick, and what did we assume about the assets?" Here is the decision framework I use.

Step 1 — Pick an internally consistent era, not the earliest or the latest

A site that lived for years almost always went through design changes. Pulling snapshots from across the redesigns gives you a CSS reset, broken image references, and a navigation that contradicts itself between pages. Pick a contiguous 12–24 month window where the design and content were most internally consistent. For most niche sites this is the second or third year of life, when the launch template had settled and the founder was still active.

How to find it: pull the CDX index, group timestamps by quarter, and look at which quarter has the most pages, the lowest count of "this page is the same as the previous snapshot" duplicates, and the longest unbroken run of weekly crawls.

Step 2 — Only include assets that exist in snapshots from that era

The biggest single failure mode I see is pulling a 2018 HTML page and a 2011 placeholder image. The HTML loads; the image is wrong; the design breaks.

Before you generate any output, audit each asset reference in the chosen era: does the referenced image, stylesheet, JS file, or font actually have at least one snapshot inside your chosen window? If not, either drop the asset reference, swap to a Wayback-archived version of the asset that does exist, or accept that the page will render with a broken image and decide explicitly whether that is OK.

Doing this once, manually, takes a Saturday. Doing it once, automated, takes 200 lines of Python and one CDX query per asset URL.

Step 3 — Preserve the canonical URL structure, not the archived path

A common mistake is keeping the on-disk path exactly as it was crawled. That means /20190105120304/http://example.com/about/ shows up in your sitemap and your redirects. Strip the timestamp prefix and the protocol mirror. Your new URL should look like the live sites URL would have looked.

If the live site had about.html and the archive only has about/, normalize — but log the normalization so you can spot the cases where the archive and the live site disagreed.

Step 4 — Verify the four recovery limits out loud before you promise anything

The restore is only as complete as the snapshots that exist. If archive.org only crawled 30% of the pages, you get 30% back, and you find out which 30% by running the job, not before you tell the customer.

  • archive.org sometimes only has the HTML, not the CSS/images, depending on crawl depth. The id_ trick helps for the HTML; the assets are what they are.
  • Outbound links to other sites are usually dead — Wayback "we also archived this" links are not guaranteed to point to a real page.
  • Any form, search, or login on the original site will not work after restore unless you have the backend, which you do not have from a public archive.

Step 5 — Decide what to do with the missing 30%

Three honest options:

  1. Ship what you have and tell the customer what is missing. This is the default for SMBs and is what wbrestore.com does by default. A live site with a documented list of "these pages did not survive" is more useful than a sitemap full of 404s.
  2. Rewrite the missing pages with new content. This is fine if the customer wants it, but it is no longer a restore; it is a rebuild.
  3. Leave them as Wayback redirects. Each missing page becomes a 302 to web.archive.org/web/<timestamp>/<original>. Honest, ugly, preserves the link equity if the page had any.

What not to do: leave them as 404 and hope nobody notices.