Website Uptime Monitoring: Catch Downtime Before Search Engines Do

A website can be down without anyone on your team noticing. The server runs out of memory at night, a certificate expires over a weekend, a deploy breaks the home page for logged-out visitors, or DNS stops answering. Customers notice first. Search engine crawlers notice too, and they react to what they see: server errors make Google slow its crawling, and URLs that keep failing can eventually be dropped from the index.

Website uptime monitoring is how you find out first. A monitor requests your pages from outside your own network every few minutes, checks the answer, and tells a person when something is wrong. This guide shows how to choose what to check, how to set up checks that catch real failures (not just "the server answered"), how to route alerts so someone actually acts, and how to take the site down for planned work without telling search engines it is gone.

It is for site owners, small teams and developers preparing a launch. You will finish with a short monitoring plan, a correct maintenance response and a second source of evidence from Search Console. It does not promise a particular uptime figure, and it will not make search engines crawl more.

Before you begin

Decide who gets woken up. A monitor with no owner is just noise. Pick at least two people or channels (for example email plus a team chat message), and write down who responds out of hours, even if the answer is "nobody until morning".

You need a list of your important URLs, access to your DNS and hosting settings, and either a monitoring service or a script you run from a machine that is not the one hosting the site. A monitor on the same server fails when the server fails. Allow an hour for setup. The risk is low, but a monitor that sends too many alerts gets ignored, so start with a few checks and add more later.

If you do not have Google Search Console yet, set it up with Set Up Google Search Console: Complete Website Owner Checklist, because Step 5 uses it.

What search engines do when your site is down

Google's documentation on HTTP status codes describes what happens at the crawler's end:

5xx errors (500, 502, 503) and 429: Google's crawlers temporarily slow down. Content from those responses is ignored. URLs already in the index are kept at first but are eventually dropped if the errors persist. Once the server returns successful responses again, the crawl rate gradually increases. Google gives no fixed number of hours or days for "eventually", so treat every outage as something to shorten.
4xx errors (except 429): Content is ignored, and previously indexed URLs are removed from the index. A broken page that returns 404 or 403 is therefore worse for search than a temporary 503.
Soft 404: A page that returns a normal 200 but shows an error or empty content is reported in Search Console as a soft 404. Your monitor should look inside the page, not only at the status code.
robots.txt that cannot be fetched (a 5xx, or a DNS, network or timeout problem): Google stops crawling the site for the first 12 hours but keeps trying. For the next 30 days it uses the last good copy it has. If the file stays unreachable after that, the result depends on whether the site is otherwise available. That makes robots.txt a file worth monitoring on its own.

None of this means a short outage will damage your visibility. It means repeated or long failures give crawlers reasons to slow down, and you want to be the first to know. For deeper diagnosis of server errors, see HTTP 500 and 503 Errors: Find the Real Website Problem.

Step-by-step instructions

Step 1: List what to monitor

Start with the URLs whose failure would hurt you most:

The home page.
One or two representative inner pages, such as a product or article.
/robots.txt and your XML sitemap.
Anything transactional: login, checkout, contact form (a check that loads the page, not one that submits data).
The DNS name itself, since a DNS failure takes everything down.
The TLS certificate, which should be checked for expiry well in advance. See SSL Certificate Errors: Diagnose Expired, Mismatched and Incomplete Certificates.

What success looks like: A list of five to ten URLs with a reason next to each.

Step 2: Make each check meaningful

A status code of 200 proves little. Configure each check to:

Request the real HTTPS URL, following redirects, and require the final status you expect (usually 200).
Look for a piece of text that only appears when the page is healthy, such as a product name or footer phrase, so that a "200 with an error page" counts as a failure.
Time out at a few seconds. A page that answers after thirty seconds is, for most visitors, down.
Run from at least two locations that are not your server's network. One failed probe may be a network blip between the monitor and you; failures from several places are more likely to be real.
Run often enough to find a problem quickly. Every minute or every few minutes is typical for a service monitor. Check your provider's options rather than assuming a value.

A short script can do the basics while you decide on a service:

curl -sS -o /dev/null -w "%{http_code} %{time_total}s\n" --max-time 10 https://example.com/
curl -sS --max-time 10 https://example.com/ | grep -q "Example Shop" && echo OK || echo MISSING

What success looks like: Stopping your web server on purpose (on staging, or during a quiet test window) turns the check red within a couple of intervals.

Step 3: Route alerts to people

Send alerts where people will see them, and not only to a mailbox hosted on the site you are monitoring. Include in each alert the URL, the failing status or error, the time and the probe locations. Add a recovery message so nobody has to guess whether the problem ended.

Avoid alerting on one failed probe. Most services let you require two or more consecutive failures or failures from multiple regions before sending. That cuts false alarms without slowing real detection by much.

What success looks like: A test alert reaches the people on your list, and each of them knows what to do first.

Step 4: Serve a proper 503 for planned downtime

When you must take the site down for a migration or maintenance, answer with HTTP 503 Service Unavailable, which signals a temporary condition, and add a Retry-After header saying when to try again. The value is either a number of seconds or an HTTP date:

HTTP/1.1 503 Service Unavailable
Retry-After: 3600
Content-Type: text/html; charset=utf-8

MDN's documentation notes that support for Retry-After is inconsistent across clients, though some crawlers such as Googlebot honor it. Serve a short human-friendly page in the body so visitors know what is happening. A 503 response should not usually be cached, so check that your CDN is not holding on to it after you finish. If requests from particular clients are being restricted, the correct status is 429, not 503.

Do not use a 200 page that says "back soon": a 200 tells crawlers the page works, and Google reports error-like content under a 200 as a soft 404. Do not use 404 or 410 either, because Google treats 4xx responses as pages that should leave the index. After the work, confirm that the normal pages return 200 again before you stop watching.

Tell your monitoring service about the maintenance window so that the 503 does not trigger a page at 3 a.m.

What success looks like: During a test, curl -I https://example.com/ shows 503 with Retry-After, and after the work the same command shows 200.

Step 5: Add Search Console's host status as a second signal

The Crawl Stats report in Search Console has a Host status section built from three things: whether robots.txt could be fetched, whether DNS resolved during crawling, and whether your server connected and returned a full response. A significant error in any one can lower the overall status. In the robots.txt measure, a valid file, an empty one, or a 403, 404 or 410 counts as success; a 429 or 5xx counts as a failure. Google marks a day as an issue when a metric rises above the dotted red line on the chart, and the status covers the last 90 days.

The report is only available for root-level properties, meaning a Domain property or a URL-prefix property at the root such as https://example.com. It shows what Google experienced, which can differ from what your monitor saw. Check it after any outage and weekly during the first month after launch.

What success looks like: You can open Crawl Stats, find Host status, and tell whether the last incident shows up there.

Step 6: Write a short runbook and test it

For each check, write the first three things to try: look at the hosting dashboard, check the latest deploy, check DNS and the certificate. Decide who can roll back a release. After an incident, record the cause, how long it lasted and what you changed. Redirect Chains, Loops, and Incorrect HTTP Status Codes is a good reference when the failure is a redirect problem rather than a dead server.

Crawvix runs scheduled scans that repeat its health checks and flag regressions, including whether the domain resolves and the home page loads. That is useful for spotting configuration drift between releases, but it is not a replacement for a dedicated monitor that probes every few minutes, so keep both.

What success looks like: A one-page runbook exists, and someone other than its author has followed it in a drill.

Final verification checklist

Monitors request real HTTPS URLs for the home page, key inner pages, robots.txt and your sitemap.
Each check verifies status and expected page content, not status alone.
Probes run from at least two locations outside your own server.
Alerts reach at least two people or channels, need more than one failed probe, and send a recovery message.
Certificate expiry and DNS resolution are monitored.
Planned maintenance returns 503 with a Retry-After header and a friendly page.
After maintenance, key pages return 200 and the maintenance window in the monitor is closed.
Search Console's Crawl Stats host status has been opened and understood.
A written runbook exists and has been tested.

Common mistakes to avoid

Monitoring from the same server or network. If the host is down, the monitor goes down with it. Probe from elsewhere.
Checking only the status code. An error page with a 200 passes. Match text that appears only on a healthy page.
Monitoring only the home page. Add inner pages, robots.txt and the sitemap.
Alerting on every blip. Require repeated or multi-location failures so people trust the alerts.
Sending alerts to one mailbox. Use at least two routes, one of them separate from the monitored site.
Using 200, 404 or 410 for maintenance. Use 503 with Retry-After for temporary downtime.
Forgetting certificate and domain expiry. A valid site becomes unreachable the moment either lapses, so monitor both well ahead.
Never testing the alert path. Trigger a failure on purpose to confirm that the message arrives.

Frequently asked questions

How often should an uptime check run?

Often enough that you would act on the delay. Many services offer intervals from one minute upward; shorter intervals find problems sooner, but you should confirm what your provider and plan allow. Pair frequent checks with a rule that needs repeated failures before alerting.

Does downtime hurt my search rankings?

Google says server errors make its crawlers slow down and that persistent errors can lead to indexed URLs being dropped. It gives no figure for how long that takes, and brief outages are normal. This guide does not claim a ranking effect; it aims to shorten the time a problem goes unnoticed.

Should I use 503 or 404 when the site is down for maintenance?

Use 503. It tells crawlers the condition is temporary. A 404 or other 4xx tells Google the content is gone, and previously indexed URLs can be removed.

Is a free monitor good enough?

For a small site, a simple external check on the home page may be enough to start. Look at the interval, number of locations, content matching and alert options. Choose based on those, not the brand.

Can Search Console replace uptime monitoring?

No. The Crawl Stats report shows what Google's crawler experienced, over days, and only for crawls that happened. A monitor probes on a schedule you control and can tell you within minutes.

Official references

Next step: See how Crawvix checks a website and capture a baseline once scanning opens. This guide provides technical information, not a guarantee of rankings, traffic or index inclusion.