jtlowry-blog

[jtlowry-blog]

Why Go Through the Trouble Analyzing Server Logs?

A server log is a simple file output made by a web server containing a record of all requests your server has received. In the context of SEO, whenever a page is requested by a bot or a searcher an entry is made. While the format can differ slightly depending on your server, they all follow The Common Logfile Format.

Server logs excel at providing insight into potential issues crawlers and searchers face when accessing your site.

For example, consider the below screenshot. Within Search Console, Google only provides “examples” of pages that they are excluding from being indexed.

This is where server logs come into play. You can access your logs and parse them to uncover potential crawler traps. You can easily filter traffic by the user agent string and look at the crawl frequency of those pages. Then you can compare it to what you’re seeing within Search Console during that same time frame.

Server logs will allow you to make better informed decisions when it comes to optimizing crawl-ability of client sites.

Additional use cases for Server Logs

Much has already been written about why you would use server logs as part of your SEO strategy.

But here are a few additional use cases.

Troubleshoot 4XX’s & 5XX’s

Audit 1:1 redirects

Identify parameter URLs

Identify crawler traps

Site speed improvements

Identify malicious bots

In this guide I want to share my process for accessing, cleaning and possible next steps to take with your data.