A log file is actually a file output made from a web server containing ‘hits’ or records of all requests that the server has received. Data are stored and deliver details about the time and date in which the request was made, the URL requested, the user agent, the request ID address, and other interesting details. To explain it quickly, log file analysis allows you to get information about SEO visits and to see what the Googlebot is actually doing on your website. You can thus cross it with your crawl data and see further.

Let’s see the advantages of what you can get with log file analysis.

The advantages of log file analysis

Why is it useful?

Log file analysis is useful for many reasons:

• For your audits :
  • You can diagnosis useful and useless pages
  • You can detect zones that Google crawls
  • You can know which pages Google does not know
• For your monitoring :
  • You can get alerts and avoid waiting for a Google Webmaster Tool message
  • You can monitor optimizations or deployment more easily
  • You can anticipate attacks

Why should you use log file analysis?

You can exactly know what Google does on your website:

• Which pages are crawled by the Googlebot? Log file analysis helps you monitor the crawl behavior and crawl frequency. You can know what Google is actually crawling and see how page popularity, page depth, load time or any other important metrics can influence Google’s crawl. It can also help you determine if specific new content has increased Google visits on your website. The more interesting your site is, the more often Google will come.

• What are my active pages? Which pages receive the most SEO visits? Are these my most valuable pages? You can know which pages are actually generating SEO traffic, value and, thus, conversions. Logs analysis can also help to determine the most popular pages to Google’s eyes and see which ones are less crawled. For instance, if a user would like to rank a specific post for a targeted query but it is located in a directory that Google only visits once a time every three months, he will miss chances to receive organic traffic from this publication for at least three months. With log analysis, he can know that it could be necessary, for example, to redefine his internal linking to increase the impact of his “most valuable pages”.
• Does Google meet errors? Do you have too many 4xx errors on your website that lower Google experience on your website? Log data analysis can also help track errors in status codes like 4xx or 5xx that compromise SEO. Analyzing a website’s status codes also helps to measure their impact on bot hits and their frequency. Too many 404 errors will limit the crawler visit.
• In every case, you need to save Google crawl budget to help spend it on the right pages. It helps you improve your money pages’ performance and be sure that Google is actually seeing them! With log file analysis you can, for example, detect if Google spends too much crawl budget crawling resources like images or .css files. This budget is linked to the authority of your domain, the sanity of your website, and is proportional to the flow of link equity through your website. You don’t want that budget to be spent on useless pages.

Imagine if you could do log analysis for free.

Actually, log file analysis can be seen as expensive for companies and many of them stick to crawl analysis. But this technology is getting more accessible.

Open source solutions also exist. One of them is the free OnCrawl open source log analyzer. It is quite easy to install even for non-tech profiles.

1- Install Docker

Install Docker Tool Box.

Choose Docker Quickstart terminal to start.

Copy/paste the IP address 192.168.99.100

Then, download oncrawl-elk release: https://github.com/cogniteev/oncrawl-elk/archive/1.1.zip

Add these lines in the terminal to create a directory and unzip the file:

• MacBook-Air:~ cogniteev$ mkdir oncrawl-elk
• MacBook-Air:~ cogniteev$ cd oncrawl-elk/
• MacBook-Air:oncrawl-elk cogniteev$ unzip ~/Downloads/oncrawl-elk-1.1.zip

And then, add:

• MacBook-Air:oncrawl-elk cogniteev$ cd oncrawl-elk-1.1/
• MacBook-Air:oncrawl-elk-1.1 cogniteev$ docker-compose -f docker-compose.yml up -d

(Note: those lines are working for Mac and Linux. If you are under Windows, the process is a little bit more complicated for non techs).

Docker-compose will download all necessary images from docker hub, this may take a few minutes. Once the docker container has started, you can enter the following address in your browser: http://DOCKER-IP:9000. *Make sure to replace DOCKER-IP with the IP you copied earlier.*

You should see the OnCrawl-ELK dashboard, but there is no data yet. Let’s get some data to analyze!

2-Import log files

Importing data is as easy as copying log access files to the right folder. Logstash starts indexing any file found at logs/apache/*.log, logs/nginx/*.log, automatically.

If your web server is powered by Apache or NGinx, make sure the format is combined for log format. They should look like this:

127.0.0.1 – – [28/Aug/2015:06:45:41 +0200] “GET /apache_pb.gif HTTP/1.0” 200 2326 “http://www.example.com/start.html” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”

Drop your .log files into the logs/apache or logs/nginx directory accordingly.

3-Play

Go back to http://DOCKER-IP:9000. You should have figures and graphs. Congratulations!

You can now start using the free open source log analyzer and daily monitor your SEO performance.

To sum up, log file analysis is a powerful partner to increase your SEO performances and as a result, helps you drive more traffic and conversions to your website!

[xyz-ihs snippet=”Hubspot-CTA-Leaderboard”]

The post A Guide to Log File Analysis – How to Open the Google Blackbox appeared first on Relevance.

Duplication refers to blocks of content that appear more than one time inside or outside a website or which are pretty similar. Duplicate content or near duplicates can lead to SEO penalties.

In fact, crawlers have trouble indexing the right content between different versions. Bots are then obliged to pick the content likely to be the best one, but it can lead to a loss of relevancy as it is not always simple to choose the right version. Moreover, bots will face difficulties to deliver the link metrics to the right page or share it between the different versions. Then, duplicate content also leads to the inability to rank the right version for a given query. At the end, you will face a drop in traffic.

The thing is, Google gives a lot of credits to user experience and focuses on delivering the best content possible to its audience. This is also why duplicate content is penalized.

While not all types of duplicate content can hurt your SEO, some of them need to be on the look out to avoid SEO penalties.

This article will cover:

• types of duplicate content
• how to deal with duplicate content
• tools to get rid of duplicate content

Types of Duplicate Content Leading to SEO Penalties

There are different types of duplicate content you should avoid.

Duplicate product forms

E-commerce websites often use manufacturer’s item descriptions to describe the products they sell. The problem is that those products are often sold to different e-commerce websites. Then, the same description appears on different websites and creates duplicate content.

Syndicated or copied content

Many bloggers use content, quotes or comments from other websites to illustrate their articles. There is nothing wrong with that if you link back to the original one. However, Google can still consider this as a duplication and will poorly value those pieces of content.

Sorting and multi-pages lists

Large e-commerce websites have filter and category options that generate unique URLs. Product pages can appear in different categories and be ordered differently depending on how the list is sorted. For instance, if you range 45 products by price or by alphabetical order, you will end up with two pages containing the same content, but with different URLs.

URL issues

Google considers URLs in www, http, https, .com and .com/index.html as different ones even if they point to identical pages and will evaluate them as duplicate content.

Session IDs

Session IDs issues refer to different session IDs stored in the same URL that are assigned to a visitor when he or she comes to the website.

Printer-friendly

Printable versions of content can lead to duplicate content issues when different versions of a page are indexed.

How to Avoid Duplicate Content?

There are different best practices to avoid duplicate content issues. The main solution with content located in different URLs is to canonicalize the original one. You can use a 301 redirect, a rel=canonical or parameters handling tool from Google Webmaster Central.

301 redirect

The 301 redirect is great for URL issues leading to duplications. It informs search engines which version of a page is the original and it links the duplications to that original one. Plus, when different well-ranked pages are linked to a single one, they are not in competition anymore and they create an overall stronger and more popular signal.

Rel=canonical

It works quite the same as 301 redirects, but it is easier to set up. That tag is located in the HTML head section of your web page and looks like this:

<link href=”http://www.mywebsite.com/canonical-version-of-page/” rel=”canonical” />

Then search engines know the above URL is a copy of the original one.

You can use it for content you integrated from other websites. It will inform search engines that you know the content is not from you and that the link metrics of that content should pass to the original one.

NoIndex, NoFollow

Use the noindex,nofollow meta tag to tell search engines not to index the content. Bots will be able to crawl the page, but won’t index it. Thus, you won’t be penalized for duplicate content.

Preferred domain

A quite simple operation is to set a preferred domain for search engines. It will inform whether a site must be displayed under ‘www’ or not in the SERPs.

Unique product description

As we said, product information on e-commerce websites can lead to duplicate content issues. Take the time to write unique ones or enrich your descriptions as it will help you rank above sites whose descriptions are duplicated.

What tools can help me to detect duplicate content?

In order to save time, you can use different qualitative tools to help you eradicate duplicate content. Here are three different ones, with some being totally free.

Siteliner

This tool detects any duplicate content on your website. You just need to add your website’s URL and it will draw a full report with your content performances.

OnCrawl

This onsite SEO semantic crawler also offers a duplicate and near duplicate detection feature. It shows you clusters of duplicates and near duplicates, types of duplication and clearly indicates which URLs are concerned.

The tool offers a 30 day-trial. If you want to enjoy all the functionalities, you just need to pick a plan and cancel your subscription before the end of the trial and you won’t pay anything.

Copyscape

Copyscape is a great partner because it detects duplicate content outside of your blog. You can thus easily know if someone has duplicated your content without your permission or without giving you credits.

[xyz-ihs snippet=”Hubspot-CTA-Leaderboard”]

The post How to Fix Duplicate Content and Improve your SEO appeared first on Relevance.