SEO Concerns and Product Update Issues with Affiliate Product Data Feeds

A visitor of my site who read some of my product data feed resources contacted me with an interesting question about how to deal with availability and SEO issues when you are utilizing merchant product feeds.

If I build a website with an affiliate data feed of an example: 10,000 different digital camera products. How would anyone but myself know that these products are available on my website?

My idea is a simple PHP script site with a CSV data feed file dynamically loaded. Because there won’t be any static pages built, how will the search engines see them?

If your pages URLs change all the time (as in appear and disappear = 404 error), you will have a hard time to get pages indexed by search engines. In order to be able to generate the same page for the same product over and over again is it necessary to have an unique product identifier, which could be the merchant SKU or in the case of digital cameras the UPC or EAN (I would use the EAN if you have that).

You don’t have to delete pages if a product is out of stock. You could (and probably should) keep the page and indicate that it is out of stock. You have one problem though. You need to know when a product is out of inventory and discontinued (= it will never come with new inventory in a future data feed file). There are three 1/2 options, which all involve the merchant (more or less)

Option 1) The merchant provides ALL products in the feed that he is selling, even the out of inventory one, but indicates the inventory level in a column of the data feed (absolute number or just an indicator e.g. available/out of stock or out-of-stock/back-order/low/available etc.

Option 2) The merchant has a product status that indicate which item is an active SKU that will be replenished or if it is a “discontinued” item. A column for product status would do the trick, like “A” for active and “D” for discontinued.

If a SKU that was in status “D” is not included in the next feed, then you have to delete it, but you keep all products that are in status “A” and just show “out of stock” for them and wait for a future feed with new inventory again.

Option 3) and 1/2 If you don’t get any of the above, you can’t automate the update properly and should keep the product in the DB until you remove it manually. You either have to check yourself, ask the merchant or the merchant comes and asks for the removal (if he complains, explain to him what the problem is and what he can do about it).

In the case of cameras does it probably make sense to check once or twice a year if old models are still manufactured or not. Call it inventory cleaning day or spring cleaning/preparing for the holidays.

Dynamic Scripts and Search Engines

Now some general notes about the use of Product Data Feeds to generate content pages in an automated fashion.

I did a lot with data feeds in the past and know that they can be a pain-in-the-neck sometimes.

While the technology behind it isn’t rocket science, did affiliate networks and merchant manage to screw up pretty much everything there is in the process, leaving it up to the affiliate to deal with those problems. You can’t assume anything and should always expect the worse. It does not only sound like a lot of work, it actually is.

I don’t want to discourage you, but I want to make sure that your expectations and goals are realistic and that you are ready and willing to spend the time and energy necessary to reach them.

I spent a lot of time to collect and write up resources and information to make it easier for other affiliates to deal with the subject. I also talked to several of the networks about the issues. Most listen, but only few are actually doing something about it.

Data Feed Resources / Suggested Reading

I suggest checking out the entry page on my site that is dedicated to affiliate data feeds.

My posts in this forum thread talk about the requirements and skills needed that are necessary to deal with data feeds from a technical point of view. It also mentions alternatives to the use of raw data feeds that should always be considered. I planned to write something up based on those posts, but did not get around it yet.

I created a write-up that is a good 101 for merchants and affiliates to the subject. It was also based on a forum thread at ABestWeb actually. I strongly suggest to you to read it.

Then I have documentations for the individual networks that provide data feeds. There is no standard, which means that it is up to you to either create a flexible solution and/or you chose to work with data feeds from one source (network in most cases) only.

Regarding SEO and the use of automated scripts to process feeds

Read this article of mine. It’s about web templates, but mentions data feed sites as well.

If your goal is to create sites based solely on data feeds and nothing else and expect them to rank anywhere in the search engines and make money off them, look for something else to do.

It is not 2001-2003 anymore. Google caused virtually a genocide among data feed affiliates with their “Florida” update in Fall 2003. Webmerge was the tool of the trade to create those sites (its still available, see my data feed page).

The sites that were build a bit more cleverly were mostly killed off by their “Jagger” update at the end of 2005, followed by their “BigDaddy” infrastructure update in early 2006. My largest data feed based site was among those. I was not doing much SEO at that time anymore, which was part of the reason why I could not prevent the site to get almost entirely washed out of Google’s index. There are only a few hundred pages left today, most of them are supplemental.

Making a Data Feed based Site Work Today

You have to add value for the user and content for the search engines in order to make it work. What you also need are a lot of inbound links to your site to be able to keep thousands of pages in the search engine index. This means that some word of mouth and social media marketing should be part of your business plan. It can be done, but it is much harder today than it was back then.

I suggest to concentrate on building a socially friendly site that does not rely as much on search engines and do the SEO work on the side and see the results of it as a nice added bonus, but not an essential part of your business.

You can do what you want. My tips are only well intended to save yourself a lot of time by learning it yourself the hard way and make mistakes that others already did before you.


Carsten Cumbrowski

About Carsten Cumbrowski

Internet Marketer, Entrepreneur and Blogger. To learn more about me and what I am doing, visit my website and check out the “about” section.

Twitter: ccumbrowski

5 Responses to SEO Concerns and Product Update Issues with Affiliate Product Data Feeds

  1. SEO says:

    I wonder if hard coding section pages as index-able pages would work? Create a html page that would link to the dynamic pages. They can show discounted products and some of the popular models. All this info would be static and indexed. You would need to update these pages weekly or monthly.

    Good solution?

    Paul R

  2. Hi Paul, thanks for the comment. From a SEO perspective I don't think that your idea makes a lot of sense and is even counter productive IMO.

    No, you have to do something with the feed content. Mash it with something else of value to the user and that is unique, add also your own content to it, step by step and/or have users generate it for you.

    Whatever the result might be, it not only has to pass the automated filters and checks by the search engine bots and algorithms, but also pass the human check as well. If it does not pass the second and only the first, then it is just a question of when and not if your site will be kicked out of the search engines… manually penalized or filtered out during a future update of SE Algos that will come for sure.

  3. Scott A. says:

    Good post Carsten. I agree that you can't just build a datafeed site for SEO anymore. It needs to have some social functionality and some noteworthy features that can garner some press. It's fine to have datafeed content, but I think a good solution for a new site would be to keep the data part of an internal search engine and only make products public if your community interacts with them. For example if you've got a review system maybe only publish the products with reviews. That way there is always a person tied to each spiderable product in one way or another.

    As for your structural suggestions I'd like to add another solution. You can assign an auto_increment product ID to each product as you import them into your database. These ID's can be used in your URL and will never change. You can simply mark them as out of stock when the product is no longer in the feed and stop linking to those pages. Then maybe after a couple months the product can be removed permanently.

  4. Hi Scott, thanks for your comments as well. The identity or auto_increment product id generates extra overhead during updates, unless you are using it as an additional key for your internal purposes. Having your own product identifier makes sense if you do price comparison, but there I would prefer using the EAN13 barcode, which is even used by several U.S. merchants themselves (Wal Mart is an example of a large retailer who uses EAN13 for their product ids).

    It's a bit slower for primary key look ups, if you keep your own auto_increment product ids smaller than 2.1 million, but you can save a lot of time during pre-processing of feeds.

  5. p.s. the EAN13 does not work for all types of products, particularly the ones that have no national nor international bar code.