Use case analysis
Let's start this by analyzing the ways this system will be used. The result will be one (or more) modules for Drupal which we can use to build product catalogs automagically populated from affiliate program datafeeds.
This section covers usage scenarios conducted by the site administrator.
Load a new datafeed: This will be performed shortly after the administrator initiates a relationship with a new merchant. The administrator will download the datafeed or otherwise access it, e.g. the data might come through a web API rather than via a data file.
Script system to take a datafeed file, and upload its contents into the drupal database such that each product becomes one node. As the data is uploaded, also insert taxonomical categorization for each product node. This must be compatible in all ways with the drupal node system, most especially ensuring a "node ID" is properly assigned.
On generated product pages show the product name in the page title, and at least one other place on the page.
Replace an existing datafeed with a new release: Merchants regularly release new versions of their data feed. They are generally updating prices, giving more information, adding products or deleting products.
For items where the merchant is updating the details, the process must determine whether the product is already existing, and replace the new data into the existing object. This must leave the same product at the same URL.
For new items, it can be entered into the content management system as described above.
For items deleted from the data feed it needs to be unlinked from all cross-referencing created in the CMS. I'll note in passing that it's going to mean an interesting implementation conundrum to detect the deleted item. A deleted item won't show up in the new datafeed at all, so how do we detect it's missing?
There should be a filter mechanism in the scripts for updating a datafeed to modify the items as they're loaded to the CMS. For example an administrator may want to block certain products from entering the CMS. Or they may want to modify items, such as their description, as they go into the CMS.
Edit or delete individual items sourced from a datafeed: The administrator may want to customize the products being displayed beyond what is in the datafeed. The administrator may want to remove a product from being displayed, or to change something about a product.
Edit the taxonomy associated with the datafeed: Similarly the administrator may want to modify the categorization (taxonomy) relationships after the items are loaded into the CMS.
Delete all items loaded from a given datafeed: When the administrator cuts off relationship with a merchant, they must remove all items from that merchant. When removing items, as already stated, all taxonomy relationships must be removed.
This section covers usage scenarios conducted by the website users.
Find product pages via search engine queries: The goal of this whole project is to connect with people looking for a given product or products in a given category. One way of doing this is ensuring the pages generated from the datafeed are indexed nicely by search engines. For example if someone searches for a specific product name, the datafeed will contain that product name (at least once) hence the product page must contain the product name prominently.
Browse catalog pages of products loaded from a datafeed: Another set of users to connect with are ones looking for products in a given category. The system must support creating index pages showing products from given categories. The categories must be easily identifiable, creatable and managed.
Facilitate ordering products: The product page must include the affiliate link. That link will direct the user to the merchant web site.
Offer reasonable behavior for search engine queries that land on stale pages: In the system we might well end up publishing a page that is later deleted. For example, if a merchant closes down or stops ordering a given product, the pages related to that merchant or product will be deleted. But the search engines can take months before they realize the page is gone and in the meantime people might still find the stale pages in search queries. ...????... TODO: Do not have a solution for this right now.
It might be nice for marketing purposes to brand pages related to a given merchant with graphics or banners from that merchant.