Feed process steps
Whether you manually trigger a feed or it is triggered by the scheduled hour, the feed process will go through steps to map, filter, categorize, customize and validate your data from your sources to feed files. Then, the files are ready to be fetched by the channel or submitted to a channel via FTP or API.
- Process is triggered by manually selecting "compile" or "submit" in the app or by a scheduled import
- Mapping data into the template from imported product fields or static values
- (Optional) Filters & manual exclusion/inclusions remove/include SKU rows from the feed
- (Optional) Category mappings and rules apply channel taxonomy values
- (Optional) Custom rules modify and update data according to their execution sequence
- Validation analyzes product data for missing required fields, invalid format, recommended fields and any optimization
- Feed preview and validation files are written as CSV files, as well as any channel specific files like XML
- (Optional) At this point, the feed file is ready to be fetched/pulled via URL
- (Optional) Finally, when submitting the feed manually or on schedule, the system, sends the feed files to the destination channel using the submission parameters
Triggering a feed
Compiling the feed builds the output files, as well as the feed preview and validation. When a feed submits, then the feed is both compiled and sent to the channel. Feeds are triggered either from the front-end of the app by clicking on buttons in the UI or by scheduled imports/submissions.
Before we continue, allow us to define some terms so you may better understand different steps better.
|Product Import||Product data is first downloaded in the cloud from your source(s), then parsed and imported into our internal database for use in one or more of your feed templates. Product imports can be manually triggered, but are most often ran nightly or on a daily schedule of your choosing.|
|Product Download||When you download the imported product data from our system to your own computer, we will provide a zipped file containing all of the desired products and columns so you can review the data in a program like Microsoft Excel or Notepad++.|
|Feed Compilation||Compiling a feed runs all steps from mapping, filters, categories, custom rules, validation, and so on to produce the feed files which are kept in a feed folder along with a CSV formatted preview and validation list file. Feed compilations can be manually triggered, but are most often ran after an import completes before being submitted or pulled.|
|Feed Download||Similar to the feed download, when you download the feed folder from our system to your own computer, we will provide a zipped folder containing a preview and validation list as well as the output file(s) for the channel so you can review the data in a program like Microsoft Excel or Notepad++.|
|Feed Submission||Submitting the feed send the output file(s) to the channel using the feed settings. Feeds which are scheduled to submit on a particular hour are first queued as "submission pending import". Once the import completes, the feed compiles and then submits to the channel. API feeds will then attempt to retrieve a processing report from the channel to get the results of the feed submission.|
|Feed Pull||After every import, our system will check for feeds that have been pulled in the last 30 days and will compile those feeds so they are ready to be pulled. Feeds can be pulled by a channel from a provided URL found in the feed settings. You will want to coordinate your import schedule to allow enough time to import and compile your feeds before the channel pulls or fetches the feed.|
Setting up your feed
Once you have added the desired channel template to your store, you will need to turn it into a feed. You will need to map your product data, choose which products to send to the channel, categorize your products according to the channel's taxonomy, and then configure to when and where the feed will submit.
The following are excerpts from our Feed setup guide: How do I set up my feed? article.
The system will automatically map system fields to required fields. By default, fields like Sku, Price, URL, ImageURL, Brand, Description and Title, will be populated from the source data. You can add or remove these default field mappings.
Then, you can map more of your product data by mapping a source field or static value.
Filters and exclusions
There are three ways products can be excluded from a feed:
- Filter rules
- Manual exclusions
- Validation errors
Using Filters to exclude groups of products
Filter rules allow you to exclude many products based on data values or empty fields. In this example, a filter rule is created to exclude gift cards. Filters are inclusion rules which means you will need to write the conditions in a manner which correctly includes only products you want to include in the feed.
In the example below, the conditions are created to only include products which page_title does not contain "gift card" and Category does not equal to "Gift Card". If you want to create the same rule, then your conditions will be different depending on your product data.
Notes about feed filters
- An unfiltered feed includes all products from your product catalog unless you manually exclude them on the product validation page
When we compile your feed, each rule is executed and products that do not meet the rule conditions are removed from the feed
You can manually exclude or include individual skus that are affected by filters using the options in the table on the Products page
You may also filter products at the product catalog import level using import filters (requires Ultimate plan)
Each channel categorizes products differently. Most channels require you to send your products pre-categorized to their taxonomy, like Google Shopping. However, there are a few channels that do not require categories in the feed like Walmart.
We've loaded channel categories into each feed template. There are two ways to categorize products in your feed: mapping and rules.
Category mapping makes categorizing products fast and easy. Type to find channel categories for each of you unique categories.
You can also use the Suggest feature to find suggested matches. We look at your imported categories from your store and suggest matching categories from the channel. Most categories will yield suggested matches, but some categories will return 'No Matches Found.' For these unmatched categories, you will need to create category rules.
The Category Rules section allows you to override category mapping using rule conditions.
If you map your category "Diamond Rings > Wedding Gifts > Gifts Above $500" to Google's category "Apparel & Accessories > Jewelry > Rings", maybe there are some products that are not correctly categorized on your store.
You can create a rule that says, categorize products into "Apparel & Accessories > Jewelry > Rings" if the product title or product type from your store data contains "rings".
Rules allow you to apply logic to your product data to update or modify field attributes, add third-party tracking, replace text in your data, calculate number values and generally improve product listing quality.
Append or prepend fields or values together, like you would using a Excel function.
For example, you could join your brands and titles or you could append colors to the end of your titles or units of measure to shipping dimensions.
- Brand + Title = "TREK Basketball Shoes"
- Title + " - " Color = "Boys shirt - Blue"
- Shipping length + " in" = "3.5 in"
Replace or remove text in your feed data with a custom rule. Here are some examples of how you can use replace text functions.
Fix duplicate values
Remove values completely by leaving the replace value empty
Apply basic arithmetic to your numeric fields in your data feed with a custom rule. Learn more: Calculator function.
Examples of calculator functions in custom feed rules
Static Text vs Fields
Add 20% to price for a markup
Multiply feed price by .7 to discount price by 30%
Subtract 2 from Quantity to create a buffer
Append UTM parameters to URLS
Use concatenation to append UTM tracking parameters to product URLS.
Use Google's Campaign builder tool to brainstorm your parameters. Then, use dynamic product data or static values to build your rule.
Learn more: Concatenate UTM parameters to product URLs
Each compilation ends with a validation of the product data. Each feed template has data quality check rules called "validation rules" that analyze the feed data for completeness, channel readiness, and optimizations.
Feed compilation and files
Every time a feed is compiled, a feed folder is created which contains a feedpreview.csv and feedvalidation.csv, as well as any other channel formatted files like XML.
The downloaded feed folder is always contained in a zipped folder.
The feed folder, preview file, and validation file all use a naming convention which follows:
This helps to identify files across accounts, stores and feeds.
In this example, the feed folder contains the actual XML file to be sent to the channel in addition to the system provided preview and validation CSV files. If the output file is CSV format, then we rename the preview file to the file name from the feed settings. If the output file is TXT, then we have the preview CSV as well as the output TXT file.
Download feed files
To download the feed folder, you may do so from the feeds page or the feed details page.
The feed pull URL provides the feed in the desired file format as specified in the feed settings. Downloading the feed folder gives you the output feed file(s) as well as any channel specific feed files which can be XML or other format.
Now that you have setup your feed, let's learn about how to publish the feed to your channels.