On my mission to create a completely automated Money Making Blog, I discovered Yahoo Pipes. Yahoo Pipes provides you with the ability to take anything from any other site and use it however you see fit.
What really excites me about Yahoo Pipes is that it provides us a way to create an automated site! Imagine a site that is constantly updating with the most interesting stories and photos, and deals from all over the web. It’s not a fantasy anymore.
In a previous article, Post to WordPress, I showed you how you can post to your WordPress site by email. Yahoo Pipes will send you an email every time new information is gathered. See where I’m going? Now you just tell Yahoo Pipes what you want gathered, it gets it and then posts it directly on your website!
I’m going to now walk you through numerous examples on how to use Yahoo Pipes. It is a drag and drop interface, but it does have a learning curve that must be overcome.
Here is a Video Where I Demonstrate Using Yahoo Pipes
Here are some neat things you can do with Yahoo Pipes:
Pull statistics from any CSV (Comma Separated Values) file and manipulate them on the fly
Pull any article from any site and present the entire article, or the exact number of words that you want
If you don’t like the ads in RSS feeds, eliminate them with ease
Want to pull an article and then populate that article with photos from Flickr? No problem!
Want to pull up a map listing every restaurant nearby? Piece of cake!
Want to monitor deals on laptops on Craigslist? Easy!
The Basics of Yahoo Pipes
Here is the interface for Yahoo Pipes. On the left side there are modules that you drag on to the working area. I’ll walk you through all of the different modules and what they do.
This is a list of the types of data that you can gather using Yahoo Pipes. Just drag the module out, and tell it from which URL you wish to gather data. Here is the types of data you can pull:
CSV Data: Comma-Separated Value files are often used to store statistics. You can pull those statistics using Yahoo Pipes and organize them as you wish.
Feed Auto Discovery: You use this module when you don’t know if a URL has a feed. Just plug in the URL and Yahoo Pipes will return all of the information from every feed available from that URL.
Fetch Feed: If you know the feed address, just plug in the URL for the RSS, Atom, or RDF formatted feed. Yahoo Pipes pulls all the data for each feed you enter.
Fetch Page: You can retrieve data from any HTML page with this module. Just provide the URL, the point to start retrieving info and the end point and Pipes retrieves the info you define.
Fetch Site Feed: Similar to Feed Auto Discovery, in that it finds the feeds available based off of the URL you provide. The difference is that it returns just the most important feed and will update the location if the feed address changes automatically.
Flickr: Returns photos from the Flickr website, based on your requested keywords or defined location. Very nice for dressing up articles with complimentary photos.
Google Base: Allows you to assign Google search results to an RSS feed, based off of the keywords you supply.
Item Builder: This module, allows you to pull individual pieces of data from any data source and assign it a given name. Then you can manipulate the content more easily.
RSS Item Builder: Allows you to easily create a custom RSS feed, by plugging in the location of the Title, Description, Link to the Main Article, Phots, and much more. Use this when you want a step by step way to create a custom feed.
Yahoo Local: Enter what your looking for and the location and this module spits out a feed specific to your search. It also will create a map, based off of the results, if you’d like.
YQL: Allows you to select very specific data from the URL’s you provide. It uses a syntax that is very similar to SQL. Here is an example on how you could use it: select * from rss where url in (‘http://digg.com/rss/index.xml’,'http://feeds.mixx.com/MixxPopular’) | sort(field=”pubDate”) This would return every article from two different feeds and then sort them based on publication date.
Yahoo Search: Provides information based off of searches done using Yahoo. You can define that you only want data from specific URL’s. You also can perform searches based on defined keywords and locations.
User Input Modules
By using User Input modules, you can easily change the data searched for by a Yahoo Pipe. You can define the following:
Number: For example, you could define the number of articles to retrieve
Text: For example, you could provide a keyword to search for
URL: Allows you to define a changing URL
Private: Used when you want to define a secret Userid or Password
You use the Operator Modules to manipulate the data that flows into your Yahoo Pipe. Here are the operator’s available:
Count: Used to count the number of items you input into a Yahoo Pipe
Filter: The Filter module allows you to define what info to include or exclude from your Yahoo Pipe. If you want to only include an article if the word “Cat” is in the title, use Filter.
Location Extractor: Will search the information for any location based information. This information is sometimes provided and other times is not. If location information is available, you can present that information on a map.
Loop: Used to perform the same action repetitively on each piece of information that enters your Yahoo Pipe. You drag other modules onto the loop module for this to work. I’ll provide many examples on how to use this module in future articles.
Regex: Allows you to manipulate data using Regular Expressions. This is a very powerful tool and I’ll provide many examples on how to use it.
Rename: You use this module to sort and rename data that flows into your Yahoo Pipe.
Reverse: This module flips the order of all items in a data feed.
Sort: This module will sort all of the items in an incoming feed, based off of any defined item.
Split: If you want to perform two different sets of operations on each feed item, this is the module for you.
Sub Element: Allows you to pull very specific pieces of data from a feed, if the data is contained between a logical set of tags.
Tail: Returns a specified number of items from the bottom of a feed.
Truncate: Returns a specified number of items from the top of a feed.
Union: Used to merge multiple feeds into one feed. This is the opposite of the split module.
Unique: Eliminates feed articles that are duplicates.
Web Service: Allows you to send your created Yahoo Pipe data to another Web Service for processing.
Create RSS: Converts a list of non RSS data into the RSS format.
URL Builder Module
You use this module to define a specific URL to pull information from. If you wanted to pull data from the results of a Google search on “dogs”, you’d manipulate the following URL:
See where you see the word dogs, in the URL. With the URL Builder module, you could assign the value to search for with a User Input module for example. You’d do this by defining the value of the variable q, to what ever the user typed in.
This is just another way to make your Yahoo Pipe’s results easy to change.
String Modules allow you to manipulate strings of text, inside of Yahoo Pipes. Here are the modules available:
Yahoo Shortcuts: This is a very cool, but hard to explain tool. It tries to categorize the information it is sent. For example if it received the string, “Paul went to Toys R Us in Tacoma”, it would try to label Paul as a name, Toys R Us as a location, and Tacoma, as a city. It’s great, when it works.
String Builder: Allows you to create a long string by combining many small strings, from multiple data sources.
String Regex: Allows you to perform Regular Expression edits on individual strings of characters
String Replace: Allows you to search for a specific string, with in another, and replace it.
String Tokenizer: Splits a long string into an array of strings, based off of a delimiter that separates the parts of the string. This would be useful for a CSV, which has the delimiter set as a comma.
Sub String: Allows you to pull a string from a larger string, by defining the starting and end of the string you want. If I just wanted the string “ape”, from the string, “That is a hairy ape.” I would ask for the character in the position 16 and then 3, for the total number of characters to retrieve. The string starts off with the first letter being assigned to the value 0 and then 1.
Term Extractor: Will return those words it considers important, from a given string. It strips off worthless words like, and, the, or, etc. This is a great tool for extracting the tags our articles will use. More on that later.
Translate: Translates text from one language to another.
Private String: Used when you are working with secret information like Userids or Passwords.
The Rest of the Yahoo Pipe Modules
There are a bunch of other modules available, but because of their simplicity, I’ll just list them. As you watch the video’s I provide and play with Yahoo Pipes on your own, you’ll get to know them better.
Date Modules: Are used to format dates with in the data you analyze
Location Modules: Used to convert strings into specific geographic locations
Number Modules: Used to perform mathematical operations on data
Favorites: Those Yahoo Pipes you have chosen as your favorites
My Pipes: Here is a list of all the Yahoo Pipes you have created
Deprecated Modules: Are modules that have been replaced, and shouldn’t be used (Ignore These)
That’s All for Now
I will be providing many examples on how you can use Yahoo Pipes to create custom RSS feeds and create automated websites in future articles. If you have any questions or comments, leave them in the comment section below.