Yahoo Pipes Tutorial Part 1

Yahoo PipesOn my mission to create a completely automated Money Making Blog, I discovered Yahoo Pipes. Yahoo Pipes provides you with the ability to take anything from any other site and use it however you see fit.

What really excites me about Yahoo Pipes is that it provides us a way to create an automated site! Imagine a site that is constantly updating with the most interesting stories and photos, and deals from all over the web. It’s not a fantasy anymore.

In a previous article, Post to WordPress, I showed you how you can post to your WordPress site by email. Yahoo Pipes will send you an email every time new information is gathered. See where I’m going? Now you just tell Yahoo Pipes what you want gathered, it gets it and then posts it directly on your website!

I’m going to now walk you through numerous examples on how to use Yahoo Pipes. It is a drag and drop interface, but it does have a learning curve that must be overcome.

Here is a Video Where I Demonstrate Using Yahoo Pipes

Note: I demonstrate how to use the Postie Plugin, in the article Post to WordPress


Here are some neat things you can do with Yahoo Pipes:

  • Pull statistics from any CSV (Comma Separated Values) file and manipulate them on the fly
  • Pull any article from any site and present the entire article, or the exact number of words that you want
  • If you don’t like the ads in RSS feeds, eliminate them with ease
  • Want to pull an article and then populate that article with photos from Flickr? No problem!
  • Want to pull up a map listing every restaurant nearby? Piece of cake!
  • Want to monitor deals on laptops on Craigslist? Easy!
  • Etc….

The Basics of Yahoo Pipes

Yahoo Pipes Interface

Here is the interface for Yahoo Pipes. On the left side there are modules that you drag on to the working area. I’ll walk you through all of the different modules and what they do.

Sources

This is a list of the types of data that you can gather using Yahoo Pipes. Just drag the module out, and tell it from which URL you wish to gather data. Here is the types of data you can pull:

  • CSV Data: Comma-Separated Value files are often used to store statistics. You can pull those statistics using Yahoo Pipes and organize them as you wish.
  • Feed Auto Discovery: You use this module when you don’t know if a URL has a feed. Just plug in the URL and Yahoo Pipes will return all of the information from every feed available from that URL.
  • Fetch Feed: If you know the feed address, just plug in the URL for the RSS, Atom, or RDF formatted feed. Yahoo Pipes pulls all the data for each feed you enter.
  • Fetch Data: Yahoo Pipes also will pull data from XML and JSON (Javascript Object Notation) data sources. Just specify the URL and Pipes does the rest.
  • Fetch Page: You can retrieve data from any HTML page with this module. Just provide the URL, the point to start retrieving info and the end point and Pipes retrieves the info you define.
  • Fetch Site Feed: Similar to Feed Auto Discovery, in that it finds the feeds available based off of the URL you provide. The difference is that it returns just the most important feed and will update the location if the feed address changes automatically.
  • Flickr: Returns photos from the Flickr website, based on your requested keywords or defined location. Very nice for dressing up articles with complimentary photos.
  • Google Base: Allows you to assign Google search results to an RSS feed, based off of the keywords you supply.
  • Item Builder: This module, allows you to pull individual pieces of data from any data source and assign it a given name. Then you can manipulate the content more easily.
  • RSS Item Builder: Allows you to easily create a custom RSS feed, by plugging in the location of the Title, Description, Link to the Main Article, Phots, and much more. Use this when you want a step by step way to create a custom feed.
  • Yahoo Local: Enter what your looking for and the location and this module spits out a feed specific to your search. It also will create a map, based off of the results, if you’d like.
  • YQL: Allows you to select very specific data from the URL’s you provide. It uses a syntax that is very similar to SQL. Here is an example on how you could use it: select * from rss where url in (‘http://digg.com/rss/index.xml’,’http://feeds.mixx.com/MixxPopular’) | sort(field=”pubDate”)
    This would return every article from two different feeds and then sort them based on publication date.
  • Yahoo Search: Provides information based off of searches done using Yahoo. You can define that you only want data from specific URL’s. You also can perform searches based on defined keywords and locations.

[adsense]

User Input Modules

By using User Input modules, you can easily change the data searched for by a Yahoo Pipe. You can define the following:

  • Date
  • Location
  • Number: For example, you could define the number of articles to retrieve
  • Text: For example, you could provide a keyword to search for
  • URL: Allows you to define a changing URL
  • Private: Used when you want to define a secret Userid or Password

Operator Modules

You use the Operator Modules to manipulate the data that flows into your Yahoo Pipe. Here are the operator’s available:

  • Count: Used to count the number of items you input into a Yahoo Pipe
  • Filter: The Filter module allows you to define what info to include or exclude from your Yahoo Pipe. If you want to only include an article if the word “Cat” is in the title, use Filter.
  • Location Extractor: Will search the information for any location based information. This information is sometimes provided and other times is not. If location information is available, you can present that information on a map.
  • Loop: Used to perform the same action repetitively on each piece of information that enters your Yahoo Pipe. You drag other modules onto the loop module for this to work. I’ll provide many examples on how to use this module in future articles.
  • Regex: Allows you to manipulate data using Regular Expressions. This is a very powerful tool and I’ll provide many examples on how to use it.
  • Rename: You use this module to sort and rename data that flows into your Yahoo Pipe.
  • Reverse: This module flips the order of all items in a data feed.
  • Sort: This module will sort all of the items in an incoming feed, based off of any defined item.
  • Split: If you want to perform two different sets of operations on each feed item, this is the module for you.
  • Sub Element: Allows you to pull very specific pieces of data from a feed, if the data is contained between a logical set of tags.
  • Tail: Returns a specified number of items from the bottom of a feed.
  • Truncate: Returns a specified number of items from the top of a feed.
  • Union: Used to merge multiple feeds into one feed. This is the opposite of the split module.
  • Unique: Eliminates feed articles that are duplicates.
  • Web Service: Allows you to send your created Yahoo Pipe data to another Web Service for processing.
  • Create RSS: Converts a list of non RSS data into the RSS format.

URL Builder Module

You use this module to define a specific URL to pull information from. If you wanted to pull data from the results of a Google search on “dogs”, you’d manipulate the following URL:

http://www.google.com/#hl=en&source=hp&q=dogs

See where you see the word dogs, in the URL. With the URL Builder module, you could assign the value to search for with a User Input module for example. You’d do this by defining the value of the variable q, to what ever the user typed in.

This is just another way to make your Yahoo Pipe’s results easy to change.

String Modules

String Modules allow you to manipulate strings of text, inside of Yahoo Pipes. Here are the modules available:

  • Yahoo Shortcuts: This is a very cool, but hard to explain tool. It tries to categorize the information it is sent. For example if it received the string, “Paul went to Toys R Us in Tacoma”, it would try to label Paul as a name, Toys R Us as a location, and Tacoma, as a city. It’s great, when it works.
  • String Builder: Allows you to create a long string by combining many small strings, from multiple data sources.
  • String Regex: Allows you to perform Regular Expression edits on individual strings of characters
  • String Replace: Allows you to search for a specific string, with in another, and replace it.
  • String Tokenizer: Splits a long string into an array of strings, based off of a delimiter that separates the parts of the string. This would be useful for a CSV, which has the delimiter set as a comma.
  • Sub String: Allows you to pull a string from a larger string, by defining the starting and end of the string you want. If I just wanted the string “ape”, from the string, “That is a hairy ape.” I would ask for the character in the position 16 and then 3, for the total number of characters to retrieve. The string starts off with the first letter being assigned to the value 0 and then 1.
  • Term Extractor: Will return those words it considers important, from a given string. It strips off worthless words like, and, the, or, etc. This is a great tool for extracting the tags our articles will use. More on that later.
  • Translate: Translates text from one language to another.
  • Private String: Used when you are working with secret information like Userids or Passwords.

The Rest of the Yahoo Pipe Modules

There are a bunch of other modules available, but because of their simplicity, I’ll just list them. As you watch the video’s I provide and play with Yahoo Pipes on your own, you’ll get to know them better.

  • Date Modules: Are used to format dates with in the data you analyze
  • Location Modules: Used to convert strings into specific geographic locations
  • Number Modules: Used to perform mathematical operations on data
  • Favorites: Those Yahoo Pipes you have chosen as your favorites
  • My Pipes: Here is a list of all the Yahoo Pipes you have created
  • Deprecated Modules: Are modules that have been replaced, and shouldn’t be used (Ignore These)

That’s All for Now

I will be providing many examples on how you can use Yahoo Pipes to create custom RSS feeds and create automated websites in future articles. If you have any questions or comments, leave them in the comment section below.

Till Next Time

Think Tank

23 Responses to “Yahoo Pipes Tutorial Part 1”

  1. Mike Joes says:

    I want to know how we can use yahoo pipes with google trends so as to make a Hot trends website. As you have mentioned that we csv, how can we use it along with google trends. This was one of the most helpful post which i have seen for Yahoo Pipes. Lookin

    • admin says:

      I’m going to get you what you want, but in a different way. I will post this weekend Sept 11th or 12th, how to create an automated site in the way that you want. I’ve tried to do this with Yahoo Pipes and have found the results to be unreliable, because Yahoo doesn’t want you to do this. I’m instead going to do this with the Python programming language. All code will be included and it should be very easy to copy. Hope that helps 🙂

  2. Mike Joes says:

    That will be great Mate, i have spend hours and hours fideling and tweaking yahoo pipes for getting the Hot trends and then further expanding them (which was the real challenge for me) and get the output. I m really looking forward to this. Thanks again a

  3. Jill says:

    Hi I found your blog searching on how to use Yahoo Pipes. Your post has helped me a lot.

    Now Im trying to make my own and I’m having a problem. I have been trying to figure it out on my own and by doing more searching and can’t find the information I need. I thought I would post here and maybe you could help me.
    I’m using the Fetch data with the path as item, My feed is an xml.

    It works great but when I look in the dubug I can see there is an image and the url. I have tried everything to get the image to show as well as the title.

    so I hooked the Sub Element and when I do it shows there is an Item.image. I selected that but the image does not show up.

    so I have fetch data with the path as item. Then that is hooked to the sub element with item.image selected, then its hooked to output. There is no image showing in the feed.

    Yahoo pipes is so advanced I know there has to be away. could you help me figure this out?

    Thank you Jill

    • admin says:

      Did you check if you eliminated the html code from the feed before you tried to access the image. This is a common problem. You want to make sure you take out the image before you scrub the feed to eliminate ads, or whatever. If the Pipe is public, send me the link and I’ll look into it closer. Thanks 🙂

      • Jill says:

        I came back to watch the video again and to see how to send it to posies. Now I need to figure out how to get these on my site.

        I was so excited to see you replyed back. I spent another 3 hrs trying to figure this out.

        I did publish this, I’m so new at this not sure if this link will work.
        http://pipes.yahoo.com/pipes/pipe.info?_id=5666ee751254685e7514495d7109b23d

        • Jill says:

          I have another one that I have been playing with, I tried all kind of different boxes with none working.

          I sent the clean one that works the other I did not publish it because it didn’t work.

        • admin says:

          Coupons.com is causing the problem because of all the crazy javascript associated with clicking on the product. I’ll have to check into this and get back to you. Just wanted to let you know I got your comment 🙂

          • Jill says:

            Cool Thanks!
            You see the image, it just seems it should be easy to pull it out. I’m new at this I tried loops and strings and nothing worked. I did learn more about pipes working with it and others.

  4. This is more better than the yahoo pipe video

  5. Jill says:

    I wanted to say this was my first stop that looked interesting, after all the searching I did on yahoo pipes. I agree with Static’s, this is an awesome tutorial!!

    Thanks for your tutorials!

  6. Jill says:

    I was wondering when adding yahoo pipes to my wordpress blog, would it effect my regular feedburner feed?

    Last night I added yahoo pipe feeds to my site for the first time. Today when I click my regular feedburner feeds the images are not there and its only displaying part of the feed. I have already been though all this about 2 weeks ago and had full feeds and images set to show both in wp-dashboard and at my feedburner account.

    Last night I made a regular post to my blog and the image and feed all went fine to facebook. Today I made 2 post and the images did not go to facebook. I then looked at my feedburner feed and there are no images.

    Just something else to figure out.

    • Jill says:

      Just wanted to say That I played with the feedburner account to try and fix my problem. I went to optimize, browserfriendly. Then I hit the save button a few times to try and get full feeds back, it did not work.

      Then I found this info
      FeedBurner will not process an original feed from your blog if it is greater than 512K in size. So I basically followed the steps. I do not think my feed was 512k. After following the steps it came up error.
      But then I had to reset the feed and soon as I did my feeds were back to normal.

      http://www.google.com/support/feedburner/bin/answer.py?answer=79626

      Needless to say problem is solved. I just wonder how it got that way in the first place.

      • admin says:

        The reason why your having problems with the coupons.com site is because it is structured so that when you click the link JavaScript is executed instead of sending you to a new page. To get this information on your site it’s probably better for you to follow their feed without pipes. Just have wpomatic post directly to your site. I think that will get you what you need. Eventually I’ll finish my automated website, but I have to fix all of the security holes

  7. Ahmad says:

    I want to know how can i email the results obtained from the yahoo search module in yahoo pipes to my email address also?
    Is this possible???

  8. Jonas says:

    Thank you for great tutorial. I will try it.

  9. Hil says:

    Thanks for the great information.
    I wanted to use yahoo pipes and get all posts in a feed that only contain images. Basically the output(final) feed should only contain posts which have images. Hence I wrote this yahoo pipe. http://pipes.yahoo.com/pipes/pipe.info?_id=2550eacace38c7377fe07dfda0e958c4 But somehow it does not work. Any ideas. Please help. Thanks

  10. Bill says:

    Hi, When using Pipes in conjunction with Google Blogger is there a way of limiting the number of items to display? I want to use descending order, but I’m concerned I will run out of space to display items. The solution would be to set the total number so that once that number is reached, the latest item would replace the oldest. Or perhaps there is another solution of which I am not aware. Thanks in advance for any advice you may offer.

  11. Tom says:

    Hey did you use Yahoo! Pipes to auto-generate this blog post, because most of the content is directly from the Pipes editor ;). That would be the biggest irony.

Trackbacks/Pingbacks

  1. RSS, Filtrage, Google Reader et Veille technologique « BenGeek blog - [...] Yahoo Pipes Tutorial Part I from NewThinkTank [...]

Leave a Reply

Your email address will not be published.

Google+