Yahoo Pipes Tutorial Part 2
In the previous Yahoo Pipes Tutorial, I briefly described what every module does. In this article, I demonstrate how to do many useful things with Yahoo Pipes.
Remember, the ultimate goal is to create great custom feeds that will automatically get posted on our websites. I showed you how to do this in a previous article How to Make Money Blogging. With those techniques, I was able to make an extra $11 bucks, after all fees, for each site I made. Each site took less than 20 minutes to put together and then required no maintanence. With Yahoo Pipes pulling even better articles, I’m pulling in $26 on average.
Pull Entire Article Using Yahoo Pipes
Normally, when you pull information from an RSS feed, you only get an abbreviated version of the article. Yahoo Pipes allow you to pull the entire article using the following steps. This is what the Yahoo Pipe page looks like after everything is set up. I’ll then describe the steps.
[adsense]
This Yahoo Pipe example, pulls every article from the current Huffington Post RSS feed, along with the whole article, if the title contains the word Hollywood. Here is how it’s built.
The Fetch Site Feed Module
- I drug this module from the left sidebar and onto the working area
- I plugged in the address, that I wanted it to pull feeds from
- It pulled all of the information from the main feed
- If I would have wanted all of the feed’s, I would have instead used the Feed Auto Discovery Module
Fetch the Whole Article for Each Feed Item
To grab the whole article, rather than just the abbreviated description in the feed, you need the Loop and Fetch Page Modules.
- Drag the Loop module, from the left side bar, under Operators
- Drag the Fetch Page Module, into the Loop Module
- Under URL in Fetch Page, select item.link
- item.link refers to the link for the entire article
- To get the whole article, I need the tags that surround the article
Finding the Tags that Surround an Article
- Go to the Huffington Post and click on any article
- Copy the first few words of the article
- View the articles page source
- Search for the article words you copied, until you find the beginning of the article
- Copy the tag that proceeds the article
- Paste it in the Cut Content From section in the Fetch Page Module
You can find the other tag, by looking at which tag follows the ending for an article. To finish off the Fetch Page Module:
- Select the radio button next to assign
- Select the all option
- Select item.description as the target
You could stop there, but I’ll explain how the Yahoo Pipes Filter Module works.
Yahoo Pipes Filter Module
With the Filter Module, you can either Block or Allow certain articles based of whether a certain word lies within some part of a feed item. Here I choose to only include articles that contained “Hollywood”, in the title. Here is how I did that.
- Pull the Filter Module out on the stage
- Choose the Permit option
- Choose the All option
- I then told the filter to only select items with Hollywood in the title
- item.title, refers to the Title of every article in the feed
I then connected all of my modules, as you see above. Then I clicked save and done.
Manipulating CSV Data with Yahoo Pipes
CSV stands for Comma Separated Values. Here I’ll show you how to pull data from a CSV, and manipulate it however you see fit.
I used the CSV file located at http://www.census.gov/popest/national/files/NST_EST2009_ALLDATA.csv
It contains population changes in each state from 2000 to 2009, among other things. It looks like this, in its raw form:
|
1 2 3 4 5 6 |
SUMLEV,REGION,DIVISION,STATE,NAME,CENSUS2000POP, ... <pre>010,0,0,00,United States,281421906, ... <pre>020,1,0,00,Northeast,53594378, ... Etc ... |
I will pull this information into Yahoo Pipes, and define that the first row should be used as column names. Here is what the Pipe will look like:
Fetch the CSV File with Yahoo Pipes
- Drag the Fetch CSV module on to the stage
- Paste in the URL for the CSV file
- Define what separates the data into columns (This will probably be a comma)
- Define what columns, or rows are used to name the data
- I then rename the data with the Rename Module
- I do this so it is easier to work with the data
Change the Value of the RSS Description with the Regex Module
I then use the Regex, or Regular Expression module, to change the value of the description for each item in the feed.
- I define that I want to manipulate the value of item.description
- (.+) is used to refer to any series of characters (Replace everything)
- I then create the text that will go in the new descriptions
- Pop in 2000: ${POPESTIMATE2000}, Pop in 2009: ${POPESTIMATE2009}, Net Pop change 08 to 09: ${popchange}
- The ${ } operator will pull the data from the CSV file based off of the name I use
Sorting the Data in the Feed
I then use the Sort Module to sort the order of the data in the new feed I created.
- Pull the Sort Module onto the stage
- Define that I want to sort based off of the value of popchange, which I defined in the Rename Module above
- Define that I want to sort in ascending order
After I do this, the final output for the feed looks like this:
Michigan
Pop in 2000: 9955308, Pop in 2009: 9969727, Net Pop change 08 to 09: -32759
Maine
Pop in 2000: 1277211, Pop in 2009: 1318301, Net Pop change 08 to 09: -1390
Rhode Island
Pop in 2000: 1050736, Pop in 2009: 1053209, Net Pop change 08 to 09: -293
Vermont
Pop in 2000: 609903, Pop in 2009: 621760, Net Pop change 08 to 09: 711
New Hampshire
Pop in 2000: 1240446, Pop in 2009: 1324575, Net Pop change 08 to 09: 2703
Etc …
Flickr and Yahoo Pipes
Here I created an RSS feed from images that I pulled from the Flickr website.
All I did was drag the Flickr Module on to the stage and asked it to return 50 photos of Pittsburgh, taken in Pittsburgh. It returned a map detailing where the photos were taken. It also returned a list of the photos and stuck them in a new RSS feed.
You could also use the Flickr module to populate your articles with interesting photos.
That’s All Folks
There are 3 interesting ways to use Yahoo Pipes. In the next article, I create a massive Yahoo Pipe that will use most of the modules. I’ll then describe how those modules work and you’ll be a Yahoo Pipe Master. Leave Questions Below
Here to Serve
Think Tank
Hey Derek – Thanks for the great tutorials on Yahoo Pipes. I have been studying them and then manipulating other peoples pipes to my own needs and saving them on my profile. One question I have:
I grabbed my pipe RSS feed url for a particular topic and slammed it into RSS Graffiti on a particular Facebook page. It ran the first day and now I receive the message “An error occurred while retrieving the feed. A feed could not be found at http://pipes.yahoo.com/pipes/pipe.run?_id=6338de6976fbe3687c30703e457c939b&_render=rss“. Am I doing something wrong? Should I be hooking up the pipe with the Notes page?
My ultimate goal was to grab a bunch of useful blogs related to a particular topic and have 1 or 2 post randomly each day. Any help will be appreciated.
Thanks!
Well the feed is up and running. I checked into a few things and found that you get this error because of DNS issues. This can happen for numerous reasons and it is normally a temporary problem that will go away on its own.
I do know personally that Yahoo gets really irritated if you retrieve data from the feed to often (Every 5 Minutes). I’d only retrieve from it twice per hour at most.
You also should check that the feed URL is listed perfectly in RSS Graffitti. I hope that helps? Here is a link that may help as well http://rssgraffiti.pbworks.com/Error:-Cannot-Resolve-Host
Excellent post! Grabbing the full article with Yahoo Pipes instead of just the summary is great for autoblogs. But use this method at your own risk because there are some obvious legal issues by scraping the content. You might tick off a few people! Great technique none the less though.
Thanks for pointing that out. I only grab the first 800 characters from sites. My goal being to provide readers with just enough information to find out if they want to continue reading. I also always place a link to the original article and leave a reference to the original poster in the title.
I only provide original content on newthinktank, but make good money off of my autoblogs. Thanks for stopping by 🙂
Is there any way to translate the full grabbed post from a language to another?
The translate-box works only for the title and the short summery description, but for the full description it don’t translate.
Any fixes on this one?
Thnx for the tutorial Derek. Really appreciate your work.
Have you seen Googles translator toolkit? It’s pretty neat 🙂
Great! Thanks Derek
You’re very welcome. I’m glad to help
Hallo Derek,
I’ve tried to accomplish the same results as it has come out for you using Y-pipes grabbing from “huffingtonpost.com”.
So when I made this firs pipe by drag and drop in the “fetch feed” and type in the site url and then drop in the loop en put the “fetch page” inside it and filled in the div information, it showed me an error as the result…
This was the error message:
===========================
Error fetching http://www.huffingtonpost.com: Invalid XML document: org.xml.sax.SAXParseException: The entity name must immediately follow the ‘&’ in the entity reference.
B.t.w. I have tried some other sites to grab from them also, but the error is showing all the time as the result.
I know that I filled in all the fields correctly, so I’m confused here because of it.
Please Derek, can you tell me how to get around this so i can get the full article anyway?
Thanks, Mike
I’ll look into it, but it seems that Yahoo has pretty much abandoned pipes. The results were very spotty so I just wrote my own website scrapers.