Yahoo Pipes Tutorial Part 3

Yahoo Pipes TutorialHere is an enormous Yahoo Pipe. You can see it in it’s full size by clicking here Giant Yahoo Pipe.

I’m going to use it to explain how to perform many different data manipulations using Yahoo Pipes. Just so you know, Yahoo Pipes is used to pull data from websites, manipulate that data, create an RSS feed with that data, and then send it where ever you would like. For more information on Yahoo Pipes see my tutorial Yahoo Pipes Tutorial.

Giant Yahoo Pipe

In this article I’ll show you:

  • How to retrieve data using the Yahoo Search Module in Yahoo Pipes
  • How to fetch an article from a foreign site and translate it with Yahoo Pipes
  • How to use the User Input Modules
  • How to filter your results
  • How to fetch the entire article that the feed refers to
  • How to make sure each article you pipe is unique
  • How to use the Truncate and Tail Modules
  • How to define how long your articles are
  • How to add a read more link to the end of your articles
  • How to join multiple feeds in Yahoo Pipes
  • How to filter out html code with Yahoo Pipes

And, a bunch more!

Remember you can open the full screen version of the Yahoo Pipe to follow along.

Using the Yahoo Search Module in Yahoo Pipes

Here I’m using the 2 User Input modules (URL Input & Text Input) to pass text to the Yahoo Search Module. I then am filtering those results, to only return articles that contain the word star. I pull the full article from the website and place that in the feed description. To finish it all off, I truncate the results to the top 5 results. This is what it looks like.

Yahoo Pipes Yahoo Search Module

Yahoo Search & User Input Modules

The Yahoo Search Module takes 2 inputs. The URL you want to restrict the search to. (This is optional) It also excepts the keywords you want to search for. It will then output the results, sorted in the order of relevance to the search term.

I’m allowing the Yahoo Pipe user to enter the chosen URL to search through, with the URL Input Module. You can define the order in which it is listed by typing a value into the Position field. I also chose the Huffington Post as the default URL to search.

Similarly I used the User Text Input Module to enter the chosen keyword, or I used the word “Fed” as a default.

Yahoo Pipe Filter Module

I then defined that I want to block any articles in which the description contains the word “star”. I could also only permit articles with the word “star”. Searches can be further refined by adding additional rules and demanding that descriptions contain all of the words I list. You can filter based off of whether the words lie in numerous locations such as the title (item.title), description (item.description), link to original article (, etc.

Yahoo Pipe Truncate Module

After all of the articles are gathered I then use the Truncate Module to eliminate all of the articles except for the top 5 relevant ones. You do this with the Truncate module.

Fetch a Feed & Fetch a Foreign Feed & Translate it

The next Yahoo Pipe modules will be used to:

  • Grab a feed from Reuters
  • Retrieve the original article, instead of the mini description found in the RSS feed
  • Use the Tail Module to eliminate all but the last 4 articles in the feed
  • Join all of the feeds into one with the Union Module
  • Retrieve a French website feed
  • Translate the feed from French to English
  • Remove all but the top 5 articles with the Truncate Module
  • Remove all of the HTML tags from every article, with the Regex (Regular Expression) Module

This is what it would look like:

Yahoo Pipes

What Am I Doing Here?

RSS Feeds are great, but not great for posting to your automated website. To get the whole article, instead of a mini description, just combine the Fetch Feed, Loop and Sub String Modules. Here is what to do:

  • Drag the Fetch Feed Module onto the screen, because I know the address for the feed.
  • You could use the Auto Discovery or Fetch Site Feed Module if you didn’t know the feed address
  • Enter the feed address
  • Send the results to the Loop Module, which performs the same action on every article that is sent to it
  • The action being performed is to grab the original article with the Fetch Page Module, just like before
  • I then use the Tail Module, to eliminate all but the last 4 articles returned
  • I then pull articles from the French website, with the Yahoo Search Module
  • That feed is then translated into English with the Translate Module
  • The Truncate Module is used again to return the top 5 relevant articles
  • All of the feeds are then joined into one, using the Union Module
  • The Regex Module is then used to strip all of the HTML tags from every article in our feed

Wrapping Up the Giant Yahoo Pipe

Yahoo Pipe Example

The Unique Module

After every feed is combined into one, I send them all to the Unique Module. This module will in this case check for duplicate titles. That is probably the only option, of those available, that will eliminate duplicate articles in the feed.

The Sort Module

I use the Sort Module, to sort all of the feeds based off of publication date, in ascending order. You can sort based off of other factors, but I believe this is the most common option chosen.

The Sub String Module

The Sub String Module, will trim the articles to a maximum length of 200 characters, in this example. You could choose to make the length of the description larger or smaller with it.

The String Builder Module

Here I’m creating a Read More link at the end of the article that will point back to the original article. This is considered good web etiquette and is required by most sites online. The String Builder Module allows you to build custom strings on the fly. To create the Read More tag I just enter:

  • The descriptions I built with this Yahoo Pipe followed by the link to the original article
  • … <a href=”
  • >
  • Read More
  • </a>

That’s All Folks

There is a giant Yahoo Pipe that demonstrates how to use most of the modules and how they are often used. If you have any questions leave them in the comment section below.

Till Next Time

Think Tank

31 Responses to “Yahoo Pipes Tutorial Part 3”

  1. Vipin J says:

    Sir, i found your tutorials very useful.
    I’m trying to use fetch page module to create custom RSS, but i have basic knowledge of HTML. I was able to split the page into no of contents, and then used the Rename module to copy the contents as link, title, description and pubDate as all these are available within the contents.

    I’m stuck after this point, how do i trim off the unwanted data from title, pubDate, link and Description. I know my solution is in regex module.., can you please tell me the symbols that i should use to search & replace the contents with respective item data (something like, i hope u understand what i mean)
    Thanks,,,eagerly waiting for reply

  2. Mike Joes says:


    Another Awesome tutorial from you 🙂

    Learned New things today.

  3. Mike Joes says:

    Hi Derek, will it be possible to use Yahoo! Pipes along with affiliates. Thus making a Pipe which show feeds for a particular Keyword product in such a way that the amazon affiliate code automatically gets added to each product thus Making a Au

    • admin says:

      There are a few things that will be hard to over come. Amazon doesn’t create an RSS feed for all of it’s pages. Also Yahoo Pipes has been really odd recently. It seems to block my feed on an off if it generates to much traffic. I’m about 80% done with a program built in Python that automates posting. You can see that in the Python tutorials here. When I get it completely done I’ll make all the code freely available to everyone. Check out the Python tutorial till I get to that. Thanks a bunch

  4. Deepak says:

    Sir, Thanks for your great tutorial. i have a problem. i want to know how to change/remove to title as this point to original feed source.
    I am waiting for your response.

    • admin says:

      I’m not sure I understand your question. refers to the link to the original article from the feed. You would use it to pull information from the original article and not just the description found in the feed. item.title is a reference to the title of the article located in the feed. Please provide more information on what you are trying to do and I’ll do my best to help you out. Thanks

  5. deepak says:

    I have created a yahoo pipe, and publish this with my favorite wp plugin(feedwordpress). When i see published post, “Post title” point to orginial source of website. i want to change this with my own title. how to do this??

    • admin says:

      Have you tried piping it through the Pipe Regex? This provides you with a great deal of editing capabilities? How do you want to change the titles? Do you want to append the same words every time? That would be easy

  6. Focuri says:


    I am trying to use a feed, take the first 400 words of the article and translate it.

    I have succeded to take the whole article using Loop+Fetch page but:

    1. I don’t know how to limit only the first 400 words of the article

    2. the translation doesn’t seem to work on the whole extracted article. It only work on title and on the excertp of the article send by thr rss feed

    Can you help?

  7. Leo says:

    Pals, thanks for amazing tutorial, I’m new about pipes. May you help me how to delete video from RSS feed? I have got the content but I don’t need the video.

    Thanks for your kind help and nice to meet you and your blog… 😉

  8. Carlo says:

    I am truly impressed by your work, wow!
    On my site I’m trying to insert a yahoo pipes to be able to take from this page (which has no feeds) and extract the contents of the table on the page (using the Fetch Page module).
    Then, then, given that the content in the table would not have the pubDate field and therefore I would not have the ability to sort contents by date, I could use your tutorial to extract the full content of each news, divide the part description from the pubdate and create the final with the sort to pubDate.
    What I ask you is: Could you help me build the first part of the pipe, the one that extracts the table from the page using Fetch Page?
    Sorry, but I just can not find solutions …

    Thanks anyway

    I hope that I explained

    • admin says:

      Hi Carlo

      I would show you how to use Yahoo Pipes if I thought it would work consistently for you. Yahoo seems to have abandoned Pipes and nothing seems to work consistently now. I have a ton of tutorials using PHP for website scraping that I assure you will work. Just look up web site scraping and regular expressions on my site. I have finished scripts that you can download.

  9. rdnp says:

    hai…ist possible to combine 2 article from 2 feed to be one article?

  10. kevin says:

    thank you for the tutorials

    hi i’m trying to remove all tags but I want the images to remain how do I do that

  11. Xen says:

    Hi its a great article..

    wanna asking.. how to merge 2 feed into 1, example: in 1st feed there’s a contain “iam” and then in the 2nd feed there’s contain “handsome” , how can u put it together into 1 feed with contain “iam handsome” ??

    any advice please mail me?

    Thanks, and sory my bad english 🙂

  12. Hi, I followed this below
    The descriptions I built with this Yahoo Pipe followed by the link to the original article

    Read More

    but i cant seem to have it working
    I can see the “Read More” text but it does have the link behind it…
    Hope you can help.

    Here is my pipe

    Thanks a lot..

  13. Dave says:

    Hi, I have created a very simple pipe to pull in a feed from twitter. When I run pipe it publishes the result as a URL and then again below it as text, so it’s duplicated. I only want it as a text so I can re tweet it. For an example go to Yahoo Pipes.

    Fetch feed

    to Pipe output and run pipe to see what I mean.

    Any help will be appreciated.

  14. Gianmarco says:

    I agree with remy etienne the link does not work, if your works give us the link of the pipe please. I’m trying to solve this problem for a long time.

  15. Anonymous says:

    Awesome tutorials!

Leave a Reply

Your email address will not be published.