Python 2.7 Tutorial Pt 19

Python 2.7 TutorialWell it has been a long trip, but I’m going to wrap up the Python Tutorial here. Before I stopped I wanted to answer one question posed to me which was to create a program that automated the process of getting demographics.

I didn’t have much time to perfect this and I plan to come back, but this program will grab tons of demographics for you including:

  • Median Age
  • Income
  • Population Information
  • Education Levels
  • Unemployment Data
  • Occupation Data

Just enter your city and state and out pops the information. Here is a sample of what you get if you type in Pittsburgh, Pennsylvania.

Estimated median household income in 2008: $36,709
Population in July 2009: 311,647. Population change since 2000: -6.8%
Median resident age:35.5 years
Pennsylvania median age:38.0 years

Population in July 2009: 311,647. Population change since 2000: -6.8%
Males: 148,220
Females: 163,427
Population density: 5607 people per square mile
High school or higher: 81.3
Bachelor’s degree or higher: 26.2
Graduate or professional degree: 12.5
Unemployed: 10.1
Mean travel time to work: 23.1 minutes

Most common occupations for males in 2008 (%):
Other management occupations except farmers and farm managers (6%)
Postsecondary teachers (5%)
Building and grounds cleaning and maintenance occupations (4%)
Other sales and related workers including supervisors (3%)
Cooks and food preparation workers (3%)
Computer specialists (3%)
Other office and administrative support workers including supervisors (3%)

Most common occupations for females in 2008 (%):
Secretaries and administrative assistants (7%)
Other office and administrative support workers including supervisors (6%)
Other management occupations except farmers and farm managers (5%)
Information and record clerks except customer service representatives (5%)
Registered nurses (4%)
Nursing, psychiatric, and home health aides (3%)
Other sales and related workers including supervisors (3%)

I hope it saves you some time.

Here is the Demographics Python Code

#! /usr/bin/python

from urllib import urlopen
from BeautifulSoup import BeautifulSoup
import re

def cleanHtmlRegex(i):
i = str(i)
regexPatClean = re.compile(r'<[^<]*?/?>’)
i = regexPatClean.sub(”, i)
i = re.sub(‘&\w+;’,”,i)
return i

def demogSearch(regexSearch,i):
demogResults = ”
demogResults =,str(i)).group().split(‘\n’)

return demogResults

# Searches multiple lines and then splits the data at the location of %
def multiLineRegex(regexSearch,i):
multiLine = ”
multiLine =,str(i)).group().split(‘%’)
return multiLine

# Input the city and state to create the web address to search
cityName = raw_input(“Enter the City Name? “).replace(‘ ‘,’-‘)
stateName = raw_input(“Enter the State Name? “).replace(‘ ‘,’-‘)
searchCity = cityName + ‘-‘ + stateName
cityWebsite = ‘’ + searchCity + ‘.html’

# Copy all of the content from the provided web page
webpage = urlopen(cityWebsite).read()

# Pass the article to the Beautiful Soup Module
soup = BeautifulSoup(webpage)

# Tell Beautiful Soup to locate all of the p tags and store them in a list
paragList = soup.findAll(‘td’)

# Set up all of the regular expressions
regexMale = re.compile(‘^Males.*$’)
regexFemale = re.compile(‘^Females.*$’)
regexAge = re.compile(‘(Median res.*)\n’)
regexIncome = re.compile(‘Estimated.*:\s\$\d*,\d*’)
regexEdu = re.compile(‘High school.*’)
regexIndust = re.compile(‘Most common occupations.*’,re.DOTALL)
regexDensity = re.compile(‘Population\s(in|de).*’)

# Print all of the paragraphs to screen
for i in paragList:
i = cleanHtmlRegex(i)

allDemogs = []


for m in allDemogs:
print m

12 Responses to “Python 2.7 Tutorial Pt 19”

  1. Jay says:

    Please continue your tutorial with more videos. They are very useful, yet to know many more things.

    • admin says:

      I plan on revisiting Python in the future. A Django tutorial is on the list

      • Charles says:

        These video tutorials are the best and most inspiring I have ever watched on Python. I used to be afraid of programming and now Python is my language of choice. My question is how can I start or design/develop a multifunction Linux [Debian] command line Tool using Python. {[something looking like the way Metasploit works]} Need guide lines or blue prints.

        • Derek Banas says:

          Thank you very much 🙂 I’m very happy that I could help. I however don’t know much about penetration testing. I have never been in a situation in which I could test security in that way. Sorry, but I’ll be the first to say when I don’t know something.

  2. Shri says:

    Pls continue your tutorial regarding how to use Gis application in conjunction with python.

    Thanks & Regards

  3. James says:

    Can you make a tutorial about threading?

  4. Raj says:

    your tutorials are very knowledgeable, can u please provide some tutorial about python-Scrapy project (in this tutorial there is only few video about that) like scrapping the web or developing the crawler..
    that would be very awesome..

    Or if you can do a web search application(crawler) based tuts..

    thanks 🙂 n waiting for you new videos.

Leave a Reply

Your email address will not be published.