This blog is defunct! New location: andrewsturges.com/blog/

Find new content (I hope) at andrewsturges.com. I won't be updating or monitoring this blog anymore.

Exploring the Ruby gem craig

I've decided to take down this post after receiving a cease and desist letter from Perkins Coie LLC, which represents craigslist.org. https://github.com/arsturges/craigslist

Exploring the Craigslist Ruby Gem

I've removed this post because I got a cease and desist letter from Perkins Coie LLC regarding a violation of the craigslist terms of use. https://github.com/arsturges/craigslist

Solving an Escape Riddle Using a Monte Carlo Simulation in Python

I recently heard this riddle from a friend at a campfire. I wasn't smart enough to come up with the answer, but when it was explained to me I wanted to know how long it would take the prisoners to be released. In this post I'll describe the riddle and the answer, and show how to use Python to set up an experiment, use a Monte Carlo simulator to run the experiment repeatedly, and briefly discuss some of the results.

The Riddle

There are 50 prisoners, and one prison keeper. The prisoners will be placed in separate cells, never able to communicate. Once per day, the prison keeper will randomly select one prisoner and take him into a room with two levers. The levers have two positions each: Up, and down. The prisoner may operate the levers however he wishes, then he must return to his cell. (The levers don't do anything; they just represent a binary state.) The prisoners have one chance to indicate to the prison keeper whether all the prisoners have been in the lever-room. If they so indicate too early, they will never be released. If they're correct (even if they're late), they will be released immediately. The prisoners get one chance to meet prior to imprisonment, to discuss strategy. How can they ensure their release?

The Answer

Only one lever is needed. Elect one prisoner to be a 'counter.' When a prisoner enters the lever room and the lever is in the 'down' position and he has never moved the lever, then and only then will he move the lever to the 'up' position. Only the 'counter' prisoner may return the lever to the 'down' position, and each time he does so he increments a counter. When the counter reaches 50, he knows that all prisoners have been into the lever room at least once. He can then inform the prison guard, and all the prisoners will be released.

A Solution in Python

I came up with the following model in Python to describe the prison scenario. There are probably better and more optimized way to represent this situation, but here's the first one I came up with.

#Iteration 1: The basic scenario
import random
day_number = 0 
prisoner_designee = 1 # chosen by prisoners to be the official counter
counter = 0 # as tracked by the prisoner_designee
lever_position = 'down' # starting position of the lever
prisoners_who_have_operated_lever = []

while counter < 50: 
  prisoner_number = random.randint(1,50)    # Select a prisoner
  if not prisoner_number in prisoners_who_have_operated_lever:
    if lever_position == 'down':
      prisoners_who_have_operated_lever.append(prisoner_number)
      lever_position = 'up'
  if prisoner_number == prisoner_designee:  # If it's the designee...
    if lever_position == 'up':              # ...and the lever is up...
        lever_position = 'down'             # ...put the lever back down...
        counter += 1                        # ...and increment the counter.
  day_number += 1

print day_number

The above code, iteration 1, uses random.randint to generate a random integer. I used if statements to describe the logic, and printed out the result. Example:

$ python prisoner1.py 
2536

This means that for this particular run, it would have taken the prisoners (or rather, the prisoners' designated counter prisoner_designee) 2,536 days to count 50 'up' levers, proving that all prisoners had entered the lever room and ensuring their release. That's just under seven years(!).

But this result depends on the order in which the prison keeper (in our case represented by random.randint) chose the prisoners. Running this code at the command line produces different results each time, usually in the range of 2500--3000 (days). To get a better picture of how long it would take a new group of prisoners to earn their release we'd have to simulate the whole range of possible outputs (or which there are nearly infinitely many) But I don't want to run this manually dozens of times and tabulate the results by hand; I want to make the code run the simulation thousands of times, with random inputs each time, and then aggregate the results. That's the Monte Carlo method.

Implementing the Monte Carlo Method

We're going to use a loop to run the experiment, so first, wrap the experiment in a function:

#Iteration 2: Wrapping the simulation in a function
import random

def run_simulation():
  day_number = 0 
  prisoner_designee = 1 # chosen by prisoners to be the official counter
  counter = 0 # as tracked by the prisoner_designee
  lever_position = 'down' # starting position of the lever
  prisoners_who_have_operated_lever = []

  while counter < 50: 
    prisoner_number = random.randint(1,50)    # Select a prisoner
    if not prisoner_number in prisoners_who_have_operated_lever:
      if lever_position == 'down':
        prisoners_who_have_operated_lever.append(prisoner_number)
        lever_position = 'up'
    if prisoner_number == prisoner_designee:  # If it's the designee...
      if lever_position == 'up':              # ...and the lever is up...
          lever_position = 'down'             # ...put the lever back down...
          counter += 1                        # ...and increment the counter.
    day_number += 1
  return day_number

print run_simulation()

Running this code should produce the same type of output (with a different result, of course):

$ python prisoner2.py 
2950

The next step is to write a loop that will call this function a thousand times:

#Iteration 3: Loop it to run 1000 times
import random

def run_simulation():
  counter = 0 
  day_number = 0 
  prisoner_chief = 1 
  lever_position = 'down'
  prisoners_who_have_operated_lever = []

  while counter < 50: 
    prisoner_number = random.randint(1,50) # Select a prisoner
    if not prisoner_number in prisoners_who_have_operated_lever:
      if lever_position == 'down':
        prisoners_who_have_operated_lever.append(prisoner_number)
        lever_position = 'up'
    if prisoner_number == prisoner_chief:
      if lever_position == 'up':
          lever_position = 'down'
          counter += 1
    day_number += 1
  return day_number 

simulation_results = []
simulations = 0 
while simulations < 1000:
  simulation_results.append(run_simulation())
  simulations += 1
print simulation_results

In this iteration we use a while loop to run the simulation 1000 times, storing the results in the list called simulation_results. Here's an example of the output:

$ python prisoner3.py 
[2444, 2119, 2818, 2253, 2586, 2543, 2490, 3388, 2034, 2739, 2554, 2585, 2498, 2689, 3180, 2760, 3145, 2698, 2196, 2769, 2400, 2783, 3091, 2258, 2952, 1730, 2974, 2656, 3059, 2000, 3222, 2186, 3114, 2618, .... ]

(List truncated.)
The last step in the Monte Carlo method is to aggregate and interpret the results. In this case, that means taking the average of all the simulation runs. A simple way to do this with lists is to sum the list :

average = sum(list)/len(list)

A more elegant way is to use Numpy's built-in mean function, which requires first converting the list to a Numpy array:

import numpy

array = numpy.array(list)

average = numpy.mean(array)

The next iteration of the code will include this small refactorization. Running this code produces the following result:

$ python prisoner3.py 
2720.895

Now, instead of one experiment, this result represents the average of 1000 experiments.

So what?

So on average, it would take 50 prisoners in this scenario 2721 days to earn their release. Cool. Neat. But this raises so many more questions: Is this number related to the probability of )1/50)*(1/50)? Where is this average value in the range of values yielded from the 1000 results? If we halve the number of prisoners, will the number of days go down by half? What does a histogram of the results look like? The answers, as well as a more in-depth discussion of other statistics, will be in the next post.

Installing Twitter Bootstrap in Flask 0.9

I have a small web application called 'dispatchtool' written in the Python microframework Flask. I want to use Twitter's Bootstrap CSS framework. In this post I will explain how to do this.

I am running Ubuntu 12.04, using Python 2.7, and Flask 0.9. This tutorial assumes you've got a basic Flask application up and running, and that you can view the development version on your local machine. It also assumes that you're using templates. (This is important!)

Step 1: Get a basic CSS file working.

Before dealing with Bootstrap, we want to ensure sure that the application can correctly find and use a simple test CSS file.
To do this, first, create a new directory in your Flask application's root directory called 'static'. My app is called 'dispatchtool', so I create 'static' inside the 'dispatchtool' directory:

$ cd dispatchtool

$ mkdir static 

Next, in that new directory, create the test css file. I'll call mine 'test.css'. Add some simple style that will be loud and obvious. It's only purpose will be to ensure that we can get the application to see this file and use it. Here's my 'test.css':

h1 {color: red}

p {color: green}

Of course, make sure that your app includes an <h1> tag and a <p> tag somewhere, or else these styles won't be applied, even if the stylesheet is properly set up.

Now that we've got our test css file in the right place, we need to tell Flask how to find it. CSS files are always linked to HTML pages in the <head> section of each page using the <link … /> tag. In my application, I'll add the following tag:

<link type="text/css" rel="stylesheet" href='/static/test.css' />

Here's the top of my page, for context:

<!DOCTYPE html>

<html lang="en">
  <head>
    <title>The Demand Response Dispatch Tool</title>
    <link type="text/css" rel="stylesheet" href='static/test.css' />
  </head>
  <body

Now open up your page in a browser, and see if the styles were applied. In my case, it suddenly looks like Christmas. If nothing changed, look at the source (ctrl-U) and click on the link you created to the stylesheet. That will show where Flask thinks the CSS file is.

Even though that's working, we're not quite done yet. Web application frameworks like Flask have a better way of creating links to static things like CSS files. In Flask, it's a method called url_for(). Methods like this make your application more flexible, allowing you to move around pages without breaking links to your CSS files. So now, replace the value of the 'href' attribute in the <link> tag with the following line:

{{ url_for('static',filename='test.css') }}

The double braces tell Flask a) that the contents are to parsed as Python code and b) to render the result directly into the HTML output. Now the whole link tag should look like this:

<link type="text/css" rel="stylesheet" href="{{ url_for('static', filename='test.css') }}" />

Now save the file and refresh your browser page. If that still renders your page using your test CSS file, then we're most of the way done. If it doesn't, check the page source, and ensure that the <link> tag is pointing to the right place.

Step 2: Download and "install" Twitter Bootstrap

The next step is simple and short. We need to get the Bootstrap files (download) and put them in the right place (install). Normal people will navigate to the Bootstrap page, click the link, and unzip the files, then put those files (as-is, with no changes to the directory structure) in the 'static' folder that we created earlier. Super elite hackers will do it the following way:

$ cd ../static/

$ wget http://twitter.github.com/bootstrap/assets/bootstrap.zip

$ unzip bootstrap.zip

$ rm -r bootstrap.zip 

Whichever way you do it, you should now have a folder called 'bootstrap' inside your 'static' folder. That's it. You have downloaded and 'installed' Bootstrap.

Step 3: Link to the boostrap.css stylesheet

The main Bootstrap stylesheet is in bootstrap/css/, and it's called 'bootstrap.css'. We want to change our link from 'test.css' to this file, so update your <link> tag as follows:

<link type="text/css" rel="stylesheet" href="{{ url_for('static', filename='bootstrap/css/bootstrap.css') }}" />

That's it. If you reload your page, you should see at least some changes reflecting Twitter's style. (When you go to production, you should switch from the general 'bootstrap.css' to the minified 'bootstrap.min.css' to save bandwidth.)

Step 4: Update your HTML to hook into the Bootstrap styles

Now that we've got our Flask app properly linking to the Boostrap css file, all that remains is to use it. That means we have to look at the boostrap.css file, and see what classes and ids we need to assign to our HTML elements. I'll go through two quick examples.

Container

The most important is probably to use Bootstrap's "container" class. Simply add the class 'container' the <body> tag:

<body class="container">

When you refresh the page, you should see your content contained in a 940 pixel-wide column.

The Hero Unit

The most famous Bootstrap style is probably the "hero unit". To see it, just add the class 'hero-unit' to any block-level element:

<div class="hero-unit"><p>Don't abuse this style!</p></div>

Conclusion

We walked through how to set up general CSS file in a Flask app, then how to acquire and link Bootstrap to the same Flask app. The next step is to check out the Bootstrap Documentation, and start choosing the styles you want to use. Thanks for reading!

Basic Unit Testing Tutorial in Python 3.2

I have a Python 3.2 program that's in a working state, but I never wrote any tests for it. Now, I want to go back and provide some code coverage so that as I go back to refactor and add functionality, I'll be more certain I'm not breaking anything. This is a fairly common case--you have existing code, and you want to write some tests for it.

Environment

I'm running Python 3.2 in a Windows 7 environment, using the IDLE Python code editor. I typically run code directly from IDLE by using the F5 key. Because of how IDLE invokes the Python interpreter, some of the code examples below will run differently depending on whether code is invoked from IDLE or from the command line. To enable Python from the Windows command line, it must be added to the PATH. To do this, open a command line window (Start -> Accessories -> Command Line) and type path <path to your python installation>. For example, since my python 3.2 installation can be found at C:\Python32, I issue the following path command:

>path C:\Python32

This adds the Python interpreter to the path so it can be invoked from within your Python project directory. Note that it only changes the path for this one command line session, so if you close and re-open the command line window you'll have to re-issue the path command.

Python's UnitTest Module

The first issue is what testing framework to use. I'm going to use Python's built-in module called UnitTest. It's a one-stop shop for all the testing most pure-Python programs will need. More advanced testing frameworks are available. The two most popular seem to be py.test and nose. I want to explore testing in a more 'raw' state, so I'll save these frameworks until I'm comfortable using unittest on its own.

Writing Testable Code

It's easiest to write a test for a small, discreet bit of code that accomplishes a well-defined task. Python best practices dictate that code be broken into small functions that perform one task at one level of abstraction. If you follow this advice and write your code using many small functions that each perform one well-defined task, then those functions will be easy to test. If instead you have written large functions that combine many steps, then first consider breaking these functions down into smaller ones.

Pick a small function to test

Start small by picking a short, well-understood function that does something simple. I'll use a function that takes one argument, uses an if statement to choose a "rate", and returns that rate:

def choose_renovation_rate(self, years_since_last_renovation):
    if years_since_last_renovation < 7:
        rate = 0
    elif years_since_last_renovation < 15:
        rate = 0.01
    elif years_since_last_renovation < 25:
        rate = 0.05
    elif years_since_last_renovation < 50:
        rate = 0.07
    else:
        rate = 0.1
    return rate

Create a new file to hold the test code

To get started with the testing itself, create a new file, and for now make sure it's in the same directory as the code you wish to test. I called mine unit_tests.py. Import the testunit module, and start a new class that describes what you will be testing. This new class will be a sub-class of unittest.TestCase. This class will hold several small test cases.

import unittest

class TestRateFunctions(unittest.TestCase):
    pass
    # test cases go here

Write a test function title and description

Next, inside the TestRateFunctions class I just defined, I will define a function and describe in words what it will test. The function names, by convention and to allow the testing machinery to run properly, should start with test_. For the moment, I will use a single assertTrue statement and pass it the condition True, just to see if everything is working.

import unittest

class TestRateFunctions(unittest.TestCase):
    def test_renovation_chooser_should_return_correct_rate(self):
        # This function should return the correct renovation rate
        self.assertTrue(True)

This is now a trivial, but functioning test.

Run the test

To actually run the test, we need to add one line to the end of the file: unittest.main(). With this line in place, we can simply call this test one of two ways.

Via the command line

After issuing the PATH commands as discussed under "Environment" above, in the command line terminal navigate to the directory where you created the file and call it:

C:\lighting_floor_space_stock_model>python unit_tests.py
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

The output consists of

1. A period. This represents one test, which in this case was test_renovation_chooser. With more tests written, these dots acts as a progress indicator for the test suite.
2. A long line of hyphens. This is just a visual separator representing the end of the execution of tests.
3. A report, which in this case Ran 1 test in 0.000s.
4. The phrase OK. We only get this if all tests pass.

Via IDLE

In the IDLE code editor, you can just press F5 to run the code directly. This produces the following output:

.
----------------------------------------------------------------------
Ran 1 test in 0.004s

OK
Traceback (most recent call last):
  File "C:\lighting_floor_space_stock_model\unit_tests.py", line 8, in <module>
    unittest.main()
  File "C:\Python32\lib\unittest\main.py", line 124, in __init__
    self.runTests()
  File "C:\Python32\lib\unittest\main.py", line 272, in runTests
    sys.exit(not self.result.wasSuccessful())
SystemExit: False

As you can see, when we ran the test suite from inside IDLE, we got the same successful output telling us the test passed (OK!), but it's followed by an ugly error and a traceback. The error is because unittest wants to "exit" but it can't, because the default IDLE behavior is to keep running. There are two simple ways around this error:

1. One solution is to just catch the error and pass around it. Simply change the last line to following:
   

   try: unittest.main()
   except SystemExit: pass

2. Alternatively, we could keep the first line as-is (without adding the try/except statements or the second line) and just add an argument to the original line, as follows:
   

   unittest.main(exit = False)

With either of those two modifications in place, the test suite should run properly from within IDLE.

Now Test Your Function

Now that we're sure we've got all the machinery operating correctly (import unittest, create a subclass, define a test case starting with test_, make sure it runs properly), we have to write code that actually tests our function.

In order to call the function in question (choose_renovation_rate) inside the testing function (test_renovation_chooser_should_return_correct_rate), we need to first make sure its parent class is available. In this case, choose_renovation_rate is a function inside the class FloorSpace(), so first I need to import the FloorSpare class with from floor_space import *. Then I can create a new variable called self.rate that holds the result of the FloorSpace.choose_renovation_rate function. The whole file now appears as follows, using assertTrue:

import unittest
from floor_space import FloorSpace

class TestRateFunctions(unittest.TestCase):
    def test_renovation_chooser_should_return_correct_rate(self):
        # A 2-year old building has a 0% chance of renovation:
        self.rate = FloorSpace.choose_renovation_rate(self, 2)
        self.assertTrue(self.rate == 0)

unittest.main(exit = False)

A Quick Refactor

Let's make one short change. Since we're testing for equality (using the == operator), we could use the assertEqual function instead of assertTrue. Furthermore, we could a string argument to the end of the function that will get printed if the test fails, allowing us to pass more detailed and useful information to the user. Change the last line of the test function as follows:

self.assertEqual(self.rate, 0, "2-year-old building should have a 0% renovation rate.")

This is such a simple test method that the message is nearly redunant to just reading the code, but it serves to illustrate the method of adding an information string to the assert method.

Refining the Test

Consider again the function we're testing:

def choose_renovation_rate(self, years_since_last_renovation):
    if years_since_last_renovation < 7:
        rate = 0
    elif years_since_last_renovation < 15:
        rate = 0.01
    elif years_since_last_renovation < 25:
        rate = 0.05
    elif years_since_last_renovation < 50:
        rate = 0.07
    else:
        rate = 0.1
    return rate

We pass it the number of years since it was last renovated (in years), and it returns a renovation rate. That renovation rate is used by other functions to perform various other computations. We could test each of the if conditions, but those numbers might change if we later tune the function to match real-world data. It might be more useful to test, for example, that the function always returns a percentage (i.e. a float between zero and one). So let's test a few of the if conditions just for completeness, and add a new test function to check whether it always returns a percentage between 0 and 1. Note that instead of testing each year we put the assert methods inside a for loop.

import unittest
from floor_space import FloorSpace

class TestRateFunctions(unittest.TestCase):
    def test_renovation_chooser_should_return_correct_rate(self):
        # A 2-year old building has a 0% chance of renovation:
        self.rate = FloorSpace.choose_renovation_rate(self, 2)
        self.assertEqual(self.rate, 0, "2-year-old building should have a 0% renovation rate.")

        # A 13-year old building has a 1% chance of renovation:
        self.rate = FloorSpace.choose_renovation_rate(self, 13)
        self.assertEqual(self.rate, 0.01, "13-year-old building should have a 1% renovation rate.")

        # A 55-year old building has a 10% chance of renovation:
        self.rate = FloorSpace.choose_renovation_rate(self, 55)
        self.assertEqual(self.rate, 0.1, "55-year-old building should have a 10% renovation rate.")

    def test_renovation_chooser_should_return_percentage(self):
        # The function should always return a rate such that 0 <= rate <= 1
        for years_since_renovation in range(100):
            self.rate = FloorSpace.choose_renovation_rate(self, years_since_renovation)
            self.assertGreaterEqual(self.rate,0, "renovation rate should be >= 0")
            self.assertLessEqual(self.rate,1, "renovation rate should be <= 1")

unittest.main(exit = False)

For a list of assert methods made available to unittest, see the documentation.

Conclusion

The basic steps to writing unit tests in Python are:

1. Write testable code by keeping functions short; each function should perform one task
2. Create a new file for the test code (e.g. unit_tests.py)
3. Import unittest
4. Subclass unittest.TestCase (e.g. class TestRateFunctions(unittest.TestCase))
5. Create new test functions with description names starting with test_ (e.g.
   def test_renovation_chooser_should_return_percentage)
   
6. Make your "work" functions available to the "test" functions by importing where necessary (e.g. from floor_space import FloorSpace)
7. Add string arguments to pass relevant information to the user in the case of a failed test.
8. Refine and refactor for readability and usefulness.

Finally, don't forget to actually run the tests frequently during development!

Basic linear algebra with Numpy on Python 3.2

I'm taking Stanford's online course on machine learning through Coursera. The second week has a good overview of linear algebra and matrix operations. The instructor has provided a useful PowerPoint deck in which he explains the basics. I'm going to go through this pdf and implement the linear algebra using NumPy. I have NumPy 1.6 installed with Python 3.2.
Ml SlideMachine Learning: Linear Algebra Reviews Lecture3
First, open an interactive Python shell and load NumPy:

>>> from numpy import *

Create a Matrix

Let's try just creating the 4x2 matrix he shows in slides 2 and 3.The basic form for creating arrays is to use the array method with parenthesis:

a = array()

Inside the parenthesis, nest some square brackets, and in those brackets just put comma-separated lists of elements in more square brackets. Each square bracket represents a row. Make sure they all have the same number of columns. So to create the 4x2 matrix from slides 2 and 3 use the following

>>> a = array([[1402,191],[1371,821],[949,1437]])

>>> print(a)
[[1402  191]
 [1371  821]
 [ 949 1437]] 

Matrix Elements

Next he talks about accessing the matrix element-wise, so a11 should access '1402.'

>>> a[1,1]
821

Instead of returning the first row of the first column, it gave us the second row of the second column. This is because NumPy is using 0-indexing (start at 0) instead of 1-indexing (start at 1). So to get the first row of the first column we index from 0:

>>> a[0,0]
1402

Matrix Addition

Next let's create two 3x2 matrices and add them together. First, instantiate the matrices:

>>> b = array([[1,0],[2,5],[3,1]])
>>> c = array([[4,0.5],[2,5],[0,1]])
>>> print(b)
[[1 0]
 [2 5]
 [3 1]]
>>> print(c)
[[ 4.   0.5]
 [ 2.   5. ]
 [ 0.   1. ]]

Add b and c and see what happens:

>>> b+c
array([[  5. ,   0.5],
       [  4. ,  10. ],
       [  3. ,   2. ]])

As you can see, NumPy correctly performed an element-wise addition.

Scalar Multiplication

Next, multiply a scalar by a 3x2 matrix. We'll use matrix 'b' from above:

>>> 3 * b
array([[ 3,  0],
       [ 6, 15],
       [ 9,  3]], dtype=int32)

Again, NumPy correctly multiplied each element of the matrix by 3. Note that this can be done in any order (i.e. scalar * matrix = matrix * scalar).
Division is just multiplication by a fraction:

>>> d / 4
array([[ 1.  ,  0.  ],
       [ 1.5 ,  0.75]])

Combination of Operands

Order of operations is important. In this slide the instructor sets up three vectors (3x1) and provides an example in which he multiples by a scalar then adds then subtracts then divides. Let NumPy do it and see what happens:

>>> e = array([[1],[4],[2]])
>>> f = array([[0],[0],[5]])
>>> g = array([[3],[0],[2]])
>>> 3 * e + f - g / 3
array([[  2.        ],
       [ 12.        ],
       [ 10.33333333]])

As before, NumPy produces the same answer as the instructor found by doing it by hand.

Matrix-vector multiplication

We can multiply a matrix by a vector as long as the number of columns of the matrix is the same as the number of rows of the vector. In other words, the matrix must be as wide as the vector is long.

>>> h = array([[1,3],[4,0],[2,1]]) # 3x2
>>> i = array([[1],[5]]) # 2x1
>>> h * i
Traceback (most recent call last):
  File "<pyshell#54>", line 1, in <module>
    h * i
ValueError: operands could not be broadcast together with shapes (3,2) (2,1)

My multiplication operation didn't work, and it helpfully gave me the shapes of the arrays for which multiplication failed. This is because the multiplication operator '*' causes element-wise multiplication. For that to work, the matrices need to be of the same shape (hence the error message shosed us the shapes were different). What we want is the dot product.

The Dot Product

To multiply two matrices using the dot product, use the dot() method:

>>> h = array([[1,3],[4,0],[2,1]]) # 3x2
>>> i = array([[1],[5]]) # 2x1

>>> dot(h,i)
array([[16],
       [ 4],
       [ 7]])

That gives us the correct answer, according to the slides. A longer example:

>>> j = array([[1,2,1,5],[0,3,0,4],[-1,-2,0,0]])
>>> k = array([[1],[3],[2],[1]])
>>> dot(j,k)
array([[14],
       [13],
       [-7]])

This one checks out with the slides as well.

Matrix-Matrix Multiplication

As far as NumPy is concerned, matrix-matrix multiplication is just like matrix-vector multiplication.

>>> l = array([[1,3,2],[4,0,1]])
>>> m = array([[1,3],[0,1],[5,2]])
>>> dot(l,m)
array([[11, 10],
       [ 9, 14]])

He goes on to give one more example with a pair of 2x2 matrices:

>>> n = array([[1,3],[2,5]]) # 2x2
>>> o = array([[0,1],[3,2]]) # 2x2
>>> dot(n,o)
array([[ 9,  7],
       [15, 12]])

Next he provides some context by using the example of home prices. He sets up a 4x2 and 2x3 matrix and multiplies them to quickly come up with price predictions:

>>> p = array([[1,2104],[1,1416],[1,1534],[1,852]]) # 4x2
>>> q = array([[-40,200,-150],[0.25,0.1,0.4]]) # 2x3
>>> dot(p,q)
array([[ 486. ,  410.4,  691.6],
       [ 314. ,  341.6,  416.4],
       [ 343.5,  353.4,  463.6],
       [ 173. ,  285.2,  190.8]])

Matrix Multiplication Properties

Show that matrix multiplication is not commutative:

>>> A = array([[1,1],[0,0]]) # 2x2
>>> B = array([[0,0],[2,0]]) # 2x2
>>> dot(A,B)
array([[2, 0],
       [0, 0]])
>>> dot(B,A)
array([[0, 0],
       [2, 2]])

To test this another way I asserted equality between the two operations and found a neat element-wise comparison:

>>> dot(A,B) == dot(B,A)
array([[False,  True],
       [False, False]], dtype=bool)

To show the associative property of arrays, create another 2x2 array C and multiply them in different groupings, but in the same order, to show that the result is always the same:

>>> C = array([[1,3],[0,2]]) # 2x2
>>> A = array([[1,1],[0,0]]) # 2x2
>>> B = array([[0,0],[2,0]]) # 2x2
>>> C = array([[1,3],[0,2]]) # 2x2
>>> dot(A,dot(B,C))
array([[2, 6],
       [0, 0]])
>>> dot(dot(A,B),C)
array([[2, 6],
       [0, 0]])

Identity Matrix

NumPy comes with a built-in function for producing an identity matrix. Just pass it the dimension (numnber of rows or columns) as the argument. Optionally tell it to output elements as integers in order to clean up the output:

>>> identity(3)
array([[ 1.,  0.,  0.],
       [ 0.,  1.,  0.],
       [ 0.,  0.,  1.]])
>>> identity(3, dtype=int)
array([[1, 0, 0],
       [0, 1, 0],
       [0, 0, 1]])

Show that for any matrix A, AI=IA=A. We can use the same A from before:

>>> A = array([[4,2,1],[4,8,3],[1,1,0]]) # 3x3
>>> I = identity(3, dtype=int)
>>> dot(A,I)
array([[4, 2, 1],
       [4, 8, 3],
       [1, 1, 0]])
>>> A
array([[4, 2, 1],
       [4, 8, 3],
       [1, 1, 0]])
>>> dot(A,I)==dot(I,A)
array([[ True,  True,  True],
       [ True,  True,  True],
       [ True,  True,  True]], dtype=bool)
>>> dot(A,I) == A
array([[ True,  True,  True],
       [ True,  True,  True],
       [ True,  True,  True]], dtype=bool)

Inverse and Transpose

Show that if A is an mxm matrix, and if it has an inverse, then A(A^-1) = (A^-1)A = I. To do this, we can use the same 3x3 matrix A from above:

>>> A = array([[4,2,1],[4,8,3],[1,1,0]]) # 3x3

>>> inv(A)
Traceback (most recent call last):
  File "<pyshell#112>", line 1, in <module>
    inv(A)
NameError: name 'inv' is not defined

We got this error because though we loaded NumPy, we need to also load the special linear algebra library:

>>> from numpy.linalg import *

Then we can try the inversion again:

>>> inv(A)
array([[ 0.3, -0.1,  0.2],
       [-0.3,  0.1,  0.8],
       [ 0.4,  0.2, -2.4]])

Now show that that A(A^-1) = (A^-1)A = I:

>>> dot(A, inv(A))
array([[  1.00000000e+00,  -2.77555756e-17,   0.00000000e+00],
       [ -2.22044605e-16,   1.00000000e+00,   0.00000000e+00],
       [ -5.55111512e-17,  -1.38777878e-17,   1.00000000e+00]])
>>> dot(inv(A), A)
array([[  1.00000000e+00,   0.00000000e+00,  -5.55111512e-17],
       [ -2.22044605e-16,   1.00000000e+00,  -5.55111512e-17],
       [  0.00000000e+00,   0.00000000e+00,   1.00000000e+00]])

Note that this was meant to return the identity matrix, but that the float operations returned not zeros but numbers very close to zero. Note also that a matrix of all zeros has no inverse:

>>> C = array([[0,0],[0,0]])
>>> C
array([[0, 0],
       [0, 0]])
>>> inv(C)
Traceback (most recent call last):
  File "<pyshell#121>", line 1, in <module>
    inv(C)
  File "C:\Python32\lib\site-packages\numpy\linalg\linalg.py", line 445, in inv
    return wrap(solve(a, identity(a.shape[0], dtype=a.dtype)))
  File "C:\Python32\lib\site-packages\numpy\linalg\linalg.py", line 328, in solve
    raise LinAlgError('Singular matrix')
numpy.linalg.linalg.LinAlgError: Singular matrix

The error message agrees with the machine learning instructor, who calls these special matrices "singular" or "degenerate."

Matrix Transpose

Lastly, show some matrix transposition, whereby the rows are flipped element-wise:

>>> A = array([[1,2,0],[3,5,9]])
>>> A.transpose()
array([[1, 3],
       [2, 5],
       [0, 9]])

If we name the transposed array 'B', then we can show that the ijth element of A is the jith element of B:

>>> B = A.transpose()
>>> A[0,2] == B[2,0]
True

>>> A[1,2] == B[2,1]
True 

Conclusion

We used NumPy and NumPy's library linalg to go through the linear algebra review slides from Coursera's Machine Learning course.

Installing NumPy for Python 3 in Windows 7

It's time to do some scientific computing, which, in the Python world, means using NumPy. I'm in a Windows 7 (64-bit) environment running Python 3.2.3 (64-bit).

Getting NumPy installed for Python 2 or Python 3 in Ubuntu was easy. Getting it to work in Windows turned out to be more tricky.

The Short Takeaway

• In a Windows 7 environment (even a 64-bit Windows 7 environment), you must install the 32-bit version of Python 3. The 64-bit version will not work with NumPy 1.6. 
• Furthermore, the 32-bit version of Python 3 must be installed 'just for me', and not 'for everyone on this computer'. 
• Finally, make sure you select the proper NumPy version (for Python 3.2), not the default version from SourceForge (which is for Python 2.6).

In this post I'm assuming you have already installed Python 3 and that you're running Windows 7. Specifically, I'm running Windows 7 Professional, 64-bit, Service Pack 1. What follows is the whole story of the troubleshooting, in case it helps out anyone else having the same issues.

Step 1: Ensure that NumPy Isn't Already Installed

Open a Python prompt and ensure that you don't already have NumPy installed:

>>> from numpy import *
Traceback (most recent call last):
  File "<pyshell#2>", line 1, in <module>
    from numpy import *
ImportError: No module named numpy

Step 2: Download NumPy from SourceForge

The SourceForge homepage for NumPy can be found at http://sourceforge.net/projects/numpy/. From there find the latest version of NumPy for Windows, and download it to your default download location. For me as write this, that means downloading NumPy 1.6.1. Note that the default SourceForge download link pointed to a version of NumPy compiled for Python 2.6. I had to navigate a bit deeper to find a version for Python 3.

Step 3: Open the installation file

Double-click the file from wherever you downloaded it to to start the installation wizard. It's trustworthy open-source software, so it's safe to click through all the prompts and allow it to be installed.


This leads us to a crucial error:

It reads: Python version 3.2 required, which was not found in the registry. Yikes. I definitely have Python 3.2, but it's telling me it looked in the Windows registry and couldn't find it. I heard somewhere that installing Python for 'just this user (me)' instead of 'for all users of this computer' is one way to get around this.

Step 4: Reinstall Python 'just for me'

So to comply, I uninstall Python, and reinstall it, being careful to install it 'just for me' as shown below.

Step 5: Try installing NumPy again

With Python 3.2.3(64-bit) reinstalled properly, I try the NumPy installer again. This time it finds Python in the registry (presumably), and installs NumPy 1.6 without issue. Now test it out.

Step 6: Test out NumPy

To make sure it installed correctly, go into the Python interpreter and try importing NumPy:

from numpy import *

This returns the following mess:

Traceback (most recent call last):
  File "<pyshell#0>", line 1, in <module>
    from numpy import *
  File "C:\Python32\lib\site-packages\numpy\__init__.py", line 137, in <module>
    from . import add_newdocs
  File "C:\Python32\lib\site-packages\numpy\add_newdocs.py", line 9, in <module>
    from numpy.lib import add_newdoc
  File "C:\Python32\lib\site-packages\numpy\lib\__init__.py", line 4, in <module>
    from .type_check import *
  File "C:\Python32\lib\site-packages\numpy\lib\type_check.py", line 8, in <module>
    import numpy.core.numeric as _nx
  File "C:\Python32\lib\site-packages\numpy\core\__init__.py", line 5, in <module>
    from . import multiarray
ImportError: DLL load failed: %1 is not a valid Win32 application

The last line is telling: "not a valid Win32 application." Some of the online forums seem to suggest that it could be a problem with 64-bit Python. Here's the exact version I'm running: "Python 3.2.3 (default, Apr 11 2012, 07:12:16) [MSC v.1500 64 bit (AMD64)] on win32".

 Step 7: Uninstall 64-bit Python, install 32-bit Python

So I'm going to uninstall NumPy and uninstall this 64-bit version of Python 3.2.3, and in its place install a 32-bit version of Python 3.2.3. Again, be careful to install 'just for me.' When this is done, I have this version installed: "Python 3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit (Intel)] on win32". Now try NumPy again.

Step 8: Try installing NumPy again

Using the same NumPy binary as every time before, re-install it. To re-cap, I'm installing this in the context of a 32-bit Python 3.2.3 installation. I get no errors from the installation of NumPy, so it's time to test it.

Step 9: Test NumPy

One more time, in the Python interpreter, try importing NumPy:

>>> from numpy import *

This time it returns nothing, meaning it worked! Try creating a NumPy array, and see if it returns the proper type:

type(array([1,6,3,7]))

<class 'numpy.ndarray'>

That worked too, which means our task of installing NumPy for Python 3 in Windows 7 has been completed.

Conclusion

Here are the condensed steps for getting NumPy to work with Python 3 in Windows 7

1. Regardless of whether you have a 32-bit or a 64-bit operating system, install the 32-bit version of Python 3.2
2. Make sure you have installed Python 3.2 'just for me', and not 'for all users of this computer'.
3. Make sure you download the correct version of NumPy from SourceForge, not the default that it offers as the latest version (which is for Python 2.6 instead of Python 3)

Installing NumPy 1.6 for Python 3 in Ubuntu 12.04

I have a fresh install of Ubuntu 12.04. I want to use NumPy with Python 3. Note that Ubuntu 12.04 ships with Python 2.7 under the 'python' namespace and also ships with Python 3 under the 'python3' namespace.

Go to terminal, check python version

$ python --version
Python 2.7.3
$ python3 --version
Python 3.2.3

We want to use Python 3, so for the rest of this tutorial, make sure you're using python3 and not just python.

Go into the Python 3 interpreter:

$ python3
Python 3.2.3 (default, Apr 12 2012, 21:55:50) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 

See if you have NumPy already installed:

>>> from numpy import *
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named numpy

Nope, it's not there, so we need to install. Exit the Python interpreter.

>>> exit()

Install NumPy using apt-get

Instead of browsing web forums and finding the right package to download and compile, just let Ubuntu's built-in package manager, apt-get, do the work. This is just like installing NumPy for the standard python installation, but because we need NumPy for the Python3 installation, we'll affix the '3' where needed:

$ sudo apt-get install python3-numpy python3-scipy

Let it have all the dependencies it wants (just type 'yes' when prompted).

See if it worked

To see if it worked, go back into the Python interpreter, import NumPy, create a NumPy array, and make sure it's a NumPy array:

>>> from numpy import *
>>> type(array([1,2,4,5]))
<type 'numpy.ndarray'>

Yup it worked.

Conclusion

In Ubuntu 12.04, just use apt-get to install NumPy for python 3 using

$ sudo apt-get install python3-numpy python3-scipy

Everything just works.

Installing NumPy 1.6 on Python 2.7 in Ubuntu 12.04

I have a fresh install of Ubuntu 12.04. I want to use NumPy, and I'm okay with using the default Python version that ships with Ubuntu 12.04, which is Python 2.7. Note that Ubuntu 12.04 also ships with Python 3 under the 'python3' namespace. 

Go to terminal, check python version

$ python --version
Python 2.7.3

Go into the Python interpreter:

$ python
Python 2.7.3 (default, Apr 20 2012, 22:44:07) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.

See if you have NumPy already installed:

>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named numpy

Nope, it's not there, so we need to install. Exit the Python interpreter.

>>> exit()

Install NumPy using apt-get

Instead of browsing web forums and finding the right package to download and compile, just let Ubuntu's built-in package manager, apt-get, do the work:

$ sudo apt-get install python-numpy python-scipy

Let it have all the dependencies it wants (just type 'yes' when prompted).

See if it worked

To see if it worked, go back into the Python interpreter, import NumPy, create a NumPy array, and make sure it's a NumPy array:

$ python

>>> from numpy import *

>>> type(array([1,2,4,5]))
<type 'numpy.ndarray'>

Yup it worked.

Conclusion

In Ubuntu 12.04, just use apt-get to install NumPy using

$ sudo apt-get install python-numpy python-scipy

Everything just works.

Abounding life in Antarctica

The following excerpt comes from Endurance: Shackleton's Incredible Voyage, by Alfred Lansing (New York: Basic Books 2007), which tells the story of a doomed 1915 attempt to cross the Antarctic. The ship entered a pack a sea-ice floes, which eventually surrounded the ship, froze, and crushed her over a period of months. Its crew, lead by the legendary Ernest Shackleton, survived in the ship, on the ice, and on an island for more than a year.

Photo by Frank Hurley

I highlighted the following passage because it disabused my own misunderstanding that the Antarctic would be a lifeless desert.

But they had not yet even crossed the Antarctic Circle, though the summer had already officially begun. It was now light twenty-four hours a day; the sun disappeared only briefly near midnight, leaving a prolonged, magnificent twilight. Often during this period, the phenomenon of an "ice shower," caused by the moisture in the air freezing and settling to earth, lent a fairlyland atmosphere to the scene. Millions of delicate crystals, frequently thin and needle-like in shape, descended in sparkling beauty through the twilight air.

And though the [ice] pack in every direction appeared to stretch in endless desolation, it abounded with life. Finner, humpback, and huge blue whales, some of them a hundred feet long, surfaced and sported in the leads of open water between the floes. There were killer whales, too, who thrust their ugly, pointed snouts above the surface of the ice to look for whatever prey they might upset into the water. Overhead, giant albatross, and several species of petrels, fulmars, and terns wheeled and dipped. On the ice itself, Weddell and crabeater seals were a common sight as they lay sleeping.

Emperor penguins. Photo by Glenn Grant, National Science Foundation

 And there were penguins, of course. Formal, stiff-necked emperors, who watched in dignified silence as the ship sailed past them. But there was nothing dignified about the little Adélies. They were so friendly they would flop down on their bellies and toboggan along, pushing with their feet and croaking what sounded like "Clark! Clark!" . . . especially, it seemed, if Robert Clark, the gaunt and taciturn Scottish biologist, happened to be at the wheel.

Endurance: Shackleton's Incredible Voyage, by Alfred Lansing (New York: Basic Books 2007), page 27.

Web Scraping: How to harvest web data using Ruby and Nokogiri

Web Scraping with Nokogiri

In this post I will walk through how to use Nokogiri to harvest data from retailer web pages and save that data into a spreadsheet, instead of copying and pasting by hand. I am using Ubuntu 10.10, Nokogiri 1.5.0, and Ruby 1.9.2. Update: I've learned that this technique is commonly called "web scraping," so I've updated the text to reflect that.

Web Scraping Background and Introduction

Recently I was assigned the task of populating a spreadsheet with fan data pulled from the retailer Industrial Fans Direct. My client needed the price, description, and serial number of a lot of fans, from each of the categories visible below (e.g. ceiling fans, exhaust fans, contractor fans). Some of these categories have sub-categories, and some of those sub-categories have further sub-categories. The point is that there are many hundreds of fans listed on this web site, and doing the traditional copy-paste into an Excel spreadsheet was going to take a long time.

Industrial Fans Direct -- Home Page

Below is a screenshot of a product summary page of ceiling fans. This page contains all the data I need: price, serial number, and description. I noticed that the formats are the same for all the ceiling fans, and it turns out that this retailer has used the same format across all categories of fans.

Industrial Fans Direct -- showing ceiling fan product summary page.

Since the format is consistent, this is a great format for using an HTML parser to gather the data.This technique is known "web scraping."

Introducing Nokogiri

Nokogiri is a Ruby gem designed to help parse HTML and XML. Its creators describe it as an "HTML, XML, SAX, & Reader parser with the ability to search documents via XPath or CSS3." Since we only want to read a simple HTML page, we can ignore the part about XML and SAX (I have no idea what SAX is). We can also ignore the part about XPath, which I'm also unfamiliar with. The takeaway is that Nokogiri can parse HTML and search it via CSS. That's how we're going to perform our web scraping. The parsing part we can largely ignore as well; it basically means Nokogiri will load the document. The really important part for us is that we can use Nokogiri to search HTML using CSS.

Searching with CSS

Searching HTML with CSS means using CSS selectors to identify parts of an HTML document. Consider the following simple HTML page (borrowed from tenderlove):

<html>
  <head>
    <title>Hello World</title>
  </head>
  <body>
    <h1>This is an awesome document</h1>
    <p>
      I am a paragraph
        <a href="http://google.ca">I am a link</a>
    </p>
  </body>
</html>

If we wanted to change that h1 heading to red text, we would use CSS. First we would select the h1 heading using the CSS selector "h1", and then we would apply the "color" property with the attribute "red". In a separate style sheet, that would look like this:

h1 { 
  color: red;
}

The point here is the selector. We use the selector "h1" to identify the discreet text string "This is an awesome document", which then turns red. Using CSS, we can identify any(?) element in an HTML document, assuming that document is properly marked-up. Using these exact selector rules from CSS, we can tell Nokogiri which elements we want to grab.

An important lesson here: know how to use CSS selectors. The CSS2 specification has a short and useful list of selectors. These will get you far.

Set up your own CSS file

Before jumping into Nokogiri, we have to know what we want to grab from the web site, and how to grab it using CSS selectors. In properly marked-up with semantic CSS, that should be fairly easy. However, the fan data I need is in Industrial Fans Direct, a web site with atrocious mark-up. That's okay--Nokogiri can handle it. It just means this will be a rather advanced lesson in selectors.

First, save a local copy of the HTML document, so that we can play around with its CSS. I started with this page of exhaust fans, and saved it onto my computer as "fans.html."

Second, create a style sheet (I called mine "andrew.css") and save it in the same location that you saved your local copy of the HTML page. I put both my local copy of the HTML and my style sheet in a folder called "nokogiri_testing".

Third, look at the source code in the browser. Specifically, look at the stylesheets. The "head" section from fans.html is below:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> 
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"> 
<head> 
<title>INDUSTRIAL I - BELT :: Industrial Fans Direct</title> 
 
<base href="http://www.industrialfansdirect.com/Merchant2/"> 
<meta http-equiv="content-type" content="text/html; charset=utf-8" /> 
<link rel="canonical" href="http://www.industrialfansdirect.com/IND-FA-EF-CM-I1.html" /> 
<link href="css/andreas09.css" rel="stylesheet" type="text/css" /> 
<link href="css/dropdown.css" rel="stylesheet" type="text/css" /> 
<link href="css/tab-view.css" rel="stylesheet" type="text/css" /> 
<link href="css/IFD_Print.css" rel="stylesheet" type="text/css" media="print"> 
<link href="file:///home/andy/nokogiri_testing/andrew.css" rel="stylesheet" type="text/css" />
<script language="javascript"> 
function cfm_calc (form) {
form.cfm.value = Math.round((form.height.value * 1.2) * form.width.value *
form.length.value) ; }
</script> 
</head> 

Notice the base tag, which sets all links relative to the root of the site. With this in mind, we know we can insert our own stylesheet into this local copy by including a full path, as I have done above, on line 13. Notice also that it comes after all the other style sheets, so that it overrides everything that comes before it.

Fourth, populate this CSS file with something obnoxious just so that we know it works. Here's mine:

body {
	background-color: blue;
}

If that turns the page blue when you render the page in a browser, you'll know that you have a working style sheet.

Fifth, open the page in a browser to see if your CSS modifications are working. Remember: load your local copy (in my case, "fans.html"), not the online version.

Once you have a working stylesheet, the next step is to start using it to figure out what CSS selectors to use.

Identify your CSS selectors

The next step is to decide what you want to grab from the web page, and then figure out how to use CSS selectors to get to it. This is where it starts to get a bit difficult, especially with a page marked up as badly as this one is, with tables nested in tables nested in tables, and with countless divs, few of which have identifiers or classes.

A local copy of the exhaust fans page. Note the address bar (local copy!) and the prices next to each product.

The first piece of data I want from the Industrial Fans Direct summary product page is the price. Looking at the HTML document, I see that the price is embedded in lines that look like this:

<div align="left"><b>Your Price: <font color="#003366">$1,349.00</font></b></div>

The piece we want is with the dollar sign. We can see that it's wrapped in a font tag, which is in turn wrapped in a <b> tag, which is in turn wrapped in a <div> tag. The CSS selector which represents this is div > b > font.

Let's use a CSS selector to grab this.

First, go back to your CSS file and add the following line:

div > b > font {
	background-color: green;
}

Second, go back and refresh the browser (the local copy!). That should turn all of the prices green. If it works correctly, then we've gained our objective, which is to discover a suitable CSS selector to grab the information we want from the web page. For the price, that selector is div > b > font. Note that there are usually several ways to drill down to the information you need. As your knowledge of CSS selectors grows, you'll discover the most efficient ways.

The exhaust fans page again, this time with prices highlighted in green using the CSS selector " div > b > font". Notice that that selector didn't pick up any other elements on the page.

 Third, pick another piece of desired information, and find a CSS selector to identify it. The next piece of information I want is the serial number.

Fourth, go back to the HTML and take a guess at how you would drill down to the serial number. Here's a line that contains the serial number.

<div align="left"><b style="font-size:8px;">LFI-XB24SLB10050</b></div>

The serial number is wrapped in a <b> tag, which is wrapped in a <div> tag. Using the same logic as above, I try out the CSS selector div > b, as shown in my style sheet, which now has two styles:

div > b > font {
	background-color: green;
}

div > b {
	background-color: red;
}

Fifth, go back to the browser again and refresh the page. I've given my serial number style a background color of red, and the result is shown in the following figure.

The exhaust fans page after a first attempt at selecting price (green) and serial number (red). Notice that the serial number selector was too liberal.

 As you can see, the div > b CSS selector picked up more than just the serial number, so we'll have to get more precise.

From this point, it's an iterative process. Keep adding more and more specificity to your CSS selector chain until you highlight exactly the elements you need, and nothing more. My completed stylesheet is shown below:

/* price */
div > b > font {
	background-color: green;
}

/* serial number */
div#contentalt1 div:first-child b {
	background-color: red;
}

/* description */
table + table tr + tr td a {
	background-color: blue;
}

The result of all this work (downloading the page, adding a CSS file, highlighting elements) are the three CSS selectors we found:

1. div > b > font
2. div#contentalt1 div:first-child b
3. table + table tr + tr td a

In the next section we will provide those CSS selectors to Nokogiri, which will use them to speed through HTML pages and pull out prices, serial numbers, and descriptions for all sorts of fans.

Dive into Nokogiri

Now that we've identified our CSS selectors, we're done with HTML and CSS. From here, we'll be in Ruby. I find it's always easiest to start in an Interactive Ruby (IRb) session. So type irb at the command prompt and type in the following commands:

$ irb
ruby-1.9.2-p180 :001 > require 'nokogiri'
 => true 
ruby-1.9.2-p180 :002 > require 'open-uri'
 => true 
ruby-1.9.2-p180 :003 > doc = Nokogiri::HTML(open('http://www.industrialfansdirect.com/IND-FA-PC-EC.html'))
[output truncated]
ruby-1.9.2-p180 :004 > doc.class
 => Nokogiri::HTML::Document 

The first two lines loaded Nokogiri and a library used by Nokogiri, respectively. The third line told Nokogiri to fetch an HTML document from the web, parse it as HTML, and save the result in an object called "doc". Since Ruby returns the result of every operation, that should result in a huge amount of output, which you can ignore. But now that you have the object called "doc", you can use Nokogiri's css method to search it. Simply pass the css method the CSS selector that you want it to use. That's it.

ruby-1.9.2-p180 :005 > > puts doc.css('div > b > font')
<font color="#003366">$739.00</font>
<font color="#003366">$1,019.00</font>
<font color="#003366">$1,779.00</font>
<font color="#003366">$2,099.00</font>
<font color="#003366">$2,329.00</font>
<font color="#003366">$2,499.00</font>
<font color="#003366">$3,849.00</font>
<font color="#003366">$3,599.00</font>
 => nil 

As you can see, Nokogiri returned the font tags in their entirety. Later we'll use the content method to return just what's inside those tags. But for the moment, the takeaway is:

1. Load Nokogiri
2. Pass it a file or a web page to parse and return a Nokogiri object
3. Use the css method to search that object

Now that we know how to use Nokogiri, let's start a Ruby script to start doing the heavy lifting.

A Nokogiri Ruby Script

First, create a Ruby file as follows. I called mine "fans.rb".

require 'nokogiri'
require 'open-uri'
doc = Nokogiri::HTML(open('http://www.industrialfansdirect.com/IND-FA-PC-EC.html'))

doc.css('div > b > font').each do |price|
  puts price.content
end

Run this file and note that the output only includes the content of the font tags.
However, we don't want to just print data to the terminal window; we want to store it. Let's take an intermediate step by filling out the program with all three attributes (price, description, serial number), and storing those attributes in Ruby arrays. To check that this is working, we can still print the output to the terminal window. Here's the new script:

require 'nokogiri'
require 'open-uri'
doc = Nokogiri::HTML(open('http://www.industrialfansdirect.com/IND-FA-PC-EC.html'))

prices = Array.new
serial_numbers = Array.new
descriptions = Array.new

doc.css('div > b > font').each do |price|
  prices << price.content
end

doc.css('div#contentalt1 table + table div:first-child b').each do |serial_number|
  serial_numbers << serial_number.content
end

doc.css('div#contentalt1 table + table tr + tr td a').each do |description|
  descriptions << description.content unless description.content.length < 2
end

(0..prices.length - 1).each do |index|
  puts "serial number: #{serial_numbers[index]}"
  puts "price: #{prices[index]}"
  puts "description: #{descriptions[index]}"
  puts ""
end

Note line 18: I had to add an unless modifier because I couldn't find a CSS selector that would select the description and nothing else. Instead, it selected the descriptions and random bits of empty tables. Since I don't want to store the random bits of empty tables (which appeared in my array as strings of length 0 or 1), I required a description to have at least 3 characters.

This Ruby script produces the following output:

$ ruby fans.rb
serial number: PC-PAC2KCYC01
price: $739.00
description: CYCLONE 3000 Portable 2 Speed Evaporative Cooler (2,400 / 3,000 CFM)

serial number: PC-PAC2K163SHD
price: $1,019.00
description: Portable 3 Speed Evaporative Cooler: 16 in Blade (2,500 / 3,280 / 3,900 CFM)

serial number: PC-PAC2K24HPVS
price: $1,779.00
description: Portable Variable Speed Evaporative Cooler: 24 in Blade (6,700 CFM)

serial number: PC-PAC2K361S
price: $2,099.00
description: Portable 1 Speed Evaporative Cooler: 36 in Blade (9,600 CFM)

serial number: PC-PAC2K363S
price: $2,329.00
description: Portable 3 Speed Evaporative Cooler: 36 in Blade (4,800 / 6,600 / 9,600 CFM)

serial number: PC-PAC2K36HPVS
price: $2,499.00
description: Portable Variable Speed Evaporative Cooler: 36 in Blade (10,100 CFM)

serial number: SCF-PROK142-2HV
price: $3,849.00
description: Portable 2 Speed Evaporative Cooler (high velocity): 42 in Blade (9,406 / 14,232 CFM)

serial number: PC-PAC2K482S
price: $3,599.00
description: Portable 2 Speed Evaporative Cooler: 48 in Blade (11,000 / 20,000 CFM)

It tells me that it knows the serial number, price and description of eight fans. I tested this script on a several different web pages from this retailer, and found that it works for each category and sub category.

Now that we know we can harvest (web scrape) and store the data in Ruby, we have to get it into a spreadsheet.

Storing the Harvested Data

For this part, we'll use Ruby's CSV class to store the data in a csv file. Simply require CSV at the top of the file, and use two loops to write the contents of our three arrays into a csv file. Below is the complete new script:

require 'nokogiri'
require 'open-uri'
require 'csv'

doc = Nokogiri::HTML(open('http://www.industrialfansdirect.com/IND-FA-PC-EC.html'))

prices = Array.new
serial_numbers = Array.new
descriptions = Array.new

doc.css('div > b > font').each do |price|
  prices << price.content
end

doc.css('div#contentalt1 div:first-child b').each do |serial_number|
  serial_numbers << serial_number.content
end

doc.css('table + table tr + tr td a').each do |description|
  descriptions << description.content unless description.content.length < 2
end

(0..prices.length - 1).each do |index|
  puts "serial number: #{serial_numbers[index]}"
  puts "price: #{prices[index]}"
  puts "description: #{descriptions[index]}"
  puts ""
end

CSV.open("fans.csv", "wb") do |row|
  row << ["serial number", "price", "description"]
  (0..prices.length - 1).each do |index|
    row << [serial_numbers[index], prices[index], descriptions[index]]
  end
end

That works correctly, which means we've completed the hard part. The script is parsing the HTML file, pulling out the data we want, and storing it in a csv file called "fans.csv". But we're not done yet; this script only takes one HTML file, and we have lots of web pages from which we want to harvest data. The next step is find a way to efficiently go through all these web pages without having to insert a new URL each time.

Running the script over multiple web pages

There are several ways to make this script "crawl" the web site. I think the simplest is to create an array of all the URLs that contain my data, and pass those URLs from the array, one at time, to the script we wrote. This means we'll establish the array of URLs and the three attribute arrays (serial numbers, prices, descriptions), and then wrap the rest of our code in a loop that goes through all the URLs. Here's the script, with the URL array and the loop. Notice that the arrays had to become instance variables so that they could be accessed outside the URL loop.

require 'nokogiri'
require 'open-uri'
require 'csv'

urls = Array[
  'http://www.industrialfansdirect.com/IND-FA-AF-S.html',
  'http://www.industrialfansdirect.com/IND-FA-AF-WE.html',
  'http://www.industrialfansdirect.com/IND-FA-AF-SS.html',
  'http://www.industrialfansdirect.com/IND-FA-AF-CF.html',
  'http://www.industrialfansdirect.com/IND-FA-BL.html',
  'http://www.industrialfansdirect.com/IND-FI-CF.html'
]

@prices = Array.new
@serial_numbers = Array.new
@descriptions = Array.new

urls.each do |url|
  doc = Nokogiri::HTML(open(url))
  doc.css('div > b > font').each do |price|
    @prices << price.content
  end

  doc.css('div#contentalt1 div:first-child b').each do |serial_number|
	  @serial_numbers << serial_number.content
  end

  doc.css('table + table tr + tr td a').each do |description|
    @descriptions << description.content unless description.content.length < 2
  end

  (0..@prices.length - 1).each do |index|
    puts "serial number: #{@serial_numbers[index]}"
    puts "price: #{@prices[index]}"
    puts "description: #{@descriptions[index]}"
    puts ""
  end
end
  
CSV.open("fans.csv", "wb") do |row|
  row << ["serial number", "price", "description"]
  (0..@prices.length - 1).each do |index|
    row << [@serial_numbers[index], @prices[index], @descriptions[index]]
  end
end

That completes the objectives of this task. With this Ruby script, using the power of Nokogiri, we can "web scrape," or harvest data from, as many pages as we want to include in the url array.

Special thanks are due to Aaron Paterson, creator of Nokogiri, and all who contribute to it.

Update

Without going into all the specifics of how I did, below is the completed script. It has a few extra features:

• URLs are stored in an external CSV file
• CSS selectors are updated to be slightly more robust
• Includes category, sub-category, and sub-sub-category

The if statements at the top of the file organize how the category and sub-categories are identified. They're pulled from "bread crumb" navigation, which changes structure depending on how deep the category hierarchy goes. Again, all my thanks go to the creators of Nokogiri. With their Ruby gem, I pulled out more than 1,700 rows of data in 68 lines of code, which runs in about one minute. Including the 5 or so hours it took me to write this script, it probably saved me about 10 hours of work, and increased the accuracy of the finished product.

require 'nokogiri'
require 'open-uri'
require 'csv'

@prices = Array.new
@serial_numbers = Array.new
@descriptions = Array.new
@urls = Array.new
@categories = Array.new
@subcategories = Array.new
@subsubcategories = Array.new

urls = CSV.read("fan_urls.csv")
(0..urls.length - 1).each do |index|
  puts urls[index][0]
  doc = Nokogiri::HTML(open(urls[index][0]))
  
  #the last bread crumb does not have an anchor tag, which allows the following logic
  bread_crumbs_length = doc.css('div[style="padding-left:10px;"] a').length + 1
  puts "bread crumbs length: #{bread_crumbs_length}"
  if bread_crumbs_length == 2
    category = doc.css('a + font')[0].content
    sub_category = "na" 
    sub_sub_category = "na" 
  elsif bread_crumbs_length == 3
    category = doc.css('div[style="padding-left:10px;"] a:first-child + a')[0].content
    sub_category = doc.css('a + font')[0].content
    sub_sub_category = "na" 
  elsif bread_crumbs_length == 4
    category = doc.css('div[style="padding-left:10px;"] a:first-child + a')[0].content
    sub_category = doc.css('div[style="padding-left:10px;"] a:first-child + a + a')[0].content
    sub_sub_category = doc.css('a + font')[0].content
  else
    category = "na"
    sub_category = "na"
    sub_sub_category = "na"
  end

  doc.css('div > b > font').each do |price|
    @prices << price.content
    @urls << urls[index][0]
    @categories << category
    @subcategories << sub_category
    @subsubcategories << sub_sub_category
  end

  doc.css('div#contentalt1 table[align] div:first-child b').each do |serial_number|
	  @serial_numbers << serial_number.content
  end

  doc.css('table + table tr + tr td a').each do |description|
    @descriptions << description.content unless description.content.length < 2
  end
end
 
CSV.open("fans.csv", "wb") do |row|
  row << ["category", "sub-category", "sub-sub-category", "serial number", "price", "description", "url"]
  (0..@prices.length - 1).each do |index|
    row << [
      @categories[index], 
      @subcategories[index], 
      @subsubcategories[index], 
      @serial_numbers[index], 
      @prices[index], 
      @descriptions[index], 
      @urls[index]]
  end
end

Installing Nokogiri with RVM on Ubuntu

I'm using Ruby 1.9.2, Ubuntu 10.10, and RVM 1.6.14.

The Nokogiri installation page provides instructions for Ubuntu/Debian users:

# ruby developer packages
sudo apt-get install ruby1.8-dev ruby1.8 ri1.8 rdoc1.8 irb1.8
sudo apt-get install libreadline-ruby1.8 libruby1.8 libopenssl-ruby

# nokogiri requirements
sudo apt-get install libxslt-dev libxml2-dev
sudo gem install nokogiri

But what if I'm using Ruby 1.9 instead of 1.8? The Nokogiri GitHub page says it requires either Ruby 1.8 or 1.9. I guess the official website hasn't been updated to reflect that.

Even so, using RVM, I have a few options. First, I could create a new Ruby environment and gemset with RVM, use Ruby 1.8 explicitly, and follow the instructions as provided by Nokogiri. My second option is to create a clean gemset and use Ruby 1.9 (which is faster than 1.8), and experiment in the safe confines on the gemset. Let's do the second option.

Create a New RVM Gemset

First, create a new gemset and call it "nokogiri":

$ rvm gemset create nokogiri
'nokogiri' gemset created (/home/andy/.rvm/gems/ruby-1.9.2-p180@nokogiri).

Then switch into that gemset:

$ rvm use 1.9.2@nokogiri
Using /home/andy/.rvm/gems/ruby-1.9.2-p180 with gemset nokogiri

Then confirm the Ruby version:

$ ruby -v
ruby 1.9.2p180 (2011-02-18 revision 30909) [i686-linux]

NB: Normally we wouldn't create a whole gemset just to install one gem. RVM is meant to create a whole environment where you would install many gems. But here we're just using it as a safe playground to see how the Nokogiri installation goes.

Now let's just follow Nokogiri's installation instructions, as copied above:

Install Nokogiri and Dependencies

$ sudo apt-get install libxml2 libxml2-dev libxslt libxslt-dev
[sudo] password for andy: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Note, selecting 'libxslt1-dev' instead of 'libxslt-dev'
E: Unable to locate package libxslt

I have no idea why it can't find the package "libxslt". Instead of worrying about that, I'm going to install the dependencies listed on tenderlove's Nokogiri GitHub page:

$ sudo apt-get install libxslt-dev libxml2-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Note, selecting 'libxslt1-dev' instead of 'libxslt-dev'
libxslt1-dev is already the newest version.
libxml2-dev is already the newest version.

That appears to have worked well enough, so I press on. The next step is install the actual Nokogiri gem. The instructions say to use "sudo gem install nokogiri", but because I'm using RVM, I drop the "sudo" part:

$ gem install nokogiri
Fetching: nokogiri-1.5.0.gem (100%)
Building native extensions.  This could take a while...
Successfully installed nokogiri-1.5.0
1 gem installed
Installing ri documentation for nokogiri-1.5.0...
Installing RDoc documentation for nokogiri-1.5.0...

A quick "gem list" shows that it's indeed there:

$ gem list

*** LOCAL GEMS ***

nokogiri (1.5.0)

Test It Out

Now how do we know that it installed successfully and actually works? Let's jump into an Interactive Ruby (IRb) session and try it out, using the "synopsis" provided by tenderlove on the Nokogiri GitHub page.

> require 'nokogiri'
 => true
> require 'open-uri'
 => true 
>doc = Nokogiri::HTML(open('http://www.google.com/search?q=nokogiri'))
[output truncated]
> doc.css('h3.r a.l').each do |link|
> puts link.content
?>   end
Nokogiri
Tutorials
Installation
Nokogiri::XML::Node
Nokogiri::HTML::Document
Parsing an HTML / XML ...
Nokogiri::XML::Document
tenderlove/nokogiri - GitHub
Nokogiri - GitHub
Nokogiri Is Released - Tender Lovemaking
Getting Started with Nokogiri | Engine Yard Ruby on Rails Blog
nokogiri | RubyGems.org | your community gem host
Nokogiri: A Faster, Better HTML and XML Parser for Ruby (than ...
#190 Screen Scraping with Nokogiri -  RailsCasts
RubyForge: nokogiri: Project Info
Nokogiri (project) - Wikipedia, the free encyclopedia
 => 0 

Conclusion

Nokogiri is incredibly easy to install. It works with Ruby 1.8 or 1.9. Simply install the necessary package dependencies, and then don't be afraid to include the Nokogiri gem in any RVM gemset. It should just work.

Special thanks to Aaron Paterson, aka Tenderlove, for his work on Nokogiri, and Wayne Seguin for his work on RVM.

Installing autotest with Rails 3.1 and RSpec on Ubuntu 10.10

My environment:
RVM 1.6.14
Ubuntu 10.10
Ruby 1.9.2
Rails 3.1.rc1
RSpec 2.6.4

First install the autotest gem (currently version 4.4.6). Since I'm in a project gemset (through RVM), I don't need "sudo":

$ gem install autotest

Next install the Rails helper gem for autotest (currently version 4.1.2):

$ gem install autotest-rails-pure

Then install the libnotify-bin package (currently version 0.5.0):

$ sudo apt-get install libnotify-bin

Then install the autotest-notification gem (currently version 2.3.1)

$ gem install autotest-notification

This gives you the executable "an-install", which you should to run:

$ an-install

Now start the regular autotest gem:

$ autotest

That's it. Now you should get automatic desktop notification of passing and failing tests each time you change a file covered by a test. For complete instructions, see the README at Carlos Brando's autotest-notification gem.

Installing RVM (Ruby Version Manager) on CentOS 5.6

In this blog post I explain how to install Wayne Seguin's RVM (Ruby Version Manager) on CentOS 5.6. I will be following the installation instructions for a "Single-User Installation as a standard user". I already have installed git 1.6 and Ruby 1.8.7.

What CentOS version are you running?

First, confirm what version of CentOS you're running:

$ cat /etc/issue
CentOS release 5.6 (Final)

As you can see, I'm running 5.6, but these instructions may work for earlier versions.

What git version are you running?

The RVM installation instructions recommend having git version 1.7 or later. What do I have?

$ git --version
git version 1.6.1

I'm okay with this for the moment. The standard CentOS repositories do not have a newer version; in the CentOS universe, this is the latest version of Git (although you could get a newer version by adding a different repo). If I run into trouble later, I'll have this potential git upgrade as a possible solution.

What terminal are you running?

I believe the RVM installation instructions assume you are using Bash as a terminal. Check what terminal you are using by running the following command:

$ ps -p$$ -ocmd=
ps -p$$ -ocmd=
bash -v

That tells me I'm running bash. Confirm by asking it for the specific version:

$ bash --version
bash --version
GNU bash, version 3.2.25(1)-release (i686-redhat-linux-gnu)
Copyright (C) 2005 Free Software Foundation, Inc.

Installation: Three steps

The installation instructions list three steps for completing the RVM installation package:

1. Download and run the RVM installation script
2. Load RVM into your shell sessions as a function
3. Reload shell configuration & test

We'll follow each of these in turn.

Download and run the RVM installation script

Copy the following line into your terminal:

$ bash < <(curl -sk https://rvm.beginrescueend.com/install/rvm)

For me, this produced the following output:

Initialized empty Git repository in /home/arsturges/.rvm/src/rvm/.git/
remote: Counting objects: 4930, done.
remote: Compressing objects: 100% (2305/2305), done.
remote: Total 4930 (delta 3194), reused 3552 (delta 1943)
Receiving objects: 100% (4930/4930), 1.60 MiB | 1646 KiB/s, done.
Resolving deltas: 100% (3194/3194), done.

  RVM:  Shell scripts enabling management of multiple ruby environments.
  RTFM: https://rvm.beginrescueend.com/
  HELP: http://webchat.freenode.net/?channels=rvm (#rvm on irc.freenode.net)
  
Installing RVM to /home/arsturges/.rvm/~/.rvm ~/.rvm/src/rvm
~/.rvm/src/rvm

    Correct permissions for base binaries in /home/arsturges/bin...
    Copying manpages into place.


Notes for Linux ( CentOS release 5.6 (Final) )

NOTE: 'ruby' represents Matz's Ruby Interpreter (MRI) (1.8.X, 1.9.X)
             This is the *original* / standard Ruby Language Interpreter
      'ree'  represents Ruby Enterprise Edition
      'rbx'  represents Rubinius

bash >= 3.2 is required
curl is required
git is required (>= 1.7 recommended)
patch is required (for ree and some ruby-head's).

If you wish to install rbx and/or Ruby 1.9 head (MRI) (eg. 1.9.2-head),
then you must install and use rvm 1.8.7 first.

If you wish to have the 'pretty colors' again,
  set 'export rvm_pretty_print_flag=1' in ~/.rvmrc.

dependencies:
  # For RVM
  rvm: yum install -y bash curl git # NOTE: For git you need the EPEL repository enabled

  # For Ruby (MRI & Ree) you should install the following OS dependencies:
  ruby: yum install -y gcc-c++ patch readline readline-devel zlib zlib-devel libyaml-devel libffi-devel openssl-devel ;
        yum install -y make bzip2 ;
        yum install -y iconv-devel # NOTE: For centos 5.4 final iconv-devel might not be available :(

  # For JRuby (if you wish to use it) you will need:
  jruby: yum install -y java

For rbx (Rubinius) more then 600MB of free RAM required.


  You must now complete the install by loading RVM in new shells.

  1) Place the folowing line at the end of your shell's loading files
     (.bashrc or .bash_profile for bash and .zshrc for zsh),
     after all PATH/variable settings:

     [[ -s "/home/arsturges/.rvm/scripts/rvm" ]] && source "/home/arsturges/.rvm/scripts/rvm"  # This loads RVM into a shell session.

     You only need to add this line the first time you install rvm.

  2) Ensure that there is no 'return' from inside the ~/.bashrc file,
     otherwise rvm may be prevented from working properly.

     
  This means that if you see something like:

    '[ -z "$PS1" ] && return'

  then you change this line to:

  if [[ -n "$PS1" ]] ; then

    # ... original content that was below the '&& return' line ...

  fi # <= be sure to close the if at the end of the .bashrc.

  # This is a good place to source rvm v v v
  [[ -s "/home/arsturges/.rvm/scripts/rvm" ]] && source "/home/arsturges/.rvm/scripts/rvm"  # This loads RVM into a shell session.

EOF - This marks the end of the .bashrc file

     Be absolutely *sure* to REMOVE the '&& return'.

     If you wish to DRY up your config you can 'source ~/.bashrc' at the bottom of your .bash_profile.

     Placing all non-interactive (non login) items in the .bashrc,
     including the 'source' line above and any environment settings.

  3) CLOSE THIS SHELL and open a new one in order to use rvm.
  

Installation of RVM to /home/arsturges/.rvm/ is complete.


Andy Sturges,

Thank you very much for using RVM! I sincerely hope that RVM helps to
make your work both easier and more enjoyable.

If you have any questions, issues and/or ideas for improvement please
join#rvm on irc.freenode.net and let me know, note you must register
(http://bit.ly/5mGjlm) and identify (/msg nickserv  ) to
talk, this prevents spambots from ruining our day.

My irc nickname is 'wayneeseguin' and I hang out in #rvm typically

  ~09:00-17:00EDT and again from ~21:00EDT-~23:00EDT

If I do not respond right away, please hang around after asking your
question, I will respond as soon as I am back.  It is best to talk in
#rvm itself as then other users can help out should I be offline.

Be sure to get head often as rvm development happens fast,
you can do this by running 'rvm get head' followed by 'rvm reload'
or opening a new shell

  w⦿‿⦿t

    ~ Wayne

Notice a few things about this comprehensive output:

• Wayne has customized his installation script (which produced this file) to produce output specific to my linux distribution (CentOS 5.6). This means he provides "yum" commands, instead of, say, Ubuntu's "apt-get".
• He notes several dependencies:
  

  dependencies:
    # For RVM
    rvm: yum install -y bash curl git # NOTE: For git you need the EPEL repository enabled
  
    # For Ruby (MRI & Ree) you should install the following OS dependencies:
    ruby: yum install -y gcc-c++ patch readline readline-devel zlib zlib-devel libyaml-devel libffi-devel openssl-devel ;
          yum install -y make bzip2 ;
          yum install -y iconv-devel # NOTE: For centos 5.4 final iconv-devel might not be available :(
  

  We can run these commands as-is, which I'll do below.
• He gives us three steps to complete the installation:
  
  1. Place the folowing line at the end of your shell's loading files
     (.bashrc or .bash_profile for bash and .zshrc for zsh),
     after all PATH/variable settings:
  2. Ensure that there is no 'return' from inside the ~/.bashrc file,
     otherwise rvm may be prevented from working properly.
  3. CLOSE THIS SHELL and open a new one in order to use rvm.
  These three steps basically take care of steps two and three in the on-line installation instructions.
  
• It talks about editing the .bashrc or .bash_profile. This can be done by using the vim text editor at the command line from your home (~) director:
  

  $ cd ~
  $ vim .bashrc
  $ vim .bash_profile

So let's get to it.

Meet RVM's listed dependencies

As noted in the script output above, we should meet RVM's and Ruby's dependencies. I don't care about JRuby or other editions, so I'll ignore their dependencies. Copying the line from above:

$ yum install -y bash curl
Loaded plugins: fastestmirror
You need to be root to perform this command.

I need to be root, so I'll re-run the command, only prepending "sudo" in front of it:

$ sudo yum install -y bash curl
[sudo] password for arsturges: 
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos.mirror.facebook.net
 * extras: mirror.nwresd.org
 * rpmforge: ftp-stud.fht-esslingen.de
 * updates: centos.mirrors.hoobly.com
Setting up Install Process
Package bash-3.2-24.el5.i386 already installed and latest version
Package curl-7.15.5-9.el5_6.2.i386 already installed and latest version
Nothing to do

This tells me those packages are all up-to-date. Next, try the Ruby OS dependencies (again adding "sudo"):

sudo yum install -y gcc-c++ patch readline readline-devel zlib zlib-devel libyaml-devel libffi-devel openssl-devel

This gives me the following output:

$ sudo yum install -y gcc-c++ patch readline readline-devel zlib zlib-devel libyaml-devel libffi-devel openssl-devel
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos.mirror.facebook.net
 * extras: mirror.5ninesolutions.com
 * rpmforge: ftp-stud.fht-esslingen.de
 * updates: centos.mirrors.hoobly.com
Setting up Install Process
Package gcc-c++-4.1.2-50.el5.i386 already installed and latest version
Package patch-2.5.4-31.el5.i386 already installed and latest version
Package readline-5.1-3.el5.i386 already installed and latest version
Package readline-devel-5.1-3.el5.i386 already installed and latest version
Package zlib-1.2.3-3.i386 already installed and latest version
Package zlib-devel-1.2.3-3.i386 already installed and latest version
Package openssl-devel-0.9.8e-12.el5_5.7.i386 already installed and latest version
Resolving Dependencies
--> Running transaction check
---> Package libffi-devel.i386 0:3.0.9-1.el5.rf set to be updated
--> Processing Dependency: libffi = 3.0.9-1.el5.rf for package: libffi-devel
--> Processing Dependency: libffi.so.5 for package: libffi-devel
---> Package libyaml-devel.i386 0:0.1.3-1.el5.rf set to be updated
--> Processing Dependency: libyaml = 0.1.3-1.el5.rf for package: libyaml-devel
--> Processing Dependency: libyaml-0.so.2 for package: libyaml-devel
--> Running transaction check
---> Package libffi.i386 0:3.0.9-1.el5.rf set to be updated
---> Package libyaml.i386 0:0.1.3-1.el5.rf set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================
 Package               Arch         Version                  Repository        Size
====================================================================================
Installing:
 libffi-devel          i386         3.0.9-1.el5.rf           rpmforge          16 k
 libyaml-devel         i386         0.1.3-1.el5.rf           rpmforge          12 k
Installing for dependencies:
 libffi                i386         3.0.9-1.el5.rf           rpmforge          87 k
 libyaml               i386         0.1.3-1.el5.rf           rpmforge         115 k

Transaction Summary
====================================================================================
Install       4 Package(s)
Upgrade       0 Package(s)

Total download size: 230 k
Downloading Packages:
(1/4): libyaml-devel-0.1.3-1.el5.rf.i386.rpm                 |  12 kB     00:00     
(2/4): libffi-devel-3.0.9-1.el5.rf.i386.rpm                  |  16 kB     00:00     
(3/4): libffi-3.0.9-1.el5.rf.i386.rpm                        |  87 kB     00:00     
(4/4): libyaml-0.1.3-1.el5.rf.i386.rpm                       | 115 kB     00:00     
------------------------------------------------------------------------------------
Total                                                84 kB/s | 230 kB     00:02     
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing     : libyaml                                                      1/4 
  Installing     : libffi                                                       2/4 
  Installing     : libffi-devel                                                 3/4 
  Installing     : libyaml-devel                                                4/4 

Installed:
  libffi-devel.i386 0:3.0.9-1.el5.rf       libyaml-devel.i386 0:0.1.3-1.el5.rf      

Dependency Installed:
  libffi.i386 0:3.0.9-1.el5.rf             libyaml.i386 0:0.1.3-1.el5.rf            

Complete!

As you can see, it found most of those packages installed and up-to-date, but it did find some new ones as well, which it installed without issue.

The next two lines I'll run together, as Wayne wrote them, split by a semi-collon:

sudo yum install -y make bzip2; sudo yum install -y iconv-devel

Without listing all the output, I'll say that these packages were found to be up-to-date.

Now that we've met all the listed dependencies, we continue by following the enumerated instructions. Step one is: Place the folowing line at the end of your shell's loading files (.bashrc or .bash_profile for bash and .zshrc for zsh), after all PATH/variable settings:

[[ -s "/home/arsturges/.rvm/scripts/rvm" ]] && source "/home/arsturges/.rvm/scripts/rvm"  # This loads RVM into a shell session.

The .bashrc file is run each time you open a Bash shell terminal window, and since we'll be running RVM througha Bash shell, this line will be invoked before we ever need to use RVM. To add it, simply copy it, open the file with vim, and paste it in as the last line in the file.

$ cd ~
$ vim .bashrc

Here's what my file looks like after I've added the line:

# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi

# User specific aliases and functions
[[ -s "/home/arsturges/.rvm/scripts/rvm" ]] && source "/home/arsturges/.rvm/scripts/rvm"  # This loads RVM into a shell session.

The next step, according the instructions:

Ensure that there is no 'return' from inside the ~/.bashrc file

The script instructs us to "Ensure that there is no 'return' from inside the ~/.bashrc file,
otherwise rvm may be prevented from working properly." I checked mine and saw no such thing, so I'll continue.

Restart the shell

The next step: "CLOSE THIS SHELL and open a new one in order to use rvm." Just close and re-open the terminal window.

That's it for the script instructions. To see what to do next, we return to the on-line instructions, where we've basically completed step three. The next thing to do is see if it worked.

Test the installation

Type the following command, which should return the phrase "rvm is a function":

$ type rvm | head -1
rvm is a function

I have no idea what that does or why it worked, but as you can see, it worked for me. Now run rvm notes as recommended:

$ rvm notes


Notes for Linux ( CentOS release 5.6 (Final) )

NOTE: 'ruby' represents Matz's Ruby Interpreter (MRI) (1.8.X, 1.9.X)
             This is the *original* / standard Ruby Language Interpreter
      'ree'  represents Ruby Enterprise Edition
      'rbx'  represents Rubinius

bash >= 3.2 is required
curl is required
git is required (>= 1.7 recommended)
patch is required (for ree and some ruby-head's).

If you wish to install rbx and/or Ruby 1.9 head (MRI) (eg. 1.9.2-head),
then you must install and use rvm 1.8.7 first.

If you wish to have the 'pretty colors' again,
  set 'export rvm_pretty_print_flag=1' in ~/.rvmrc.

dependencies:
  # For RVM
  rvm: yum install -y bash curl git # NOTE: For git you need the EPEL repository enabled

  # For Ruby (MRI & Ree) you should install the following OS dependencies:
  ruby: yum install -y gcc-c++ patch readline readline-devel zlib zlib-devel libyaml-devel libffi-devel openssl-devel ;
        yum install -y make bzip2 ;
        yum install -y iconv-devel # NOTE: For centos 5.4 final iconv-devel might not be available :(

  # For JRuby (if you wish to use it) you will need:
  jruby: yum install -y java

For rbx (Rubinius) more then 600MB of free RAM required.

This just reminds us of the dependencies we've already met for Ruby, and of the dependencies we'd need to meet if we wanted to run JRuby or rbx. Otherwise, we should be done with the installation, and ready to use RVM on CentOS 5.6.

Try it out

$ rvm list known

# MRI Rubies
[ruby-]1.8.6[-p420]
[ruby-]1.8.6-head
[ruby-]1.8.7[-p334]
[ruby-]1.8.7-head
[ruby-]1.9.1-p378
[ruby-]1.9.1[-p431]
[ruby-]1.9.1-head
[ruby-]1.9.2[-p180]
[ruby-]1.9.2-head
ruby-head

# GoRuby
goruby

# JRuby
jruby-1.2.0
jruby-1.3.1
jruby-1.4.0
jruby-1.6.0
jruby-1.6.1
jruby[-1.6.2]
jruby-head

# Rubinius
rbx-1.0.1
rbx-1.1.0
rbx-1.1.1
rbx-1.2.0
rbx-1.2.1
rbx-1.2.2
rbx-1.2.3
rbx[-head]

# Ruby Enterprise Edition
ree-1.8.6
ree[-1.8.7][-2011.03]
ree-1.8.6-head
ree-1.8.7-head

# Kiji
kiji

# MagLev
maglev[-25913]
maglev-head

# Mac OS X Snow Leopard Only
macruby[-0.10]
macruby-nightly
macruby-head

# IronRuby -- Not implemented yet.
ironruby-0.9.3
ironruby-1.0-rc2
ironruby-head

This shows a list of all the versions of Ruby RVM is capable of installing. For the moment, I just want a standard installation of the latest Ruby version, which is Ruby 1.9.2:

$ rvm install 1.9.2
Installing Ruby from source to: /home/arsturges/.rvm/rubies/ruby-1.9.2-p180, this may take a while depending on your cpu(s)...

ruby-1.9.2-p180 - #fetching 
ruby-1.9.2-p180 - #downloading ruby-1.9.2-p180, this may take a while depending on your connection...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 8609k  100 8609k    0     0   828k      0  0:00:10  0:00:10 --:--:-- 1078k
ruby-1.9.2-p180 - #extracting ruby-1.9.2-p180 to /home/arsturges/.rvm/src/ruby-1.9.2-p180
ruby-1.9.2-p180 - #extracted to /home/arsturges/.rvm/src/ruby-1.9.2-p180
Fetching yaml-0.1.3.tar.gz to /home/arsturges/.rvm/archives
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  455k  100  455k    0     0   586k      0 --:--:-- --:--:-- --:--:--  730k
--no-same-owner
Configuring yaml in /home/arsturges/.rvm/src/yaml-0.1.3.
Compiling yaml in /home/arsturges/.rvm/src/yaml-0.1.3.
Installing yaml to /home/arsturges/.rvm/usr
ruby-1.9.2-p180 - #configuring 
ruby-1.9.2-p180 - #compiling 
ruby-1.9.2-p180 - #installing 
Removing old Rubygems files...
Installing rubygems dedicated to ruby-1.9.2-p180...
Retrieving rubygems-1.6.2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  236k  100  236k    0     0  1053k      0 --:--:-- --:--:-- --:--:-- 9025k
Extracting rubygems-1.6.2 ...
Installing rubygems for /home/arsturges/.rvm/rubies/ruby-1.9.2-p180/bin/ruby
Installation of rubygems completed successfully.
ruby-1.9.2-p180 - adjusting #shebangs for (gem irb erb ri rdoc testrb rake).
ruby-1.9.2-p180 - #importing default gemsets (/home/arsturges/.rvm/gemsets/)
Install of ruby-1.9.2-p180 - #complete 

This took about 10 minutes on this particular server. Now that it's done and installed, play around with it:

$ ruby -v
ruby 1.8.7 (2009-12-24 patchlevel 248) [i686-linux], MBARI 0x8770, Ruby Enterprise Edition 2010.01
$ which ruby
/usr/local/bin/ruby
$ rvm use 1.9.2
Using /home/arsturges/.rvm/gems/ruby-1.9.2-p180
$ ruby -v
ruby 1.9.2p180 (2011-02-18 revision 30909) [i686-linux]
$ which ruby
~/.rvm/rubies/ruby-1.9.2-p180/bin/ruby

As you can see, it's important to use the "rvm use" command to actually switch to a version of Ruby installed by RVM.

Conclusion

We followed the on-line RVM installation instructions on CentOS 5.6, first running the bash script, then following the instructions output by the script, including installing dependencies and adding one line to the .bashrc file. We did this without needing to upgrade git beyond what the CentOS default repositories offer.

Summary of commands for power users

$ cat /etc/issue
$ git --version
$ bash --version
$ bash < <(curl -sk https://rvm.beginrescueend.com/install/rvm)
$ sudo yum install -y bash curl git ; 
sudo yum install -y gcc-c++ patch readline readline-devel zlib zlib-devel libyaml-devel libffi-devel openssl-devel ; 
sudo yum install -y make bzip2 ; 
yum install -y iconv-devel

Now place the following line in your .bashrc (but copy the one generated by your own installation, not mine):

[[ -s "/home/arsturges/.rvm/scripts/rvm" ]] && source "/home/arsturges/.rvm/scripts/rvm"  # This loads RVM into a shell session.

Next, make sure that there is no 'return' from inside the ~/.bashrc file.
Now restart the shell, and test the installation:

$ type rvm | head -1
rvm is a function

This should return the phrase "rvm is a function".

$ rvm notes
$ rvm list known
$ rvm install 1.9.2

Now try it out:

$ ruby -v
ruby 1.8.7 (2009-12-24 patchlevel 248) [i686-linux], MBARI 0x8770, Ruby Enterprise Edition 2010.01
$ which ruby
/usr/local/bin/ruby
$ rvm use 1.9.2
Using /home/arsturges/.rvm/gems/ruby-1.9.2-p180
$ ruby -v
ruby 1.9.2p180 (2011-02-18 revision 30909) [i686-linux]
$ which ruby
~/.rvm/rubies/ruby-1.9.2-p180/bin/ruby

Thank you to Wayne Seguin for RVM, and for all who contribute code, time, and money to open source software. Donate to RVM to show your support.
~fin~