Saturday, September 24, 2011

Installing Nokogiri with RVM on Ubuntu

I'm using Ruby 1.9.2, Ubuntu 10.10, and RVM 1.6.14.

The Nokogiri installation page provides instructions for Ubuntu/Debian users:
# ruby developer packages
sudo apt-get install ruby1.8-dev ruby1.8 ri1.8 rdoc1.8 irb1.8
sudo apt-get install libreadline-ruby1.8 libruby1.8 libopenssl-ruby

# nokogiri requirements
sudo apt-get install libxslt-dev libxml2-dev
sudo gem install nokogiri

But what if I'm using Ruby 1.9 instead of 1.8? The Nokogiri GitHub page says it requires either Ruby 1.8 or 1.9. I guess the official website hasn't been updated to reflect that.

Even so, using RVM, I have a few options. First, I could create a new Ruby environment and gemset with RVM, use Ruby 1.8 explicitly, and follow the instructions as provided by Nokogiri. My second option is to create a clean gemset and use Ruby 1.9 (which is faster than 1.8), and experiment in the safe confines on the gemset. Let's do the second option.

Create a New RVM Gemset

First, create a new gemset and call it "nokogiri":
$ rvm gemset create nokogiri
'nokogiri' gemset created (/home/andy/.rvm/gems/ruby-1.9.2-p180@nokogiri).
Then switch into that gemset:
$ rvm use 1.9.2@nokogiri
Using /home/andy/.rvm/gems/ruby-1.9.2-p180 with gemset nokogiri
Then confirm the Ruby version:
$ ruby -v
ruby 1.9.2p180 (2011-02-18 revision 30909) [i686-linux]

NB: Normally we wouldn't create a whole gemset just to install one gem. RVM is meant to create a whole environment where you would install many gems. But here we're just using it as a safe playground to see how the Nokogiri installation goes.

Now let's just follow Nokogiri's installation instructions, as copied above:

Install Nokogiri and Dependencies


$ sudo apt-get install libxml2 libxml2-dev libxslt libxslt-dev
[sudo] password for andy: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Note, selecting 'libxslt1-dev' instead of 'libxslt-dev'
E: Unable to locate package libxslt
I have no idea why it can't find the package "libxslt". Instead of worrying about that, I'm going to install the dependencies listed on tenderlove's Nokogiri GitHub page:
$ sudo apt-get install libxslt-dev libxml2-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Note, selecting 'libxslt1-dev' instead of 'libxslt-dev'
libxslt1-dev is already the newest version.
libxml2-dev is already the newest version.
That appears to have worked well enough, so I press on. The next step is install the actual Nokogiri gem. The instructions say to use "sudo gem install nokogiri", but because I'm using RVM, I drop the "sudo" part:
$ gem install nokogiri
Fetching: nokogiri-1.5.0.gem (100%)
Building native extensions.  This could take a while...
Successfully installed nokogiri-1.5.0
1 gem installed
Installing ri documentation for nokogiri-1.5.0...
Installing RDoc documentation for nokogiri-1.5.0...
A quick "gem list" shows that it's indeed there:
$ gem list

*** LOCAL GEMS ***

nokogiri (1.5.0)

Test It Out

Now how do we know that it installed successfully and actually works? Let's jump into an Interactive Ruby (IRb) session and try it out, using the "synopsis" provided by tenderlove on the Nokogiri GitHub page.
> require 'nokogiri'
 => true
> require 'open-uri'
 => true 
>doc = Nokogiri::HTML(open('http://www.google.com/search?q=nokogiri'))
[output truncated]
> doc.css('h3.r a.l').each do |link|
> puts link.content
?>   end
Nokogiri
Tutorials
Installation
Nokogiri::XML::Node
Nokogiri::HTML::Document
Parsing an HTML / XML ...
Nokogiri::XML::Document
tenderlove/nokogiri - GitHub
Nokogiri - GitHub
Nokogiri Is Released - Tender Lovemaking
Getting Started with Nokogiri | Engine Yard Ruby on Rails Blog
nokogiri | RubyGems.org | your community gem host
Nokogiri: A Faster, Better HTML and XML Parser for Ruby (than ...
#190 Screen Scraping with Nokogiri -  RailsCasts
RubyForge: nokogiri: Project Info
Nokogiri (project) - Wikipedia, the free encyclopedia
 => 0 

Conclusion

Nokogiri is incredibly easy to install. It works with Ruby 1.8 or 1.9. Simply install the necessary package dependencies, and then don't be afraid to include the Nokogiri gem in any RVM gemset. It should just work.

Special thanks to Aaron Paterson, aka Tenderlove, for his work on Nokogiri, and Wayne Seguin for his work on RVM.

Tuesday, June 7, 2011

Installing autotest with Rails 3.1 and RSpec on Ubuntu 10.10

My environment:
RVM 1.6.14
Ubuntu 10.10
Ruby 1.9.2
Rails 3.1.rc1
RSpec 2.6.4

First install the autotest gem (currently version 4.4.6). Since I'm in a project gemset (through RVM), I don't need "sudo":

$ gem install autotest

Next install the Rails helper gem for autotest (currently version 4.1.2):
$ gem install autotest-rails-pure

Then install the libnotify-bin package (currently version 0.5.0):
$ sudo apt-get install libnotify-bin

Then install the autotest-notification gem (currently version 2.3.1)
$ gem install autotest-notification

This gives you the executable "an-install", which you should to run:
$ an-install

Now start the regular autotest gem:
$ autotest

That's it. Now you should get automatic desktop notification of passing and failing tests each time you change a file covered by a test. For complete instructions, see the README at Carlos Brando's autotest-notification gem.

Wednesday, June 1, 2011

Installing RVM (Ruby Version Manager) on CentOS 5.6

In this blog post I explain how to install Wayne Seguin's RVM (Ruby Version Manager) on CentOS 5.6. I will be following the installation instructions for a "Single-User Installation as a standard user". I already have installed git 1.6 and Ruby 1.8.7.

What CentOS version are you running?

First, confirm what version of CentOS you're running:
$ cat /etc/issue
CentOS release 5.6 (Final)

As you can see, I'm running 5.6, but these instructions may work for earlier versions.

What git version are you running?

The RVM installation instructions recommend having git version 1.7 or later. What do I have?
$ git --version
git version 1.6.1
I'm okay with this for the moment. The standard CentOS repositories do not have a newer version; in the CentOS universe, this is the latest version of Git (although you could get a newer version by adding a different repo). If I run into trouble later, I'll have this potential git upgrade as a possible solution.

What terminal are you running?

I believe the RVM installation instructions assume you are using Bash as a terminal. Check what terminal you are using by running the following command:
$ ps -p$$ -ocmd=
ps -p$$ -ocmd=
bash -v
That tells me I'm running bash. Confirm by asking it for the specific version:
$ bash --version
bash --version
GNU bash, version 3.2.25(1)-release (i686-redhat-linux-gnu)
Copyright (C) 2005 Free Software Foundation, Inc.

Installation: Three steps

The installation instructions list three steps for completing the RVM installation package:
  1. Download and run the RVM installation script
  2. Load RVM into your shell sessions as a function
  3. Reload shell configuration & test
We'll follow each of these in turn.

Download and run the RVM installation script

Copy the following line into your terminal:
$ bash < <(curl -sk https://rvm.beginrescueend.com/install/rvm)
For me, this produced the following output:
Initialized empty Git repository in /home/arsturges/.rvm/src/rvm/.git/
remote: Counting objects: 4930, done.
remote: Compressing objects: 100% (2305/2305), done.
remote: Total 4930 (delta 3194), reused 3552 (delta 1943)
Receiving objects: 100% (4930/4930), 1.60 MiB | 1646 KiB/s, done.
Resolving deltas: 100% (3194/3194), done.

  RVM:  Shell scripts enabling management of multiple ruby environments.
  RTFM: https://rvm.beginrescueend.com/
  HELP: http://webchat.freenode.net/?channels=rvm (#rvm on irc.freenode.net)
  
Installing RVM to /home/arsturges/.rvm/~/.rvm ~/.rvm/src/rvm
~/.rvm/src/rvm

    Correct permissions for base binaries in /home/arsturges/bin...
    Copying manpages into place.


Notes for Linux ( CentOS release 5.6 (Final) )

NOTE: 'ruby' represents Matz's Ruby Interpreter (MRI) (1.8.X, 1.9.X)
             This is the *original* / standard Ruby Language Interpreter
      'ree'  represents Ruby Enterprise Edition
      'rbx'  represents Rubinius

bash >= 3.2 is required
curl is required
git is required (>= 1.7 recommended)
patch is required (for ree and some ruby-head's).

If you wish to install rbx and/or Ruby 1.9 head (MRI) (eg. 1.9.2-head),
then you must install and use rvm 1.8.7 first.

If you wish to have the 'pretty colors' again,
  set 'export rvm_pretty_print_flag=1' in ~/.rvmrc.

dependencies:
  # For RVM
  rvm: yum install -y bash curl git # NOTE: For git you need the EPEL repository enabled

  # For Ruby (MRI & Ree) you should install the following OS dependencies:
  ruby: yum install -y gcc-c++ patch readline readline-devel zlib zlib-devel libyaml-devel libffi-devel openssl-devel ;
        yum install -y make bzip2 ;
        yum install -y iconv-devel # NOTE: For centos 5.4 final iconv-devel might not be available :(

  # For JRuby (if you wish to use it) you will need:
  jruby: yum install -y java

For rbx (Rubinius) more then 600MB of free RAM required.


  You must now complete the install by loading RVM in new shells.

  1) Place the folowing line at the end of your shell's loading files
     (.bashrc or .bash_profile for bash and .zshrc for zsh),
     after all PATH/variable settings:

     [[ -s "/home/arsturges/.rvm/scripts/rvm" ]] && source "/home/arsturges/.rvm/scripts/rvm"  # This loads RVM into a shell session.

     You only need to add this line the first time you install rvm.

  2) Ensure that there is no 'return' from inside the ~/.bashrc file,
     otherwise rvm may be prevented from working properly.

     
  This means that if you see something like:

    '[ -z "$PS1" ] && return'

  then you change this line to:

  if [[ -n "$PS1" ]] ; then

    # ... original content that was below the '&& return' line ...

  fi # <= be sure to close the if at the end of the .bashrc.

  # This is a good place to source rvm v v v
  [[ -s "/home/arsturges/.rvm/scripts/rvm" ]] && source "/home/arsturges/.rvm/scripts/rvm"  # This loads RVM into a shell session.

EOF - This marks the end of the .bashrc file

     Be absolutely *sure* to REMOVE the '&& return'.

     If you wish to DRY up your config you can 'source ~/.bashrc' at the bottom of your .bash_profile.

     Placing all non-interactive (non login) items in the .bashrc,
     including the 'source' line above and any environment settings.

  3) CLOSE THIS SHELL and open a new one in order to use rvm.
  

Installation of RVM to /home/arsturges/.rvm/ is complete.


Andy Sturges,

Thank you very much for using RVM! I sincerely hope that RVM helps to
make your work both easier and more enjoyable.

If you have any questions, issues and/or ideas for improvement please
join#rvm on irc.freenode.net and let me know, note you must register
(http://bit.ly/5mGjlm) and identify (/msg nickserv  ) to
talk, this prevents spambots from ruining our day.

My irc nickname is 'wayneeseguin' and I hang out in #rvm typically

  ~09:00-17:00EDT and again from ~21:00EDT-~23:00EDT

If I do not respond right away, please hang around after asking your
question, I will respond as soon as I am back.  It is best to talk in
#rvm itself as then other users can help out should I be offline.

Be sure to get head often as rvm development happens fast,
you can do this by running 'rvm get head' followed by 'rvm reload'
or opening a new shell

  w⦿‿⦿t

    ~ Wayne

Notice a few things about this comprehensive output:
  • Wayne has customized his installation script (which produced this file) to produce output specific to my linux distribution (CentOS 5.6). This means he provides "yum" commands, instead of, say, Ubuntu's "apt-get".
  • He notes several dependencies:
    dependencies:
      # For RVM
      rvm: yum install -y bash curl git # NOTE: For git you need the EPEL repository enabled
    
      # For Ruby (MRI & Ree) you should install the following OS dependencies:
      ruby: yum install -y gcc-c++ patch readline readline-devel zlib zlib-devel libyaml-devel libffi-devel openssl-devel ;
            yum install -y make bzip2 ;
            yum install -y iconv-devel # NOTE: For centos 5.4 final iconv-devel might not be available :(
    
    We can run these commands as-is, which I'll do below.
  • He gives us three steps to complete the installation:
    1. Place the folowing line at the end of your shell's loading files
      (.bashrc or .bash_profile for bash and .zshrc for zsh),
      after all PATH/variable settings:
    2. Ensure that there is no 'return' from inside the ~/.bashrc file,
      otherwise rvm may be prevented from working properly.
    3. CLOSE THIS SHELL and open a new one in order to use rvm.
    These three steps basically take care of steps two and three in the on-line installation instructions.
  • It talks about editing the .bashrc or .bash_profile. This can be done by using the vim text editor at the command line from your home (~) director:
    $ cd ~
    $ vim .bashrc
    $ vim .bash_profile
So let's get to it.

Meet RVM's listed dependencies

As noted in the script output above, we should meet RVM's and Ruby's dependencies. I don't care about JRuby or other editions, so I'll ignore their dependencies. Copying the line from above:
$ yum install -y bash curl
Loaded plugins: fastestmirror
You need to be root to perform this command.
I need to be root, so I'll re-run the command, only prepending "sudo" in front of it:
$ sudo yum install -y bash curl
[sudo] password for arsturges: 
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos.mirror.facebook.net
 * extras: mirror.nwresd.org
 * rpmforge: ftp-stud.fht-esslingen.de
 * updates: centos.mirrors.hoobly.com
Setting up Install Process
Package bash-3.2-24.el5.i386 already installed and latest version
Package curl-7.15.5-9.el5_6.2.i386 already installed and latest version
Nothing to do
This tells me those packages are all up-to-date. Next, try the Ruby OS dependencies (again adding "sudo"):
sudo yum install -y gcc-c++ patch readline readline-devel zlib zlib-devel libyaml-devel libffi-devel openssl-devel
This gives me the following output:
$ sudo yum install -y gcc-c++ patch readline readline-devel zlib zlib-devel libyaml-devel libffi-devel openssl-devel
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos.mirror.facebook.net
 * extras: mirror.5ninesolutions.com
 * rpmforge: ftp-stud.fht-esslingen.de
 * updates: centos.mirrors.hoobly.com
Setting up Install Process
Package gcc-c++-4.1.2-50.el5.i386 already installed and latest version
Package patch-2.5.4-31.el5.i386 already installed and latest version
Package readline-5.1-3.el5.i386 already installed and latest version
Package readline-devel-5.1-3.el5.i386 already installed and latest version
Package zlib-1.2.3-3.i386 already installed and latest version
Package zlib-devel-1.2.3-3.i386 already installed and latest version
Package openssl-devel-0.9.8e-12.el5_5.7.i386 already installed and latest version
Resolving Dependencies
--> Running transaction check
---> Package libffi-devel.i386 0:3.0.9-1.el5.rf set to be updated
--> Processing Dependency: libffi = 3.0.9-1.el5.rf for package: libffi-devel
--> Processing Dependency: libffi.so.5 for package: libffi-devel
---> Package libyaml-devel.i386 0:0.1.3-1.el5.rf set to be updated
--> Processing Dependency: libyaml = 0.1.3-1.el5.rf for package: libyaml-devel
--> Processing Dependency: libyaml-0.so.2 for package: libyaml-devel
--> Running transaction check
---> Package libffi.i386 0:3.0.9-1.el5.rf set to be updated
---> Package libyaml.i386 0:0.1.3-1.el5.rf set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================
 Package               Arch         Version                  Repository        Size
====================================================================================
Installing:
 libffi-devel          i386         3.0.9-1.el5.rf           rpmforge          16 k
 libyaml-devel         i386         0.1.3-1.el5.rf           rpmforge          12 k
Installing for dependencies:
 libffi                i386         3.0.9-1.el5.rf           rpmforge          87 k
 libyaml               i386         0.1.3-1.el5.rf           rpmforge         115 k

Transaction Summary
====================================================================================
Install       4 Package(s)
Upgrade       0 Package(s)

Total download size: 230 k
Downloading Packages:
(1/4): libyaml-devel-0.1.3-1.el5.rf.i386.rpm                 |  12 kB     00:00     
(2/4): libffi-devel-3.0.9-1.el5.rf.i386.rpm                  |  16 kB     00:00     
(3/4): libffi-3.0.9-1.el5.rf.i386.rpm                        |  87 kB     00:00     
(4/4): libyaml-0.1.3-1.el5.rf.i386.rpm                       | 115 kB     00:00     
------------------------------------------------------------------------------------
Total                                                84 kB/s | 230 kB     00:02     
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing     : libyaml                                                      1/4 
  Installing     : libffi                                                       2/4 
  Installing     : libffi-devel                                                 3/4 
  Installing     : libyaml-devel                                                4/4 

Installed:
  libffi-devel.i386 0:3.0.9-1.el5.rf       libyaml-devel.i386 0:0.1.3-1.el5.rf      

Dependency Installed:
  libffi.i386 0:3.0.9-1.el5.rf             libyaml.i386 0:0.1.3-1.el5.rf            

Complete!
As you can see, it found most of those packages installed and up-to-date, but it did find some new ones as well, which it installed without issue.

The next two lines I'll run together, as Wayne wrote them, split by a semi-collon:
sudo yum install -y make bzip2; sudo yum install -y iconv-devel
Without listing all the output, I'll say that these packages were found to be up-to-date.

Now that we've met all the listed dependencies, we continue by following the enumerated instructions. Step one is: Place the folowing line at the end of your shell's loading files (.bashrc or .bash_profile for bash and .zshrc for zsh), after all PATH/variable settings:
[[ -s "/home/arsturges/.rvm/scripts/rvm" ]] && source "/home/arsturges/.rvm/scripts/rvm"  # This loads RVM into a shell session.
The .bashrc file is run each time you open a Bash shell terminal window, and since we'll be running RVM througha Bash shell, this line will be invoked before we ever need to use RVM. To add it, simply copy it, open the file with vim, and paste it in as the last line in the file.
$ cd ~
$ vim .bashrc
Here's what my file looks like after I've added the line:
# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi

# User specific aliases and functions
[[ -s "/home/arsturges/.rvm/scripts/rvm" ]] && source "/home/arsturges/.rvm/scripts/rvm"  # This loads RVM into a shell session.
The next step, according the instructions:

Ensure that there is no 'return' from inside the ~/.bashrc file

The script instructs us to "Ensure that there is no 'return' from inside the ~/.bashrc file,
otherwise rvm may be prevented from working properly." I checked mine and saw no such thing, so I'll continue.

Restart the shell

The next step: "CLOSE THIS SHELL and open a new one in order to use rvm." Just close and re-open the terminal window.

That's it for the script instructions. To see what to do next, we return to the on-line instructions, where we've basically completed step three. The next thing to do is see if it worked.

Test the installation

Type the following command, which should return the phrase "rvm is a function":
$ type rvm | head -1
rvm is a function
I have no idea what that does or why it worked, but as you can see, it worked for me. Now run rvm notes as recommended:
$ rvm notes


Notes for Linux ( CentOS release 5.6 (Final) )

NOTE: 'ruby' represents Matz's Ruby Interpreter (MRI) (1.8.X, 1.9.X)
             This is the *original* / standard Ruby Language Interpreter
      'ree'  represents Ruby Enterprise Edition
      'rbx'  represents Rubinius

bash >= 3.2 is required
curl is required
git is required (>= 1.7 recommended)
patch is required (for ree and some ruby-head's).

If you wish to install rbx and/or Ruby 1.9 head (MRI) (eg. 1.9.2-head),
then you must install and use rvm 1.8.7 first.

If you wish to have the 'pretty colors' again,
  set 'export rvm_pretty_print_flag=1' in ~/.rvmrc.

dependencies:
  # For RVM
  rvm: yum install -y bash curl git # NOTE: For git you need the EPEL repository enabled

  # For Ruby (MRI & Ree) you should install the following OS dependencies:
  ruby: yum install -y gcc-c++ patch readline readline-devel zlib zlib-devel libyaml-devel libffi-devel openssl-devel ;
        yum install -y make bzip2 ;
        yum install -y iconv-devel # NOTE: For centos 5.4 final iconv-devel might not be available :(

  # For JRuby (if you wish to use it) you will need:
  jruby: yum install -y java

For rbx (Rubinius) more then 600MB of free RAM required.

This just reminds us of the dependencies we've already met for Ruby, and of the dependencies we'd need to meet if we wanted to run JRuby or rbx. Otherwise, we should be done with the installation, and ready to use RVM on CentOS 5.6.

Try it out

$ rvm list known

# MRI Rubies
[ruby-]1.8.6[-p420]
[ruby-]1.8.6-head
[ruby-]1.8.7[-p334]
[ruby-]1.8.7-head
[ruby-]1.9.1-p378
[ruby-]1.9.1[-p431]
[ruby-]1.9.1-head
[ruby-]1.9.2[-p180]
[ruby-]1.9.2-head
ruby-head

# GoRuby
goruby

# JRuby
jruby-1.2.0
jruby-1.3.1
jruby-1.4.0
jruby-1.6.0
jruby-1.6.1
jruby[-1.6.2]
jruby-head

# Rubinius
rbx-1.0.1
rbx-1.1.0
rbx-1.1.1
rbx-1.2.0
rbx-1.2.1
rbx-1.2.2
rbx-1.2.3
rbx[-head]

# Ruby Enterprise Edition
ree-1.8.6
ree[-1.8.7][-2011.03]
ree-1.8.6-head
ree-1.8.7-head

# Kiji
kiji

# MagLev
maglev[-25913]
maglev-head

# Mac OS X Snow Leopard Only
macruby[-0.10]
macruby-nightly
macruby-head

# IronRuby -- Not implemented yet.
ironruby-0.9.3
ironruby-1.0-rc2
ironruby-head
This shows a list of all the versions of Ruby RVM is capable of installing. For the moment, I just want a standard installation of the latest Ruby version, which is Ruby 1.9.2:
$ rvm install 1.9.2
Installing Ruby from source to: /home/arsturges/.rvm/rubies/ruby-1.9.2-p180, this may take a while depending on your cpu(s)...

ruby-1.9.2-p180 - #fetching 
ruby-1.9.2-p180 - #downloading ruby-1.9.2-p180, this may take a while depending on your connection...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 8609k  100 8609k    0     0   828k      0  0:00:10  0:00:10 --:--:-- 1078k
ruby-1.9.2-p180 - #extracting ruby-1.9.2-p180 to /home/arsturges/.rvm/src/ruby-1.9.2-p180
ruby-1.9.2-p180 - #extracted to /home/arsturges/.rvm/src/ruby-1.9.2-p180
Fetching yaml-0.1.3.tar.gz to /home/arsturges/.rvm/archives
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  455k  100  455k    0     0   586k      0 --:--:-- --:--:-- --:--:--  730k
--no-same-owner
Configuring yaml in /home/arsturges/.rvm/src/yaml-0.1.3.
Compiling yaml in /home/arsturges/.rvm/src/yaml-0.1.3.
Installing yaml to /home/arsturges/.rvm/usr
ruby-1.9.2-p180 - #configuring 
ruby-1.9.2-p180 - #compiling 
ruby-1.9.2-p180 - #installing 
Removing old Rubygems files...
Installing rubygems dedicated to ruby-1.9.2-p180...
Retrieving rubygems-1.6.2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  236k  100  236k    0     0  1053k      0 --:--:-- --:--:-- --:--:-- 9025k
Extracting rubygems-1.6.2 ...
Installing rubygems for /home/arsturges/.rvm/rubies/ruby-1.9.2-p180/bin/ruby
Installation of rubygems completed successfully.
ruby-1.9.2-p180 - adjusting #shebangs for (gem irb erb ri rdoc testrb rake).
ruby-1.9.2-p180 - #importing default gemsets (/home/arsturges/.rvm/gemsets/)
Install of ruby-1.9.2-p180 - #complete 
This took about 10 minutes on this particular server. Now that it's done and installed, play around with it:
$ ruby -v
ruby 1.8.7 (2009-12-24 patchlevel 248) [i686-linux], MBARI 0x8770, Ruby Enterprise Edition 2010.01
$ which ruby
/usr/local/bin/ruby
$ rvm use 1.9.2
Using /home/arsturges/.rvm/gems/ruby-1.9.2-p180
$ ruby -v
ruby 1.9.2p180 (2011-02-18 revision 30909) [i686-linux]
$ which ruby
~/.rvm/rubies/ruby-1.9.2-p180/bin/ruby
As you can see, it's important to use the "rvm use" command to actually switch to a version of Ruby installed by RVM.

Conclusion

We followed the on-line RVM installation instructions on CentOS 5.6, first running the bash script, then following the instructions output by the script, including installing dependencies and adding one line to the .bashrc file. We did this without needing to upgrade git beyond what the CentOS default repositories offer.

Summary of commands for power users

$ cat /etc/issue
$ git --version
$ bash --version
$ bash < <(curl -sk https://rvm.beginrescueend.com/install/rvm)
$ sudo yum install -y bash curl git ; 
sudo yum install -y gcc-c++ patch readline readline-devel zlib zlib-devel libyaml-devel libffi-devel openssl-devel ; 
sudo yum install -y make bzip2 ; 
yum install -y iconv-devel
Now place the following line in your .bashrc (but copy the one generated by your own installation, not mine):

[[ -s "/home/arsturges/.rvm/scripts/rvm" ]] && source "/home/arsturges/.rvm/scripts/rvm"  # This loads RVM into a shell session.
Next, make sure that there is no 'return' from inside the ~/.bashrc file.
Now restart the shell, and test the installation:
$ type rvm | head -1
rvm is a function
This should return the phrase "rvm is a function".
$ rvm notes
$ rvm list known
$ rvm install 1.9.2
Now try it out:
$ ruby -v
ruby 1.8.7 (2009-12-24 patchlevel 248) [i686-linux], MBARI 0x8770, Ruby Enterprise Edition 2010.01
$ which ruby
/usr/local/bin/ruby
$ rvm use 1.9.2
Using /home/arsturges/.rvm/gems/ruby-1.9.2-p180
$ ruby -v
ruby 1.9.2p180 (2011-02-18 revision 30909) [i686-linux]
$ which ruby
~/.rvm/rubies/ruby-1.9.2-p180/bin/ruby
Thank you to Wayne Seguin for RVM, and for all who contribute code, time, and money to open source software. Donate to RVM to show your support.
~fin~

Monday, May 23, 2011

Installing RVM (Ruby Version Manager) on Ubuntu 10.10

Wayne E. Seguin's RVM is the new hotness, so I decided to install it. Here I describe how I did it on my Dell laptop running Ubuntu 10.10. I assume you already have Ruby and git installed. I'll be following the installation instructions on the RVM page for a single-user install, not a multi-user install.

First, make sure you have git installed, and then find out what shell you're using:
$ ps -p$$ -ocmd=
bash -v
This tells me I'm using the Bash shell, which I can confirm by asking Bash explicitly what version it is:
$ bash --version
GNU bash, version 4.1.5(1)-release (i686-pc-linux-gnu)

The RVM installation instructions show three steps to follow:
  1. Download and run the RVM installation script
  2. Load RVM into your shell sessions as a function
  3. Reload shell configuration & test
We'll go through each of these in turn.

Step one: Download and run the RVM installation script

Run the installation script by copying exactly the command specified in the instructions:
$ bash < <(curl -s https://rvm.beginrescueend.com/install/rvm)
curl -s https://rvm.beginrescueend.com/install/rvm
Successfully checked out branch ''
remote: Counting objects: 430, done.
remote: Compressing objects: 100% (361/361), done.
remote: Total 384 (delta 263), reused 0 (delta 0)
Receiving objects: 100% (384/384), 49.60 KiB, done.
Resolving deltas: 100% (263/263), completed with 36 local objects.
From git://github.com/wayneeseguin/rvm
   07698bc..8bbab99  master     -> origin/master
From git://github.com/wayneeseguin/rvm
 * [new tag]         1.6.10     -> 1.6.10
 * [new tag]         1.6.11     -> 1.6.11
 * [new tag]         1.6.12     -> 1.6.12
 * [new tag]         1.6.13     -> 1.6.13
 * [new tag]         1.6.14     -> 1.6.14
 * [new tag]         1.6.6      -> 1.6.6
 * [new tag]         1.6.7      -> 1.6.7
 * [new tag]         1.6.8      -> 1.6.8
 * [new tag]         1.6.9      -> 1.6.9
First, rewinding head to replay your work on top of it...
Fast-forwarded master to 8bbab996123063aaeb5f7025f34aa742fc107b57.
Successfully pulled (rebased) from origin 

  RVM:  Shell scripts enabling management of multiple ruby environments.
  RTFM: https://rvm.beginrescueend.com/
  HELP: http://webchat.freenode.net/?channels=rvm (#rvm on irc.freenode.net)
  
Upgrading the RVM installation in /home/andy/.rvm/~/.rvm ~/.rvm/src/rvm
~/.rvm/src/rvm

    Correct permissions for base binaries in /home/andy/bin...
    Copying manpages into place.


Upgrade Notes

  * rvm_trust_rvmrcs has been changed to rvm_trust_rvmrcs_flag for consistency

  * Ruby package dependency list for your OS is given by:
    rvm notes

  * If you encounter any issues with a ruby 'X' your best bet is to:
    rvm remove X ; rvm install X

  * If you wish to have the 'pretty colors' again, set in ~/.rvmrc:
    export rvm_pretty_print_flag=1

  * If you see the following error message: Unknown alias name: 'default'
    re-set your default ruby, this is due to a change in how default works.


WARNING: You have RUBYOPT set in your current environment.
This may cause rubies to not work as you expect them to as it is not supported
by all of them If errors show up, please try un-setting RUBYOPT first.


Upgrade of RVM in /home/andy/.rvm/ is complete.


Andy Sturges,

Thank you very much for using RVM! I sincerely hope that RVM helps to
make your work both easier and more enjoyable.

If you have any questions, issues and/or ideas for improvement please
join#rvm on irc.freenode.net and let me know, note you must register
(http://bit.ly/5mGjlm) and identify (/msg nickserv  ) to
talk, this prevents spambots from ruining our day.

My irc nickname is 'wayneeseguin' and I hang out in #rvm typically

  ~09:00-17:00EDT and again from ~21:00EDT-~23:00EDT

If I do not respond right away, please hang around after asking your
question, I will respond as soon as I am back.  It is best to talk in
#rvm itself as then other users can help out should I be offline.

Be sure to get head often as rvm development happens fast,
you can do this by running 'rvm get head' followed by 'rvm reload'
or opening a new shell

  w⦿‿⦿t

    ~ Wayne

It appears that everything worked, but we got one big warning:
WARNING: You have RUBYOPT set in your current environment.
This may cause rubies to not work as you expect them to as it is not supported
by all of them If errors show up, please try unsetting RUBYOPT first.
What is RUBYOPT, what is my current environment, and how do I unset it? First, try this:

$ echo $RUBYOPT
rubygems

RUBYOPT is an environment variable that I set a long time ago because I got tired of requiring rubygems in all of my ruby scripts. But I guess it's time to remove it. RubyGems is a gem management system, but it's not the only one; a particular library might prefer to use another one, so by setting it as an environment variable, I force all libraries to use my choice, and those libraries might have issues with that. I had set this environment variable by adding the following line to the file ~./bashrc:

export RUBYOPT="rubygems"
To unset it, I'll just delete or comment out that line, and restart the shell. Now if I try echoing the value of that variable again, I get nothing:

$ echo $RUBYOPT

With that taken care of, I'll run the RVM installation script again, and this time I get a much shorter output with no warnings:
$ bash < <(curl -s https://rvm.beginrescueend.com/install/rvm)
Successfully checked out branch ''
Current branch master is up to date.
Successfully pulled (rebased) from origin 

  RVM:  Shell scripts enabling management of multiple ruby environments.
  RTFM: https://rvm.beginrescueend.com/
  HELP: http://webchat.freenode.net/?channels=rvm (#rvm on irc.freenode.net)
  
Upgrading the RVM installation in /home/andy/.rvm/~/.rvm ~/.rvm/src/rvm
~/.rvm/src/rvm

    Correct permissions for base binaries in /home/andy/bin...
    Copying manpages into place.


Upgrade Notes

  * rvm_trust_rvmrcs has been changed to rvm_trust_rvmrcs_flag for consistency

  * Ruby package dependency list for your OS is given by:
    rvm notes

  * If you encounter any issues with a ruby 'X' your best bet is to:
    rvm remove X ; rvm install X

  * If you wish to have the 'pretty colors' again, set in ~/.rvmrc:
    export rvm_pretty_print_flag=1

  * If you see the following error message: Unknown alias name: 'default'
    re-set your default ruby, this is due to a change in how default works.

Upgrade of RVM in /home/andy/.rvm/ is complete.


Andy Sturges,

Thank you very much for using RVM! I sincerely hope that RVM helps to
make your work both easier and more enjoyable.

If you have any questions, issues and/or ideas for improvement please
join#rvm on irc.freenode.net and let me know, note you must register
(http://bit.ly/5mGjlm) and identify (/msg nickserv  ) to
talk, this prevents spambots from ruining our day.

My irc nickname is 'wayneeseguin' and I hang out in #rvm typically

  ~09:00-17:00EDT and again from ~21:00EDT-~23:00EDT

If I do not respond right away, please hang around after asking your
question, I will respond as soon as I am back.  It is best to talk in
#rvm itself as then other users can help out should I be offline.

Be sure to get head often as rvm development happens fast,
you can do this by running 'rvm get head' followed by 'rvm reload'
or opening a new shell

  w⦿‿⦿t

    ~ Wayne

That's it for step one. Now on to step 2.

Step 2: Load RVM into your shell sessions as a function

The shell session (remember our shell is called "bash," which is the default shipped with Ubuntu) can be changed by modifying the file ~/.bash_profile, which is loaded each time a shell is started. We could edit it with the command:
$ vim ~/.bash_profile
but the RVM installation instructions give us a one-line terminal command that will append the correct line to the correct file, so just do it their way:
$ echo '[[ -s "$HOME/.rvm/scripts/rvm" ]] && . "$HOME/.rvm/scripts/rvm" # Load RVM function' >> ~/.bash_profile
This adds the line "[[ -s "$HOME/.rvm/scripts/rvm" ]]" to the file ~/.bash_profile. That's it for step two.

Step 3. Reload shell configuration & test

As in the instructions, reload your bash profile:
$ source .bash_profile
Then type the following command, after which you hope to see the output "rvm is a function":
$ type rvm | head -1
rvm is a function
This worked for me once, but after I closed and re-opened the terminal, I got the following output:
$ type rvm | head -1
rvm is /home/andy/bin/rvm
The problem, I believe, lies in the difference between .bash_profile and .bashrc: the first runs only when you log in, and the second runs every time you open a terminal (shell) window. If we add the line in question to the file .bashrc, I get the desired output every time I run the type command. We can add line to the .bashrc file by slightly modifying and then re-running the command, replacing ".bash_profile" with ".bashrc":
$ echo '[[ -s "$HOME/.rvm/scripts/rvm" ]] && . "$HOME/.rvm/scripts/rvm" # Load RVM function' >> ~/.bashrc
Now if I run the command type rvm | head -1, I get the correct output each time. Not sure if this is the proper "fix", but there it is. [Edit: Wayne (RVM author) points out that some developers place non-interactive environment setup in .bashrc and then source it from .bash_profile.]

Now run the command rvm notes to see if RVM has any dependencies:
$ rvm notes


Notes for Linux ( DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=10.10
DISTRIB_CODENAME=maverick
DISTRIB_DESCRIPTION="Ubuntu 10.10" )

NOTE: 'ruby' represents Matz's Ruby Interpreter (MRI) (1.8.X, 1.9.X)
             This is the *original* / standard Ruby Language Interpreter
      'ree'  represents Ruby Enterprise Edition
      'rbx'  represents Rubinius

bash >= 3.2 is required
curl is required
git is required (>= 1.7 recommended)
patch is required (for ree and some ruby-head's).

If you wish to install rbx and/or Ruby 1.9 head (MRI) (eg. 1.9.2-head),
then you must install and use rvm 1.8.7 first.

If you wish to have the 'pretty colors' again,
  set 'export rvm_pretty_print_flag=1' in ~/.rvmrc.

dependencies:
# For RVM
  rvm: bash curl git

# For Ruby (MRI & ree)  you should install the following OS dependencies:
  ruby: /usr/bin/apt-get install build-essential bison openssl libreadline6 libreadline6-dev curl git-core zlib1g zlib1g-dev libssl-dev libyaml-dev libsqlite3-0 libsqlite3-dev sqlite3 libxml2-dev libxslt-dev autoconf libc6-dev ncurses-dev

# For JRuby (if you wish to use it) you will need:
  jruby: /usr/bin/apt-get install curl g++ openjdk-6-jre-headless
  jruby-head: /usr/bin/apt-get install ant openjdk-6-jdk

# In addition to ruby: dependencies,
  ruby-head: subversion

# For IronRuby (if you wish to use it) you will need:
  ironruby: /usr/bin/apt-get install curl mono-2.0-devel
As you can see, I'm running Ubuntu 10.10, and it lists the following dependencies:
  • build-essential
  • bison
  • openssl
  • libreadline6
  • libreadline6-dev
  • curl
  • git-core
  • zlib1g
  • zlib1g-dev
  • libssl-dev
  • libyaml-dev
  • libsqlite3-0
  • libsqlite3-dev
  • sqlite3
  • libxml2-dev
  • libxslt-dev
  • autoconf
  • libc6-dev
  • ncurses-dev
That's a lot, but it gives us the full command, and if a package is already installed, then apt-get will just skip it, so we can copy the whole command and run it, prepending "sudo" and removing the "/usr/bin/" part:
$ sudo apt-get install build-essential bison openssl libreadline6 libreadline6-dev curl git-core zlib1g zlib1g-dev libssl-dev libyaml-dev libsqlite3-0 libsqlite3-dev sqlite3 libxml2-dev libxslt-dev autoconf libc6-dev ncurses-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Note, selecting 'libxslt1-dev' instead of 'libxslt-dev'
Note, selecting 'libncurses5-dev' instead of 'ncurses-dev'
build-essential is already the newest version.
build-essential set to manually installed.
curl is already the newest version.
libncurses5-dev is already the newest version.
libncurses5-dev set to manually installed.
libreadline6 is already the newest version.
libreadline6-dev is already the newest version.
libreadline6-dev set to manually installed.
libxslt1-dev is already the newest version.
zlib1g is already the newest version.
zlib1g-dev is already the newest version.
zlib1g-dev set to manually installed.
git-core is already the newest version.
libc6-dev is already the newest version.
libsqlite3-0 is already the newest version.
libsqlite3-dev is already the newest version.
libxml2-dev is already the newest version.
libxml2-dev set to manually installed.
openssl is already the newest version.
sqlite3 is already the newest version.
The following extra packages will be installed:
  automake autotools-dev libyaml-0-2 m4
Suggested packages:
  autoconf2.13 autoconf-archive gnu-standards autoconf-doc libtool gettext bison-doc
The following NEW packages will be installed:
  autoconf automake autotools-dev bison libssl-dev libyaml-0-2 libyaml-dev m4
0 upgraded, 8 newly installed, 0 to remove and 0 not upgraded.
Need to get 4,122kB of archives.
After this operation, 12.9MB of additional disk space will be used.
Do you want to continue [Y/n]? y
Get:1 http://us.archive.ubuntu.com/ubuntu/ maverick/main m4 i386 1.4.14-3 [276kB]
Get:2 http://us.archive.ubuntu.com/ubuntu/ maverick/main autoconf all 2.67-2ubuntu1 [569kB]
Get:3 http://us.archive.ubuntu.com/ubuntu/ maverick/main autotools-dev all 20100122.1 [70.7kB]
Get:4 http://us.archive.ubuntu.com/ubuntu/ maverick/main automake all 1:1.11.1-1 [608kB]
Get:5 http://us.archive.ubuntu.com/ubuntu/ maverick/main bison i386 1:2.4.1.dfsg-3 [468kB]
Get:6 http://us.archive.ubuntu.com/ubuntu/ maverick-updates/main libssl-dev i386 0.9.8o-1ubuntu4.4 [2,013kB]
Get:7 http://us.archive.ubuntu.com/ubuntu/ maverick/main libyaml-0-2 i386 0.1.3-1 [53.8kB]
Get:8 http://us.archive.ubuntu.com/ubuntu/ maverick/main libyaml-dev i386 0.1.3-1 [63.3kB]
Fetched 4,122kB in 3s (1,227kB/s)   
Selecting previously deselected package m4.
(Reading database ... 169680 files and directories currently installed.)
Unpacking m4 (from .../archives/m4_1.4.14-3_i386.deb) ...
Selecting previously deselected package autoconf.
Unpacking autoconf (from .../autoconf_2.67-2ubuntu1_all.deb) ...
Selecting previously deselected package autotools-dev.
Unpacking autotools-dev (from .../autotools-dev_20100122.1_all.deb) ...
Selecting previously deselected package automake.
Unpacking automake (from .../automake_1%3a1.11.1-1_all.deb) ...
Selecting previously deselected package bison.
Unpacking bison (from .../bison_1%3a2.4.1.dfsg-3_i386.deb) ...
Selecting previously deselected package libssl-dev.
Unpacking libssl-dev (from .../libssl-dev_0.9.8o-1ubuntu4.4_i386.deb) ...
Selecting previously deselected package libyaml-0-2.
Unpacking libyaml-0-2 (from .../libyaml-0-2_0.1.3-1_i386.deb) ...
Selecting previously deselected package libyaml-dev.
Unpacking libyaml-dev (from .../libyaml-dev_0.1.3-1_i386.deb) ...
Processing triggers for install-info ...
Processing triggers for man-db ...
Processing triggers for doc-base ...
Processing 1 added doc-base file(s)...
Registering documents with scrollkeeper...
Setting up m4 (1.4.14-3) ...
Setting up autoconf (2.67-2ubuntu1) ...
Setting up autotools-dev (20100122.1) ...
Setting up automake (1:1.11.1-1) ...
update-alternatives: using /usr/bin/automake-1.11 to provide /usr/bin/automake (automake) in auto mode.
Setting up bison (1:2.4.1.dfsg-3) ...
update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode.
Setting up libssl-dev (0.9.8o-1ubuntu4.4) ...
Setting up libyaml-0-2 (0.1.3-1) ...
Setting up libyaml-dev (0.1.3-1) ...
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
As you can see, most of the packages were already installed, but it did find and install some new ones:
  • autoconf
  • automake
  • autotools-dev
  • bison
  • libssl-dev
  • libyaml-0-2
  • libyaml-dev
  • m4
I have no idea what those are, but who cares. They're free. [Edit: Wayne tells me they're for installing head versions of Ruby 1.9]

That's it for setting up RVM. Now we can start using it.

Install a new version of Ruby using RVM

I already had Ruby 1.8.7 installed on my laptop. Now I want to use RVM to install 1.9.2:
$ rvm install 1.9.2
Installing Ruby from source to: /home/andy/.rvm/rubies/ruby-1.9.2-p180, this may take a while depending on your cpu(s)...

ruby-1.9.2-p180 - #fetching 
ruby-1.9.2-p180 - #downloading ruby-1.9.2-p180, this may take a while depending on your connection...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 8609k  100 8609k    0     0   891k      0  0:00:09  0:00:09 --:--:-- 1642k
100 8609k  100 8609k    0     0   871k      0  0:00:09  0:00:09 --:--:--  871kruby-1.9.2-p180 - #extracting ruby-1.9.2-p180 to /home/andy/.rvm/src/ruby-1.9.2-p180
ruby-1.9.2-p180 - #extracted to /home/andy/.rvm/src/ruby-1.9.2-p180
ruby-1.9.2-p180 - #configuring 
ruby-1.9.2-p180 - #compiling 
ruby-1.9.2-p180 - #installing 
Removing old Rubygems files...
Installing rubygems dedicated to ruby-1.9.2-p180...
Retrieving rubygems-1.6.2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  236k  100  236k    0     0  1629k      0 --:--:-- --:--:-- --:--:-- 1957k
Extracting rubygems-1.6.2 ...
Installing rubygems for /home/andy/.rvm/rubies/ruby-1.9.2-p180/bin/ruby
Installation of rubygems completed successfully.
ruby-1.9.2-p180 - adjusting #shebangs for (gem irb erb ri rdoc testrb rake).
ruby-1.9.2-p180 - #importing default gemsets (/home/andy/.rvm/gemsets/)
Install of ruby-1.9.2-p180 - #complete 
So now I have installed Ruby 1.9.2. Next we have to tell RVM to switch to that version of Ruby:
$ which ruby
/usr/bin/ruby
andy@andy-laptop:~$ ruby -v
ruby 1.8.7 (2010-06-23 patchlevel 299) [i686-linux]
andy@andy-laptop:~$ rvm use 1.9.2
Using /home/andy/.rvm/gems/ruby-1.9.2-p180
andy@andy-laptop:~$ which ruby
/home/andy/.rvm/rubies/ruby-1.9.2-p180/bin/ruby
andy@andy-laptop:~$ ruby -v
ruby 1.9.2p180 (2011-02-18 revision 30909) [i686-linux] 

That's it. From here, head over the RVM page and start reading the RVM documentation.

Conclusion and summary for power users

On Ubuntu 10.10 running Ruby 1.8.7 and git, we installed and configured RVM, then used RVM to download and run Ruby 1.9.2. We can switch between Ruby versions by using the command rvm use. We followed the installation instructions exactly, except we added a line to the file .bashrc, instead of to the file .bash_profile, as was instructed. We also had to unset the RUBYOPT variable to remove an installation warning.

Summary of commands for power uses:

$ bash < <(curl -s https://rvm.beginrescueend.com/install/rvm)
$ echo '[[ -s "$HOME/.rvm/scripts/rvm" ]] && . "$HOME/.rvm/scripts/rvm" # Load RVM function' >> ~/.bashrc
$ source .bashrc
$ type rvm | head -1
rvm is a function
$ rvm notes
# For Ruby (MRI & ree)  you should install the following OS dependencies:
  ruby: /usr/bin/apt-get install build-essential bison openssl libreadline6 libreadline6-dev curl git-core zlib1g zlib1g-dev libssl-dev libyaml-dev libsqlite3-0 libsqlite3-dev sqlite3 libxml2-dev libxslt-dev autoconf libc6-dev ncurses-dev
sudo apt-get install build-essential bison openssl libreadline6 libreadline6-dev curl git-core zlib1g zlib1g-dev libssl-dev libyaml-dev libsqlite3-0 libsqlite3-dev sqlite3 libxml2-dev libxslt-dev autoconf libc6-dev ncurses-dev
$ rvm install 1.9.2
$ rvm use 1.9.2
$ ruby -v
ruby 1.9.2p180 (2011-02-18 revision 30909) [i686-linux]
Special thanks go to Wayne Seguin for RVM, and all who contribute code to it. Throw some money Wayne's way if you find RVM useful.
~fin~

Wednesday, May 4, 2011

Installing NCView 2.0beta4 on Ubuntu 10.10

This post explains how I installed David Pierce's Ncview utility (version 2.0beta4) on Ubuntu 10.10. NB: Power users should scroll to the bottom for a summary of commands.

What is Ncview?

According to its author David Pierce of the Scripps Institute of Oceanography, Ncview is a visual browser for NetCDF files.

What is a NetCDF file?

A NetCDF file is a self-describing, machine-independent array file commonly used by the climate science community to store climate data. If you're interested in Ncview, you probably know about NetCDF files.

Why do we need Ncview?

NetCDF files can have dozens of dimensions, hundreds of variables, and millions of data points. Ncview can help a user get a quick picture of what the file holds, how to read it, and where to find the parts of the data in which he or she is interested.

Installing Ncview on Ubuntu 10.10

This post shows how to install Ncview 2.0 beta4 on Ubuntu (10.10) specifically. For instructions on how to install Ncview generally on Linux, see David Pierce's Ncview site.

Get Ncview source files

First we need to download the Ncview source files. Navigate to ftp://cirrus.ucsd.edu/pub/ncview/ and select the most recent package. As of this writing, the most recent package is ncview-2.0beta4.tar.gz, updated on 3/4/2010. Download this tar.gz file and open it with Ubuntu's Archive Manager.

Next, extract the files to your home directory. From Archive Manager, I just click on the "Extract" button, select my home directory, and click on "Extract." This creates the folder "ncview-2.0beta4" in my home directory.

Browse the readme and the installation instructions

In the newly created folder, take a look at the files README and INSTALL. These are general instructions that provide good background. This installation tutorial borrows heavily from these two files.

Installation overview

In a perfect world, i.e. one where all dependencies were met and all files on your local machine were exactly where the installation scripts expected them to be, the installation would follow three steps, entered from the terminal from within the ncview-2.0beta4 directory:
  1. $ ./configure
  2. $ make
  3. $ make install
As such, this is a good starting point, because it just might work. But on my machine, I get the following errors when I try the first step:
andy@andy-laptop:~$ cd ncview-2.0beta4/
andy@andy-laptop:~/ncview-2.0beta4$ ./configure
checking for nc-config... yes
Netcdf library version: netCDF 4.1.1
Netcdf library has version 4 interface present: yes
Netcdf library was compiled with C compiler: gcc
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables... 
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for library containing strerror... none required
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking for style of include used by make... GNU
checking dependency style of gcc... gcc3
checking for X... no
------------------------------------------------------------------------------------
Error, the X libraries and development headers must be installed for ncview to work!
------------------------------------------------------------------------------------
More information: You are trying to compile ncview. The machine you are compiling on
probably already has the X windows *runtime* libraries installed, but to *compile*
a program you need more than just the runtime libraries.  You need what are usually
called the 'development headers', named because they are used when developing or
compiling X windows programs.  The best advice I can give you is to use your package
manager to look for a package whose name is something along the lines of x11-devel,
or xorg-x11-proto-devel, or something along those lines that indicates the package
contains the X windows development headers.  Install that package first, then try to
remake ncview.  
Note: If that still fails, even after you've installed the X windows *development*
headers, then you may be on a machine where the automatic configuration system is not
set up quite as it probably should be.  In that case, you might have to specify the
location of the X libraries and X headers manually.  For example, on some machines
the following will work:
   ./configure  --x-libraries=/usr/lib64 --x-includes=/usr/include/X11
                              ^^^^^^^^^^              ^^^^^^^^^^^^^^^^ these are what
you want to set to reflect the location of files such as libX11.so and X.h on your
particular system.

The script output tells us that it performed about a dozen checks which passed, then reached one it failed (the line "checking for X. . . no"). Then we get the error "the X libraries and development headers must be installed for ncview to work!" Lastly, we get a very thorough message from Ncview author David Pierce advising us how to meet the "X" dependency. As he notes, we need the X windows *development* headers.

Install xorg-dev dependencies

We need the X Windows development headers. The brute force way to do this on Ubuntu is to install the package "xorg-dev," which includes many (all?) development header files related to the X11 windows system. Here's how I did it:
andy@andy-laptop:~$ sudo apt-get install xorg-dev
[sudo] password for andy: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  libdmx-dev libdmx1 libexpat1-dev libfontconfig1-dev libfontenc-dev libfreetype6-dev libfs-dev libice-dev libpciaccess-dev
  libpixman-1-dev libpthread-stubs0 libpthread-stubs0-dev libsm-dev libx11-dev libxau-dev libxaw7-dev libxcb1-dev libxcomposite-dev
  libxcursor-dev libxdamage-dev libxdmcp-dev libxext-dev libxfixes-dev libxfont-dev libxft-dev libxi-dev libxinerama-dev libxkbfile-dev
  libxmu-dev libxmu-headers libxmuu-dev libxpm-dev libxrandr-dev libxrender-dev libxres-dev libxss-dev libxt-dev libxtst-dev libxv-dev
  libxvmc-dev libxxf86dga-dev libxxf86vm-dev x11proto-bigreqs-dev x11proto-composite-dev x11proto-core-dev x11proto-damage-dev
  x11proto-dmx-dev x11proto-dri2-dev x11proto-fixes-dev x11proto-fonts-dev x11proto-gl-dev x11proto-input-dev x11proto-kb-dev
  x11proto-randr-dev x11proto-record-dev x11proto-render-dev x11proto-resource-dev x11proto-scrnsaver-dev x11proto-video-dev
  x11proto-xcmisc-dev x11proto-xext-dev x11proto-xf86bigfont-dev x11proto-xf86dga-dev x11proto-xf86dri-dev x11proto-xf86vidmode-dev
  x11proto-xinerama-dev xserver-xorg-dev xtrans-dev
The following NEW packages will be installed:
  libdmx-dev libdmx1 libexpat1-dev libfontconfig1-dev libfontenc-dev libfreetype6-dev libfs-dev libice-dev libpciaccess-dev
  libpixman-1-dev libpthread-stubs0 libpthread-stubs0-dev libsm-dev libx11-dev libxau-dev libxaw7-dev libxcb1-dev libxcomposite-dev
  libxcursor-dev libxdamage-dev libxdmcp-dev libxext-dev libxfixes-dev libxfont-dev libxft-dev libxi-dev libxinerama-dev libxkbfile-dev
  libxmu-dev libxmu-headers libxmuu-dev libxpm-dev libxrandr-dev libxrender-dev libxres-dev libxss-dev libxt-dev libxtst-dev libxv-dev
  libxvmc-dev libxxf86dga-dev libxxf86vm-dev x11proto-bigreqs-dev x11proto-composite-dev x11proto-core-dev x11proto-damage-dev
  x11proto-dmx-dev x11proto-dri2-dev x11proto-fixes-dev x11proto-fonts-dev x11proto-gl-dev x11proto-input-dev x11proto-kb-dev
  x11proto-randr-dev x11proto-record-dev x11proto-render-dev x11proto-resource-dev x11proto-scrnsaver-dev x11proto-video-dev
  x11proto-xcmisc-dev x11proto-xext-dev x11proto-xf86bigfont-dev x11proto-xf86dga-dev x11proto-xf86dri-dev x11proto-xf86vidmode-dev
  x11proto-xinerama-dev xorg-dev xserver-xorg-dev xtrans-dev
0 upgraded, 69 newly installed, 0 to remove and 0 not upgraded.
Need to get 8,853kB of archives.
After this operation, 26.6MB of additional disk space will be used.
Do you want to continue [Y/n]? Y
This will download and install 27 megabytes worth of files, which we hope includes whatever it is that Ncview needs. Of course we could have tried to identify which handful of files ncview specifically needs and installed just those, but this is easier and faster.

Try ./configure again

After installing the xorg-dev packages, I run the ./configure command again and get the following (truncated) result:
andy@andy-laptop:~/ncview-2.0beta4$ ./configure
checking for nc-config... yes
Netcdf library version: netCDF 4.1.1
[...]
checking for X... libraries , headers 
checking for gethostbyname... yes
[...]
checking udunits2.h usability... no
checking udunits2.h presence... no
checking for udunits2.h... no
[...]
************************************************************************
Note: udunits2 support is NOT enabled, because I could not find the 
location of the udunits2 include file 'udunits2.h' or library file
'libudunits2.a'.  Ncview uses the udunits2 package to format date strings
with units of the form 'days since 1900-01-01'.  If you do not use
these udunits2-standard date formats, then don't worry about the lack
of udunits2 support.  If you DO use udunits2 format date strings, and
you want the udunits2 support, then you must tell me where to find
the udunits2 package by giving arguments to configure, as follows:
  ./configure -with-udunits2_incdir=include_directory -with-udunits2_libdir=library_directory
************************************************************************
checking /usr/local/include/ppm.h usability... no
checking /usr/local/include/ppm.h presence... no
[...]
************************************************************************
Note: the -frames option is NOT enabled, because I could not find the 
location of the PPM include file 'ppm.h' or library file
'libppm.a'.  Ncview uses the ppm package to dump out the frames viewed,
which is an easy way to make an mpeg video of the data if you want.
If you do not want this feature, then don't worry about the lack
of ppm support.  If you DO want this, then you must tell me where to find
the ppm package by giving arguments to configure, as follows:
  ./configure -with-ppm_incdir=include_directory -with-ppm_libdir=library_directory
************************************************************************
checking for a BSD-compatible install... /usr/bin/install -c
configure: creating ./config.status
config.status: creating Makefile
config.status: creating src/Makefile
config.status: creating config.h
config.status: executing depfiles commands
 
----------- Configure Summary ----------
Compiler:
        CC                = gcc
 
UDUNITS:
        UDUNITS2_LIBS     = 
        UDUNITS2_CPPFLAGS = 
        UDUNITS2_LDFLAGS  = 
 
 
NETCDF:
        VERSION          = netCDF 4.1.1
        COMPILER USED    = gcc
        NETCDF_CPPFLAGS  = -I/usr/include
        NETCDF_LDFLAGS   = -L/usr/lib -lnetcdf
        NETCDF_V4        = yes
 
X:
        X_CFLAGS         = 
        X11_LIBS         = -lX11 
        XAW_LIBS         = -lXaw -lXt 
        X_PRE_LIBS       =  -lSM -lICE
        X_LIBS           = 
        X_EXTRA_LIBS     = 
A few things to note:
  1. This time the line "Checking for X ... " succeeded, so we know that installing xorg-dev worked;
  2. It started to fail again when looking for files called "udunits";
  3. Then it started to fail again when it couldn't find PPM files, which we are told are used for making movies of the data.
I want to be able to make movies of the data, so let's tackle the PPM issue.

Install the PPM dependency

The ./configure script output told us it's missing the files ppm.h, among others. Let's figure out what package that belongs to using the program apt-file. I don't have it installed, so I'll install it and then use it to find packages that include the file ppm.h. The following is my truncated commands and output:
andy@andy-laptop:~$ sudo apt-get install apt-file
[sudo] password for andy: 
Reading package lists... Done
[...]
Need to get 508kB of archives.
After this operation, 1,434kB of additional disk space will be used.
Do you want to continue [Y/n]? y
[...]
The system-wide cache is empty. You may want to run 'apt-file update'
as root to update the cache. You can also run 'apt-file update' as
normal user to use a cache in the user's home directory.
And then update the cache as it recommends (this took about 20 minutes with my Comcast Internet connection):
andy@andy-laptop:~$ sudo apt-file update
Downloading complete file http://us.archive.ubuntu.com/ubuntu/dists/maverick/Contents-i386.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 17.4M  100 17.4M    0     0  16574      0  0:18:23  0:18:23 --:--:-- 16731
[...]
Ignoring source without Contents File:
  http://extras.ubuntu.com/ubuntu/dists/maverick/Contents-i386.gz
Ignoring source without Contents File:
  http://dl.google.com/linux/talkplugin/deb/dists/stable/Contents-i386.gz
Now we want to use this program to find the packages with the file ppm.h:
andy@andy-laptop:~$ apt-file search ppm.h
gamgi-doc: /usr/share/doc/gamgi-doc/doc/formats/introduction/ppm.html
grass-doc: /usr/share/doc/grass-doc/html/r.out.ppm.html
gromacs-dev: /usr/include/gromacs/pppm.h
libnetpbm10-dev: /usr/include/libppm.h
libnetpbm10-dev: /usr/include/ppm.h
libnetpbm9-dev: /usr/include/libppm.h
libnetpbm9-dev: /usr/include/ppm.h
libtachyon-dev: /usr/include/tachyon/ppm.h
libtk-img-doc: /usr/share/doc/libtk-img-doc/html/img-ppm.html
tau-examples: /usr/share/doc/tau-examples/examples/opari/c++/ppm.h
tau-examples: /usr/share/doc/tau-examples/examples/openmp/c++/ppm.h
tau-examples: /usr/share/doc/tau-examples/examples/openmp/c/ppm.h
tbb-examples: /usr/share/doc/tbb-examples/examples/parallel_for/tachyon/src/ppm.h
I'm guessing it's the package libnetpbm that we want, so we'll install it:
andy@andy-laptop:~$ sudo apt-get install libnetpbm10-dev
[sudo] password for andy: 
[...]
Setting up libnetpbm10-dev (2:10.0-12.2) ...
Now we can try ./configure again (still heavily truncated output):
andy@andy-laptop:~$ cd ncview-2.0beta4/
andy@andy-laptop:~/ncview-2.0beta4$ ./configure
checking for nc-config... yes
Netcdf library version: netCDF 4.1.1
[...]
checking udunits2.h usability... no
[...]
checking /usr/local/include/ppm.h usability... no
checking /usr/local/include/ppm.h presence... no
checking for /usr/local/include/ppm.h... no
checking /usr/include/ppm.h usability... yes
checking /usr/include/ppm.h presence... yes
checking for /usr/include/ppm.h... yes
checking for ppm_writeppm in -lppm... no
checking for /usr/local/lib/libppm.so... no
checking for /usr/lib/libppm.so... no
checking for /lib/libppm.so... no
checking for /home/andy/lib/libppm.so... no
checking for ppm_writeppm in -lnetpbm... yes
checking for a BSD-compatible install... /usr/bin/install -c
configure: creating ./config.status
config.status: creating Makefile
config.status: creating src/Makefile
config.status: creating config.h
config.status: config.h is unchanged
config.status: executing depfiles commands
 
----------- Configure Summary ----------
Compiler:
        CC                = gcc
 
UDUNITS:
        UDUNITS2_LIBS     = 
        UDUNITS2_CPPFLAGS = 
        UDUNITS2_LDFLAGS  = 
 
 
NETCDF:
        VERSION          = netCDF 4.1.1
        COMPILER USED    = gcc
        NETCDF_CPPFLAGS  = -I/usr/include
        NETCDF_LDFLAGS   = -L/usr/lib -lnetcdf
        NETCDF_V4        = yes
 
X:
        X_CFLAGS         = 
        X11_LIBS         = -lX11 
        XAW_LIBS         = -lXaw -lXt 
        X_PRE_LIBS       =  -lSM -lICE
        X_LIBS           = 
        X_EXTRA_LIBS     = 
This time it found the PPM files it wanted, and we're left with just one remaining error: the UDunits thing. Let's tackle that now.

Install UDUnits dependency

Let's use apt-file search again to find which packages we should install to get our missing files. From the output above, we see that we need the file "udunits2.h", among others. Let's search for it:
andy@andy-laptop:~$ apt-file search udunits2.h
libudunits2-0: /usr/share/doc/libudunits2-0/udunits2.html
libudunits2-dev: /usr/include/udunits2.h
So there's a package called udunits; let's see if there's a broader collection of packages, using aptitude:
andy@andy-laptop:~$ aptitude search udunits
p   libudunits2-0                                                 - Library for handling of units of physical quantities                    
p   libudunits2-dev                                               - Development files for the libunits physical units package               
p   udunits-bin                                                   - Utility for handling units of physical quantities
Let's install all three of these packages:
andy@andy-laptop:~$ sudo apt-get install libudunits2-0 libudunits2-dev udunits-bin

Try ./configure again

Now that we've met the UDUnits and PPM and X Windows dependencies, let's try configuring again:
andy@andy-laptop:~/ncview-2.0beta4$ ./configure
checking for nc-config... yes
Netcdf library version: netCDF 4.1.1
[...]
checking udunits2.h usability... yes
checking udunits2.h presence... yes
[...]
****************************************************************************
Udunits library version 2 support enabled.   
udunits2 dirs: include: .  library: .  libname: udunits2
****************************************************************************
[...] 
----------- Configure Summary ----------
Compiler:
        CC                = gcc
 
UDUNITS:
        UDUNITS2_LIBS     = -ludunits2
        UDUNITS2_CPPFLAGS = -I.
        UDUNITS2_LDFLAGS  = -L. -ludunits2 -lexpat -L. -ludunits2
 
 
NETCDF:
        VERSION          = netCDF 4.1.1
        COMPILER USED    = gcc
        NETCDF_CPPFLAGS  = -I/usr/include
        NETCDF_LDFLAGS   = -L/usr/lib -lnetcdf
        NETCDF_V4        = yes
 
X:
        X_CFLAGS         = 
        X11_LIBS         = -lX11 
        XAW_LIBS         = -lXaw -lXt 
        X_PRE_LIBS       =  -lSM -lICE
        X_LIBS           = 
        X_EXTRA_LIBS     = 
Success. The configure summary shows that it is aware of Compiler, UDUNITS, NETCDF, and X.

Run the make command

The next step is to run the make command.
andy@andy-laptop:~/ncview-2.0beta4$ make
This runs on my machine with copious warnings, but it appears to complete without any fatal errors. Next run make install:
andy@andy-laptop:~/ncview-2.0beta4$ sudo make install

Test the installation

Try testing the installation by running the program:
andy@andy-laptop:~/ncview-2.0beta4$ ncview
Ncview 2.0beta4 David W. Pierce  3 March 2010
Success. That completes the installation of Ncview. It's now ready to be used to browse NetCDF files.

Summary of commands for power users

  1. Download the Ncview source files: ftp://cirrus.ucsd.edu/pub/ncview/ (select most recent package)
  2. Install X.org development headers:
    andy@andy-laptop:~$ sudo apt-get install xorg-dev
  3. Install LibnetPBM development headers:
    andy@andy-laptop:~$ sudo apt-get install libnetpbm10-dev
  4. Install the UDUnits packages:
    andy@andy-laptop:~$ sudo apt-get install libudunits2-0 libudunits2-dev udunits-bin
    
  5. Run the configure script:
    andy@andy-laptop:~/ncview-2.0beta4$ ./configure
  6. Run make:
    andy@andy-laptop:~/ncview-2.0beta4$ make
    
  7. Run make install:
    andy@andy-laptop:~/ncview-2.0beta4$ sudo make install
    
  8. Test the installation:
    andy@andy-laptop:~/ncview-2.0beta4$ ncview
    Ncview 2.0beta4 David W. Pierce  3 March 2010

Sunday, April 17, 2011

Using RubyNetCDF to read NetCDF Files: Part 2

See also Using RubyNetCDF to read NetCDF Files: Part 1.

I'm using Ubuntu Linux, Ruby 1.8.7, NetCDF 3.6, ruby-netcdf-0.6.5.

First, download and open the file as described in Part 1. Start an IRb (Interactive Ruby) session and open the file:

$ irb --simple-prompt
>> require 'rubygems' #this will return false because I already loaded RubyGems
=> false
>> require 'numru/netcdf'
=> true
>> file = NumRu::NetCDF.open("tas_A1.020101-022012.nc")
=> NetCDF:tas_A1.020101-022012.nc

to motive this discussion, drill down to the actual temperature data (the variable is "tas") and look around:
>> file.var("tas").class
=> NumRu::NetCDFVar
>> file.var("tas").get[0]
=> 248.853698730469
>> file.var("tas").get[0].class
=> Float
>> file.var("tas").get.class
=> NArray
A few things to notice, in order:
  • The variable itself is of the class NetCDFVar. No surprises there.
  • The actual value of the very first temperature data point (get[0]) is 248.853698730469.This implies the temperature readings are in Kelvin. I'll also note that it's a little strange to report temperature to 12 decimal places.
  • The actual data is float object, meaning a number with a decimal point.
  • The command get without an index returns an object of class NArray.
To deal with NetCDF objects, we should look in the RubyNetCDF Reference Manual. To deal with NArray objects, we need the NArray Reference Manual. When working with NetCDF data, always be aware of exactly what object class you're working with.

This file (tas_A1.020101-022012.nc) is part of a NOAA climate model. It holds surface temperature data. Each data point has a longitude, a latitude, a time, and a temperature. To begin, I would like to know two things:
  1. how many longitude/latitude points are we talking about, and
  2. for each geographical point, how many temperature readings do we have?

The "tas" NArray Object

First, how many points are we talking about? We can know that by asking how large is the NArray object returned by the get command on the variable "tas":

>> file.var("tas").get.size
=> 3110400
That's 3,110,400. But are those all temperature readings? What exactly are we dealing with? First notice that this NArray object has 3 dimensions:

>> file.var("tas").get.dim
=> 3
Why should temperature data be in a 3-dimensional array? I have no idea. Let's dig deeper. What exactly are the lengths of these 3 dimensions? That's the shape method (of an NArray object):

>> file.var("tas").get.shape
=> [144, 90, 240]
Notice that 144 * 90 * 240 = 3,110,400, so that's where the size of this array comes from. But why those lengths? Who knows. But since we know it's dimensions, we can call individual elements by specifying the index of each dimension:
>> file.var("tas").get[0,0,0]
=> 248.853698730469
>> file.var("tas").get[1,0,0]
=> 248.853637695312
>> file.var("tas").get[82,23,42]
=> 283.665496826172

That's a dead end for me, so let's dive into a different variable.

Time


>> file.dim_names
=> ["lon", "lat", "time", "bnds"]
>> file.dim("time")
=> NetCDFDim:time
>> file.dim("time").length
=> 240
That's a great clue: the length of the variable "time" is 240, the same length as one of the dimensions of the "tas" (temperature) variable. Keep digging:

>> file.var("time").vartype
=> "float"
>> file.var("time").natts
=> 6
>> file.var("time").att_names
=> ["standard_name", "long_name", "units", "axis", "calendar", "bounds"]
>> file.var("time").att("long_name").get
=> "time"
>> file.var("time").att("units").get
=> "days since 0001-01-01 00:00:00"
This is the next important clue: the units of the "time" variable are days since...the beginning (whenever that is). Next, we use the shape_current command to reaffirm the length of this dimension, and then check that we indeed have only 240 values:

>> file.var("time").shape_current
=> [240]
>> file.var("time").get[0]
=> 73015.5
>> file.var("time").get[239]
=> 80284.5
>> file.var("time").get[240]
IndexError: index out of range
 from (irb):194:in `[]'
 from (irb):194
 from :0

As expected, when we try for an index of 240, it throws an error. So we have 240 points in time, measured in days since the beginning. We can divide by 365 to get a better feel for what we're dealing wtih. (Note that I use the get method to return an NArray object, which includes the methods min and max.)
>> file.var("time").get.min
=> 73015.5
>> file.var("time").get.min/365
=> 200.042465753425
>> file.var("time").get.max/365
=> 219.957534246575
So the starting time is year 200, and the ending time is about year 220. We have 240 data points, so assuming they're evenly distributed, we have 240 / 20 = 12, or one data point every month. Makes sense. It seems like we have one temperature reading per month; so as we move along the 240 axis, we're moving along in time. Now what about the other two axes? Let's look at "lon".

Longitude

>> file.var_names
=> ["lon", "lon_bnds", "lat", "lat_bnds", "time", "time_bnds", "height", "tas"]
>> file.var("lon").att_names
=> ["standard_name", "long_name", "units", "axis", "bounds"]
>> file.var("lon").att("long_name").get
=> "longitude"
>> file.var("lon").att("units").get
=> "degrees_east"
>> file.var("lon").att("axis").get
=> "X"
>> file.var("lon").att("bounds").get
=> "lon_bnds"

We have longitute, measured in degrees east along the X axis, bounded by "lon_bounds." Not sure what that last one means.
>> file.var("lon").get.shape
=> [144]
>> file.var("lon").get.min
=> 1.25
>> file.var("lon").get.max
=> 358.75
So we have 144 longitude points, from 1.25 to 358. We know there are 360 degrees around the Earth, so to find our precision, we divide 360 by 144:
>> 360.0/144
=> 2.5
>> 360.0/144.0
=> 2.5
So we essentially have an average of one data point every 2.5 degrees. At the equator that translates to one data point approximately every 170 miles. Now let's look at latitude.

Latitude


>> file.var_names
=> ["lon", "lon_bnds", "lat", "lat_bnds", "time", "time_bnds", "height", "tas"]
>> file.var("lat").att_names
=> ["standard_name", "long_name", "units", "axis", "bounds"]
>> file.var("lat").att("long_name").get
=> "latitude"
>> file.var("lat").att("units").get
=> "degrees_north"
>> file.var("lat").att("axis").get
=> "Y"

We have latitude, measured in degrees north, represented by the Y axis. More detail:
>> file.var("lat").get.shape
=> [90]
>> file.var("lat").get.min
=> -89.494384765625
>> file.var("lat").get.max
=> 89.494384765625

We have 90 data points, stretching from about -90 to +90 degrees north; that's about one every two degrees, or about one data point every 140 miles. Note that these are appoximations, since we can't map a square grid onto a sphere. The geographical data resolution will shift across the topology of a sphere.

Now let's see if we can get at the data itself.

Put it all together


We know from before that the temperature data (in variable "tas") is in a 3-dimensional grid with lengths [144, 90, 240]. We now know that those surely correspond to [longitude, latitude, months]. Let's say we want to find the temperature at longitude 0 (corresponds to England, Morroco, and Ghana), latitude 0 (the equator), month 0 (the beginning). First we need to know which index in 0 to 90 corresponds to latitude 0. If we print out all the latitudes, we see that they start at -90 and increase to +90:
>> file.var("lat").get.to_a.each_with_index{|value, index| puts "#{index}   #{value}"}
0   -89.494384765625
1   -87.9775314331055
2   -85.9550552368164
3   -83.9325866699219
4   -81.9101104736328
5   -79.8876419067383
6   -77.8651657104492
7   -75.8426971435547
8   -73.8202209472656
9   -71.7977523803711
10   -69.7752838134766
11   -67.7528076171875
12   -65.730339050293
13   -63.7078666687012
14   -61.6853942871094
15   -59.6629219055176
16   -57.6404495239258
17   -55.617977142334
18   -53.5955047607422
19   -51.5730323791504
20   -49.5505599975586
21   -47.5280914306641
22   -45.5056190490723
23   -43.4831466674805
24   -41.4606742858887
25   -39.4382019042969
26   -37.4157295227051
27   -35.3932571411133
28   -33.3707847595215
29   -31.3483142852783
30   -29.3258419036865
31   -27.3033714294434
32   -25.2808990478516
33   -23.2584266662598
34   -21.235954284668
35   -19.2134838104248
36   -17.191011428833
37   -15.1685390472412
38   -13.1460676193237
39   -11.1235952377319
40   -9.10112380981445
41   -7.07865190505981
42   -5.05618000030518
43   -3.03370785713196
44   -1.01123595237732
45   1.01123595237732
46   3.03370785713196
47   5.05618000030518
48   7.07865190505981
49   9.10112380981445
50   11.1235952377319
51   13.1460676193237
52   15.1685390472412
53   17.191011428833
54   19.2134838104248
55   21.235954284668
56   23.2584266662598
57   25.2808990478516
58   27.3033714294434
59   29.3258419036865
60   31.3483142852783
61   33.3707847595215
62   35.3932571411133
63   37.4157295227051
64   39.4382019042969
65   41.4606742858887
66   43.4831466674805
67   45.5056190490723
68   47.5280914306641
69   49.5505599975586
70   51.5730323791504
71   53.5955047607422
72   55.617977142334
73   57.6404495239258
74   59.6629219055176
75   61.6853942871094
76   63.7078666687012
77   65.730339050293
78   67.7528076171875
79   69.7752838134766
80   71.7977523803711
81   73.8202209472656
82   75.8426971435547
83   77.8651657104492
84   79.8876419067383
85   81.9101104736328
86   83.9325866699219
87   85.9550552368164
88   87.9775314331055
89   89.494384765625
The midway point, then, corresponds to latitude zero, the equator. We access that with
>> file.var("lat").get[45]
=> 1.01123595237732
This is a close enough approximation to zero latitude. Now we can find the temperature at longitude 0, latitude 0, and month 0:
>> file.var("tas").get[0, 45, 0]
=> 300.574279785156
>> file.var("tas").get[0, 45, 0]-273
=> 27.5742797851562
We get an answer of 300 Kelvin, or about 27 degrees Celsius. 0 longitude and 0 latitude is a spot off the southern coast of Ghana, Africa. I suppose 27 Celsius sounds about right for January. Let's see what temperature England reports, at the same time and longitude, but at England's latitude (which is about 52). We look up latitude 52 in the list above, and see that it corresponds to an index of 70.
>> file.var("tas").get[0,70,0]-273
=> 1.03604125976562
Despite the absurd number of significant digits, that number looks about right for England in January. How about England in July?
>> file.var("tas").get[0,70,6]-273
=> 17.9317932128906
Much more pleasant.

Conclusion

We've discovered that for this particular file from one of NOAA's climate models, we can get temperature data by longitude, latitude, and month. We did this by digging around the self-describing NetCDF file, and letting it tell us what it is.

Saturday, March 26, 2011

How to Access Data From NOAA's Climate Model CM2.1

NOAA's climate model CM2.1, developed by GFDL, is one of about a dozen climate models considered in IPCC's 4th Assessment Report. This blog post describes how to access the model's data. For background on the model itself and how it is structured, see my blog post titled An Introduction to NOAA's Climate Change Model CM2.1.

Where is CM2.1 data stored?

NOAA's climate model is available in two places:
  1. The GFDL data portal
  2. The PCMDI data portal
PCMDI (Program for Climate Model Diagnosis and Intercomparison) is the group that formed essentially to host the huge climate data sets that IPCC uses. It's a place to find all the models referenced in IPCC reports, as well as information about IPCC's standardization of variables, time spans, etc.

However, for NOAA's GFDL model CM2.1, it's probably easier just to download it straight from NOAA. There are several pathways through the GFDL site to do this; I'll describe two. For this tutorial, I'll focus on the 1PctTo2X experiment, monthly data, and the surface temperature variable "tas."

How to download the data

Start from the GFDL data portal.
The GFDL data portal

Pathway one: Folders

  1. Scroll down to the CM2.1 bullet points, and choose Download CM2.1 netCDF files via http
  2. Find the experiment name from among the available folders, and open that folder. Each is prepended with "CM2.1" and may have some other characters in the file name. The one we're looking for is called "CM2.1U-D4_1PctTo2X_I1/" and it's about four folders down from the top.
  3. The next level is a single folder called "pp." This stands for "post-processed," meaning the data is not raw; it has been cleaned up.
  4. Next we must choose from folders called "atmos/", "ice_tripolar/", and other names. Each of these represents different parts of the model. The surface temperature variable we want, "tas," will be under the folder atmos/.
  5. At the next level we can choose between "av" (reserved for future, but currently empty), "static" (variables that don't depend on time), and "ts" (time series). Choose ts/.
  6. Next, pick the time series. Choose monthly/.
  7. Now we have a list of files. Each file name begins with the name of the variable it represents, so scroll down to "tas." Notice there are three files, relating to three different time periods (YYYYMM-YYYYMM:
    1. tas_A1.000101-010012.nc (year 1 to year 100)
    2. tas_A1.010101-020012.nc (year 101 to year 200)
    3. tas_A1.020101-022012.nc (year 201 to year 220)
    Pick the 20-year file, since it's smaller: tas_A1.020101-022012.nc
That's it. If you know the variable and time series you want, as well as which part of the model it comes from (atmos, land, etc), you should be able to find any file you need from this method. However, if you're not sure where variables are, then the second pathway is easier.

Pathway two: Tables

Start from the GFDL data portal:
  1. Click on the link to Info on CM2.1 Data Variables Available by Experiment
  2. From here you have a list of data tables (surface temperature "tas" is in Table A0a), and lower down, a grid relating tables of variables to individual experiments. Take a look at the grid.
  3. Find the experiment name across the top ("1%to 2x(run1)"), and click on the GFDL link in the first row (corresponding to data table A1a).
  4. Using this table, you can find the variable you want by its human-readable name, and download it directly.
  5. Click on either http or ftp links for row 6 to download the air_temperature variable "tas" for the 20-year period between year 200 and 220.

How to open the file

Now that we have downloaded the file we want, how do we open it? This reaches beyond the scope of this blog post, but briefly, it's a NetCDF file, with the file extension .nc. The easiest method is to install NetCDF and use the command line utility "ncdump." For detailed instructions on how to open and read these files, see my blog post on using RubyNetCDF.

How to access other climate models

To get information about and access to data from other climate models used by the IPCC, including NOAA's CM2.1, use the PCMDI data portal, which is the official repository of all IPCC-related climate model information.