How to install RubyNetCDF on Ubuntu: The Long Version

What is NetCDF?

The best introduction to NetCDF, originally developed for the Earth science community, comes from UCAR's intro page. NetCDF stands for Network Common Data Format. According to its home site, NetCDF is "a set of software libraries and machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data." It seems to be popular among research organizations dealing with large climate data. For more details information, see the NetCDF page at unidata.ucar.edu.

Basically, NetCDF is a file format, like .xlsx for Excel files or .csv for commas separated value files. NetCDF files have the file extension .nc. NetCDF is also a set of software libraries. Once these libraries are on your computer, you can open the NetCDF files.

We know how to open Excel files; how do we open NetCDF files?

How do we open NetCDF files?

There is no desktop application like Excel to open NetCDF files. Instead, they must be read directly with another program. We'd like to do it with Ruby. For example, in a Ruby environment, one can use the CSV class to open, read, modify, and write to CSV files. So we need a class for opening, reading, modifying, and writing to NetCDF files. That's what RubyNetCDF is for.

What is RubyNetCDF?

According to its creators, "RubyNetCDF is the Ruby interface to the NetCDF library built on the NArray library, which is an efficient multi-dimensional numeric array class for Ruby." So it's an interface, built on a library. What's an interface?

An interface is just the user-facing part of an application that a user interacts with. A command-line is an interface; Microsoft Windows is a graphical user interface (GUI). So RubyNetCDF will be the interface that the user interacts with to engage with a NetCDF file.

What's a library?
In Ruby, a library is a piece of additional information that the Ruby language incorporates into itself in order to make more functionality available to the user. Why wasn't it included to begin with? Probably because its use is esoteric and only useful to a small number of users.

Installing RubyNetCDF

I follow the RubyNetCDF installation instructions provided by the program creators. I am using Linux Ubuntu 10.04 (Lucid) with Ruby 1.8.7 on a Dell Latitude E6400. This probably won't work on Windows, except perhaps through Cyg-Win.

Install dependencies

The first steps are to install Ruby, install NArray, and install NetCDF.

1. Install Ruby
1. I assume this is done since you already chose Ruby as the NetCDF interface
2. Check your version by typing

   $ ruby -v

   at the command prompt. Mine is 1.8.7. NB: Ruby versions prior to 1.9 require explicitly requiring Rubygems before requiring other files. See the following example in Ruby 1.8.7:
   

   $ irb --simple-prompt
   >> require 'narray'
   LoadError: no such file to load -- narray
    from (irb):1:in `require'
    from (irb):1
    from :0
   >> require 'rubygems'
   => true
   >> require 'narray'
   => true
   >> exit
   

   Note that I had to include 'rubygems' before Ruby knew how to include the file 'narray'. In versions of Ruby later than 1.9, requiring 'rubygems' explicitly is not required.
2. Install NArray 
1. At the command prompt type

   $ sudo gem install narray

2. I got a whole bunch of "No definition for ..." errors. I ignored them.
3. Check the installation version by typing

   $ gem list narray

   Mine returns version 0.5.9.9. Now check to make sure it actually is working by jumping into an interactive Ruby session and creating an NArray object:
   

   $ irb --simple-prompt
   >> require 'rubygems' #this line not required for Ruby > 1.9
    => true
   >> require 'narray' 

   => true
   >> a = NArray[2,3,4]
   => NArray.int(3): 
   [ 2, 3, 4 ]
   >> exit
   

3. Install NetCDF version 3. I'm running Ubuntu 10.04, so I go for the simplest solution possible:
1. Go to Applications->Ubuntu Software Center
2. Type "netcdf" (without the quotes) in the search field.
3. Select "Programs for reading and writing NetCDF files" (netcdf-bin)
4. Click "Install." Notice that it will automatically install the required dependency "libnetcdf4", which it calls "An interface to scientific data access to large binary data."
5. Check that the installation was successful typing

   $ ncdump

   at the command line. The last line returned should give you the version number. Mine returns

   netcdf library version "3.6.3" of Dec 22 2009 06:10:17 $

So that completes the first three necessary steps to using RubyNetCDF. Now we must download RubyNetCDF and install it.

Download and install ruby-netcdfManually

We'll do it manually, following the installation instructions.
To begin, download the tar file. I let my browser put it in the default /home/andy/Downloads folder. From there I double clicked on the file ruby-netcdf-0.6.5.tar.gz, and used the default Ubunutu archive manager extract it. I extracted it to my home folder, /home/andy. Now I have a folder there called ruby-netcdf-0.6.5. Next I just follow the steps in the installation page:

Navigate to the folder where you extracted the file:

$ cd ruby-netcdf-0.6.5

Then, type the following command:

$ ruby extconf.rb

This runs the ruby script called extconf.rb.When I run this command, I get the following error output:

checking for narray.h... no
** configure error **  
   Header narray.h or narray_config.h is not found. If you have these files in 
   /narraydir/include, try the following:

   % ruby extconf.rb --with-narray-include=/narraydir/include

*** extconf.rb failed ***

The extconf script couldn't find the file narray.h (which is part of the NArray gem we installed earlier), so we need to tell the program where to find the file. Fortunately, it tells us how to do that, but first we need to actually find the file on our system. I did it in Ubuntu

$ sudo find / -name narray.h

Both methods gave me the path to the file: /usr/lib/ruby/gems/1.8/gems/narray-0.5.9.9. So now we use that in the command line, appending it to the first part:

$ ruby extconf.rb --with-narray-include=/usr/lib/ruby/gems/1.8/gems/narray-0.5.9.9

Now I get a slightly different error:

$ ruby extconf.rb --with-narray-include=/usr/lib/ruby/gems/1.8/gems/narray-0.5.9.9
checking for narray.h... yes
checking for narray_config.h... yes
checking for netcdf.h... no
    ** configure error **  
       Header netcdf.h or the compiled netcdf library is not found. 
       If you have the library installed under /netcdfdir (that is, netcdf.h is
       in /netcdfdir/include and the library in /netcdfdir/lib/),
       try the following:

       % ruby extconf.rb --with-netcdf-dir=/netcdfdir

This time, it found the first file it was looking for (narray.h), and the second file it was looking for (narray_config.h), and then failed on the third file (netcdf.h). So we need to repeat the steps above to find the missing file, and then pass the path to the ruby script at the command line.

First, find the file:

$ sudo find / -name netcdf.h

But this time it doesn't find anything. So what happened? We need to install an additional package, called libnetcdf-dev. The easiest way to do this in Ubuntu is through the package manager. System->Administration->Synaptec Package Manager, and search for "libnetcdf-dev." Install this file. Now try finding the file from the command line:

$ sudo find / -name netcdf.h

This time it should find the file. Mine is located at /usr/include/netcdf.h.

Now that we know the file exists in our system, try running the RubyNetCDF installer again:

$ ruby extconf.rb --with-narray-include=/usr/lib/ruby/gems/1.8/gems/narray-0.5.9.9

This time, it should succeed. I get the following output:

andy@andy-laptop:~/ruby-netcdf-0.6.5$ ruby extconf.rb --with-narray-include=/usr/lib/ruby/gems/1.8/gems/narray-0.5.9.9
checking for narray.h... yes
checking for narray_config.h... yes
checking for netcdf.h... yes
checking for main() in -lnetcdf... yes
creating Makefile

Now let Linux compile all the code and create the shared libraries it wants:

$ make
gcc -I. -I. -I/usr/lib/ruby/1.8/i486-linux -I. -DHAVE_NARRAY_H -DHAVE_NARRAY_CONFIG_H -DHAVE_NETCDF_H -I/usr/local/include -I/usr/lib/ruby/gems/1.8/gems/narray-0.5.9.9  -D_FILE_OFFSET_BITS=64  -fPIC -fno-strict-aliasing -g -g -O2  -fPIC   -c netcdfraw.c
gcc -shared -o netcdfraw.so netcdfraw.o -L. -L/usr/lib -L/usr/local/lib -L/usr/local/lib/site_ruby/1.8/i486-linux -L. -Wl,-Bsymbolic-functions -rdynamic -Wl,-export-dynamic    -lruby1.8 -lnetcdf  -lpthread -lrt -ldl -lcrypt -lm   -lc

Finally, install it:

$ sudo make install
[sudo] password for andy: 
mkdir -p /usr/local/lib/site_ruby/1.8/i486-linux/numru
/usr/bin/install -c -m 0755 netcdfraw.so /usr/local/lib/site_ruby/1.8/i486-linux/numru
mkdir -p /usr/local/lib/site_ruby/1.8/numru
/usr/bin/install -c -m 644 ./lib/netcdf_miss.rb /usr/local/lib/site_ruby/1.8/numru/
/usr/bin/install -c -m 644 ./lib/netcdf.rb /usr/local/lib/site_ruby/1.8/numru/

This should complete the installation of RubyNetCDF. The installation instructions recommend running a test:

$ make test
test.rb:3:in `require': no such file to load -- narray (LoadError)
 from test.rb:3
make: *** [test] Error 1

This tells us that it tried to run a ruby script called test.rb, and in line 3 it failed to execute a 'require' command on the file "narray." Adding the line

require 'rubygems'

to the top of this file will solve the problem and allow the test to run successfully.

Alternative/better method: Install ruby-netcdf as a gem

RubyNetCDF exists as a gem. If at the command line you type the command

$ gem list ruby-netcdf --remote

you'll see ruby-netcdf (0.6.5) listed. So I tried

sudo gem install ruby-netcdf

but that failed as follows:

Building native extensions.  This could take a while...
ERROR:  Error installing ruby-netcdf:
 ERROR: Failed to build gem native extension.

/usr/bin/ruby1.8 extconf.rb
extconf.rb:3: uninitialized constant Gem (NameError)

Gem files will remain installed in /usr/lib/ruby/gems/1.8/gems/ruby-netcdf-0.6.5 for inspection.
Results logged to /usr/lib/ruby/gems/1.8/gems/ruby-netcdf-0.6.5/gem_make.out

Note that it left the files in the directory where gems would go, so we can navigate to that directory and install it manually.

$ cd /usr/lib/ruby/gems/1.8/gems/ruby-netcdf-0.6.5

Next, run the Ruby script with the second argument telling it where to find the NArray files (I use sudo just to preempt potential permission denied errors):

$ ruby -rubygems extconf.rb --with-narray-include=/usr/lib/ruby/gems/1.8/gems/narray-0.5.9.9

That produces the following output:

checking for narray.h... yes
checking for narray_config.h... yes
checking for netcdf.h... yes
checking for main() in -lnetcdf... yes
creating Makefile

If you get the error extconf.rb:3: uninitialized constant Gem (NameError), then make sure you're including the argument -rubygems above. That's the equivalent of inserting at the top of the file the line

require 'rubygems'

The next step is to run the make command, which lets Linux compile libraries and do a few other things:

$ sudo make
gcc -I. -I. -I/usr/lib/ruby/1.8/i486-linux -I. -DHAVE_NARRAY_H -DHAVE_NARRAY_CONFIG_H -DHAVE_NETCDF_H -I/usr/local/include -I/usr/lib/ruby/gems/1.8/gems/narray-0.5.9.9  -D_FILE_OFFSET_BITS=64  -fPIC -fno-strict-aliasing -g -g -O2  -fPIC   -c netcdfraw.c
gcc -shared -o netcdfraw.so netcdfraw.o -L. -L/usr/lib -L/usr/local/lib -L/usr/lib/ruby/gems/1.8/gems/narray-0.5.9.9/ -L. -Wl,-Bsymbolic-functions -rdynamic -Wl,-export-dynamic    -lruby1.8 -lnetcdf  -lpthread -lrt -ldl -lcrypt -lm   -lc

Finally, run sudo make install to finish:

$ sudo make install
/usr/bin/install -c -m 0755 netcdfraw.so /usr/local/lib/site_ruby/1.8/i486-linux/numru
/usr/bin/install -c -m 644 ./lib/netcdf_miss.rb /usr/local/lib/site_ruby/1.8/numru/
/usr/bin/install -c -m 644 ./lib/netcdf.rb /usr/local/lib/site_ruby/1.8/numru/

The final step is run a test to see if the installation of ruby-netcdf was successful, but mine got an error:

$ sudo make test
test.rb:3:in `require': no such file to load -- narray (LoadError)
 from test.rb:3
make: *** [test] Error 1

What's going on here? It's calling the file test.rb, located in the subdirectory "/test." This fails because line 3 of the file test.rb is as follows:

require 'narray'

For this line to work with Ruby 1.8.7, it needs to have RubyGems initialized. This is usually done by requiring RubyGems at the top of the file:

require 'rubygems'
require 'narray'

This is why we included the -rubygems argument above, but we can't do that here because we're not calling ruby, we're calling the Linux command make.

To get around this, I resorted to common hackery: open the test.rb file, and add the line manually to the top. This is what the first few lines of my file look like after I insert the new line 3:

##require 'numru/netcdf' 
## // to test before make install -->
require 'rubygems'
require 'narray'
require '../netcdfraw'  
require '../lib/netcdf'
## <-- to test before make install //

include NumRu

Now try running the test again:

$ sudo make test
/usr/local/lib/site_ruby/1.8/i486-linux/numru/netcdfraw.so: warning: already initialized constant NC_NOWRITE
/usr/local/lib/site_ruby/1.8/i486-linux/numru/netcdfraw.so: warning: already initialized constant NC_WRITE
/usr/local/lib/site_ruby/1.8/i486-linux/numru/netcdfraw.so: warning: already initialized constant NC_SHARE
/usr/local/lib/site_ruby/1.8/i486-linux/numru/netcdfraw.so: warning: already initialized constant NC_CLOBBER
/usr/local/lib/site_ruby/1.8/i486-linux/numru/netcdfraw.so: warning: already initialized constant NC_NOCLOBBER
creating test.nc...
test did not fail :-p (please ignore the warnings)

The message at the end, along with the friendly emoticon, tell us that the test succeeded. We have completed the installation of RubyNetCDF.

Confirm installation succeeded

We can confirm that everything is working by opening a Ruby session (but not in the protected gem directory)...

$ cd ~
$ irb

...and interacting with NetCDF.

irb(main):001:0> require 'numru/netcdf'
=> true
irb(main):002:0> file = NumRu::NetCDF.create("test.nc")
=> NetCDF:test.nc
irb(main):003:0> file.close
=> nil
irb(main):004:0> exit

That confirms that everything is working.

Conclusion

To summarize, we completed four steps:

1. We installed Ruby 1.8.7
2. Then we installed NArray 0.5.9.9 
3. Then we installed NetCDF 
4. Then we installed Ruby-NetCDF 0.6.5 

However, we are left with one problem: the ruby-netcdf gem is not listed when we ask RubyGems for a list of installed gems:

$ gem list ruby-netcdf

*** LOCAL GEMS ***

However, the program works, so I will worry about this later.
~Fin~

Lessons Learned

• The gem version of ruby-netcdf (gem install ruby-netcdf) is different from the tarball version available from the website (http://ruby.gfd-dennou.org/products/ruby-netcdf/#download). Specifically, the files extconf.rb and extconf.rb.orig.
• Installation instructions seem to assume Ruby 1.9 (which obviates RubyGems issue below).
• If you're using Ruby 1.8, you must ensure that Ruby includes RubyGems. For details on how to do this, see the RubyGems documentation on this issue.
• Installation instructions do not specify that the program requires NetCDF to have the additional package "libnetcdf-dev."
• Even after successful installation of all parts, ruby-netcdf doesn't show up as a gem after a call to

  $ gem list ruby-netcdf