Mirroring RubyGems, and Ruby 9.2 on CentOS 5.5
This is one of those troubleshooting scenarios that starts out being about one thing (in this case, mirroring RubyForge) and ends up being about something entirely different (getting a new Ruby running in an old CentOS). To wit:
One of the most frustrating things about running new code in stable production environments is the assumption by many developers that all of your servers will have access to the outside world, and to their repositories.
For general security, unless a host is an edge web server, we don’t allow outbound connections to untrusted subnets from our systems, let alone to repos on the outside world. And even if we did allow it, I really don’t need my hosts phoning out to third-party repos all over the place and asking for data. Those hosts are internal and should stay that way.
To get around this we tend to mirror repositories locally with Cobbler, and point our hosts at these repos. Generally that’s all well and good since we are mostly a CentOS shop and use RPM based repos. But as part of our recent efforts to deploy cucumber-puppet, I decided to mirror RubyGems in order to allow a small number of test hosts to install via gem instead of RPM (another post later on the perils of mixing those two…). The saga of getting this running was interesting and worth noting. (I should note the considerable assistance of Sun’s BigAdmin pages, which I never thought would ever come in handy again, but http://www.sun.com/bigadmin/content/submitted/ruby_http.jsp — pretty good starting point).
To begin, if you’re not able to route out but can get to a proxy from the box hosting your mirrored repos (how else are you going to get them?), you went the builder gem installed. You can get this via “gem install builder,” or build/download an RPM, whichever suits your package management scheme. WIth this done, create a file in /root/.gemmirrorrc containing:
--- - from: http://gems.rubyforge.org/ to: /gemrepo/dev
Replace the to: field with the location of your target repo.
Rubygems are nice enough to obey your proxy settings, so go ahead and execute:
and with this done, simply run
To download the gems from RubyForge. Be prepared for a very, very long wait — gem files are small, but there are around 40,000. Go get coffee, or better yet dinner.
With that done, the BigAdmin page asks you to generate an index, a process similar to the createrepo command. This sounds good in practice — just run
gem generate_index -d /gemrepo/dev
Again substituting your directory for /gemrepo/dev. And here is where we ran into the fun, after just a few minutes. I’ll be happy to post the mountain of stack trace fail if anyone is interested but the critical bits, from the beginning and end of the trace, are here:
...............................................................*** buffer overflow detected ***: /usr/bin/ruby terminated ======= Backtrace: ========= /lib64/libc.so.6(__chk_fail+0x2f)[0x33e12e803f] /usr/lib64/ruby/1.8/x86_64-linux/syck.so(rb_syck_mktime+0x48e)[0x2b5b11b36a7e] /usr/lib64/ruby/1.8/x86_64-linux/syck.so(yaml_org_handler+0x860)[0x2b5b11b37390] /usr/lib64/ruby/1.8/x86_64-linux/syck.so(syck_defaultresolver_node_import+0x39)[0x2b5b11b37599] /usr/lib64/libruby.so.1.8[0x33e1a34a11] /usr/lib64/libruby.so.1.8[0x33e1a34f28] /usr/lib64/libruby.so.1.8[0x33e1a3552a] /usr/lib64/libruby.so.1.8(rb_funcall+0x85)[0x33e1a357f5] /usr/lib64/ruby/1.8/x86_64-linux/syck.so(rb_syck_load_handler+0x47)[0x2b5b11b36527] /usr/lib64/ruby/1.8/x86_64-linux/syck.so(syck_hdlr_add_node+0x39)[0x2b5b11b30d49] /usr/lib64/ruby/1.8/x86_64-linux/syck.so(syckparse+0xb45)[0x2b5b11b305d5] /usr/lib64/ruby/1.8/x86_64-linux/syck.so(syck_parse+0x19)[0x2b5b11b38169] /usr/lib64/ruby/1.8/x86_64-linux/syck.so(syck_parser_load+0xed)[0x2b5b11b3639d] ... 00400000-00401000 r-xp 00000000 fd:00 1291456 /usr/bin/ruby 00600000-00602000 rw-p 00000000 fd:00 1291456 /usr/bin/ruby 1a46b000-296d6000 rw-p 1a46b000 00:00 0 [heap] 33e0e00000-33e0e1c000 r-xp 00000000 fd:00 1998852 /lib64/ld-2.5.so 33e101b000-33e101c000 r--p 0001b000 fd:00 1998852 /lib64/ld-2.5.so 33e101c000-33e101d000 rw-p 0001c000 fd:00 1998852 /lib64/ld-2.5.so 33e1200000-33e134e000 r-xp 00000000 fd:00 1998859 /lib64/libc-2.5.so 33e134e000-33e154e000 ---p 0014e000 fd:00 1998859 /lib64/libc-2.5.so 33e154e000-33e1552000 r--p 0014e000 fd:00 1998859 /lib64/libc-2.5.so 33e1552000-33e1553000 rw-p 00152000 fd:00 1998859 /lib64/libc-2.5.so 33e1553000-33e1558000 rw-p 33e1553000 00:00 0 33e1600000-33e1602000 r-xp 00000000 fd:00 1998863 /lib64/libdl-2.5.so 33e1602000-33e1802000 ---pAborted
This was not really the outcome I was hoping for, and some quick Googling seemed to indicate that this was a deeper problem than one I wanted to solve. One of my major concerns is that I’m using a somewhat dated version of Ruby, 1.8.6, caused by our desire to keep our Ruby stack stable across several different CentOS distributions. However this often is the type of error released in the version after the one I’m currently running…what’s more I suspected strongly that this was an old generate_index gem running up against package metadata that it couldn’t handle properly. To determine whether this was the problem, or whether it was something else altogether, I decided to try and get the latest and greatest Ruby up and running on a dev box and see if I could index the same repo there. If I could, it would be worth it to work backwards and figure out if what had changed and perhaps get my Ruby version working with it as well.
In a desire to keep as many of the variables the same as possible, I did this on a CentOS 5.5 x86_64 host nearly identical to the Cobbler host, with the exception that Ruby 1.8.6 (and all the goodies that depend on it, including Puppet) were uninstalled after the host was built. I then grabbed and installed the Ruby 1.9.2 code from http://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.2-p180.tar.gz, dropped it on my test host, and did a good ol ./compile; make; sudo make install.
This went swimmingly and, as with all things Ruby, faster than expected. Ruby 1.9 even comes with RubyGems pre-installed…fantastic. But also as with all things Ruby, it wasn’t as simple as it initially appeared.
For a test, I try to get a list of local gems:
# gem list --local ERROR: Loading command: list (LoadError) no such file to load -- zlib ERROR: While executing gem ... (NameError) uninitialized constant Gem::Commands::ListCommand
OK, zlib’s missing. I go and check and sure enough zlib-devel package isn’t installed. Go ahead and install it, and came across another helpful tip for fixing this without a total rebuild (http://stackoverflow.com/questions/3928965/installing-ruby-on-ubuntu-10-10-using-rvm-problem-with-gem). So inside the Ruby source tree, I execute:
cd ext/zlib ruby extconf.rb make sudo make install
This is lovely and rebuilds exactly what I need without having to repeat the process. So now, with everything installed, I cd to the proper directory, execute my index command, and get:
# gem generate_index -d . ERROR: While executing gem ... (RuntimeError) Gem::Indexer requires that the XML Builder library be installed: gem install builder
Oh good! Well, that’s easy to fix. Except it wasn’t:
# gem install builder Successfully installed builder-3.0.0 1 gem installed Installing ri documentation for builder-3.0.0... Installing RDoc documentation for builder-3.0.0... # gem generate_index -d . ERROR: While executing gem ... (RuntimeError) Gem::Indexer requires that the XML Builder library be installed: gem install builder
# gem list *** LOCAL GEMS *** builder (3.0.0) minitest (1.6.0) rake (0.8.7) rdoc (2.5.8)
Some more quick Googling (you’ll find this about me…I’m pretty quick to google before investigating. I feel like less of a geek for it…and I also feel 10x more productive when I don’t waste an entire afternoon poking at something other people figured out already) revealed turned up http://stackoverflow.com/questions/4456127/ruby-gem-generate-indexer-broken-with-xml-builder-3-0-0, which indicates it’s a problem with the current XML Builder gem.
I decided not to follow the author’s directions in their entirety and rather than hack indexer.rb, just remove the current builder gem and replace with an older one. To wit:
# gem uninstall builder Successfully uninstalled builder-3.0.0 # gem install builder -v 2.1.2 Successfully installed builder-2.1.2 1 gem installed Installing ri documentation for builder-2.1.2... Installing RDoc documentation for builder-2.1.2...
Once this was done I restarted the indexer, and it appears to be running smoothly. I do however see a number of the following errors pop up during the processing:
ERROR: Unable to process /mnt/tmp/gem/gems/FaceToFace-0.1.0.gem invalid byte sequence in UTF-8 (ArgumentError)
Followed by yet another stack trace. This very much suggests to me that the problem is what I thought it was — bad metadata in a user-contributed gem that absolutely broke the 1.8 indexer, a problem fixed in 1.9.
I’m still waiting for the index to complete, but it’s looking good. Part II will cover getting this repo into production, and (hopefully) getting this process working on the older Ruby.
Some takeaways from this:
- RubyGems are fickle, not unlike CPAN. Running those built as part of a base distro or repository like EPEL where they have been repackaged and tested for your OS is easy…anything else can be unexpectedly difficult
- Watch out for unclear cross-dependencies between gems and packages. This is why it makes sense to use one package system. If what I have here ultimately has to be the way I get things to work, I will gem2rpm the gems I need, fix them up so they work properly, and install…the danger of breaking gems by, for instance, upgrading zlib-devel won’t alert me as to the problem if I don’t use one management system
- Google Is Your Friend. I could have wasted a lot of time trying to fix the stack trace I initially hit, and I ultimately will if this turns out to be a viable way to mirror RubyGems. But it was a waste of my time to do so without trying to find a working solution first and determining if the problem was really mine to begin with. Now I can work backwards to a solution that works within my framework