I recently had some fun debugging a little application of mine written in JRuby which crawls the twitter search API looking for specific tweets. The problem was, every now and again, the crawler would hang even if I set appropriate timeouts. The crawler consisted of two threads, one periodically issuing a search to twitter, and another one writing the results into the database.
The first surprise to me was that in Ruby, by default, if there is an
exception in a thread, the thread silently dies, not even issuing an
error message. Only after you joined the thread, you get the error
message. You can set Thread.abort_on_exception = true which completely
kills your application, however. This meant that what appeared to be
missed timeouts could just as well be uncaught exceptions.
So when working with multithreaded applications, enclosing the whole
thread in a begin .. rescue Exception => e ... end is important if you
want to get noticed about errors at all.
But still, the thread would misteriously die without properly handling the timeout. Some digging deeper revealed an old bug report and an interesting article about Ruby and timeouts in general which seemed to imply that timeouts might not always work, in particular if system calls are involved. It was unsure, though, whether the situation is the same for JRuby which uses native threads instead of Green threads (simulated threads by doing explicit time-slicing in the interpreter).
So I started to read the JRuby
sources, to understand
where and how timeouts are implemented in Net::HTTP. Which lead to my
next surprise: Net::HTTP completely handles timeouts through the
Timeout module, not on a socket level. It does not use the
possibilities to set timeouts on reads or writes but encapsulates all
the significant portions of code in Timeout::timeout { ... } calls. I
also found a nice old post by Charles Nutter (a.k.a. headius)
explaining the implementation in depth.
I guess based on that post, JRuby has started to implement Timeout again in Java (source). And some first tests revealed that this timeout plays well within Net::HTTP, but my crawler was still hanging every once and again.
Finally, I found the last missing piece of information: From the sources, it seems that JRuby’s implementation of the Timeout module raises the Timeout::ExitException when the timeout happened. However, that is a ruby 1.9 feature, in ruby 1.8, the exception was named Timeout::Error. So basically, I was catching the wrong exception, in fact not catching it at all, thereby killing the thread (silently). Interestingly, some more testing showed that if JRuby raises a Timeout::Error if you use the Timeout module yourself, but raises a Timeout::ExitException when you’re using Net::HTTP.
In the end, I just enclose the whole HTTP request section with a Timeout::timeout of my own, catching both Timeout::Error and Timeout::ExitException and finally everything was running robustly.
So in summary
I guess I should post a bug report on the latter… .
Update: I submitted a bug report, and it’s already fixed! Those jruby people are really incredibly fast!
Posted by at 2009-07-08 15:43:00 +0000
blog comments powered by Disqus