Python vs java in threading

September 14, 2009 10:54 AM

How do you respond to a dude who says that python ain't _really_ a multithreading language, and you should write it in Java- and you really think that's wrong?
posted by xmutex to technology (24 comments total)

First ask yourself if he could be right?
posted by jockc at 10:55 AM on September 14 [3 favorites]

uh... they have different built-in threading support mechanisms but this is basically a religious issue. You like blue, he likes red. There's no right answer.
posted by GuyZero at 10:56 AM on September 14

"That isn't really an X" could possibly be a No True Scotsman fallacy. with all of the fun that entails.
posted by adipocere at 11:06 AM on September 14

Consider your motivation.

Java programming is worth a lot more money than Python programming.
posted by rokusan at 11:08 AM on September 14

C'mon where's the bloodbath?
posted by xmutex at 11:11 AM on September 14 [1 favorite]

a) show the dude cherrypy : http://cherrypy.org/

b) ask him if it will be magically improved by using jython


absent objective data, language choices are subjective decisions, In those cases, the person who owns the project (pays the bills or sponsors development) gets to pick the language.
posted by jenkinsEar at 11:14 AM on September 14

I'll synthesis a combination of the above and say:

Java and python have two different mechanisms for threading and whether or not that matters depends solely on your problem domain and your requirements. I'm sure there is probably an academic opinion that can be said that one is better or more flexible or more pure or what not but none of that matters except where this overlaps with your needs.
posted by mmascolino at 11:15 AM on September 14

According to This article on python threading:

The Global Interpretor Lock refers to the fact that the Python interpreter is not thread safe. There is a global lock that the current thread holds to safely access Python objects. Because only one thread can aquire Python Objects/C API, the interpreter regularly releases and reacquires the lock every 100 bytecode of instructions. The frequency at which the interpreter checks for thread switching is controlled by the sys.setcheckinterval() function.

Not very encouraging.

If you want to get into hard core concurrency, and you want to use something new with functional programming capabilities, I would encourage you to look into Scala it's a new and pretty innovative language full of modern goodness like closures and 'actors' (which I think are like python generators, but I'm not sure what the differences are).

Scala is designed for super high concurrency and scalability (thus the name) Twitter is actually built with it now, after they switched from ruby due to performance issues. Erlang is another language built for lots of concurrency, but it sounds kind of lame to me. Clojure is another option as well, which is pretty popular with language nerds.

Both Scala and Clojure run on the Java virtual machine, so you not only get all the benefits of a stable, well-tested VM, but you also have access to all java libraries. Speaking of which, you could also look at Jython, which is a version of python that runs on the JVM, my guess is it might be thread safe, I don't know.
posted by delmoi at 11:15 AM on September 14 [3 favorites]

He's right about the first part.

Python ain't a multi-threaded language. Its global interpreter lock (GIL) effectively serializes your program, so only one Thread instance is ever running at any given time. The benefit of this is that you don't have to think about synchronization, and the down side is that you're not going to be able to exploit multi-core machines unless you run multiple Python processes or drop down to writing C extensions that do multi-threading.

About the second part: If your code is performance intensive, then he may be right. If it's not (e.g., a GUI whose threads spend most of their time sleeping anyway), then it's probably simpler to write in Python exactly because you don't need to worry about synchronization to the same extent you would in Java.
posted by qxntpqbbbqxl at 11:27 AM on September 14

Python 2.6 added the multiprocessing library which bypasses the Global Interpreter Lock. But it's a recent addition for an old design problem in python. Vanilla CPython threads are single-process. The interpreter might switch between threads while waiting for IO, but it still does only one instruction at a time. I don't know how multiprocessing works under Java, but multiprocessing has never been a selling point of CPython.
posted by KirkJobSluder at 11:31 AM on September 14 [1 favorite]

Threading support in Python is useful only in certain circumstances. For example, it can be useful to wait for some I/O or other external stimulus in one thread and keep processing in another thread. But the GIL will keep you from working on multiple cores at once.

As KirkJobSluder says, Python 2.6 has the multiprocessing module, which allows you to pretend you are using the threading interface, but the implementation uses multiple subprocesses which can truly run concurrently, not sharing a single GIL.

The Unladen Swallow project aims to remove the GIL from Python altogether. I don't think it's ready yet.
posted by grouse at 11:35 AM on September 14

Well you guys are just crapping all over my parade.

Thanks though :)
posted by xmutex at 11:39 AM on September 14

I don't know how multiprocessing works under Java

The JVM is multithreaded and uses OS threads. In theory on very old systems (like MacOS 9.x) without preemptive multithreading, java threads would also not be premtive, but that's really a non-issue these days.

java used Green Threads in Java 1.1 which were effectively single threaded but got around the limitations of OSes without preemptive multithreading, but that's long gone.

Your threads in Java or any JVM languages are real threads.
posted by delmoi at 11:40 AM on September 14

And new JVMs even have decent concurrent GC which is a huge deal on multicore.
posted by GuyZero at 11:43 AM on September 14

First thing I would do is repost this on StackOverflow.

Then I would rewrite it in C#
posted by blue_beetle at 12:07 PM on September 14 [1 favorite]

Look at Stackless Python. It's what Eve Online uses, and it was built for true concurrency and a lot of advanced features like support for the actor model and super-lightweight threads.

The short answer is that your friend is right in a dictionary-definition sense of the word, thanks to Python's GIL. But of course, whether that matters depends on what you're programming. If you actually need true multi-threading (and not many projects do), then it's worth looking at Java (which hasn't been without its own threading issues). But of course, the standard trade-off applies: longer, more involved development time, and a huge increase in complexity if the project could be, but isn't, architected to not require true multi-threading.

If you need true multi-threading, you could also look at Erlang, which was developed at Nokia and is widely deployed as part of their infrastructure. Fully functional, and designed from the ground up as a language for concurrent programming.
posted by fatbird at 12:12 PM on September 14

Erlang was developed by Ericsson , not Nokia.
posted by delmoi at 12:14 PM on September 14

Also, Ericsson never really used Erlang, despite spending, I guess, a lot of money on it:

In 1998, the AXD301 switch was announced, containing over a million lines of Erlang, and reported to achieve a reliability of nine "9"s. Shortly thereafter, Erlang was banned within Ericsson Radio Systems for new products, citing a preference for non-proprietary languages. The implementation was open sourced at the end of the year.[1] The ban at Ericsson was eventually lifted, and Armstrong was re-hired by Ericsson in 2004.[4]

I don't think Nokia uses it.
posted by delmoi at 12:16 PM on September 14

Ouch. Sucks to be Erlang, I guess. Too bad. I bought Joe Armstrong's book, and really liked Erlang, but never had the time to get into it.

My bad on Nokia instead of Ericsson. Armstrong does refer in the book to it mainly being planned to be used for a lot switching equipment.
posted by fatbird at 12:31 PM on September 14

Hmmm... if Erlang's wikipedia page is to be believed, it is being used by some significant projects: Delicious, Facebook's chat system, and Amazon's SimpleDB (the pseudo-relational database they offer as part of their cloud computing system).
posted by fatbird at 12:34 PM on September 14

Yeah sorry man, your friend is right. I program in Python professionally and it is my most favorite thing in the world, but its threading support is broken in the cpython reference implementation. In fact, running a CPU-intensive task in threads in an attempt to leverage multiple cores will slow it down significantly. Not only is there a GIL, but it's not written very well. It defers scheduling to the kernel instead of handling it natively, causing them to fight for the lock. If you want to understand the issue, everything you need to know about it is covered by David Beazley in this excellent talk.

For what it's worth, this is only a problem for a very specific domain, and there are other ways of solving it besides threads. It is a bit ridiculous to use this argument as a basis of language choice. Kind of a low blow too, usually java evangelists stick to the dynamic vs. static typing arguments -- which have merit two reasonable people can debate -- rather than pick a specific wart and claim it invalidates the language. You should counter: Python should be used instead of java because its vm starts up faster. That'll show him.
posted by cj_ at 1:04 PM on September 14

I didn't mean to say no one used Erlang, just that it didn't really happen to be used by the people who invented it. It's probably nice but the fact that it uses it's own VM means you can't leverage all the libraries and code out there that's already written for the JVM.
posted by delmoi at 1:15 PM on September 14

it didn't really happen to be used by the people who invented it.

If this HOPL paper (::pdf) is to be believed, that is not quite the case.

Just when we thought everything was going well, in 1998, Erlang was banned within Ericsson Radio AB (ERA) for new product development. ... In addition, projects that were already using Erlang were allowed to continue but had to make a plan as to how dependence upon Erlang could be eliminated. Although the ban was only within ERA, the damage was done. The ban was supported by the Ericsson technical directorate and flying the Erlang flag was thereafter not favored by middle management.

Shortly after the open source release, the majority of the original Erlang development team resigned from Ericsson and started a new company called Bluetail AB with Jane as the chief executive. In retrospect the Erlang ban had the opposite effect and stimulated the long-term growth of Erlang.

Later...

The plans within Ericsson to wean existing projects off Erlang did not materialise and Erlang is slowly winning ground due to a form of software Darwinism. Erlang projects are being delivered on time and within budget, and the managers of the Erlang projects are reluctant to make any changes to functioning and tested software.
The usual survival strategy within Ericsson during this time period was to call Erlang something else. Erlang had been banned but OTP hadn’t. So for a while no new projects using Erlang were started, but it was OK to use OTP. Then questions about OTP were asked: “Isn’t OTP just a load of Erlang libraries?”—and so it became “Engine,” and so on.

So developers at Ericsson were still using Erlang; they were just going "underground". And the main inventors had left for greener pastures, but were still using Erlang.
posted by Monday, stony Monday at 7:47 PM on September 14

As far as jython is concerned, it is implemented with native JVM threads and does not have the GIL problem. It would also appear that the Jython developers don't intend to add it:

Jython 2.5rc3 (Release_2_5rc3:6384:6385, May 26 2009, 15:51:41)
[Java HotSpot(TM) Client VM (Sun Microsystems Inc.)] on java1.6.0_07
Type "help", "copyright", "credits" or "license" for more information.
>>> from __future__ import GIL
File "", line 1
SyntaxError: Never going to happen!
>>>
posted by whir at 10:52 AM on September 15 [1 favorite]