Comments on: on Zope, multiple cores, and the GIL http://david.wglick.org/2010/on-zope-multiple-cores-and-the-gil/ Fri, 30 Jul 2010 20:31:23 +0000 http://wordpress.org/?v=2.2.3 By: Musings on an Assortment of Subjects » Blog Archive » Using HAProxy with Zope via Buildout http://david.wglick.org/2010/on-zope-multiple-cores-and-the-gil/#comment-7316 Musings on an Assortment of Subjects » Blog Archive » Using HAProxy with Zope via Buildout Wed, 20 Jan 2010 07:14:56 +0000 http://david.wglick.org/2010/on-zope-multiple-cores-and-the-gil/#comment-7316 [...] my post on reducing GIL contention by using fewer Zope threads, Lee Joramo asked for more information on setting up HAProxy, so let me share my configuration. [...] […] my post on reducing GIL contention by using fewer Zope threads, Lee Joramo asked for more information on setting up HAProxy, so let me share my configuration. […]

]]> By: Lee Joramo http://david.wglick.org/2010/on-zope-multiple-cores-and-the-gil/#comment-7300 Lee Joramo Mon, 18 Jan 2010 17:14:32 +0000 http://david.wglick.org/2010/on-zope-multiple-cores-and-the-gil/#comment-7300 David, Thanks for the data. Your 'before' chart looks rather the current charts for several of my servers. I have been planning to check out HAProxy since seeing Elizabeth Leddy's presentation at the Plone 2009 Conference in Budapest http://www.slideshare.net/eleddy/unloading-plone Your data really helps me see that I need to invest the time in this approach. HAProxy looks a little daunting. Do you have any advice or configuration files that you can share? David, Thanks for the data. Your ‘before’ chart looks rather the current charts for several of my servers. I have been planning to check out HAProxy since seeing Elizabeth Leddy’s presentation at the Plone 2009 Conference in Budapest http://www.slideshare.net/eleddy/unloading-plone Your data really helps me see that I need to invest the time in this approach.

HAProxy looks a little daunting. Do you have any advice or configuration files that you can share?

]]> By: Laurence Rowe http://david.wglick.org/2010/on-zope-multiple-cores-and-the-gil/#comment-7298 Laurence Rowe Mon, 18 Jan 2010 14:43:02 +0000 http://david.wglick.org/2010/on-zope-multiple-cores-and-the-gil/#comment-7298 When configuring HAProxy it's important to remember that traffic is not only served by Zope's threads but also by filestream iterators As Blob and skin resource images get served by filestream iterators outside of the thread, think carefully before setting 'maxconn 1' on your backend server. FWIW, I'm no longer convinced you get much benefit from cache affinity on individual zope servers unless you have complex membrane or possibly ldap users. There's a lot to be said for just using varnish's inbuilt load balancing. When configuring HAProxy it’s important to remember that traffic is not only served by Zope’s threads but also by filestream iterators As Blob and skin resource images get served by filestream iterators outside of the thread, think carefully before setting ‘maxconn 1′ on your backend server. FWIW, I’m no longer convinced you get much benefit from cache affinity on individual zope servers unless you have complex membrane or possibly ldap users. There’s a lot to be said for just using varnish’s inbuilt load balancing.

]]> By: Hanno Schlichting http://david.wglick.org/2010/on-zope-multiple-cores-and-the-gil/#comment-7295 Hanno Schlichting Mon, 18 Jan 2010 10:34:10 +0000 http://david.wglick.org/2010/on-zope-multiple-cores-and-the-gil/#comment-7295 Did you have the obligatory "zope-conf-additional = python-check-interval 1000" line in your instance configs? Or did you use the Python default of 100? With the default you get much more trashing, but with a setting like 1000 ticks you usually get that under control. Did you have the obligatory “zope-conf-additional = python-check-interval 1000″ line in your instance configs? Or did you use the Python default of 100?

With the default you get much more trashing, but with a setting like 1000 ticks you usually get that under control.

]]> By: Matt Hamilton http://david.wglick.org/2010/on-zope-multiple-cores-and-the-gil/#comment-7294 Matt Hamilton Mon, 18 Jan 2010 10:33:37 +0000 http://david.wglick.org/2010/on-zope-multiple-cores-and-the-gil/#comment-7294 If you want to see some older info on this, take a look at the report I did back in 2002 on Zope's performance on Solaris: http://www.zope.org/Members/glpb/solaris This was mainly looking at Solaris's different threading systems. The default threading back then was NxM in which they mapped N userspace threads to M kernel processes. This was due to kernel process context switches being that much heavier on Solaris (maybe due to massive multi-processor designs). They have now by default switched to what is referred in that report as the 'alternate threading model' which mapped user threads 1 to 1 to kernel processes. This worked better for Zope as it meant the OS could schedule the threads better. In the case of Solaris there was also issues with not just the GIL but locks on I/O that could end up with deadlocks on the NxM threading model. I guess what you are seeing now is really an extension of that same issue, but with zope threads and cores replacing the OS threads and processes in that report. Another thing that I've not seem mentioned at all in recent years is the python-check-interval setting in zope.conf. It used to be said that this was set too low for faster processors and should be raised. I was going to say, I don't even know if that setting still exists, but a quick search and of course Jarn are on the ball as usual: http://pypi.python.org/pypi/jarn.checkinterval -Matt If you want to see some older info on this, take a look at the report I did back in 2002 on Zope’s performance on Solaris:

http://www.zope.org/Members/glpb/solaris

This was mainly looking at Solaris’s different threading systems. The default threading back then was NxM in which they mapped N userspace threads to M kernel processes. This was due to kernel process context switches being that much heavier on Solaris (maybe due to massive multi-processor designs).

They have now by default switched to what is referred in that report as the ‘alternate threading model’ which mapped user threads 1 to 1 to kernel processes. This worked better for Zope as it meant the OS could schedule the threads better. In the case of Solaris there was also issues with not just the GIL but locks on I/O that could end up with deadlocks on the NxM threading model.

I guess what you are seeing now is really an extension of that same issue, but with zope threads and cores replacing the OS threads and processes in that report.

Another thing that I’ve not seem mentioned at all in recent years is the python-check-interval setting in zope.conf. It used to be said that this was set too low for faster processors and should be raised. I was going to say, I don’t even know if that setting still exists, but a quick search and of course Jarn are on the ball as usual:

http://pypi.python.org/pypi/jarn.checkinterval

-Matt

]]>