thread limit in python

[danieldsmith]

hello all,

i have run into a problem where i cannot start more than 1021 threads
in a python program, no matter what my ulimit or kernel settings are.
my program crashes with 'thread.error: can't start new thread' when it
tries to start the 1021st thread.

in addition to tweaking my ulimit settings, i have also verified that
the version of glibc i'm using has PTHREAD_THREADS_MAX defined to be
16374.  i have posted my test program below as well as the ulimit
settings i was running it with.  any help would be greatly appreciated.


thanks in advance,
dan smith


import os
import sys
import time
import threading

num_threads = int(sys.argv[1])

def run ():
   # just sleep
   time.sleep(1)

for i in range(1,num_threads+1):

   print 'starting thread %d' %i
   t=threading.Thread (None, run)

   t.start()

os._exit(1)



core file size(blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) 4
max memory size   (kbytes, -m) unlimited
open files(-n) 100
pipe size  (512 bytes, -p) 8
stack size(kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes(-u) unlimited
virtual memory(kbytes, -v) unlimited


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: thread limit in python

[danieldsmith]

also, on the same box a similar C program (posted below) has no problem
starting 5000+ threads.

#include 
#include 
#include 
#include 


void *
run (void *arg) {

   sleep(1000);
}

int main(int argc, char *argv[]) {

   int j;
   pthread_t tid;
   int num_threads = atoi(argv[1]);

   for (j=0; j < num_threads; j++) {

  pthread_create (&tid, NULL, run, NULL);
  printf("created thread %d\n",j);
  fflush(stdout);

   }

   sleep(1000);

}

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: thread limit in python

[danieldsmith]

disregard the C example.  wasn't checking the return code of
pthread_create.  the C program breaks in the same place, when creating
the 1021st thread.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: thread limit in python

[danieldsmith]

i modified my C test program (included below) to explicitly set the
default thread stack size, and i'm still running into the same
problem.  can you think of any other thing that would possibly be
limiting me?

and sorry to continue to post here.  since this is occurring in both c
and python, i think there's no question i'm running into an os limit.


#include 
#include 
#include 
#include 

void *
run (void *arg) {

   sleep(1000);
}

int main(int argc, char *argv[]) {

   int j;
   int ret;
   pthread_t tid;
   int num_threads = atoi(argv[1]);
   pthread_attr_t attr;
   int stacksize;

   pthread_attr_init(&attr);
  pthread_attr_getstacksize (&attr, &stacksize);
  printf("Default stack size = %d\n", stacksize);

   // set stack size to 64K
   pthread_attr_setstacksize (&attr, 0x1);

  pthread_attr_getstacksize (&attr, &stacksize);
  printf("New stack size = %d\n", stacksize);

   for (j=0; j < num_threads; j++) {

   ret = pthread_create (&tid, NULL, run, NULL);

   if (ret != 0) {
   printf("thread create failed\n",j);
   fflush(stdout);
   exit(0);
   }
   printf("created thread %d\n",j);
   fflush(stdout);

   }

   sleep(1000);
}

-- 
http://mail.python.org/mailman/listinfo/python-list