help in rnopen in bsddb?

2007-06-19 Thread Squzer 
How ca i create a databse file using rnopen?

How to set the key for this file?
i am tried but i got error.. please can any one tell...

-- 
http://mail.python.org/mailman/listinfo/python-list

Mirror Urls

2007-06-25 Thread Squzer 
I am involving in the development of a crawler. i need my script to
detect the mirror urls of the page. so that i can ignor the urls.
Please tell me any idea to detect..

-- 
http://mail.python.org/mailman/listinfo/python-list

Memory problem with Python

2007-06-17 Thread Squzer Crawler 


i am developing distributed environment in my college using Python. I
am using therads in client for downloading wepages. Even though i am
reusing the thread, memory usage get increased. I don know why.? I am
using BerkelyDB for URLQueue, BeautifulShop for Parsing the webpages.

Any idea of redusing the memory usage.. please tell me

I want my program to run in bouded Memory.. Please..

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Memory problem with Python

2007-06-17 Thread Squzer Crawler 
On Jun 18, 11:06 am, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote:
> On Jun 17, 8:51 pm, Squzer Crawler <[EMAIL PROTECTED]> wrote:
>
> > i am developing distributed environment in my college using Python. I
> > am using therads in client for downloading wepages. Even though i am
> > reusing the thread, memory usage get increased. I don know why.? I am
> > using BerkelyDB for URLQueue, BeautifulShop for Parsing the webpages.
>
>   Isn't the increased memory resulted from storing the already
> processed pages?
>
>   Look first at all places where your code instantiates new
> objects - and make sure you don't keep references to such objects that
> are not needed anymore.
>
>   Also, reusing threads has nothing to do with saving memory - but
> with saving on thread creation time, if I understand your problem
> description.

what about the cyclic reference.. can i use GC in my program..

if so, please tell me how to implement.. i am calling the gc.collect()
at the enf of the fetching.. Will it reduce my program speed. Else in
which way i can call it..?

please tell me

-- 
http://mail.python.org/mailman/listinfo/python-list