Re: [Tutor] IDLE shell indentation?
On 19/10/12 03:11, boB Stepp wrote: if zero: print(zero) else: print(phrase) This is a long standing bug/feature of IDLE and gets debated regularly on the IDLE mailing list. BTW There is a more feature rich unofficial version of Idle called IdleX by Roger Serwy here: http://www.ews.illinois.edu/~serwy/idlex/ which has: * Terminal-like behavior for PyShell. The cursor is confined to the prompt and up/down arrow keys navigate the command history, similar to the original Python interpreter. * Matplotlib support for interactive figures when using the subprocess. * Tabbed editor windows with drag'n'drop reordering. * SubCodes, similar to Matlab cell mode and Sagemath cells, for quick code prototyping without restarting the shell. * Integrated reindent.py support. * Improved code navigation with Code Browser. * Cython editing and execution support. (Version 0.15.1) * Line numbers for the Editor. * Clear PyShell Window without restarting PyShell. * Simple interface for enabling/disabling extensions. * SearchBar, Squeezer, and IDLE2HTML included. (Originally by Tal Einat, Noam Raphael, and Michael Haubenwallner) But it still has the broken indentation. The last thread I saw on the topic said: - On 06/21/2012 03:39 AM, Alan Gauld wrote: > On 21/06/12 02:16, Roger Serwy wrote: > >> I could write a patch to allow ps2 prompts if there's any serious interest. > > I would love that. > > This is one of the most common issues I get from users of my web tutorial. At least one email per month from beginners confused by IDLEs indentation. There is an issue for supporting ps1 and ps2. See http://bugs.python.org/issue13657 I'll start writing a patch later today. ... --- -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] How to unpack python-dateutil-2.0.tar.gz
On 19/10/2012 02:55, Richard D. Moores wrote: On Thu, Oct 18, 2012 at 6:08 PM, Mark Lawrence wrote: Could you please take a training course on how to use a search engine. First hit on google for "python dateutils install windows" is http://stackoverflow.com/questions/879156/how-to-install-python-dateutil-on-windows You're assuming I haven't searched and\or don't know how to. If you don't provide data people have to assume, guess or whatever. Still stuck. 2 problems it seems. Your link is 3 years old. It applies to python 2.6. So what? Why should the way that Python software gets installed vary from version to version? Dick ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor -- Cheers. Mark Lawrence. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] How to unpack python-dateutil-2.0.tar.gz
On Fri, Oct 19, 2012 at 1:55 AM, Mark Lawrence wrote: > On 19/10/2012 02:55, Richard D. Moores wrote: >> >> On Thu, Oct 18, 2012 at 6:08 PM, Mark Lawrence >> wrote: >> >>> Could you please take a training course on how to use a search engine. >>> First >>> hit on google for "python dateutils install windows" is >>> >>> http://stackoverflow.com/questions/879156/how-to-install-python-dateutil-on-windows >> >> >> You're assuming I haven't searched and\or don't know how to. > > > If you don't provide data people have to assume, guess or whatever. No need to rush into snarkiness, however. >> Still stuck. 2 problems it seems. Your link is 3 years old. It applies >> to python 2.6. > > So what? Why should the way that Python software gets installed vary from > version to version? Try it yourself and you'll see. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] How to unpack python-dateutil-2.0.tar.gz
On Fri, Oct 19, 2012 at 5:53 AM, Richard D. Moores wrote: > On Fri, Oct 19, 2012 at 1:55 AM, Mark Lawrence > wrote: >> >> So what? Why should the way that Python software gets installed vary from >> version to version? > > Try it yourself and you'll see. The setup.py needs setuptools as per jcoon's answer. For 3.x you need to install Distribute, which is a fork of setuptools. The line "from setuptools import setup" extensively patches distutils. It then honors the option 'install_requires = ["six"]' to fetch and install the dependency (this dependency is new to version 2.1) when you run "setup.py install". Alternatively, once you have Distribute installed, you can add the Scripts directory to the Windows PATH and just "easy_install python-dateutil" in an elevated console. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] How to unpack python-dateutil-2.0.tar.gz
On Fri, Oct 19, 2012 at 3:47 AM, eryksun wrote: > On Fri, Oct 19, 2012 at 5:53 AM, Richard D. Moores wrote: >> On Fri, Oct 19, 2012 at 1:55 AM, Mark Lawrence >> wrote: >>> >>> So what? Why should the way that Python software gets installed vary from >>> version to version? >> >> Try it yourself and you'll see. > > The setup.py needs setuptools as per jcoon's answer. For 3.x you need > to install Distribute, which is a fork of setuptools. The line "from > setuptools import setup" extensively patches distutils. It then honors > the option 'install_requires = ["six"]' to fetch and install the > dependency (this dependency is new to version 2.1) when you run > "setup.py install". Alternatively, once you have Distribute installed, > you can add the Scripts directory to the Windows PATH and just > "easy_install python-dateutil" in an elevated console. Yes. Thanks to your earlier post I was able to do the installation. Dick ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Help Passing Variables
Thanks David. This has been helpful in understanding a bit more on how parameters are passed through. > Date: Thu, 18 Oct 2012 04:44:55 -0400 > Subject: Re: [Tutor] Help Passing Variables > From: dwightdhu...@gmail.com > To: dangu...@hotmail.com > CC: tutor@python.org > > #A little more complex in terms of params: > > def SwapCaseAndCenter(*kwargs): > > if upper_or_lower == "upper": > print a_string.center(center_num).upper() > > if upper_or_lower == "lower": > print a_string.center(center_num).lower() > > a_string = raw_input("Give me a word, or letter: ") > upper_or_lower = raw_input("upper, or lower character(s): ") > center_num = int(raw_input("Where should number be centered?: ")) > SwapCaseAndCenter(a_string, upper_or_lower, center_num) > > > > -- > Best Regards, > David Hutto > CEO: http://www.hitwebdevelopment.com ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Objects, object references, object values and memory addresses
Great explanation Alan. I am a newbie at Python but descriptions like this really help me better understand. Thanks again. > To: tutor@python.org > From: alan.ga...@btinternet.com > Date: Thu, 18 Oct 2012 08:32:41 +0100 > Subject: Re: [Tutor] Objects, object references, object values and > memory addresses > > On 18/10/12 04:41, boB Stepp wrote: > > From Programming in Python 3, 2nd edition (p. 22-23): > > > a = ["Retention", 3, None] > b = ["Retention", 3, None] > a is b > > False > b = a > a is b > > True > > > > My current understanding is as follows: On the first two lines, two > > separate objects are defined, stored in two separate blocks of memory. > > Two separate list objects are created. In Python it is rarely helpful to > think about memory usage. Python is abstracted so far from the physical > machine that the connection usually depends on the creator of the > interpreter rather than the language. > > > These two objects just happen to have the same value, ["Retention", 3, > > None], stored in two separate locations. > > Yes, but note that its the fact that they are two separate lists that > matters. Even if the contents were the same objects they would still be > two lists: > > >>> x = [42] > >>> y = 'spam' > >>> z = None > >>> a = [x,y,z] > >>> b = [x,y,z] # different list, exact same content > >>> a is b > False > >>> a == b > True > >>> a = b > >>> a is b > True > >>> a == b > True > >>> > > > a and b, object references (Variables are what I used to call these. > > And most folks still do... > > > I ask: Which implementations of Python do this? In trying to make any > > code I write portable across as many platforms as possible, should I > > avoid using the identity operator, is (and its opposite, is not), > > except when I wish to compare to None? > > The point is that you don't know. And, even if you did, the very next > release might change it so you cannot rely on it. That's the real > message - do not assume a particular implementation technique because it > is not guaranteed to be that way or to stay that way. > > -- > Alan G > Author of the Learn to Program web site > http://www.alan-g.me.uk/ > > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] How to unpack python-dateutil-2.0.tar.gz
eryksun wrote: > On Thu, Oct 18, 2012 at 8:01 PM, Richard D. Moores wrote: > > > > Python 3.2.3 64 bit > > MS Windows 7 Home Premium 64-bit SP1 > > > > have forgotten how to unpack a .tar.gz file. Please remind me. > > shutil.unpack_archive(filename, extract_dir, 'gztar') > > http://docs.python.org/release/3.2.3/library/shutil.html#shutil.unpack_archive Neat! I usually just use `tar xzvf filename.tar.gz`. Ramit This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Help Passing Variables
On 10/19/2012 11:40 AM, Daniel Gulko wrote: > Thanks David. This has been helpful in understanding a bit more on how > parameters are passed through. Please don't top-post. it ruins the sequence of events. Your comments above happened after the parts you quote below. So why are they not after the things they follow? There is a long-standing convention in this forum, and many others, and why let Microsoft ruin it for all of us? >> Date: Thu, 18 Oct 2012 04:44:55 -0400 >> Subject: Re: [Tutor] Help Passing Variables >> From: dwightdhu...@gmail.com >> To: dangu...@hotmail.com >> CC: tutor@python.org >> >> #A little more complex in terms of params: >> >> def SwapCaseAndCenter(*kwargs): By convention kwargs is used for keyword arguments, while the * means positional arguments. The function uses none of them, so it's all bogus. You can't learn anything useful about argument passing from this example code, except what NOT to do. >> if upper_or_lower == "upper": >> print a_string.center(center_num).upper() >> >> if upper_or_lower == "lower": >> print a_string.center(center_num).lower() >> >> To understand the most elementary aspects of function passing, let's write a whole new function, and call it a few times. Paste this into a file, and try various things, and make sure you see how they work. Or reread Alan's email, which was also correct and clear. First point, a simple function may take zero arguments, or one, or two, or fifty. We'll start with positional arguments, which means order matters. Python also supports keyword arguments, default values, and methods, none of which I'll cover here. When you define a function, you specify its formal parameters. You do that by putting the parameter names inside the parentheses. Unlike most languages, you do NOT specify the types of any of these. But the order matters, and the number matters. So if we define a function: def truncate_string(a_string, width): temp = a_string[:width] return temp That function needs to be called with two arguments, which match the two formal parameters. They do NOT have to have the same names, and in fact they might well be literal strings or ints, or variables with string and int values. print truncate_string("this is a long string", 4) should print out "this" (without the quotes, of course) name = raw_input("give me a name") trunc_name = truncate_string(name, 8) print trunc_name truncate_string("aaa") #gives error, because it has the wrong number of arguments Does this help? -- DaveA ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
[Tutor] Segmentation Fault shenanigans with wx Gui
Hi all i have written a wx GUI which downloads json data from a server, and populate a listbox. Every time i populate a listbox, i am receiving Segmentation Faults. I have tried to retrieve data from the URL via separate thread, and to use events, but i am still getting a segmentation fault could anyone assist pls? here's relevant code. I m running python on Ubuntu 10.04 class WxSharesGUI(wx.Frame): def __init__(self, parent, id, title): wx.Frame.__init__(self, parent, id, title, size=(300, 300)) self.InitUI() def InitUI(self): self.load_button = wx.Button(self, wx.ID_ANY, "load ListBox") self.clear_button = wx.Button(self, wx.ID_ANY, "clear ListBox") self.load_button.Bind(wx.EVT_BUTTON, self.load_buttonClick) self.clear_button.Bind(wx.EVT_BUTTON, self.clear_buttonClick) self.Bind(EVT_RESULT, self.OnResult) self.lock = threading.RLock() # Creating Menu bar menubar = wx.MenuBar() newsMenu = wx.Menu() newsMenu.Append(ID_EDGAR, 'Edgar News', 'Edgar news') newsMenu.Append(ID_GLOBAL, 'Global Economy', 'Global economy') newsMenu.Append(ID_ECONOMY, 'Economy Data', 'Economy data') newsMenu.Append(ID_FX, 'FX', 'Fx data') futuresItem = newsMenu.Append(ID_FUTURES, 'FUTURES', 'FUTURES') exitItem = newsMenu.Append(wx.ID_EXIT, 'Quit', 'Quit application') menubar.Append(newsMenu, '&News') mgmtMenu = wx.Menu() shareMgmtItem = mgmtMenu.Append(ID_UPDATE_SHARE, 'Share Management', 'Share Management') ptfMgmtItem = mgmtMenu.Append(ID_UPDATE_PORTFOLIO, 'Portfolio Management', 'Portfolio Mgmt') resetItem = mgmtMenu.Append(ID_RESET, 'Reset Listbox', 'Portfolio Mgmt') menubar.Append(mgmtMenu, '&Management') self.SetMenuBar(menubar) self.Bind(wx.EVT_MENU, self.OnQuit, exitItem) self.Bind(wx.EVT_MENU, self.OnFutures, futuresItem) self.Bind(wx.EVT_MENU, self.OnShareMgmt, shareMgmtItem) self.Bind(wx.EVT_MENU, self.OnPtfMgmt, ptfMgmtItem) self.listBox1 = wx.ListBox(self, wx.ID_ANY, choices=[], style=wx.LB_EXTENDED) self.listBox1.Append('MAIN') self.listBox1.Append('INDEXES') self.listBox1.Append('CURRENCIES') self.listBox1.Append('HEDGE') self.listBox1.Append('CDS SPREADS') self.listBox1.Bind(wx.EVT_LISTBOX, self.OnListBox1Listbox, id=ID_FRAME1LISTBOX1) self.Bind(wx.EVT_MENU, self.OnResetList, resetItem) hbox = wx.BoxSizer(wx.HORIZONTAL) panel = wx.Panel(self, -1) self.list = AutoWidthListCtrl(panel) self.list.InsertColumn(0, 'Ticker')#, width=150) self.list.InsertColumn(1, 'Name')#', width=100) self.list.InsertColumn(2, 'MovingAvg200')#', width=100) self.list.InsertColumn(3, 'MovingAvg50')#', width=100) self.list.InsertColumn(4, 'Price')#, width=80) self.list.InsertColumn(5, 'Previous')#, width=80) self.list.InsertColumn(6, 'Change')#, width=80) self.list.InsertColumn(7, 'TotalValue')#, width=80) self.list.InsertColumn(8, 'Position')#', width=100) hbox.Add(self.list, 2, wx.EXPAND) panel.SetSizer(hbox) self.SetSize((900, 500)) self.SetTitle('Camel App Python Gui') self.Centre() sizer = wx.GridBagSizer(vgap=8, hgap=10) # pos=(row, column) span=(rowspan, columnspan) # wx.ALL puts the specified border on all sides sizer.Add(self.load_button, pos=(0, 0), flag=wx.ALL, border=5) # listbox spans 6 rows and 2 columns sizer.Add(self.clear_button, pos=(0, 25), flag=wx.ALL, border=5) sizer.Add(self.listBox1, pos=(2, 4), span=(8, 8), flag=wx.ALL|wx.EXPAND, border=5) sizer.Add(panel, pos=(10,1),span=(10, 30), flag=wx.ALL|wx.EXPAND, border=5) self.SetSizer(sizer) #self.Fit() self.Show(True) def OnQuit(self, e): self.Close() def OnResult(self, event): print 'onResult' ptf_data = event.data for item in ptf_data['portfolio']['items']: index = self.list.InsertStringItem(sys.maxint, item['ticker']) self.list.SetStringItem(index, 1, item['name']) self.list.SetStringItem(index, 2, str(item['movingAvg200'])) self.list.SetStringItem(index, 3, str(item['movingAvg50'])) self.list.SetStringItem(index, 4, str(item['price'])) self.list.SetStringItem(index, 5, str(item['previous'])) self.list.SetStringItem(index, 6, str(item['price'] - item['previous']) ) self.list.SetStringItem(index, 7, str(item['totalValue'])) self.list.SetStringItem(index, 8, str(item['position'])) def load_buttonClick(self,e): idx = self.listBox1.GetSelections()[0] ptf = self.listBox1.GetStrings()[idx] thread.start_new_thread(self.fetch, (ptf,)) self.list.ClearAll() def fetch(self, ptf): data= {'action':'portfolio', 'portfolioName':ptf} ptf_data = jsonLoader.openJsonRequest(data) print ptf_data['portfolio']['cash']
Re: [Tutor] Segmentation Fault shenanigans with wx Gui
Hello i found the problem. It's calling self.list.ClearAll that causes the segmentation fault. removing the call to ClearAll fixed my problem , but i still want to clear the list before i load new data.. could anyone assist? wkr marco On Fri, Oct 19, 2012 at 11:05 PM, Marco Mistroni wrote: > Hi all > i have written a wx GUI which downloads json data from a server, and > populate a listbox. > Every time i populate a listbox, i am receiving Segmentation Faults. > I have tried to retrieve data from the URL via separate thread, and to use > events, but i am still getting a segmentation fault > > could anyone assist pls? > > here's relevant code. I m running python on Ubuntu 10.04 > > class WxSharesGUI(wx.Frame): > > > def __init__(self, parent, id, title): > wx.Frame.__init__(self, parent, id, title, size=(300, 300)) > self.InitUI() > > def InitUI(self): > self.load_button = wx.Button(self, wx.ID_ANY, "load ListBox") > self.clear_button = wx.Button(self, wx.ID_ANY, "clear ListBox") > self.load_button.Bind(wx.EVT_BUTTON, self.load_buttonClick) > self.clear_button.Bind(wx.EVT_BUTTON, self.clear_buttonClick) > self.Bind(EVT_RESULT, self.OnResult) > self.lock = threading.RLock() > > # Creating Menu bar > menubar = wx.MenuBar() > newsMenu = wx.Menu() > newsMenu.Append(ID_EDGAR, 'Edgar News', 'Edgar news') > newsMenu.Append(ID_GLOBAL, 'Global Economy', 'Global economy') > newsMenu.Append(ID_ECONOMY, 'Economy Data', 'Economy data') > newsMenu.Append(ID_FX, 'FX', 'Fx data') > futuresItem = newsMenu.Append(ID_FUTURES, 'FUTURES', 'FUTURES') > exitItem = newsMenu.Append(wx.ID_EXIT, 'Quit', 'Quit application') > menubar.Append(newsMenu, '&News') > > mgmtMenu = wx.Menu() > shareMgmtItem = mgmtMenu.Append(ID_UPDATE_SHARE, 'Share Management', > 'Share Management') > ptfMgmtItem = mgmtMenu.Append(ID_UPDATE_PORTFOLIO, 'Portfolio > Management', 'Portfolio Mgmt') > resetItem = mgmtMenu.Append(ID_RESET, 'Reset Listbox', 'Portfolio > Mgmt') > menubar.Append(mgmtMenu, '&Management') > > > self.SetMenuBar(menubar) > > self.Bind(wx.EVT_MENU, self.OnQuit, exitItem) > self.Bind(wx.EVT_MENU, self.OnFutures, futuresItem) > self.Bind(wx.EVT_MENU, self.OnShareMgmt, shareMgmtItem) > self.Bind(wx.EVT_MENU, self.OnPtfMgmt, ptfMgmtItem) > > self.listBox1 = wx.ListBox(self, wx.ID_ANY, choices=[], > style=wx.LB_EXTENDED) > self.listBox1.Append('MAIN') > self.listBox1.Append('INDEXES') > self.listBox1.Append('CURRENCIES') > self.listBox1.Append('HEDGE') > self.listBox1.Append('CDS SPREADS') > self.listBox1.Bind(wx.EVT_LISTBOX, self.OnListBox1Listbox, > id=ID_FRAME1LISTBOX1) > self.Bind(wx.EVT_MENU, self.OnResetList, resetItem) > > hbox = wx.BoxSizer(wx.HORIZONTAL) > > panel = wx.Panel(self, -1) > > self.list = AutoWidthListCtrl(panel) > self.list.InsertColumn(0, 'Ticker')#, width=150) > self.list.InsertColumn(1, 'Name')#', width=100) > self.list.InsertColumn(2, 'MovingAvg200')#', width=100) > self.list.InsertColumn(3, 'MovingAvg50')#', width=100) > self.list.InsertColumn(4, 'Price')#, width=80) > self.list.InsertColumn(5, 'Previous')#, width=80) > self.list.InsertColumn(6, 'Change')#, width=80) > self.list.InsertColumn(7, 'TotalValue')#, width=80) > self.list.InsertColumn(8, 'Position')#', width=100) > > hbox.Add(self.list, 2, wx.EXPAND) > panel.SetSizer(hbox) > > > self.SetSize((900, 500)) > self.SetTitle('Camel App Python Gui') > self.Centre() > > sizer = wx.GridBagSizer(vgap=8, hgap=10) > # pos=(row, column) span=(rowspan, columnspan) > # wx.ALL puts the specified border on all sides > sizer.Add(self.load_button, pos=(0, 0), flag=wx.ALL, border=5) > # listbox spans 6 rows and 2 columns > sizer.Add(self.clear_button, pos=(0, 25), flag=wx.ALL, border=5) > > > sizer.Add(self.listBox1, pos=(2, 4), span=(8, 8), > flag=wx.ALL|wx.EXPAND, border=5) > > sizer.Add(panel, pos=(10,1),span=(10, 30), > flag=wx.ALL|wx.EXPAND, border=5) > self.SetSizer(sizer) > #self.Fit() > > self.Show(True) > > def OnQuit(self, e): > self.Close() > > def OnResult(self, event): > print 'onResult' > ptf_data = event.data > for item in ptf_data['portfolio']['items']: > index = self.list.InsertStringItem(sys.maxint, item['ticker']) > self.list.SetStringItem(index, 1, item['name']) > self.list.SetStringItem(index, 2, str(item['movingAvg200'])) > self.list.SetStringItem(index, 3, str(item['movingAvg50'])) > self.list.SetStringItem(index, 4, str(item['price'])) > self.list.SetStringItem(index, 5, str(item['previous'])) > self.list.SetStringItem(index, 6, str(item['price'] - > item
Re: [Tutor] Segmentation Fault shenanigans with wx Gui
On 20/10/12 09:10, Marco Mistroni wrote: Hello i found the problem. It's calling self.list.ClearAll that causes the segmentation fault. removing the call to ClearAll fixed my problem , but i still want to clear the list before i load new data.. could anyone assist? This is not a question about learning Python, it is a specialised wxPython question. I expect that it is a bug in wxPython. (As a segmentation fault, it's a pretty serious bug too.) I suggest you report it on the appropriate wxPython mailing list or bug tracker. Good luck! -- Steven ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Segmentation Fault shenanigans with wx Gui
On 19/10/12 23:05, Marco Mistroni wrote: Hi all i have written a wx GUI which downloads json data from a server, and populate a listbox. ... use events, but i am still getting a segmentation fault A segmentation fault is usually due to a fault down in the C code. With wxPython that means most likely a bug in the underlying wxWidgets libraries. Have a look in the wxWidgets forums for a known issue with your version. Or try asking there. I doubt it will be a Python problem - except in that it might be passing an integer (especially zero) where wxWidgets expects a pointer. HTH, -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] managing memory large dictionaries in python
On Oct 16, 2012, at 12:57 PM, Abhishek Pratap wrote: > Hi Guys > > For my problem I need to store 400-800 million 20 characters keys in a > dictionary and do counting. This data structure takes about 60-100 Gb > of RAM. > I am wondering if there are slick ways to map the dictionary to a file > on disk and not store it in memory but still access it as dictionary > object. Speed is not the main concern in this problem and persistence > is not needed as the counting will only be done once on the data. We > want the script to run on smaller memory machines if possible. > > I did think about databases for this but intuitively it looks like a > overkill coz for each key you have to first check whether it is > already present and increase the count by 1 and if not then insert > the key into dbase. > > Just want to take your opinion on this. > > Thanks! > -Abhi > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > http://mail.python.org/mailman/listinfo/tutor With apologies for coming to this late. Just a couple of comments - 1) If the keys were sorted before the counting operation was to take place, the problem could be run in successive batches in as small a foot print as desired. 2) Sorting in limited memory space was a highly developed art back in the early days of IBM mainframe computing (when one had to use tape rather than disk for temp store and 64k, yes "k", was a lot of memory). If you look at Knuth, Volume III: Sorting and Searching you will find several algorithms that would allow the sort to also run in as small a foot print as needed. Possibly not interesting, but just saying… Bill ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor