Stephen Nelson-Smith wrote:
> Martin,
>
>>def __iter__(self):
>>while True:
>>for logline in self.logfile:
>>heappush(self.heap, (timestamp(logline), logline))
>>if len(self.heap) >= self.jitter:
>>break
>>try:
Martin,
> def __iter__(self):
> while True:
> for logline in self.logfile:
> heappush(self.heap, (timestamp(logline), logline))
> if len(self.heap) >= self.jitter:
> break
> try:
> yield heappop(self.he
Stephen Nelson-Smith wrote:
> Nope - but I can look it up. The problem I have is that the source
> logs are rotated at 0400 hrs, so I need two days of logs in order to
> extract 24 hrs from to 2359 (which is the requirement). At
> present, I preprocess using sort, which works fine as long as
Marc Tompkins wrote:
On Sun, Nov 15, 2009 at 4:11 PM, Stephen Nelson-Smith wrote:
import gzip
from heapq import heappush, heappop, merge
Is this a preferred method, rather than just 'import heapq'?
It has a couple of advantages:
- convenience: if you "import heapq", then to
On Sun, Nov 15, 2009 at 8:57 PM, Marc Tompkins wrote:
> On Sun, Nov 15, 2009 at 4:11 PM, Stephen Nelson-Smith
> wrote:
>>
>> > import gzip
>> > from heapq import heappush, heappop, merge
>>
>> Is this a preferred method, rather than just 'import heapq'?
>>
> It has a couple of advantages:
...
> -
On Sun, Nov 15, 2009 at 4:11 PM, Stephen Nelson-Smith wrote:
> > import gzip
> > from heapq import heappush, heappop, merge
>
> Is this a preferred method, rather than just 'import heapq'?
>
> It has a couple of advantages:
- convenience: if you "import heapq", then to do a push you need to type
Hi Marty,
Thanks for a very lucid reply!
> Well, you haven't described the unreliable behavior of unix sort so I
> can only guess, but I assume you know about the --month-sort (-M) flag?
Nope - but I can look it up. The problem I have is that the source
logs are rotated at 0400 hrs, so I need t
Stephen Nelson-Smith wrote:
>> It's unclear from your previous posts (to me at least) -- are the
>> individual log files already sorted, in chronological order?
>
> Sorry if I didn't make this clear. No they're not. They are *nearly*
> sorted - ie they're out by a few seconds, every so often, bu
Hi Martin,
Thanks for a very detailed response. I'm about to head out, so I
can't put your ideas into practice yet, or get down to studying for a
while.
However, I had one thing I felt I should respond to.
> It's unclear from your previous posts (to me at least) -- are the
> individual log file
Stephen Nelson-Smith wrote:
> I think I'm having a major understanding failure.
Perhaps this will help ...
http://www.learningpython.com/2009/02/23/iterators-iterables-and-generators-oh-my/
> So in essence this:
>
> logs = [ LogFile( "/home/stephen/qa/ded1353/quick_log.gz", "04/Nov/2009" ),
>
"Stephen Nelson-Smith" wrote
List 1List 2List 3
(1, cat) (2, fish) (1, cabbage)
(4, dog) (5, pig) (2, ferret)
(5, phone) (6, horse) (3, sausage)
Won't this result in the lowest number *per row* being added to the
new list? Or am I misunderstanding how it w
On Sat, Nov 14, 2009 at 8:49 AM, Stephen Nelson-Smith wrote:
> Hi Wayne,
>
> > Just write your own merge:
> > (simplified and probably inefficient and first thing off the top of my
> head)
> > newlist = []
> > for x, y, z in zip(list1, list2, list3):
>
> I think I need something like izip_longest
Hi Wayne,
> Just write your own merge:
> (simplified and probably inefficient and first thing off the top of my head)
> newlist = []
> for x, y, z in zip(list1, list2, list3):
I think I need something like izip_longest don't I, since the list wil
be of varied length?
Also, where do these lists c
Gah! Failed to reply to all again!
On Sat, Nov 14, 2009 at 1:43 PM, Stephen Nelson-Smith
wrote:
> Hi,
>> I'm not 100% sure to understand your needs and intention; just have a try.
>> Maybe what you want actually is rather:
>>
>> for log in logs:
>> for line in log:
>> print l
>
> Assuming yo
On Sat, Nov 14, 2009 at 7:34 AM, Stephen Nelson-Smith wrote:
>
> Well... what I want to do is create a single, sorted list by merging a
> number of other sorted lists.
>
>
Just write your own merge:
(simplified and probably inefficient and first thing off the top of my head)
newlist = []
for x, y
Hi,
>> for log in logs:
>> l = log.getline()
>> print l
>>
>> This gives me three loglines. How do I get more? Other than while True:
>>
> I presume that what you want is to get all lines from each log.
Well... what I want to do is create a single, sorted list by merging a
number of other sor
Le Fri, 13 Nov 2009 17:58:30 +,
Stephen Nelson-Smith s'exprima ainsi:
> I think I'm having a major understanding failure.
>
> So having discovered that my Unix sort breaks on the last day of the
> month, I've gone ahead and implemented a per log search, using heapq.
>
> I've tested it with
I think I'm having a major understanding failure.
So having discovered that my Unix sort breaks on the last day of the
month, I've gone ahead and implemented a per log search, using heapq.
I've tested it with various data, and it produces a sorted logfile, per log.
So in essence this:
logs = [
18 matches
Mail list logo
