Re: [Tutor] airflow dag
Does anybody have answer? On May 27, 2017 1:53 PM, "Cameron Simpson" wrote: > On 25May2017 18:02, Alan Gauld wrote: > >> On 25/05/17 13:15, shubham goyal wrote: >> >>> I want to ask that can we pass the parameters as commandline arguments in >>> airflow when we are triggering the dag and access them inside the dag's >>> python script/file. >>> >> >> I've no idea what a dag is. >> > > It's a directed acyclic graph, a very standard computer science term. > > Determining that that is what AirFlow means when it says "DAG" was > surprisingly hard, but here we are: > > http://airflow.apache.org/ > > (So the hardness comes from a web search not dropping me on their front > page:-) > > Looks like a tool for data driven computation using a graph of data > pipelines. By restricting it to a DAG on can be certain the computation > will finish and the termination condition is trivially satisfied. No > feedback loops. > > Cheers, > Cameron Simpson > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] airflow dag
On Sun, May 28, 2017 at 09:07:10AM +0530, shubham goyal wrote: > Does anybody have answer? Answer to what question? Peter has already answered your question about passing parameters as command line arguments. Do you have another question? -- Steve ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] airflow dag
On Sat, May 27, 2017 at 02:25:24PM +1000, Cameron Simpson wrote: > On 25May2017 18:02, Alan Gauld wrote: > >On 25/05/17 13:15, shubham goyal wrote: > >>I want to ask that can we pass the parameters as commandline arguments in > >>airflow when we are triggering the dag and access them inside the dag's > >>python script/file. > > > >I've no idea what a dag is. > > It's a directed acyclic graph, a very standard computer science term. When spelled as "DAG". When spelled as "dag" in an email about airflow, it looks like a typo for "drag". Or possibly: a lock of matted or dung-coated wool; a person with a good sense of humour (Australian and New Zealand slang); a flap along the edge of a garment (used in medieval clothing). -- Steve ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] airflow dag
On 28/05/17 04:37, shubham goyal wrote: > Does anybody have answer? You received two answers, both of which asked you to try something and get back to us for more information. Did you try printing sys.argv? What was the result? And did you try Peter's argparse code? You still haven't explained what your dag is? What library are you using? It may treat args in a non standard way, but we can't tell if you don't give us the information we need to help you. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] airflow dag
My mailbox if full of similar stories: companies dumping airflow on their ETL (or similar) group. Those who knew Python succeeded, those who didn't failed, and some even moved to other companies because they couldn't cope with all this complexity dumped on them all at once. Moral of the story, it is best to learn python first, become good at it, then get into specific tools. And, of course, this is not specific to a relatively straightforward module / application like airflow. I've seen the same with scikit-learn. Same also in other languages. And to make matters worse, the curricula (and books) always starts with a crash course in a specific language, then go on and on about the application, perpetuating the myth that learning the programming language well is totally unimportant. Francois On Sun, May 28, 2017 at 9:46 AM, Alan Gauld via Tutor wrote: > On 28/05/17 04:37, shubham goyal wrote: > > Does anybody have answer? > > You received two answers, both of which asked > you to try something and get back to us for more > information. Did you try printing sys.argv? > What was the result? > > And did you try Peter's argparse code? > > You still haven't explained what your dag is? > What library are you using? It may treat args > in a non standard way, but we can't tell if > you don't give us the information we need to > help you. > > -- > Alan G > Author of the Learn to Program web site > http://www.alan-g.me.uk/ > http://www.amazon.com/author/alan_gauld > Follow my photo-blog on Flickr at: > http://www.flickr.com/photos/alangauldphotos > > > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > -- raspberry-python.blogspot.com - www.pyptug.org - www.3DFutureTech.info - @f_dion ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] How to deploy seamless script updates to your "clients"?
On 05/27/2017 05:33 AM, M Hashmi wrote: > That's where Git or other version control systems come in. You can edit or > upgrade creating a branch and when a branch tested at your side. You can > push it with new tag like "some module changed". I guess this is how it > works for everyone. All your team will get replicated code once you merge > branch. Its simple and its reliable.all your resource only may need to > work around a dozen shell commands. Letting git (or mercurial, or some other version control system) handle this is a nice approach. It doesn't have to be a public server, it can of course be an inside-company git server. Endorsed. === I did end up curious how hard it would be to write the check code, so after a little hunting around I put together a simple one (might become the source for a blog entry someday, or not). The way this one works is an update file in yaml format is in a Known Location (I used github), it's fetched, and saved in a local cache file. The update file describes whatever different products are covered, with a version, checksum, release date, download url, whatever else you need to describe. In the simpleminded version it just returns true/false as to whether an update is needed (probably would want to return the url to download in the "true" case). Each script could import this and call it with its name as the argument to see if it needs to update itself. It's not huge, but too big for an email; lives here if anyone is curious: https://github.com/mwichmann/PyBlog/tree/master/updater.d no warranties, etc. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] Counting a string backwards
Dear Python list, I am having trouble understanding the following. If I do case 1, great = "Machine learning is awesome!" > print(great[-1]) > ! Now if I do case 2, > print(great[-3:-1]) > me Where did the exclamation mark go in case 2? I was told the count begins at zero, that's true going forward, but not backwards. What is wrong? Thank you! Mike ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Problem with if statements and else statements
On 05/27/2017 06:14 PM, boB Stepp wrote:
> Hello Jalen!
>
> On Sat, May 27, 2017 at 4:19 PM, Jalen Barr wrote:
>>
>> In this code it always changes the PlaceHolder to 0 no matter what Month is
>> set to
>>
>> Month ="September"
>>
>> if Month == "January" or "1":
>> PlaceHolder = 0
>
> This must be written as:
>
> if Month == "January" or Month == "1":
> PlaceHolder = 0
>
> The way you wrote it is interpreted as:
>
> if (Month == "January") or ("1"):
> PlaceHolder = 0
and since "1" is always True, PlaceHolder is always set to 0.
FWIW, if checking for multiples, you could also write:
if Month in ['January', '1']:
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Problem with if statements and else statements
On 2017-05-28 13:13, Mats Wichmann wrote:
FWIW, if checking for multiples, you could also write:
if Month in ['January', '1']:
Would
if Month in {'January', '1'}:
be even better? (regarding efficiency perhaps? Trivial point, I know,
but just wondering.)
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Counting a string backwards
On 28/05/17 18:58, C W wrote: > Now if I do case 2, >> print(great[-3:-1]) >> me > > Where did the exclamation mark go in case 2? > > I was told the count begins at zero, that's true going forward, but not > backwards. Its not about where the count starts its about where it finishes. It finishes 1 item before the second index. so in this case you get the items at -3 and -2. If you specify a third parameter you can change the direction of the count thus: >>> great[-1:-3:-1] '!e' So now you get the characters at -1 and -2 (ie in reverse order) If you want them in original order starting at -3 just omit the -1: >>> great[-3:] 'me!' HTH, -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Problem with if statements and else statements
On 29/05/17 00:12, Alex Kleider wrote:
> On 2017-05-28 13:13, Mats Wichmann wrote:
>
>> FWIW, if checking for multiples, you could also write:
>>
>> if Month in ['January', '1']:
>
> Would
>
if Month in {'January', '1'}:
>
> be even better? (regarding efficiency perhaps? Trivial point, I know,
> but just wondering.)
If in doubt try it out and profile/time it.
But I don't think it will make much difference since ultimately
it still has to test each value (although a hashing algorithm
may be involved that works on partial matches...) But if in
doubt...
--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Counting a string backwards
On Sun, May 28, 2017 at 01:58:22PM -0400, C W wrote: > Dear Python list, > > I am having trouble understanding the following. [...] The way to think about string indexing and slicing is that the index positions mark *between* the characters. Take your string: Machine learning is awesome! For brevity, I'll just use the first word: Machine Imagine slicing it between the characters. I'll mark the cuts with a vertical bar: |M|a|c|h|i|n|e| and add indexes. The indexes will only line correctly if you use a monspaced or fixed width font like Courier, otherwise things may not line up correctly. |M|a|c|h|i|n|e| 0 1 2 3 4 5 6 7 Here they are again starting from the right, I've spread things out a bit to fit in the minus signs: |M |a |c |h |i |n |e | -7 -6 -5 -4 -3 -2 -1 0 Notice that 0 gets used twice. Of course, that's impossible, because it would be ambiguous. If you give 0 as an index, how does Python know whether you mean 0 at the start or 0 or the end? So the simple rule Python uses is that 0 *always* means the start. When you give a single index, Python always uses the character immediately to the right of the cut: s = "Machine" s[0] => returns "M" s[-1] => returns "e" s[7] => raises an exception, because there is no character to the right When you give two indexes, using slice notation, Python returns the characters BETWEEN those cuts: s[0:7] => returns "Machine" s[1:-1] => returns "achin" Because 0 always means the start of the string, how do you slice to the end? You can use the length of the string (in this case, 7) or you can leave the ending position blank, and it defaults to the length of the string: s[1:] # means the same as [1:len(s)] You can leave the starting position blank too, it defaults to 0: s[:] # means the same as [0:len(s)] So remember that slices always cut *between* the index positions. Things get complicated when you include a step (or stride), especially when the step is negative. For step sizes other than 1, it is probably best to think of looping over the string: py> s = "Nobody expects the Spanish Inquisition!" py> s[-1:1:-2] '!otsun snp h tex db' is somewhat like: for i in range(len(s)-1, 1, -2): print s[i] -- Steve ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Problem with if statements and else statements
Alex Kleider wrote:
> On 2017-05-28 13:13, Mats Wichmann wrote:
>
>> FWIW, if checking for multiples, you could also write:
>>
>> if Month in ['January', '1']:
>
> Would
>
if Month in {'January', '1'}:
>
> be even better? (regarding efficiency perhaps? Trivial point, I know,
> but just wondering.)
Yes, according to timeit there is already a significant difference:
$ python3 -m timeit -s 'month = "September"' 'month == "January" or month ==
"1"'
1000 loops, best of 3: 0.126 usec per loop
$ python3 -m timeit -s 'month = "September"' 'month in ["January", "1"]'
1000 loops, best of 3: 0.108 usec per loop
$ python3 -m timeit -s 'month = "September"' 'month in {"January", "1"}'
1000 loops, best of 3: 0.0684 usec per loop
As the size of the list/set grows you'll see the effects of O(N) versus
O(1):
$ python3 -m timeit -s 'N = 10**6; numbers = list(range(N))' 'N in numbers'
10 loops, best of 3: 32.5 msec per loop
$ python3 -m timeit -s 'N = 10**6; numbers = set(range(N))' 'N in numbers'
1000 loops, best of 3: 0.0921 usec per loop
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Counting a string backwards
Steven D'Aprano wrote:
> Because 0 always means the start of the string, how do you slice to the
> end? You can use the length of the string (in this case, 7) or you can
> leave the ending position blank, and it defaults to the length of the
> string:
For completeness, there's a third option, None, which is equivalent to a
missing value:
>>> "Machine!"[None:None]
'Machine!'
Sometimes this is even moderately useful:
>>> def clip(s, n):
... return s[:-n or None]
...
>>> for i in reversed(range(5)):
... print(i, clip("Machine!", i))
...
4 Mach
3 Machi
2 Machin
1 Machine
0 Machine!
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
