Re: [Tutor] airflow dag
Hello,
See this is the code:
from airflow import DAG
from datetime import datetime,timedelta
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime.now(),
'email': ['airf...@airflow.com'],
'email_on_failure': False,
'email_on_retry': False
}
MAIN_DAG='check_dag'
dag = DAG(dag_id=MAIN_DAG, default_args=default_args,
schedule_interval=None)
with open(file, "r") as f:
payload = f.read() # Reading the json data from a file
SimpleHttpOperator( # creating cluster using SimpleHttpOperator
task_id='cluster_create',
method='POST',
http_conn_id='qubole_default',
# for directing to https://qa.qubole.net/api
endpoint='/v2/clusters?auth_token=%s' % (passwd),
data=payload,
headers={"Content-Type": "application/json"},
params={'auth_token': passwd},
response_check=lambda response: True if response.status_code == 200
else False,
dag=dag
)
So this code use simplehttpopeator(
https://airflow.incubator.apache.org/code.html) which is used to create the
cluster or post the information on this url(qa.qubole.net/api) this is same
as requests.post() in python.
As you can see i am using passwd here
>endpoint='/v2/clusters?auth_token=%s' % (passwd),
and i am making many dags like that which are connected and they all use
the password(auth_token) to send the data so i want to pass passwd as
command line argument.
Argument parser is not working here
i tried argument parser and also sys.argv but it doesn't work.
from airflow import DAG
from airflow.operators import SimpleHttpOperator
from datetime import datetime,timedelta
import argparse
import os
parser = argparse.ArgumentParser()
parser.add_argument("passwd")
args = parser.parse_args()
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime.now(),
'email': ['airf...@airflow.com'],
'email_on_failure': False,
'email_on_retry': False
}
print args.passwd
MAIN_DAG='check_dag'
dag = DAG(dag_id=MAIN_DAG, default_args=default_args,
schedule_interval=None)
file=os.path.join(os.path.dirname(os.path.abspath(__file__)),"check.json")
with open(file, "r") as f:
payload = f.read() # Reading the json data from a file
SimpleHttpOperator( # creating cluster using SimpleHttpOperator
task_id='cluster_create',
method='POST',
http_conn_id='qubole_default',
# for directing to https://qa.qubole.net/api
endpoint='/v2/clusters?auth_token=%s' % (args.passwd),
data=payload,
headers={"Content-Type": "application/json"},
params={'auth_token': args.passwd},
response_check=lambda response: True if response.status_code == 200
else False,
dag=dag
)
if i use this its not able to trigger the dag.
for triggering the dag we use
airflow trigger_dag dagid -r runid
argumentparser work in python
python script.py --passwd passwd #this work
but this is not a python script
airflow trigger_dag dagid -r runid --passwd passwd
sorry for my english.
On Sun, May 28, 2017 at 9:17 PM, shubham goyal wrote:
> Hello all,
>
>
> You are not understanding my point I understand command line argument
> ,sys.argv[],argparser to pass arguments via command line but I want to do
> these things when I trigger the dag
> And how I trigger the dag
> airflow trigger_dag dagname -s runid
>
> This triggering of dag doesn't support to pass command line argument but I
> want to pass authentication token as command line argument when I trigger
> the dag(you can see in airflow docs) so I am asking is there a way to do it.
>
> I tried argument parser but when I use that it's not able to trigger the
> dag.
>
> I want something like this (in cli)
>
> airflow trigger_dag dagname -r runid --passwd auth_token
>
> And want to access this authtoken in Python script.
>
>
> On May 28, 2017 8:18 PM, "Francois Dion" wrote:
>
>> My mailbox if full of similar stories: companies dumping airflow on their
>> ETL (or similar) group. Those who knew Python succeeded, those who didn't
>> failed, and some even moved to other companies because they couldn't cope
>> with all this complexity dumped on them all at once.
>>
>> Moral of the story, it is best to learn python first, become good at it,
>> then get into specific tools.
>>
>> And, of course, this is not specific to a relatively straightforward
>> module
>> / application like airflow. I've seen the same with scikit-learn. Same
>> also
>> in other languages. And to make matters worse, the curricula (and books)
>> always starts with a crash course in a specific language, then go on and
>> on
>> about the application, perpetuating the myth that learning the programming
>> language well is totally unimportant.
>>
>> Francois
>>
>> On Sun, May 28, 2017 at 9:46 AM, Alan Gauld via Tutor
>> wrote:
>>
>> > On 28/05/17 04:37, shubham goyal wrote:
>> > > Does anybody have answer?
>> >
>> > You received two answers, b
Re: [Tutor] Counting a string backwards
Hi Alan Thank you very much, I got it. So in this case, there is no need to specify where it ends. In fact, even if I wanted to specify the ending, I can't! Thank you! On Sun, May 28, 2017 at 7:19 PM, Alan Gauld via Tutor wrote: > On 28/05/17 18:58, C W wrote: > > > Now if I do case 2, > >> print(great[-3:-1]) > >> me > > > > Where did the exclamation mark go in case 2? > > > > I was told the count begins at zero, that's true going forward, but not > > backwards. > > Its not about where the count starts its about where it finishes. > It finishes 1 item before the second index. so in this case > you get the items at -3 and -2. > > If you specify a third parameter you can change the > direction of the count thus: > > >>> great[-1:-3:-1] > '!e' > > So now you get the characters at -1 and -2 (ie in reverse order) > > If you want them in original order starting at -3 just omit > the -1: > > >>> great[-3:] > 'me!' > > HTH, > > -- > Alan G > Author of the Learn to Program web site > http://www.alan-g.me.uk/ > http://www.amazon.com/author/alan_gauld > Follow my photo-blog on Flickr at: > http://www.flickr.com/photos/alangauldphotos > > > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Counting a string backwards
Wow, that's the best explanation I've seen so far, now it's gonna stick with me! Thank you! On Sun, May 28, 2017 at 10:00 PM, Steven D'Aprano wrote: > On Sun, May 28, 2017 at 01:58:22PM -0400, C W wrote: > > Dear Python list, > > > > I am having trouble understanding the following. > [...] > > > The way to think about string indexing and slicing is that the index > positions mark *between* the characters. Take your string: > > Machine learning is awesome! > > For brevity, I'll just use the first word: > > Machine > > Imagine slicing it between the characters. I'll mark the cuts with a > vertical bar: > > |M|a|c|h|i|n|e| > > and add indexes. The indexes will only line correctly if you use a > monspaced or fixed width font like Courier, otherwise things may not > line up correctly. > > |M|a|c|h|i|n|e| > 0 1 2 3 4 5 6 7 > > Here they are again starting from the right, I've spread things out a > bit to fit in the minus signs: > >|M |a |c |h |i |n |e | >-7 -6 -5 -4 -3 -2 -1 0 > > Notice that 0 gets used twice. Of course, that's impossible, because it > would be ambiguous. If you give 0 as an index, how does Python know > whether you mean 0 at the start or 0 or the end? So the simple rule > Python uses is that 0 *always* means the start. > > When you give a single index, Python always uses the character > immediately to the right of the cut: > > s = "Machine" > s[0] > => returns "M" > > s[-1] > => returns "e" > > s[7] > => raises an exception, because there is no character to the right > > When you give two indexes, using slice notation, Python returns the > characters BETWEEN those cuts: > > s[0:7] > => returns "Machine" > > s[1:-1] > => returns "achin" > > Because 0 always means the start of the string, how do you slice to the > end? You can use the length of the string (in this case, 7) or you can > leave the ending position blank, and it defaults to the length of the > string: > > s[1:] # means the same as [1:len(s)] > > You can leave the starting position blank too, it defaults to 0: > > s[:] # means the same as [0:len(s)] > > So remember that slices always cut *between* the index positions. > > > Things get complicated when you include a step (or stride), especially > when the step is negative. For step sizes other than 1, it is > probably best to think of looping over the string: > > py> s = "Nobody expects the Spanish Inquisition!" > py> s[-1:1:-2] > '!otsun snp h tex db' > > is somewhat like: > > for i in range(len(s)-1, 1, -2): > print s[i] > > > > -- > Steve > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] How to deploy seamless script updates to your "clients"?
What does a webpage has to do with it? (topic : web scraping) well i meant you put your script in a webpage no need to put any html tags if you wish. then your program has two files. an updater and the script the updater on startup scrape your web page and see if the text is same as the script file if yes : run script file if not : override the script file with the text of the web page Abdur-Rahmaan Janhangeer, Mauritius abdurrahmaanjanhangeer.wordpress.com On 25 May 2017 04:11, "Juan C." wrote: > I have some Python 3.6.0 scripts that my co-workers use for some small > and medium tasks. Whenever I have time I fix some bugs and add some > features to said scripts to make their lives (and mine :D) easier, but > there's a problem: I need to send a new script via email/chat/whatever > and they have to replace it wherever they use it, such a hassle. > > How would I go to put a "update module" inside my script? I was > thinking about using Git, because those scripts are already in > personal repositories so that I can keep track of changes. Will I have > to setup any special permissions so that the scripts can pull requests > from my private repositories? > > I was thinking of something like "check for update before start", if > an update is found the script would replace itself with the newer > version and restart, is that possible? For example, 'cool-tool.py' > v0.2 starts and find that version v0.3 is out, so it downloads and > replace 'cool-tool.py' code with newer code and restart as v0.3. > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] 3D plotting
Hello!
I am very new to python and am using it for some astronomy data analysis.
What I am trying to is plot a 3D scatter plot of three variables that are
being read from a file. I've tried using code from other examples of 3d
plots but, having little idea how the code functions, I have had very
little luck. Here is the relevant code I am hoping to augment.
>
import numpy as np
import scipy as sp
import math
from astropy.io import fits
import matplotlib.pyplot as plot
from pylab import *
import matplotlib as mpl
from mpl_toolkits.mplot3d import Axes3D
def sread(infile):
hdulist = fits.open(infile)
tdata = hdulist[1].data
ra = tdata['ra'] # degrees
dec = tdata['dec'] # degree
p = tdata['parallax'] # mas
return ra,dec,p
ra,dec,p = sread('TgasSource_000-000-001.fits')
for i in range(len(ra)):
if ra[i] > 164 and ra[i] < 166 and dec[i] > 49 and dec[i] < 51 :
plot(ra[i],dec[i],'bo')
show()
>
so I'm hoping to plot ra and dec on the x and y axes with p on the z axis.
Any way you can give me an idea how to implement this?
Thanks!
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] 3D plotting
On 29/05/17 21:40, Jack Lambert wrote: > I am very new to python and am using it for some astronomy data analysis. > What I am trying to is plot a 3D scatter plot of three variables that are > being read from a file. I've tried using code from other examples of 3d > plots but, having little idea how the code functions, I have had very > little luck. I'm not sure where you got the examples but the official site contains a wealth of samples. http://matplotlib.org/users/screenshots.html If you find one that's close to what you want, grab the source code and try to map your data to that code. That may be what you've already done, in which case you are going to have to spend a bit of time learning the tools. There is a specific matplotlib support forum however who may be better placed than this list to help: https://mail.python.org/mailman/listinfo/matplotlib-users -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Problem with if statements and else statements
On 29May2017 01:17, Alan Gauld wrote:
On 29/05/17 00:12, Alex Kleider wrote:
Would
if Month in {'January', '1'}:
be even better? (regarding efficiency perhaps? Trivial point, I know,
but just wondering.)
If in doubt try it out and profile/time it.
But I don't think it will make much difference since ultimately
it still has to test each value (although a hashing algorithm
may be involved that works on partial matches...) But if in
doubt...
Hi Alex,
As written it should be a bit slower: to construct a set each member get tested
for presence. The cost is in making the set, not in searching it.
_However_, supposing your program were doing this a lot. You might well have a
global (or, better, long lived shared object) containing a set that has already
been constructed. Then:
if Month in the_set:
is very fast; constant time. Whereas as you would expect, checking a list is
linear with the size of the list.
So, using a list:
seen = []
for item in some_item_generator():
if item in seen:
continue
seen.append(item)
... do stuff with item, which is new ...
The cost of the "if" goes up linearly as you add more items.
Using a set:
seen = {}
for item in some_item_generator():
if item in seen:
continue
seen.add(item)
... do stuff with item, which is new ...
The second version will be much more effiient as the "seen" set grows; the
lookup time on the set is essentially O(1) (constant time).
But for an ad hoc 2 element list as in your original example the difference
will be pretty small; making the 2 element set _should_ be slightly more
expensive, and isn't the common idiom (==> less readable). Personally I use:
if value in ('a', 'b', 'c'):
BTW, in Python we tend to use named like "Fred" for classes (or factories), and
"fred" for regular variables. And "FRED" for things that would be constants in
other languages. Eg:
MAX_THINGS = 16
class Foo:
def FooBah(x):
return Foo(x, style="bah")
for fred in :
Cheers,
Cameron Simpson
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Problem with if statements and else statements
On 2017-05-29 16:08, Cameron Simpson wrote: snip BTW, in Python we tend to use named like "Fred" for classes (or factories), and "fred" for regular variables. And "FRED" for things that would be constants in other languages. Eg: MAX_THINGS = 16 class Foo: def FooBah(x): return Foo(x, style="bah") Yes, I've been aware of these conventions except for the last- I haven't knowingly used 'factories.' I assume from the context that factories return class instances so it makes sense to use the same conventions that apply to class names. ...and thanks again for explaining. Alex ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Using venv
Jim writes: > It has been suggested to me that I should use a virtual environment > and venv would be a good choice. I've read through PEP405 and this > link [1]. One possible confusion is the terminology. You have “a virtual environment” when you create one. The ‘venv’ module is not itself a virtual environment; it is what you use to create one :-) > Though some of it seems a little confusing to me, I'm sure I can get > it up and running. This question seems a little dumb and maybe I am > being a little dense, but then what? * In each shell where you want to be working in that virtual Python environment, activate the virtualenv (by running the commands from its corresponding ‘activate’ script; e.g. ‘source $VENV/bin/activate’). * Do things requiring Python. > Your program/script is done how do you run it? Do you always have to > activate your venv and run it from there? To get the benefits of that particular virtualenv, yes. > I like to run Tkinter programs from a launcher. Would that be possible > and what would the command look like? How are the programs installed? Can you customise how they're launched? > Lately I have been writing some Libreoffice calc macros and evaluating > pyspread for it's macro capability. Would modules installed in my venv > be available to the spreadsheet programs? The trick is that the environment variables for a process need to be set either when the program starts, or within the program; it can't be done externally. That isn't special to Python, it is how processes work. The virtualenv is activated by setting particular shell environment variables to specific values. If you can do that, the answer is yes. The ‘source $VENV/bin/activate’ command just runs shell commands to set those environment variables. It's not the only way to set those environment variables. Other Python-bundled programs, like LibreOffice or Blender, will likely have their own ways of activating a virtualenv; or at least you'll probably find people in those communities discussing how to do it. -- \ “I distrust those people who know so well what God wants them | `\to do to their fellows, because it always coincides with their | _o__) own desires.” —Susan Brownell Anthony, 1896 | Ben Finney ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Using venv
On 05/29/2017 08:09 PM, Ben Finney wrote: You should probably disregard this message as I have since solved the problem I was asking about. I originally wrote this message on 01/27/17, how it make it back to the list I don't know. Regards, Jim Jim writes: It has been suggested to me that I should use a virtual environment and venv would be a good choice. I've read through PEP405 and this link [1]. One possible confusion is the terminology. You have “a virtual environment” when you create one. The ‘venv’ module is not itself a virtual environment; it is what you use to create one :-) Though some of it seems a little confusing to me, I'm sure I can get it up and running. This question seems a little dumb and maybe I am being a little dense, but then what? * In each shell where you want to be working in that virtual Python environment, activate the virtualenv (by running the commands from its corresponding ‘activate’ script; e.g. ‘source $VENV/bin/activate’). * Do things requiring Python. Your program/script is done how do you run it? Do you always have to activate your venv and run it from there? To get the benefits of that particular virtualenv, yes. I like to run Tkinter programs from a launcher. Would that be possible and what would the command look like? How are the programs installed? Can you customise how they're launched? Lately I have been writing some Libreoffice calc macros and evaluating pyspread for it's macro capability. Would modules installed in my venv be available to the spreadsheet programs? The trick is that the environment variables for a process need to be set either when the program starts, or within the program; it can't be done externally. That isn't special to Python, it is how processes work. The virtualenv is activated by setting particular shell environment variables to specific values. If you can do that, the answer is yes. The ‘source $VENV/bin/activate’ command just runs shell commands to set those environment variables. It's not the only way to set those environment variables. Other Python-bundled programs, like LibreOffice or Blender, will likely have their own ways of activating a virtualenv; or at least you'll probably find people in those communities discussing how to do it. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
