[Tutor] cvxopt install problem
Hi All-- I posted this question to another group but have not gotten any input. I needed to install cvxopt on a 64 bit W7. I found out that cvxopt is incompatible with a 64 bit. It only compatible with a 32 bit which can be installed on a 64 bit computer and it works. I stumbled on this on another discussion group posting. The suggestion was to install python 32 bit on a 64 bit computer, then download cvxopt and install it via conda. Here is the suggestion from the posting [If you are using Windows, an easy solution could be the Anaconda distribution (http://continuum.io/downloads). Select and install the 32 bit version. It will be easier to install the cvxopt package. Search it in the binstar repository ( https://binstar.org/search?q=cvxopt). Open a console terminal and install it (e. g. "conda install -chttps://conda.binstar.org/omnia cvxopt").?] I did everything except the last command to install cvxopt, that’s “conda install -chttps://conda.binstar.org/omnia cvxopt". Here is the command I used: conda install c:/Users/eesawi0001/Downloads/cvxopt-1.1.7-py34.tar I kept getting invalid syntax Any help is greatly appreciated. EKE ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] cvxopt install problem
Thank you Alan and Michael for your input. Alan was right on what happened; that’s my mistake was that I used IDEL instead of CMD to install cvx. Once I used CMD all went well. I am hoping that cvx will work now w/o any issue. I used it on matlab. I am an experienced numerical programmer but new to python. At the moment I am using cvx just for very small linear system, so speed is not important. Thanks again EK ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] Extract several arrays from a large 2D array
Hi All-- I have a 2D array (2000, 4); an example is given abelow. On the 1st column I have 15 variables, on the 2nd 4 variables. Ignore column 3 for now. I want a code that generate 4 arrays for each variable on the 1st column; each array contains all the values from column 4. Then I want to find the average of each array. For example; pick the 1st variable in column 1 which is 1; then pick the 1st variable on the 2nd column which is 5. Now I want an array that contains all the values on the 4th column that match variable 1 on the 1st and variable 5 on the 2nd column. I need to get the average of each array. A b c d 1 5 3 4 1 3 2 7 2 5 7 5 3 2 8 5 2 3 2 3 Thanks in advance—EKE ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] Extract several arrays from a large 2D array
Thank you all for your help. I am a decent programmer in another language but new to Python and I have some issues with a project I am working on. Some suggested using pandas but I am barley starting on Numpy. The suggestions were very helpful, however, I decided to replace the 2D array with several single arrays b/c the arrays are of different data types. I ran into another problems and am almost done but got stuck. Part of my code is below. The question is how to put the variables for each j into a 14 by 6 array by a statement at the end of this code. I was hoping to get an array like this one below: [2013 TT1 TT2 TT3 TT4 TT5 TT6] [2012TT1 TT2 TT3 TT4 TT5 TT6] . . [1999TT1 TT2 TT3 TT4 TT5 TT6] for j in range(14) for i in range(200): if TYear[i]==2013-j: if TTrea[i]=='T1': TT1+=TTemp[i] elif TTrea[i]=='T2': TT2+=TTemp[i] elif TTrea[i]=='T3': TT3+=TTemp[i] elif TTrea[i]=='T4': TT4+=TTemp[i] elif TTrea[i]=='T5': TT5+=TTemp[i] elif TTrea[i]=='T6': TT6+=TTemp[i] ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] Change datatype for specific columns in an 2D array & computing the mean
Hi All---
Sorry for posting again, but I have a problem that I tried several
different ways to solve w/o success. I approached the problem from one
angle and asked about it here; I got some good input using pandas, and
structured array, but I am new to python and not very familiar with either
to use at this moment. I decided to go about in a different direction. I
am hoping for a simpler solution using Numpy.
I have a csv file with 4 columns and 2000 rows. There are 10 variables in
column 1 and 4 variables on each column, 2 and 3. I read the csv file and
converted it to arrays. The problem I ran into and could not resolve is
2-fold: (1) change the datatype for columns 1 and 4 to float and (2) then,
I want to use Numpy-or simpler method- to calculate the mean of the data
points on column 4 based on each variable on column 1 and column 2. Below
is my code and sample data file.
Here is part of my code:
import numpy as np
import csv
TMatrix=[]
np.set_printoptions(precision=2)
" Converting csv to lists "
with open('c:/Users/My Documents/AAA/temp1.csv') as temp:
reader = csv.reader(temp, delimiter=',', quoting=csv.QUOTE_NONE)
for row in reader:
TMatrix.append(row)
" converting lists to arrays "
TMatrix=np.array(TMatrix)
TMatrix=np.array(4,TMatrix[1:,::],dtype='float,int,int,float')#
this statement is not working
+++ This is a sample of my file +
['19' 'A4' 'B2' '2']
['19' 'A5' 'B1' '12']
['18' 'A5' 'B2' '121']]
Thanks in advance
EK Esawi
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Change datatype for specific columns in an 2D array & computing the mean
Thanks for the input. Columns 2 and 3 are strings and i assume that they don't need to be converted. Because all i need is to compute the mean for data on column 4 based on each variable in column 1 and each in column 2.. BTW, is it possible to send you what i did off list? That way you see what i am doing. EKE ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Change datatype for specific columns in an 2D array & computing the mean
Thank you all for your patience and dedication to this noble effort-the
list. I actually completed my project using Excel and python; unfortunately
my python code is not a python-like at all b/c I used way too many loops
and if statements. Plus I really want to learn python b/c I am a decent
programmer in other languages.
Here is a copy of my code and the csv file. It’s similar to the one I
posted earlier. For now I am not even concerned with mean or other
computations. All I want is to be able read a csv file which has string,
float, integer, date, and etc. datatypes. These are the majority of file
types I work with. I want to transfer it to a python array with proper
datatype. For example on my file, I want the 1st column integer, the 4th
float, the 2nd and 3rd string and the 5th date. I spent a few hours looking
online and in python documentation and did not find anything simple for me
to use.
I tried the function posted here but it’s too complicated for me-you can
see what I know about python. If there is no simpler way to do this then
that’s OK. I will keep on reading and see what happens.
I even tried to do some of it manually but did not work, as you see in my
code. Thank you all for your continuous help and dedication to this great
siteEKE
++ Code++
import numpy as np
import csv
DataMat=[]
np.set_printoptions(precision=2)
with open('c:/Users/EK Esawi/My Documents/Python Scripts/TestFile1.csv') as
Test1:
reader = csv.reader(Test1, delimiter=',')
#next(Test1)
for row in reader:
DataMat.append(row)
DataMat=np.array(DataMat)
col1=np.array(DataMat[1::,0], dtype=int)
col2=np.array(DataMat[1::,1])
col3=np.array(DataMat[1::,1])
col4=np.array(DataMat[1::,3], dtype=float)
col5=np.array(DataMat[1::,4])
++ csv file++
AA
BB
CC
DD
EE
1
A1
B1
11.2
11/20/2011
2
A2
B2
2.5
10/21/2011
3
A3
B3
13.67
9/21/2011
4
A4
B4
14.2
8/22/2011
5
A5
B5
20
7/23/2011
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Change datatype for specific columns in an 2D array & computing the mean
Sorry! but it's a csv file. I just copied it directly from an opened csv file in excel. And you're correct it looks like what you wrote. EKE ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Change datatype for specific columns in an 2D array & computing the mean
Here is the file. AA,BB,CC,DD,EE 1,A1,B1,11.2,11/20/2011 2,A2,B2,2.5,10/21/2011 3,A3,B3,13.67,9/21/2011 4,A4,B4,14.2,8/22/2011 5,A5,B5,20,7/23/2011 ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Change datatype for specific columns in an 2D array & computing the mean
Sorry for the mishap. There are no blank lines on the file. I just did not delete the blank lines when i copied and pasted. Here it's again. EK *AA,BB,CC,DD,EE* *1,A1,B1,11.2,11/20/2011* *2,A2,B2,2.5,10/21/2011* *3,A3,B3,13.67,9/21/2011* *4,A4,B4,14.2,8/22/2011* *5,A5,B5,20,7/23/2011* ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Change datatype for specific columns in an 2D array & computing the mean
Ops..here is the text file.; previously i copied and pasted from either Word or Excel. AA,BB,CC,DD,EE 1,A1,B1,11.2,11/20/2011 2,A2,B2,2.5,10/21/2011 3,A3,B3,13.67,9/21/2011 4,A4,B4,14.2,8/22/2011 5,A5,B5,20,7/23/2011 ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Change datatype for specific columns in an 2D array & computing the mean
Thank you SO MUCH Oscar and others. This is exactly what i wanted to get. I think i can take it form here. I spent hours looking at different ways to figure it out and i came close. I tried numpy.genfromtxt, numpy.loadtext, json, etc. I am at the moment learning how to open various data files with different format such as csv, pdf, html, etc. and then convert them to arrays or list. Thanks again for your patience and help EK ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] Enumerate vs DictReader object manipulation:
Hi All I have a code that reads a csv file via DictReader. I ran into a peculiar problem. The python interpreter ignores the 2nd code. That is if I put the reader iterator 1st, like the code below, the enumerate code is ignored; if I put the enumerate code 1st, the reader code is ignored. I am curious to know the nature of such behavior. EKE Here part of my code: . . . reader = csv.DictReader(MyFile) for row in reader: list_values = list(row.values()) print (list_values) for i,j in enumerate(reader): print(j) ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Enumerate vs DictReader object manipulation:
Thank you all. The only reason i tried both ways is to experiment with Python. They made sense to me and thought why not try them both. And i am relatively new to Python. Thanks again--EKE ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] genfromtxt vs. reading csv to a list or dictionary
Hi all— I normally need to convert csv and text files to a Numpy array. I tried to do the same thing using (1) reader=DictReader(MyFile), (2) reader=csv.readre(MyFile), or (3) genfromtxt (MyFile ,……). The first two is after I open the file. They produce a list of lists, list of tuples or list of dictionaries which later are converted to an array. The question is which one is the fastest and/or most efficient most flexible? The genformtext function is nice b/c you can setup formatting of various column which I often need to do b/c my files contain both string and numeric values (integer and float). Thanks in advance--EKE ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] genfromtxt vs. reading csv to a list or dictionary
Thanks Danny! I realized that I should not be concerned with speed/efficiency since I am a beginner in python and giving the data I work with, speed is not that important now. I am learning how to read files via several ways and transform them to numpy arrays. I just thought genfromtxt provides many options and one could use it to read and convert a file into 2d array with proper format; instead of reading it then format it. But genfromtxt gives byte strings which i could not convert to integer, float, string, etc. Thanks again--EKE ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Python for Grade 12 Calculus
I am not sure of the level of your students but computing the Jacobin and directional derivative where they can use both calculus and linear algebra; numerical differentiation and integration would be another topic. There are several books with nice numerical recipes that can be programmed using python. EKE ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] genfromtx- date and time conversions
I am using genfromtxt to read a csv file with data types that include string, float, integer, date, and time. I was able to accomplish such a task using reader. I thought there might be easier ways and the one I thought about is genfromtxt. I have 2 questions: (1) Is there a function (method) to convert time in H:M:S, format, e.g. 12:30:30 to decimal or 24 format e.g.12.508 w/o writing my own function to convert it ? (2) Given a date of the form 12/20/2005, can such date in such format be converted with converters in numpy using datetime to ‘2005-20-15’? I had to reformat the date on my csv file to the later form for python to read it correctly. Thanks-EKE ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] Convert structured 1D array to 2D array
Hi All— I used genfromtxt to read a file with multiple data types. The result was a 1D array of tuples; for example [(1, 2, 3), (‘a’, b’, ’c’), (‘12/12/2009’, ’2/4/2014’, ‘3/4/200)’] I am trying to convert this structured array to a 2D array. Is this possible where the data types in the 1D array are preserved? Thanks in advance-EKE ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Convert structured 1D array to 2D array
Thanks for the input. I tried your idea but i did not work; l got 1D array of tuples; the same as the original array. I think b/c genfromtxt creates an array and so it's already an array object-if i understand it correctly. Your idea works well for a list which i tested but i want to an n x n 2D array from 1D n x 1 tuples, each tuple has n elements,. Thanks again-_EK ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Convert structured 1D array to 2D array
The result i am looking for is a 2D array (matrix) instead of 1D array of tuples. That's when i read a file using genfromtxt, it generated a 1D array of n tuples (like the example shown below); each tuple has one row from the original file. The file contains multiple data types and so i used genfromtxt to specify data types for each column. In another way, i just want to convert a 1D array of tuples to 2D array where each tuple is converted multiple columns. Here is an example: the output from genfromtxt is [(1, 2, 3), (‘a’, b’, ’c’), (‘12/12/2009’, ’2/4/2014’, ‘3/4/200’)] I want this be converted to: [[1, 2, 3], [‘a’,b’, ’c’], [‘12/12/2009’, ’2/4/2014’, ‘3/4/200’]] Thanks again--EKE ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Convert structured 1D array to 2D array
Thanks Peter. It works now. I am trying to be creative but i am still getting my way around pythony. EKE ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] genfromtxt and dtype to convert data to correct format
Hi All—
I am reading data from a file using genfromtxt. A part of my data (input,
output, and desired output) is shown below which consists of string,
integer, time, and date type data. I want to read the data so that the
output comes out in the correct format (the desired output as shown below).
I used converters to covert time and date values, but all came out in
string format (output below).
I tried to use structured dtype but without success and I don’t want to use
pandas. What is the best remedy to achieve the desired output shown below?
Thanks in advance—EK
++
Code
CF = lambda date: datetime.strptime(bytes.decode(date),
%m/%d/%Y).strftime(%Y-%m-%d) # convert data
CF1 = lambda time: datetime.strptime(bytes.decode(time),
%H:%M).strftime(%H:%M:%S) # convert time
MyFile=New_file
CRNs={Date: CF,Time:CF1,In:CF1,Dur:CF1,PP:CF1,Pre:CF1}
#converters
data = np.genfromtxt(MyFile,
names=True,delimiter=,,converters=CRNs,dtype=None)
Input
ODateName TimeInt
Dur PP H Pred
312171 7/1/1995 Old 13:37 1:434:42
13:16 162 13:19
358237 5/25/1993New 12:22 1:314:16
12:03 160 12:13
Output
[ (312171, 1995-07-01, bOld, 13:37:00, 01:43:00, 04:42:00, 13:16:00, 162,
13:19:00)
(358237, 1993-05-25, bNew, 12:22:00, 01:31:00, 04:16:00, 12:03:00, 160,
12:13:00)]
Desired output
[ (312171, 1995-07-01, bOld, 13:37:00, 01:43:00, 04:42:00, 13:16:00, 162,
13:19:00)
(358237, 1993-05-25, bNew, 12:22:00, 01:31:00, 04:16:00, 12:03:00, 160,
12:13:00)]
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] genfromtxt and dtype to convert data to correct format
OPS! Sorry, I made a mistake on the posting. My output is in a string format as shown below. I want the 1st and 8th integers, the 3rd string which is OK as, the 2nd date and the rest time. I tried several combinations of dtype but could not get the date and time data to without the single quote -string format. [ (b'312171', '1995-07-01', b'Old', '13:37:00', '01:43:00', '04:42:00', '13:16:00', b'162', '13:19:00') (b'358237', '1993-05-25', b'Old', '12:22:00', '01:31:00', '04:16:00', '12:03:00', b'160', '12:13:00') ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] genfromtxt and dtype to convert data to correct format
Thanks Sydney. The print statement prints items w/o quotation marks, but doesn't change the type. I want the datatype to be changed. I am thinking it can eb done with astructured array dtyep but have not been able to get to wort. Thanks again--EK ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] genfromtxt and dtype to convert data to correct format
Thanks Alan! Taking the strtime off did not work either and printed dattime.dattime(very long date format). I was able to get the integer and string parts fixed via dtype, but could not do the same for date and time. I have a feeling that structured arrays might do the trick but don't know yet how. Thanks again--EK ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] genfromtxt and dtype to convert data to correct format
Thanks again!
I tried a combination of the suggestions and still not getting what i want.
Here are the original code, file, and output.
CODE
mport csv; import numpy as np; from datetime import datetime, date, time
CF = lambda date: datetime.strptime(bytes.decode(date),
'%m/%d/%Y').strftime('%Y-%m-%d')
CF1 = lambda time: datetime.strptime(bytes.decode(time),
'%H:%M').strftime('%H:%M:%S')
MyFile='c:/Users/EK Esawi/My Documents/Temp/GOSA-3.csv'
CRNs={'Date':
CF,'Time':CF1,'Interval':CF1,'Duration':CF1,'Preplay':CF1,'Prediction':CF1}
data = np.genfromtxt(MyFile,
names=True,delimiter=',',converters=CRNs,dtype=None)
print(data)
INPUT
O Date Geyser Time Interval Duration Preplay Heght Prediction 312171
7/1/1995 Position 13:37 1:43 4:42 13:16 162 13:19 358237 5/25/1993 Position
12:22 1:31 4:16 12:03 160 12:13 339971 7/17/1994 Position 15:54 1:23 4:36
15:43 160 15:51
OUTPUT
[ (312171, '1995-07-01', b'Old Faithful', '13:37:00', '01:43:00',
'04:42:00', '13:16:00', 162, '13:19:00')
(358237, '1993-05-25', b'Old Faithful', '12:22:00', '01:31:00',
'04:16:00', '12:03:00', 160, '12:13:00')
(339971, '1994-07-17', b'Old Faithful', '15:54:00', '01:23:00',
'04:36:00', '15:43:00', 160, '15:51:00')
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] genfromtxt and dtype to convert data to correct format
Thanks again Alan for your help and patience. Your earlier suggestion works; i just now realized that the output was indeed correct. Thanks again--EK ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Loop in pre-defined blocks
Here is a beginner code that might work for you. Best of luck. EK b=[12, 20, 35] for i in range(len(b)): if i==0: c=0 else: c=b[i-1] for j in range(c, b[i]): print(i+1,j+1) ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Loop in pre-defined blocks
OPS! This code now produces desired results. I suppose that this works for smaller blocks. For larger blocks, it might be cumbersome. EK b=[12, 20, 35] for i in range(len(b)): if i==0: c=0 elif i==2: c=24 else: c=b[i-1] for j in range(c, b[i]): print(i+1,j+1) ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] Converting a sequence of dictionaries to a single dictionary
Hi All--
I am trying to read a csv file via DictReader and produce a single
dictionary without fieldnames; that is, I just want a single dictionary
that contains tests as keys and grades as values as shown below in my
sample file.
If I loop through the reader, it produces a sequence of 2 entries
dictionaries; each starts with the fieldnames, e.g. {Test:Test1,Grade:A},
etc.
I want to create a single dictionary with only Tests and Grades; that’s
{Test1:A,Test2:b,Test3:c etc.}. I tried several ways but nothing worked.
File
Test Grade
Test1 A
Test2 B
Test3 C
Code;
import csv
with open(Tests.csv, 'r') as Temp:
reader = csv.DictReader(Temp)
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
[Tutor] Most efficient way to read large csv files with properly converted mixed data types.
Hi All-- My work involves reading large csv files with mixed data types (integer, float, string, time and date). I was able to accomplish the task using (1) genfromtxt or (2) looping through each line in the file and split, strip, and assign data type to each entry. I am wondering if there is a better and more efficient alternative, especially to method 2 without using numpy or pandas. Alan Gauld mentioned namedtuples for another question. I read a little about collections and in particular namedtuples but was not sure how to apply theme here, if they are applicable to begin with. Thanks in advance--EKE An example of a file: A B C D E 1 2.3 ‘aa’ 10/01/2016 12:30 4 25.6 ‘bb’ 02/02/2015 1:30 ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Most efficient way to read large csv files with properly converted mixed data types.
The first step of my plan is to do basic statistical analysis. The 2nd step is to chose a sample from each file and do more advanced statistical analysis for which i plan to use R. EKE ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
