Dear Alan, Thank you very much for answering the question. If you don't mind, please kindly let me know which library I should focus on among beautifulsoup, selenium + PhantomJS, and dryscrape.
Have a good day regards, Hank On Sun, Dec 13, 2015 at 5:11 PM, <tutor-requ...@python.org> wrote: > Send Tutor mailing list submissions to > tutor@python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/tutor > or, via email, send a message with subject or body 'help' to > tutor-requ...@python.org > > You can reach the person managing the list at > tutor-ow...@python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Tutor digest..." > > > Today's Topics: > > 1. Re: Calculation error with a simple program (Steven D'Aprano) > 2. Re: Calculation error with a simple program (Jim Gallaher) > 3. Re: Calculation error with a simple program (Todd Purple) > 4. Re: Tutor Digest, Vol 142, Issue 10 (Ken Hammer) > 5. Beautiful Soup (Crusier) > 6. Re: Beautiful Soup (Alan Gauld) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 13 Dec 2015 04:00:25 +1100 > From: Steven D'Aprano <st...@pearwood.info> > To: tutor@python.org > Subject: Re: [Tutor] Calculation error with a simple program > Message-ID: <20151212170025.gk3...@ando.pearwood.info> > Content-Type: text/plain; charset=us-ascii > > On Sat, Dec 12, 2015 at 01:03:05AM -0600, Jim Gallaher wrote: >> Hi everyone. I'm reading through a beginners Python book and came up >> with a super simple program. I'm not getting any errors and everything >> runs through, but there's a logical calculation error. What the program >> does is take an amount and calculate a couple percentages and add a >> couple fees. >> >> For example, if I put in a value of 1, it will output 752.12 as the sub >> total and 753.12 as the grand total. It's off by 1 on sub total and 2 on >> grand total. > > Check your arithmetic -- with a base price of $1, the sub total is > $752.12 and the grand total is $753.12, exactly as Python calculates. > > > But I wonder whether you have made a logic mistake in your code. You > say: > > dealerPrep = basePrice + 500 > destinationCharge = basePrice + 250 > > This means that the car will cost more than TRIPLE the base price. > Instead of $1, enter a base price of $40,000 and you will see what I > mean: now the dealer prep is $40500 and the destination charge is > $40250, which added to the base price gives $120750 (plus tax and > licence). Surely that's not right, the deal charges more than the cost > of the car for preparation? I think what you want is: > > dealerPrep = 500 > destinationCharge = 250 > > > > -- > Steve > > > ------------------------------ > > Message: 2 > Date: Sat, 12 Dec 2015 11:34:27 -0600 > From: Jim Gallaher <jcgallahe...@gmail.com> > To: tutor@python.org > Subject: Re: [Tutor] Calculation error with a simple program > Message-ID: <8774a398-4a0f-41df-bbd2-941b49a3c...@gmail.com> > Content-Type: text/plain; charset=us-ascii > > Hi Alan, > > I'm 100 percent sure I'm wrong. :-) I verified it when I fixed the mistake. > The problem was it was adding in the basePrice and the fixed > rates/percentages each time. So I figured it out when Ian said something > about that. > > Thanks for everyone's help! :-) > > ------------------------------ > > Message: 3 > Date: Sat, 12 Dec 2015 08:13:23 -0500 > From: Todd Purple <todd_pur...@yahoo.com> > To: Jim Gallaher <jcgallahe...@gmail.com> > Cc: tutor@python.org > Subject: Re: [Tutor] Calculation error with a simple program > Message-ID: <6ec6d759-3c19-4a6d-8dd7-6efae1958...@yahoo.com> > Content-Type: text/plain; charset=us-ascii > > > >> On Dec 12, 2015, at 2:03 AM, Jim Gallaher <jcgallahe...@gmail.com> wrote: >> >> Hi everyone. I'm reading through a beginners Python book and came up with a >> super simple program. I'm not getting any errors and everything runs >> through, but there's a logical calculation error. What the program does is >> take an amount and calculate a couple percentages and add a couple fees. >> >> For example, if I put in a value of 1, it will output 752.12 as the sub >> total and 753.12 as the grand total. It's off by 1 on sub total and 2 on >> grand total. >> >> Thanks in advance! Jim Gallaher >> >> # Car Salesman Calculator >> >> # User enters the base price of the car and the program adds tax, license, >> dealer prep, and destination charge. >> >> print("Car Sales Calculator") >> basePrice = int(input("Please enter in the price of the car: ")) >> >> # Misc charges to be added to the total cost of the car >> tax = basePrice * .07 >> license = basePrice * .05 >> dealerPrep = basePrice + 500 >> destinationCharge = basePrice + 250 >> > > I think your main problem is right here. Why do dealerPrep and > destinationCharge include the basePrice in their equation? This will add the > basePrice in multiple times. I would simply set it like this: > > dealerPrep = 500 > destinationCharge = 250 > > If these charges ever change, you can put in some sort of equation. However, > it looks like these are meant to be a flat fee. > >> # Add the total misc charges together >> subTotal = float(tax + license + dealerPrep + destinationCharge) >> >> # Add all the misic charges and include the base pricem >> grandTotal = float(subTotal + basePrice) >> >> # Display the results >> print("\nThe sub total is", subTotal) >> print("\nYour grand Total is", grandTotal) >> >> input("\nPress the enter key to close the program.") >> _______________________________________________ >> Tutor maillist - Tutor@python.org >> To unsubscribe or change subscription options: >> https://mail.python.org/mailman/listinfo/tutor > > > ------------------------------ > > Message: 4 > Date: Sat, 12 Dec 2015 12:23:32 -0500 > From: "Ken Hammer"<kfh...@earthlink.net> > To: tutor@python.org > Subject: Re: [Tutor] Tutor Digest, Vol 142, Issue 10 > Message-ID: <e1a7ntk-0001en...@elasmtp-mealy.atl.sa.earthlink.net> > > > I have a more simple-minded solution. The error appears to occur in testing > without revealing the problem by use of $1 base price in the test. Using > $10,000 as the base price in the test reveals absurd numbers. I think the > real problem lies in the inclusion of base price inappropriately, twice as in: > >>dealerPrep = basePrice + 500 >>destinationCharge = basePrice + 250 > > "basePrice" has no place in those two lines. 500 and 250 alone express the > definition of the challenge. > > Ken > > > > In <mailman.13.1449939602.12987.tu...@python.org>, on 12/12/15 > at 12:00 PM, tutor-requ...@python.org said: > > > >>Send Tutor mailing list submissions to >> tutor@python.org > >>To subscribe or unsubscribe via the World Wide Web, visit >> https://mail.python.org/mailman/listinfo/tutor >>or, via email, send a message with subject or body 'help' to >> tutor-requ...@python.org > >>You can reach the person managing the list at >> tutor-ow...@python.org > >>When replying, please edit your Subject line so it is more specific than >>"Re: Contents of Tutor digest..." > > >>Today's Topics: > >> 1. Calculation error with a simple program (Jim Gallaher) >> 2. Re: Calculation error with a simple program (Alan Gauld) >> 3. Re: Calculation error with a simple program (Laura Creighton) > > >>---------------------------------------------------------------------- > >>Message: 1 >>Date: Sat, 12 Dec 2015 01:03:05 -0600 >>From: Jim Gallaher <jcgallahe...@gmail.com> >>To: tutor@python.org >>Subject: [Tutor] Calculation error with a simple program >>Message-ID: <566bc6a9.6090...@gmail.com> >>Content-Type: text/plain; charset=utf-8; format=flowed > >>Hi everyone. I'm reading through a beginners Python book and came up with >>a super simple program. I'm not getting any errors and everything runs >>through, but there's a logical calculation error. What the program does is >>take an amount and calculate a couple percentages and add a couple fees. > >>For example, if I put in a value of 1, it will output 752.12 as the sub >>total and 753.12 as the grand total. It's off by 1 on sub total and 2 on >>grand total. > >>Thanks in advance! Jim Gallaher > >># Car Salesman Calculator > >># User enters the base price of the car and the program adds tax, license, >>dealer prep, and destination charge. > >>print("Car Sales Calculator") >>basePrice = int(input("Please enter in the price of the car: ")) > >># Misc charges to be added to the total cost of the car >>tax = basePrice * .07 >>license = basePrice * .05 >>dealerPrep = basePrice + 500 >>destinationCharge = basePrice + 250 > >># Add the total misc charges together >>subTotal = float(tax + license + dealerPrep + destinationCharge) > >># Add all the misic charges and include the base price >>grandTotal = float(subTotal + basePrice) > >># Display the results >>print("\nThe sub total is", subTotal) >>print("\nYour grand Total is", grandTotal) > >>input("\nPress the enter key to close the program.") > > >>------------------------------ > >>Message: 2 >>Date: Sat, 12 Dec 2015 08:04:52 +0000 >>From: Alan Gauld <alan.ga...@btinternet.com> >>To: tutor@python.org >>Subject: Re: [Tutor] Calculation error with a simple program >>Message-ID: <n4gkf4$e99$1...@ger.gmane.org> >>Content-Type: text/plain; charset=utf-8 > >>On 12/12/15 07:03, Jim Gallaher wrote: > >>> For example, if I put in a value of 1, it will output 752.12 as the sub >>> total and 753.12 as the grand total. It's off by 1 on sub total and 2 on >>> grand total. > >>Are you sure? Lets check the values... > >>> basePrice = int(input("Please enter in the price of the car: ")) > >>=> 1 > >>> tax = basePrice * .07 > >>=> 0.07 > >>> license = basePrice * .05 > >>=> 0.05 >>> dealerPrep = basePrice + 500 > >>=> 501 > >>> destinationCharge = basePrice + 250 > >>=> 251 > >>> # Add the total misc charges together >>> subTotal = float(tax + license + dealerPrep + destinationCharge) > >>=> 0.07 + 0.05 + 501 + 251 => 752.12 > >>> # Add all the misic charges and include the base price >>> grandTotal = float(subTotal + basePrice) > >>=> 752.12 + 1 => 753.12 > >>Looks like Python got it right to me? > > > > > -- > ----------------------------------------------------------- > K.F.Hammer Associates Ken Hammer > management consultations Saint Johnsbury, VT 05819 > ----------------------------------------------------------- > > > > ------------------------------ > > Message: 5 > Date: Sun, 13 Dec 2015 15:44:33 +0800 > From: Crusier <crus...@gmail.com> > To: tutor@python.org > Subject: [Tutor] Beautiful Soup > Message-ID: > <cac7hcj_dtvm6tb7s2rrm4ktpwutbyv+wwhdj7sdcqqsm8_8...@mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > Dear All, > > I am trying to scrap the following website, however, I have > encountered some problems. As you can see, I am not really familiar > with regex and I hope you can give me some pointers to how to solve > this problem. > > I hope I can download all the transaction data into the database. > However, I need to retrieve it first. The data which I hope to > retrieve it is as follows: > > " > 15:59:59 A 500 6.790 3,395 > 15:59:53 B 500 6.780 3,390................ > > Thank you > > Below is my quote: > > from bs4 import BeautifulSoup > import requests > import re > > url = > 'https://bochk.etnet.com.hk/content/bochkweb/eng/quote_transaction_daily_history.php?code=6881&time=F&timeFrom=090000&timeTo=160000&turnover=S&sessionId=44c99b61679e019666f0570db51ad932&volMin=0&turnoverMin=0' > > def turnover_detail(url): > response = requests.get(url) > html = response.content > soup = BeautifulSoup(html,"html.parser") > data = soup.find_all("script") > for json in data: > print(json) > > turnover_detail(url) > > Best Regards, > Henry > > > ------------------------------ > > Message: 6 > Date: Sun, 13 Dec 2015 09:10:57 +0000 > From: Alan Gauld <alan.ga...@btinternet.com> > To: tutor@python.org > Subject: Re: [Tutor] Beautiful Soup > Message-ID: <n4jcn1$qrm$1...@ger.gmane.org> > Content-Type: text/plain; charset=utf-8 > > On 13/12/15 07:44, Crusier wrote: >> Dear All, >> >> I am trying to scrap the following website, however, I have >> encountered some problems. As you can see, I am not really familiar >> with regex and I hope you can give me some pointers to how to solve >> this problem. > > I'm not sure why you mention regex because your script doesn't > use regex. And for html that's a good thing. > >> I hope I can download all the transaction data into the database. >> However, I need to retrieve it first. The data which I hope to >> retrieve it is as follows: >> >> " >> 15:59:59 A 500 6.790 3,395 >> 15:59:53 B 500 6.780 3,390................ >> > > Part of your problem is that the data is not in html format but > is in fact part of the Javascript code on the page. And > BeautifulSoup is not so good at parsing Javascript. > > The page code looks like > > <script type="text/javascript" > src="../js/jquery.js?verID=20150826_153700"></script> > <script type="text/javascript" > src="../js/common_eng.js?verID=20150826_153700"></script> > <script type="text/javascript" > src="../js/corsrequest.js?verID=20150826_153700"></script> > <script type="text/javascript" > src="../js/wholedaytran.js?verID=20150826_153700"></script> > <script type="text/javascript"> > var json_result = > {"content":{"0":{"code":"6,881","timestamp":"15:59:59","order":"1175","transaction_type":"","bidask":"...{"code":"6,881","timestamp":"15:59:53","order":"1174","transaction_type":"",...{"code":"6,881","timestamp":"15:59:53","order":"1173",... > > followed by a bunch of function definitions and other stuff. > > >> def turnover_detail(url): >> response = requests.get(url) >> html = response.content >> soup = BeautifulSoup(html,"html.parser") >> data = soup.find_all("script") >> for json in data: >> print(json) > > The name json here is misleading because it's not really > json data at this point but javascript code. You will > need to further filter the code lines down to the ones > containing data and then convert them into pure json. > > You don't show us the output but I'm assuming it's > the full javascript program? If so you need a second > level of parsing to extract the data from that. > > It shouldn't be too difficult since the data you want > all starts with the string {"code": apart from the > first line which will need a little bit extra work. > But I don't think you really need any regex to do > this, regular string methods should suffice. > > I suggest you should write a helper function to do > the data extraction and experiment in the interpreter > using some cut n pasted sample data till you get it right! > > -- > Alan G > Author of the Learn to Program web site > http://www.alan-g.me.uk/ > http://www.amazon.com/author/alan_gauld > Follow my photo-blog on Flickr at: > http://www.flickr.com/photos/alangauldphotos > > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Tutor maillist - Tutor@python.org > https://mail.python.org/mailman/listinfo/tutor > > > ------------------------------ > > End of Tutor Digest, Vol 142, Issue 11 > ************************************** _______________________________________________ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
