RE: curl getting different html

2025-01-10 Thread Dick Brooks via curl-users
users Sent: Friday, January 10, 2025 11:20 AM To: curl-users@lists.haxx.se Cc: paste...@gmx.com Subject: Re: curl getting different html Thanks Hans for the code and your help i don't get the gz'ed page with curl at all (no matter what i send - headers/no header -L/no -L etc) - just the &#

Re: curl getting different html

2025-01-10 Thread toby via curl-users
Thanks Hans for the code and your help i don't get the gz'ed page with curl at all (no matter what i send - headers/no header -L/no -L etc) - just the 'needs javascript and cookies' page' a line from 'band of brothers' comes to mind here https://www.youtube.com/watch?v=cQE1YIJDNWo substitue 'clo

Re: curl getting different html

2025-01-10 Thread Hans Henrik Bergan via curl-users
>can curl give me a gzipped page if i use the correct header? (hint hint - can >you tell me what to use for curl) if you want to witness the compressed version first-hand, run curl --header 'Accept-Encoding: gzip, deflate, br, zstd' -o - https://www.podchaser.com/podcasts/crypto-corner-bitcoin-a

Re: curl getting different html

2025-01-10 Thread toby via curl-users
On Wed, 8 Jan 2025 22:09:41 +0100 Hans Henrik Bergan via curl-users wrote: > >i don;t get the 'are you a human' page with any of them > > lucky you: cloudflare's automated systems deems it unnecessary to > challenge you. A variation of a "trustworthy IP range" and > "correct-looking headers" > a

Re: curl getting different html

2025-01-10 Thread toby via curl-users
On Wed, 8 Jan 2025 12:27:00 -0800 Dan Fandrich via curl-users wrote: > On Wed, Jan 08, 2025 at 10:59:15AM -0700, toby via curl-users wrote: > > the page is actually gzipped - dillo can gunzip but curl can't by itself > > (found out using w3m) > > curl will uncompress if given the --compressed opt

Re: curl getting different html

2025-01-08 Thread Hans Henrik Bergan via curl-users
>i don;t get the 'are you a human' page with any of them lucky you: cloudflare's automated systems deems it unnecessary to challenge you. A variation of a "trustworthy IP range" and "correct-looking headers" and "an SSL/TLS implementation behaving like a web browser" cause people to not get chall

Re: curl getting different html

2025-01-08 Thread Dan Fandrich via curl-users
On Wed, Jan 08, 2025 at 10:59:15AM -0700, toby via curl-users wrote: > the page is actually gzipped - dillo can gunzip but curl can't by itself > (found out using w3m) curl will uncompress if given the --compressed option. -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users Etiquett

Re: curl getting different html

2025-01-08 Thread toby via curl-users
hi the page is actually gzipped - dillo can gunzip but curl can't by itself (found out using w3m) On Wed, 8 Jan 2025 16:45:12 +0100 Hans Henrik Bergan wrote: > The website is using the Cloudflare "Bot Fight Mode" thing which is > "protecting the website against bots", > dillo, w3c, and curl ar

Re: curl getting different html

2025-01-08 Thread toby via curl-users
On Wed, 8 Jan 2025 16:45:12 +0100 Hans Henrik Bergan wrote: thanks Henrik, > The website is using the Cloudflare "Bot Fight Mode" thing which is > "protecting the website against bots", > dillo, w3c, and curl are all triggering the "are you a human?" w3c -> w3m i don;t get the 'are you a hum

Re: curl getting different html

2025-01-08 Thread Hans Henrik Bergan via curl-users
The website is using the Cloudflare "Bot Fight Mode" thing which is "protecting the website against bots", dillo, w3c, and curl are all triggering the "are you a human?" challenge page, and none of them are capable of passing it. It is strange that your dillo is not triggering it, probably somethi

curl getting different html

2025-01-08 Thread toby via curl-users
Maybe someone can help me with this dillo https://www.podchaser.com/podcasts/crypto-corner-bitcoin-and-bloc-950963/episodes/recent gives a good page curl -b cookies.txt -c cookies.txt -A "Mozilla/5.0" -k -L -o crypto.html https://www.podchaser.com/podcasts/crypto-corner-bitcoin-and-bloc-950963/