Re: status reason-phrase missing for H2

2024-08-23 Thread Dan Fandrich via curl-users
On Fri, Aug 23, 2024 at 11:44:07AM -0500, Yang Yu via curl-users wrote:
> Compare HTTP 1.1 response header output with H/2
> 
> $ curl -I http://www.http2demo.io/
> HTTP/1.1 200 OK
> 
> $ curl -I https://www.cloudflare.com
> HTTP/2 200
> 
> For H2, the reason phrase is not shown in curl output.

The reason phrase isn't even available in the HTTP/2 protocol. According to
RFC7540:

   HTTP/2 does not define a way to carry the version or reason phrase
   that is included in an HTTP/1.1 status line.

Dan
-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users
Etiquette:   https://curl.se/mail/etiquette.html


Re: %{filename_effective but not transmit file?

2024-09-11 Thread Dan Fandrich via curl-users
On Wed, Sep 11, 2024 at 05:47:50PM -0600, Paul Gilmartin via curl-users wrote:
> Is it possible to get %{filename_effective without transmitting
> the file?  I'm trying to make the ETag name pend on that,
> something like:
> "--etag-compare  %{filename_effectiv.etag"
> 
> So far, the best I can do is perform a trial fetch, directing
> the output to a read-only directory, failing but returning
> %{filename_ effective and using that in the real fetch.

Adding --head should basically do that in a clean way.

> is there a server-side TOCTTOU timing hazard?  I'm imagining

I imagine there could be one. It's theoretically possible for the domain name
to expire, be snapped up by scammers and start serving malware between the
initial check and final download. Worst case.

> a busy server which:
> o declines to send a resource because of ETag match
> o Updates the resource concurrently
> o returns the ETag matching the new version.

I don't think you can ever completely rely on ETag matching. The ETag of a
resource can change even if the resource hasn't, although the reverse isn't
supposed to also be true. I would consider a server to be broken that uses two
different ETags while handling a single request, but I haven't looked at the
specs to see if it's technically allowed.

Dan
-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users
Etiquette:   https://curl.se/mail/etiquette.html


Re: Need help on how to upgrade the curl.exe and libcurl.dll file versions on Windows

2024-09-13 Thread Dan Fandrich via curl-users
On Fri, Sep 13, 2024 at 03:53:58PM -0400, Jody Sherwin via curl-users wrote:
> During our monthly Nessus Security Vulnerability Scan we have received a few
> separate results on needing to upgrade the version of the [curl.exe] and the
> [libcurl.dll] files on a few Windows machines, which I had a few questions on
> this...
> 
> I was wondering how do I go about theseĀ upgrades as it seems the files are
> installed in a few separate locations?

https://curl.se/docs/faq.html#How_do_I_upgrade_curl_exe_in_Win

> From my understanding , the [curl.exe] and [libcurl.dll] files are used to 
> help
> transfer data from these machines in the scan report like http / https and sql
> db traffic and such, is that correct??

curl/libcurl can be used for all kinds of Internet transfers. You can't tell by
looking at it how it's being used.

> If so, do I perhaps reach out to you guys on this, or is this something that
> the manufactures like HPE, Microsoft, SAP BusinessObjects, and the Shibboleth
> Support folks would assist on instead??

It's at best dangerous and at worst impossible to upgrade curl/libcurl that
some other entity has installed. If you didn't install it to begin with, you
need to contact the entity that *did* install it to arrange for an upgrade.
Note that packagers can patch curl so the presence of a specific curl version
number doesn't necessarily imply the presence of a security issue (but it
often does).

Dan
-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users
Etiquette:   https://curl.se/mail/etiquette.html


Re: Any way to set connect-timeout per-IP, not global?

2024-10-11 Thread Dan Fandrich via curl-users
On Fri, Oct 11, 2024 at 06:47:09PM +0300, ValdikSS via curl-users wrote:
> Currently curl, when used with --connect-timeout option, uses it as a global 
> timeout for the whole
> connection set, decreasing the timeout for each IP address in half every 
> connection attempt.
> 
> If several IP addresses from the top of DNS resolution are not working, the 
> others (potentially
> working ones) just won't get enough timeout to establish the connection.

Why set a --connect-timeout shorter than what you're willing to accept? If you
don't mind if the connect takes longer because it has to check more addresses,
just set a longer connect timeout.

Dan
-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users
Etiquette:   https://curl.se/mail/etiquette.html


Re: (Feature request) Ignore if file exist

2024-11-13 Thread Dan Fandrich via curl-users
On Wed, Nov 13, 2024 at 10:54:58AM -0700, Paul Gilmartin via curl-users wrote:
> On 11/13/24 07:44, Tobias Wendorff via curl-users wrote:
> But that would seem never to refresh an outdated existing file.
> I'd welcome something such as:
> --skip-if-uptodate
> or:
> --time-cond %{filename_effective}

You could probably make it work similarly using the --etag-* options.
-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users
Etiquette:   https://curl.se/mail/etiquette.html


Re: curl getting different html

2025-01-08 Thread Dan Fandrich via curl-users
On Wed, Jan 08, 2025 at 10:59:15AM -0700, toby via curl-users wrote:
> the page is actually gzipped - dillo can gunzip but curl can't by itself
> (found out using w3m)

curl will uncompress if given the --compressed option.
-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users
Etiquette:   https://curl.se/mail/etiquette.html


Re: CURL errors codes and messages - are they globalized?

2025-01-16 Thread Dan Fandrich via curl-users
On Thu, Jan 16, 2025 at 02:25:37PM +, Michael Conners via curl-users wrote:
> If a CURLE_SSL_CONNECT_ERROR is returned from curl_easy_perform, the error 
> text
> I see is "SSL connect error". What error text would a German user see?
> 
> Are CURL error messages globalized? If so, which languages?

curl messages are not localized, although if you restrict yourself to libcurl
and avoid debug logs, there aren't actually very many full text messages that
you can get back. Actually, CURLOPT_ERRORBUFFER and the *strerror functions are
the only exceptions that I can think of. Both of those use a technical
vocabulary that generally don't make a lot of sense to nontechnical users, no
matter the language they appear in. I suspect that's why yours is the first
request I can remember about this topic.

The CURLcode errors are quite fine-grained, so applications can return a
useful localized message based on those codes quite easily.  Digging in to a
error much further often means looking at not just the ERRORBUFFER messages
but debug logs as well, which poses a scalability problem for translation.

To get an idea of the total number of translatable messages that there might
be, there are currently about 102 curl_easy_strerror messages, 14 for
curl_multi_strerror, 6 for curl_share_strerror, 32 for curl_url_strerror, for a
total of 154 error codes.  There are approximately 910 different messages that
can show up in CURLOPT_ERRORBUFFER. There are approximately 745 additional
debug logging messages. There are approximately 562 additional trace logging
messages. This also doesn't messages passed as-is from dependent libraries
(like OpenSSL's detailed errors). That totals about 2371 messages that would
need to be translated to provide an almost fully localized (text only) libcurl
experience. And that doesn't even include the curl front-end which has many
thousands more lines of text that it can output. Or the documentation, which wc
-l tells me is now over 102,000 additional lines.

Fully translating curl would be a huge undertaking on its own, not just in the
initial translation but more importantly in keeping the hundreds or thousands
(or a hundred thousand, if you want to go full in) of messages up-to-date for
each release in every language that we'd want to localize. Going half-way is
not a viable option because libcurl is used in so many applications that depend
on providing a polished result; leaving a few messages untranslated in each new
release would give the wrong impression. Once people start depending on
translations they are going to expect them to stay up-to-date. I doubt this is
something we'd want to take on as a project without a committed team available
to manage it. And given the dearth of interest in the topic, I don't see that
happening.

If users were to submit localized versions of curl_easy_strerror() (& the other
*strerror functions) in various languages, I could see them potentially being
included in the repo as unofficial examples. These functions are well-bounded
and don't change very often (only when new curl error codes are defined) and
are the ones that are the most likely to be displayed to users in localized
applications. Collecting them within the curl repo seems like it would be
useful to avoid many people translating the same things. I suspect there are
many cases out there of these specific messages already having been translated
for specific apps, so maybe there are people out there willing to submit them
for the benefit of others.

Dan
-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users
Etiquette:   https://curl.se/mail/etiquette.html


Re: issues with CURL using RSA keys + SFTP

2025-02-13 Thread Dan Fandrich via curl-users
On Fri, Feb 14, 2025 at 12:08:15AM +, Ian Veach via curl-users wrote:
> However, the curl is pretty modern and should (I assume) be using sha256 or 
> sha512 for signing.

curl uses a SSH library to perform all the SSH protocol options. The curl
version probably won't make as much of a difference as you think in this
regard.

> curl 8.6.0 (powerpc-ibm-aix7.1.5.0) libcurl/8.6.0 OpenSSL/1.1.1v 
> zlib/1.2.13 libssh2/1.10.0 nghttp2/1.58.0 OpenLDAP/2.5.16

This shows that curl is using libssh2 in your case, and that the version is
over 3 years old. I'd try upgrading that before investigating too much deeper.

Dan
-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users
Etiquette:   https://curl.se/mail/etiquette.html


Re: Anyone using --raw?

2025-04-05 Thread Dan Fandrich via curl-users
On Sat, Apr 05, 2025 at 11:46:51AM +0200, Daniel Stenberg via curl-users wrote:
> If there is anyone around who uses --raw with a decent use case? I would not 
> mind learning how/why
> because I'm about to break it:

Debian codesearch shows a number of projects using it:
https://sourcegraph.com/search?q=context:global+/curl.*--raw%5B%5E-%5D/&patternType=keyword&sm=0

xrootd (in tests):
curl -v -L --raw -H "X-Transfer-Status: true" -H "TE: trailers" 
"${HOST}/$alphabetFilePath" --output - | tr -d '\r' > "$outputFilePath"

uwsgi (in examples):
# curl -D /dev/stdout -N --raw http://localhost:9090/

bashtop:
curl -m 4 --raw -r 0-5000 
https://raw.githubusercontent.com/aristocratos/bashtop/master/bashtop
curl -m 3 --raw 
https://raw.githubusercontent.com/aristocratos/bashtop/master/themes/index.txt
curl -m 3 --raw 
"https://raw.githubusercontent.com/aristocratos/bashtop/master/themes/${git_theme}";

Plus, many other hits via sourcegraph:
https://sourcegraph.com/search?q=context:global+/curl.*--raw%5B%5E-%5D/&patternType=keyword&sm=0

A few examples:
$ curl -i -0 --raw http://fretboard.tardate.com/status.csv
$ curl -v --raw --http2-prior-knowledge -X POST  -H "content-type: 
application/grpc" -H "TE: trailers" --data-binary @hello.mspack --output - 
http://localhost:8080/greeter.api.GreeterApi/hello
curl -s --raw 
https://raw.githubusercontent.com/bcgov/DITP-DevOps/main/code/snippets/getDockerHost
curl -s --raw -L "${kf_url}"


Fortunately, the use of --raw in the vast majority of these examples looks on
first glance to be completely wrong.  It looks like people haven't read the
documentation and assume that --raw is like jq's -r (raw strings) and are
fortunate that the servers aren't actually using chunked encoding or gzip
compression so the results just work. Some other examples appear to be for
debugging purposes where --raw might show more information, but are likely to
once again, be simply unneeded.

Interestingly, a lot of these --raw uses are also using the equally unnecessary
and potentially harmful -X as well showing how people are simply not
understanding these options. Maybe you need another blog post like
https://daniel.haxx.se/blog/2015/09/11/unnecessary-use-of-curl-x/ but for
--raw.

Dan
-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users
Etiquette:   https://curl.se/mail/etiquette.html


Re: Release candidate 3: curl 8.15.0-rc3

2025-07-09 Thread Dan Fandrich via curl-users
On Wed, Jul 09, 2025 at 03:44:56PM +0200, Daniel Stenberg via curl-users wrote:
> On Wed, 9 Jul 2025, Dick Brooks wrote:
> > Congratulations. Any chance we will see an SBOM for curl in the future?
> 
> The "normal" curl release does not need an SBOM. It is just one thing and 
> this one thing comes
> only from us: the curl release.
> 
> curl releases are done as source code tarballs with no third party code 
> included.

The curl source is fully marked up with SPDX license tags, so you can generate
your own accurate source-level SBOM in SPDX format with license information
using the "reuse" tool. Just run "reuse spdx". See https://reuse.readthedocs.io
for more information.

Dan
-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users
Etiquette:   https://curl.se/mail/etiquette.html