[RFE] Improve etag handling with --etag-compare-update
Hello, I'm trying to implement a "download only if updated on a remote resource" functionality with a curl script using etags. Currently the etag file saved with --etag-save becomes useless after a single update on a remote resource. This can be worked around on the client side by saving etag to a temporary file, comparing it with previously stored etag file etc. But it would be nice to have a --etag-compare-update option that would work like --etag-compare plus overwrite the with the new etag if one was returned by server. Thanks. -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users Etiquette: https://curl.se/mail/etiquette.html
Re: [RFE] Improve etag handling with --etag-compare-update
On 2025-04-29, Daniel Stenberg wrote: On Tue, 29 Apr 2025, Aleksei wrote: curl --etag-compare file0.etag --etag-save file0.etag $URL ... which only downloads the URL again if it is different than the last download. If they work together in a single invocation then great, no enhancement needed. Man page describes ETag usage in separate requests: Use the option --etag-save to first save the ETag from a response, and then use this option to compare against the saved ETag in a **subsequent request**. It saves the etag from this transfer for the purpose of using in the next command line. The compare option however uses the existing file contents for *this* command line. -- / daniel.haxx.se || https://rock-solid.curl.dev Thanks for explanations Daniel, all clear now. Perhaps these things should be mentioned in the man page, the current --etag-save and --etag-compare sections gave me a clear impression these options are to be used in separate curl invocations. -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users Etiquette: https://curl.se/mail/etiquette.html
Re: [RFE] Improve etag handling with --etag-compare-update
On 2025-04-28, Daniel Stenberg wrote: On Mon, 28 Apr 2025, Aleksei via curl-users wrote: I'm trying to implement a "download only if updated on a remote resource" functionality with a curl script using etags. Currently the etag file saved with --etag-save becomes useless after a single update on a remote resource. I don't understand. Can you elaborate? If the remote resource is indeed updated, surely it should download again and update the etag file? / daniel.haxx.se || https://rock-solid.curl.dev I'm asking for "and update the etag file" part to be done by curl. Scenario: 1) curl downloads a webpage, saving etag file in file0.etag 2) website updates a webpage (update 1) 3) curl uses "--etag-compare file0.etag" and re-downloads the page - so far so good 4) website updates a webpage (update 2) 5) How do I check that webpage is updated now? "--etag-compare file0.etag" will re-download even if update 2 has been downloaded Currently I would have to add a check at #3 to save the etag in a temp file, compare to file0.etag and copy it over if necessary - a script-side workaround I'm trying to avoid. -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users Etiquette: https://curl.se/mail/etiquette.html
Re: [RFE] Improve etag handling with --etag-compare-update
On 2025-04-29, Daniel Stenberg wrote: curl --etag-compare file0.etag --etag-save file0.etag $URL / daniel.haxx.se || https://rock-solid.curl.dev If they work together in a single invocation then great, no enhancement needed. Man page describes ETag usage in separate requests: Use the option --etag-save to first save the ETag from a response, and then use this option to compare against the saved ETag in a **subsequent request**. -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users Etiquette: https://curl.se/mail/etiquette.html
Re: [RFE] Improve etag handling with --etag-compare-update
On 2025-04-29, Daniel Stenberg wrote: On Tue, 29 Apr 2025, Aleksei wrote: Thanks for explanations Daniel, all clear now. Perhaps these things should be mentioned in the man page, the current --etag-save and --etag-compare sections gave me a clear impression these options are to be used in separate curl invocations. How about adding this paragraph to the --etag-save documentation? In many situations you want to use an existing etag in the request to avoid downloading the same resource again but also save the new etag if it has indeed changed, by using both etag options --etag-save and --etag-compare, in the same command line. -- / daniel.haxx.se || https://rock-solid.curl.dev Looks good to me, I would only add "with the same filename" after "--etag-compare". -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users Etiquette: https://curl.se/mail/etiquette.html
