Moved Permanently

This advisory document should ideally be written in cooperation with the reporter to make sure all the angles and details of the problem are gathered and described correctly and succinctly. ## New document A security advisory for curl is created in the `docs/` folder in the [curl-www](https://github.com/curl/curl-www) repository. It should be named `$CVEID.md` where `$CVEID` is the full CVE Id that has been registered for the flaw. Like `CVE-2016-0755`. The `.md` extension of course means that the document is written using markdown. The standard way to go about this is to first write the `VULNERABILITY` section for the document, so that there is description of the flaw available, then paste this description into the CVE Id request. ### `vuln.pm` The new issue should be entered at the top of the list in the file `vuln.pm` in the same directory. It holds a large array with all published curl vulnerabilities. All fields should be filled in accordingly, separated by a pipe character (`|`). The fields for every CVE in `vuln.pm` are, in order: 1. HTML page name 2. first vulnerable version 3. last vulnerable version 4. name of the issue 5. CVE Id 6. announce date (`YYYYMMDD`) 7. report to the project date (`YYYYMMDD`) 8. CWE 9. awarded reward amount (USD) 10. area (single word) 11. C-issue (`-` if not a C issue at all, `OVERFLOW` , `OVERREAD`, `DOUBLE_FREE`, `USE_AFTER_FREE`, `NULL_MISTAKE`, `UNINIT`, `BAD_FREE`) 12. affected components: `both`, `lib` or `tool` 13. severity: `low`, `medium`, `high` or `critical` 14. URL to the initial report (often on HackerOne) ### `Makefile` The new CVE webpage filename needs to be added in the `Makefile`'s `CVELIST` macro. When the markdown is in place and the `Makefile` and `vuln.pm` are updated, all other files and metadata for all curl advisories and versions get generated automatically using those files. ## Document format The easy way is to start with a recent previously published advisory and blank out old texts and save it using a new name. Save the subtitles and general layout. Some details and metadata are extracted from this document so it is important to stick to the existing format. The first list must be the title of the issue. ### VULNERABILITY The first subtitle should be `VULNERABILITY`. That should then include a through and detailed description of the flaw. Including how it can be triggered and maybe something about what might happen if triggered or exploited. ### INFO The next section is `INFO` which adds meta data information about the flaw. It specifically mentions the official CVE Id for the issue and it must list the CWE Id, starting on its own line. We write CWE identifiers in advisories with the full (official) explanation on the right side of a colon. Like this: `CWE-305: Authentication Bypass by Primary Weakness` ### AFFECTED VERSIONS The third section first lists what versions that are affected, then adds clarity by stressing what versions that are *not* affected. A third line adds information about which specific git commit that introduced the vulnerability. The `Introduced-in` commit should be a full URL that displays the commit, but should work as a stand-alone commit hash if everything up to the last slash is cut out. An example using the correct syntax: ~~~ - Affected versions: curl 7.16.1 to and including 7.88.1 - Not affected versions: curl < 7.16.1 and curl >= 8.0.0 - Introduced-in: https://github.com/curl/curl/commit/2147284cad ~~~ ### THE SOLUTION This section describes and discusses the fix. The only mandatory information here is the link to the git commit that fixes the problem. The `Fixed-in` value should be a full URL that displays the commit, but should work as a stand-alone commit hash if everything up to the last slash is cut out. Example: `- Fixed-in: https://github.com/curl/curl/commit/af369db4d3833272b8ed` ### RECOMMENDATIONS This section lists the recommended actions for the users in a top to bottom priority order and should ideally contain three items but no less than two. The top two are almost always `upgrade curl to version XXX` and `apply the patch to your local version`. ### TIMELINE Detail when this report was received in the project. When package distributors were notified (via the distros mailing list or similar) When the advisory and fixed version are released. ### CREDITS Mention the reporter and patch author at least, then everyone else involved you think deserves a mention. If you want to mention more than one name, separate the names with comma (`,`). ~~~ - Reported-by: Full Name - Patched-by: Full Name ~~~ # curl sponsors A sponsor is someone who donates money or resources to the curl project for no specific service in return. curl accepts donations via [GitHub sponsors](https://github.com/sponsors/curl) and [Open Collective](https://opencollective.com/curl). An even better way to contribute to the project might be to pay an engineer or two to spend work hours on curl related tasks. We promise to use donated funds for things and activities that we believe are beneficial for the project and its development. That includes but is not limited to developer conferences, infrastructure, development, services and hardware. Recurring donations above a certain amount of money puts the sponsor at a named sponsor level: **Silver**, **Gold**, **Platinum** or **Top**. Sponsors on a named level can provide their logo image and preferred URL and get recognition on the curl website's [sponsor page](https://curl.se/sponsors.html), assuming they meet the project's standards and requirements. - **Silver Sponsor** at least 100 USD/month - **Gold Sponsor** at least 500 USD/month - **Platinum Sponsor** at least 1000 USD/month - **Top Sponsor** outstanding extra valuable help ## Sponsor requirements A named level sponsor is entitled a logo and link on the curl website assuming the company, brand and link are not deemed unsuitable. The curl team reserves the right to make that decision at its own discretion. Sponsors may be denied a website presence for example if involved with drugs, gambling, pornography, social media manipulation etc. ## Past Sponsors Sponsors that stop paying are considered *Past Sponsors* and are not displayed on the sponsor page anymore. We thank you for your contributions. ## Donations Please note that sponsorship and donations are exactly that: donations to the curl project. They are used to help and further the project as the project leadership deems best. No goods or services are expected or promised in return. Requests for refunds for such purposes are rejected. # TLS Certificate Verification ## Native vs file based If curl was built with Schannel support, then curl uses the Windows native CA store for verification. On Apple operating systems, it is possible to use Apple's "SecTrust" services for certain TLS backends, details below. All other TLS libraries use a file based CA store by default. ## Verification Every trusted server certificate is digitally signed by a Certificate Authority, a CA. In your local CA store you have a collection of certificates from *trusted* certificate authorities that TLS clients like curl use to verify servers. curl does certificate verification by default. This is done by verifying the signature and making sure the certificate was crafted for the server name provided in the URL. If you communicate with HTTPS, FTPS or other TLS-using servers using certificates signed by a CA whose certificate is present in the store, you can be sure that the remote server really is the one it claims to be. If the remote server uses a self-signed certificate, if you do not install a CA cert store, if the server uses a certificate signed by a CA that is not included in the store you use or if the remote host is an impostor impersonating your favorite site, the certificate check fails and reports an error. If you think it wrongly failed the verification, consider one of the following sections. ### Skip verification Tell curl to *not* verify the peer with `-k`/`--insecure`. We **strongly** recommend this is avoided and that even if you end up doing this for experimentation or development, **never** skip verification in production. ### Use a custom CA store Get a CA certificate that can verify the remote server and use the proper option to point out this CA cert for verification when connecting - for this specific transfer only. With the curl command line tool: `--cacert [file]` If you use the curl command line tool without a native CA store, then you can specify your own CA cert file by setting the environment variable `CURL_CA_BUNDLE` to the path of your choice. `SSL_CERT_FILE` and `SSL_CERT_DIR` are also supported. If you are using the curl command line tool on Windows, curl searches for a CA cert file named `curl-ca-bundle.crt` in these directories and in this order: 1. application's directory 2. current working directory 3. Windows System directory (e.g. C:\Windows\System32) 4. Windows Directory (e.g. C:\Windows) 5. all directories along %PATH% curl 8.11.0 added a build-time option to disable this search behavior, and another option to restrict search to the application's directory. curl 8.19.0 added a build-time option to enable Native CA by default on Windows. This build-time option by default also disables searching for a `curl-ca-bundle.crt` on disk. ### Use the native store In several environments, in particular on Microsoft and Apple operating systems, you can ask curl to use the system's native CA store when verifying the certificate. Depending on how curl was built, this may already be the default. With the curl command line tool: `--ca-native`. ### Modify the CA store Add the CA cert for your server to the existing default CA certificate store. Usually you can figure out the path to the local CA store by looking at the verbose output that `curl -v` shows when you connect to an HTTPS site. ### Change curl's default CA store The default CA certificate store curl uses is set at build time. When you build curl you can point out your preferred path. ### Extract CA cert from a server curl -w %{certs} https://example.com > cacert.pem The certificate has `BEGIN CERTIFICATE` and `END CERTIFICATE` markers. ### Get the Mozilla CA store Download a version of the Firefox CA store converted to PEM format on the [CA Extract](https://curl.se/docs/caextract.html) page. It always features the latest Firefox bundle. ## Native CA store ### Windows + Schannel If curl was built with Schannel, then curl uses the certificates that are built into the OS. These are the same certificates that appear in the Internet Options control panel (under Windows). Any custom security rules for certificates are honored. Schannel runs CRL checks on certificates unless peer verification is disabled. ### Apple + OpenSSL/GnuTLS When curl is built with Apple SecTrust enabled and uses an OpenSSL compatible TLS backend or GnuTLS, the default verification is handled by that Apple service. As in: curl https://example.com You may still provide your own certificates on the command line, such as: curl --cacert mycerts.pem https://example.com In this situation, Apple SecTrust is **not** used and verification is done **only** with the trust anchors found in `mycerts.pem`. If you want **both** Apple SecTrust and your own file to be considered, use: curl --ca-native --cacert mycerts.pem https://example.com #### Other Combinations How well the use of native CA stores work in all other combinations depends on the TLS backend and the OS. Many TLS backends offer functionality to access the native CA on a range of operating systems. Some provide this only on specific configurations. Specific support in curl exists for Windows and OpenSSL compatible TLS backends. It tries to load the certificates from the Windows "CA" and "ROOT" stores for transfers requesting the native CA. Due to Window's delayed population of those stores, this might not always find all certificates. ## HTTPS proxy curl can do HTTPS to the proxy separately from the connection to the server. This TLS connection is handled and verified separately from the server connection so instead of `--insecure` and `--cacert` to control the certificate verification, you use `--proxy-insecure` and `--proxy-cacert`. With these options, you make sure that the TLS connection and the trust of the proxy can be kept totally separate from the TLS connection to the server. # SSL problems First, let's establish that we often refer to TLS and SSL interchangeably as SSL here. The current protocol is called TLS, it was called SSL a long time ago. There are several known reasons why a connection that involves SSL might fail. This is a document that attempts to detail the most common ones and how to mitigate them. ## CA certs CA certs are used to digitally verify the server's certificate. You need a "ca bundle" for this. See lots of more details on this in the `SSLCERTS` document. ## CA bundle missing intermediate certificates When using said CA bundle to verify a server cert, you may experience problems if your CA store does not contain the certificates for the intermediates if the server does not provide them. The TLS protocol mandates that the intermediate certificates are sent in the handshake, but as browsers have ways to survive or work around such omissions, missing intermediates in TLS handshakes still happen that browser users do not notice. Browsers work around this problem in two ways: they cache intermediate certificates from previous transfers and some implement the TLS "AIA" extension that lets the client explicitly download such certificates on demand. ## Protocol version Some broken servers fail to support the protocol negotiation properly that SSL servers are supposed to handle. This may cause the connection to fail completely. Sometimes you may need to explicitly select an SSL version to use when connecting to make the connection succeed. An additional complication can be that modern SSL libraries sometimes are built with support for older SSL and TLS versions disabled. All versions of SSL and the TLS versions before 1.2 are considered insecure and should be avoided. Use TLS 1.2 or later. ## Ciphers Clients give servers a list of ciphers to select from. If the list does not include any ciphers the server wants/can use, the connection handshake fails. curl has recently disabled the user of a whole bunch of seriously insecure ciphers from its default set (slightly depending on SSL backend in use). You may have to explicitly provide an alternative list of ciphers for curl to use to allow the server to use a weak cipher for you. Note that these weak ciphers are identified as flawed. For example, this includes symmetric ciphers with less than 128-bit keys and RC4. Schannel in Windows XP is not able to connect to servers that no longer support the legacy handshakes and algorithms used by those versions, so we advise against building curl to use Schannel on really old Windows versions. Reference: [Prohibiting RC4 Cipher Suites](https://datatracker.ietf.org/doc/html/draft-popov-tls-prohibiting-rc4-01) ## Allow BEAST BEAST is the name of a TLS 1.0 attack that surfaced 2011. When adding means to mitigate this attack, it turned out that some broken servers out there in the wild did not work properly with the BEAST mitigation in place. To make such broken servers work, the --ssl-allow-beast option was introduced. Exactly as it sounds, it re-introduces the BEAST vulnerability but on the other hand it allows curl to connect to that kind of strange servers. ## Disabling certificate revocation checks Some SSL backends may do certificate revocation checks (CRL, OCSP, etc) depending on the OS or build configuration. The --ssl-no-revoke option was introduced in 7.44.0 to disable revocation checking but currently is only supported for Schannel (the native Windows SSL library), with an exception in the case of Windows' Untrusted Publishers block list which it seems cannot be bypassed. This option may have broader support to accommodate other SSL backends in the future. References: https://curl.se/docs/ssl-compared.html --- c: Copyright (C) Daniel Stenberg, , et al. SPDX-License-Identifier: curl Title: testcurl.pl Section: 1 Source: testcurl See-also: - runtests.pl Added-in: 7.11.2 --- # NAME testcurl.pl - (automatically) test curl # SYNOPSIS **testcurl.pl [options] [dir] \> output** # DESCRIPTION *testcurl* is the master script to use for automatic distributed testing of curl from git or daily snapshots. It is written for the purpose of being run from a crontab job or similar at a regular interval. The output is suitable to be mailed to **curl-autocompile@haxx.se** to be dealt with automatically (make sure the subject includes the word "autobuild" as the mail gets silently discarded otherwise). The most current build status (with a reasonable backlog) is published on the curl site, at https://curl.se/dev/builds.html *options* may be omitted. See *--setup* for what happens then. *dir* is a curl source directory, possibly a daily snapshot one. Using this makes *testcurl* skip the *autoreconf* stage and thus it removes the dependency on automake, autoconf, libtool, GNU m4 and possibly a few other things. *testcurl* runs `autoreconf` (or similar), configure, builds curl and libcurl in a separate build directory and then runs `make test` to test the fresh build. # OPTIONS ## `--configure=[options]` Configure options passed to configure. ## `--crosscompile` This is a cross-compile. Makes *testcurl* skip a few things. ## `--desc=[desc]` Description of your test system. Displayed on the build summary page on the website. ## `--email=[email]` Set email address to report as. Displayed in the build logs on the site. ## `--mktarball=[command]` Generic command to run after completed test. ## `--name=[name]` Set name to report as. Displayed in the build summary on the site. ## `--nobuildconf` Do not run autoreconf. Useful when many builds use the same source tree, as then only one need to do this. Also, if multiple processes run tests simultaneously on the same source tree (like several hosts on a NFS mounted directory), simultaneous autoreconf invokes may cause problems. (Added in 7.14.1) ## `--nogitpull` Do not update from git even though it is a git tree. Useful to still be able to test even though your network is down, or similar. ## `--runtestopts=[options]` Options that is passed to the runtests script. Useful for disabling valgrind by force, and similar. ## `--setup=[filename]` filename to read setup from (deprecated). The old style of providing info. If info is missing when *testcurl* is started, it prompts you and then stores the info in a 'setup' file, which it looks for on each invoke. Use *--name*, *--email*, *--configure* and *--desc* instead. ## `--target=[your os]` Specify your target environment. Recognized strings include `vc`, `mingw32`, and `borland`. # INITIAL SETUP First, make a checkout from git (or you write a script that downloads daily snapshots automatically): $ mkdir curl-testing $ cd curl-testing $ git clone https://github.com/curl/curl With the curl sources checked out, or downloaded, you can start testing right away. If you want to use *testcurl* without command line arguments and to have it store and remember the config in its 'setup' file, then start it manually now and fill in the answers to the questions it prompts you for: $ ./curl/tests/testcurl Now you are ready to go. If you let the script run, it performs a full cycle and spit out lots of output. Mail us that output as described above. # CRONTAB EXAMPLE The crontab could include something like this: # autobuild curl: 0 4 * * * cd curl-testing && ./testit.sh Where `testit.sh` is a shell script that could look similar to this: mail="mail -s autobuild curl-autocompile@haxx.se" name="--name=whoami" email="--email=iamme@nowhere" desc='"--desc=supermachine Turbo 2000"' testprog="perl ./curl/tests/testcurl.pl $name $email $desc" opts1="--configure=--enable-debug" opts2="--configure=--enable-ipv6" # run first test $testprog $opts1 | $mail # run second test $testprog $opts2 | $mail # The Art Of Scripting HTTP Requests Using curl ## Background This document assumes that you are familiar with HTML and general networking. The increasing amount of applications moving to the web has made "HTTP Scripting" more frequently requested and wanted. To be able to automatically extract information from the web, to fake users, to post or upload data to web servers are all important tasks today. curl is a command line tool for doing all sorts of URL manipulations and transfers, but this particular document focuses on how to use it when doing HTTP requests for fun and profit. This documents assumes that you know how to invoke `curl --help` or `curl --manual` to get basic information about it. curl is not written to do everything for you. It makes the requests, it gets the data, it sends data and it retrieves the information. You probably need to glue everything together using some kind of script language or repeated manual invokes. ## The HTTP Protocol HTTP is the protocol used to fetch data from web servers. It is a simple protocol that is built upon TCP/IP. The protocol also allows information to get sent to the server from the client using a few different methods, as is shown here. HTTP is plain ASCII text lines being sent by the client to a server to request a particular action, and then the server replies a few text lines before the actual requested content is sent to the client. The client, curl, sends an HTTP request. The request contains a method (like GET, POST, HEAD etc), a number of request headers and sometimes a request body. The HTTP server responds with a status line (indicating if things went well), response headers and most often also a response body. The "body" part is the plain data you requested, like the actual HTML or the image etc. ## See the Protocol Using curl's option [`--verbose`](https://curl.se/docs/manpage.html#-v) (`-v` as a short option) displays what kind of commands curl sends to the server, as well as a few other informational texts. `--verbose` is the single most useful option when it comes to debug or even understand the curl<->server interaction. Sometimes even `--verbose` is not enough. Then [`--trace`](https://curl.se/docs/manpage.html#-trace) and [`--trace-ascii`](https://curl.se/docs/manpage.html#--trace-ascii) offer even more details as they show **everything** curl sends and receives. Use it like this: curl --trace-ascii debugdump.txt https://www.example.com/ ## See the Timing Many times you may wonder what exactly is taking all the time, or you want to know the amount of milliseconds between two points in a transfer. For those, and other similar situations, the [`--trace-time`](https://curl.se/docs/manpage.html#--trace-time) option is what you need. It prepends the time to each trace output line: curl --trace-ascii d.txt --trace-time https://example.com/ ## See which Transfer When doing parallel transfers, it is relevant to see which transfer is doing what. When response headers are received (and logged) you need to know which transfer these are for. [`--trace-ids`](https://curl.se/docs/manpage.html#--trace-ids) option is what you need. It prepends the transfer and connection identifier to each trace output line: curl --trace-ascii d.txt --trace-ids https://example.com/ ## See the Response By default curl sends the response to stdout. You need to redirect it somewhere to avoid that, most often that is done with `-o` or `-O`. # URL ## Spec The Uniform Resource Locator format is how you specify the address of a particular resource on the Internet. You know these, you have seen URLs like https://curl.se/ or https://example.com/ a million times. RFC 3986 is the canonical spec. The formal name is not URL, it is **URI**. ## Host The hostname is usually resolved using DNS or your /etc/hosts file to an IP address and that is what curl communicates with. Alternatively you specify the IP address directly in the URL instead of a name. For development and other trying out situations, you can point to a different IP address for a hostname than what would otherwise be used, by using curl's [`--resolve`](https://curl.se/docs/manpage.html#--resolve) option: curl --resolve www.example.org:80:127.0.0.1 https://www.example.org/ ## Port number Each protocol curl supports operates on a default port number, be it over TCP or in some cases UDP. Normally you do not have to take that into consideration, but at times you run test servers on other ports or similar. Then you can specify the port number in the URL with a colon and a number immediately following the hostname. Like when doing HTTP to port 1234: curl https://www.example.org:1234/ The port number you specify in the URL is the number that the server uses to offer its services. Sometimes you may use a proxy, and then you may need to specify that proxy's port number separately from what curl needs to connect to the server. Like when using an HTTP proxy on port 4321: curl --proxy http://proxy.example.org:4321 https://remote.example.org/ ## Username and password Some services are setup to require HTTP authentication and then you need to provide name and password which is then transferred to the remote site in various ways depending on the exact authentication protocol used. You can opt to either insert the user and password in the URL or you can provide them separately: curl https://user:password@example.org/ or curl -u user:password https://example.org/ You need to pay attention that this kind of HTTP authentication is not what is usually done and requested by user-oriented websites these days. They tend to use forms and cookies instead. ## Path part The path part is sent off to the server to request that it sends back the associated response. The path is what is to the right side of the slash that follows the hostname and possibly port number. # Fetch a page ## GET The simplest and most common request/operation made using HTTP is to GET a URL. The URL could itself refer to a webpage, an image or a file. The client issues a GET request to the server and receives the document it asked for. If you issue the command line curl https://curl.se/ you get a webpage returned in your terminal window. The entire HTML document this URL identifies. All HTTP replies contain a set of response headers that are normally hidden, use curl's [`--include`](https://curl.se/docs/manpage.html#-i) (`-i`) option to display them as well as the rest of the document. ## HEAD You can ask the remote server for ONLY the headers by using the [`--head`](https://curl.se/docs/manpage.html#-I) (`-I`) option which makes curl issue a HEAD request. In some special cases servers deny the HEAD method while others still work, which is a particular kind of annoyance. The HEAD method is defined and made so that the server returns the headers exactly the way it would do for a GET, but without a body. It means that you may see a `Content-Length:` in the response headers, but there must not be an actual body in the HEAD response. ## Multiple URLs in a single command line A single curl command line may involve one or many URLs. The most common case is probably to use one, but you can specify any amount of URLs. Yes any. No limits. You then get requests repeated over and over for all the given URLs. Example, send two GET requests: curl https://url1.example.com https://url2.example.com If you use [`--data`](https://curl.se/docs/manpage.html#-d) to POST to the URL, using multiple URLs means that you send that same POST to all the given URLs. Example, send two POSTs: curl --data name=curl https://url1.example.com https://url2.example.com ## Multiple HTTP methods in a single command line Sometimes you need to operate on several URLs in a single command line and do different HTTP methods on each. For this, you might enjoy the [`--next`](https://curl.se/docs/manpage.html#-:) option. It is a separator that separates a bunch of options from the next. All the URLs before `--next` get the same method and get all the POST data merged into one. When curl reaches the `--next` on the command line, it resets the method and the POST data and allow a new set. Perhaps this is best shown with a few examples. To send first a HEAD and then a GET: curl -I https://example.com --next https://example.com To first send a POST and then a GET: curl -d score=10 https://example.com/post.cgi --next https://example.com/results.html # HTML forms ## Forms explained Forms are the general way a website can present an HTML page with fields for the user to enter data in, and then press some kind of 'OK' or 'Submit' button to get that data sent to the server. The server then typically uses the posted data to decide how to act. Like using the entered words to search in a database, or to add the info in a bug tracking system, display the entered address on a map or using the info as a login-prompt verifying that the user is allowed to see what it is about to see. Of course there has to be some kind of program on the server end to receive the data you send. You cannot invent something out of the air. ## GET A GET-form uses the method GET, as specified in HTML like: ```html