Using utls with `http.Transport`

[AxbB36]

I'm using commit a89e7e6. The examples I have found of using utls with HTTPS all make a single request on a single connection, then throw the connection away. For example, httpGetOverConn in examples.go.

I'm trying to use utls with http.Transport, to take advantage of persistent connections and reasonable default timeouts. To do this, I'm hooking into the DialTLS callback. There is a problem when using a utls fingerprint that includes h2 in ALPN and a server that supports HTTP/2. The server switches to HTTP/2 mode, but the client stays in HTTP/1.1 mode, because net/http disables automatic HTTP/2 support whenever DialTLS is set. The end result is an HTTP/1.1 client speaking to an HTTP/2 server; i.e, a similar problem as what was reported in golang/go#14275 (comment). The error message differs depending on the fingerprint:

HelloFirefox_63

net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00d\x00\x04\x00\x10\x00\x00\x00\x06\x00\x00@\x00\x00\x00\x04\b\x00\x00\x00\x00\x00\x00\x0f\x00\x01\x00\x00\x1e\a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01http2_handshake_failed"

HelloChrome_70

local error: tls: unexpected message

HelloIOS_11_1

2019/01/11 14:48:56 Unsolicited response received on idle HTTP channel starting with "\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00d\x00\x04\x00\x10\x00\x00\x00\x06\x00\x00@\x00\x00\x00\x04\b\x00\x00\x00\x00\x00\x00\x0f\x00\x01"; err=<nil>
readLoopPeekFailLocked: <nil>

I get the same results even if I pre-configure the http.Transport with HTTP/2 support by calling http2.ConfigureTransport(tr).

I wrote a test program to reproduce these results. It takes a -utls option to select a utls client hello ID, and a -callhandshake option to control whether to call UConn.Handshake within DialTLS, or allow it to be called implicitly by the next Read or Write. I included the latter option because I found that not calling UConn.Handshake inside DialTLS avoids the HTTP version mismatch; however it also results in a client hello that lacks ALPN and differs from the requested one in other ways, so it's not an adequate workaround.

Click to expand program

package main

import (
	"flag"
	"fmt"
	"io"
	"io/ioutil"
	"net"
	"net/http"
	"os"
	"strings"

	utls "github.com/refraction-networking/utls"
)

func main() {
	utlsClientHelloIDName := flag.String("utls", "", "use utls with the given ClientHelloID (e.g. HelloGolang)")
	callHandshake := flag.Bool("callhandshake", false, "call UConn.Handshake inside DialTLS")
	flag.Parse()

	if *callHandshake && *utlsClientHelloIDName == "" {
		fmt.Fprintf(os.Stderr, "error: -callhandshake only makes sense with -utls\n")
		os.Exit(1)
	}

	if flag.NArg() != 1 {
		fmt.Fprintf(os.Stderr, "error: need a URL\n")
		os.Exit(1)
	}
	url := flag.Arg(0)

	utlsClientHelloID, ok := map[string]*utls.ClientHelloID{
		"":                      nil,
		"HelloGolang":           &utls.HelloGolang,
		"HelloRandomized":       &utls.HelloRandomized,
		"HelloRandomizedALPN":   &utls.HelloRandomizedALPN,
		"HelloRandomizedNoALPN": &utls.HelloRandomizedNoALPN,
		"HelloFirefox_Auto":     &utls.HelloFirefox_Auto,
		"HelloFirefox_55":       &utls.HelloFirefox_55,
		"HelloFirefox_56":       &utls.HelloFirefox_56,
		"HelloFirefox_63":       &utls.HelloFirefox_63,
		"HelloChrome_Auto":      &utls.HelloChrome_Auto,
		"HelloChrome_58":        &utls.HelloChrome_58,
		"HelloChrome_62":        &utls.HelloChrome_62,
		"HelloChrome_70":        &utls.HelloChrome_70,
		"HelloIOS_Auto":         &utls.HelloIOS_Auto,
		"HelloIOS_11_1":         &utls.HelloIOS_11_1,
	}[*utlsClientHelloIDName]
	if !ok {
		fmt.Fprintf(os.Stderr, "unknown client hello ID %q\n", *utlsClientHelloIDName)
		os.Exit(1)
	}

	tr := http.DefaultTransport.(*http.Transport)
	if utlsClientHelloID != nil {
		tr.DialContext = nil
		tr.Dial = func(network, addr string) (net.Conn, error) { panic("Dial should not be called") }
		tr.DialTLS = func(network, addr string) (net.Conn, error) {
			fmt.Printf("DialTLS(%q, %q)\n", network, addr)
			if tr.TLSClientConfig != nil {
				fmt.Printf("warning: ignoring TLSClientConfig %v\n", tr.TLSClientConfig)
			}
			conn, err := net.Dial(network, addr)
			if err != nil {
				return nil, err
			}
			uconn := utls.UClient(conn, nil, *utlsClientHelloID)
			colonPos := strings.LastIndex(addr, ":")
			if colonPos == -1 {
				colonPos = len(addr)
			}
			uconn.SetSNI(addr[:colonPos])
			if *callHandshake {
				err = uconn.Handshake()
			}
			return uconn, err
		}
	}

	for i := 0; i < 4; i++ {
		resp, err := get(tr, url)
		if err != nil {
			fmt.Printf("%2d err %v\n", i, err)
		} else {
			fmt.Printf("%2d %s %s\n", i, resp.Proto, resp.Status)
		}
	}
}

func get(rt http.RoundTripper, url string) (*http.Response, error) {
	req, err := http.NewRequest("GET", url, nil)
	if err != nil {
		return nil, err
	}
	resp, err := rt.RoundTrip(req)
	if err != nil {
		return nil, err
	}
	// Read and close the body to enable connection reuse with HTTP/1.1.
	_, err = io.Copy(ioutil.Discard, resp.Body)
	if err != nil {
		return nil, err
	}
	err = resp.Body.Close()
	if err != nil {
		return nil, err
	}
	return resp, nil
}

Sample usage:

test -utls HelloFirefox_63 -callhandshake https://golang.org/robots.txt

The output of the program appears in the following table. Things to notice:

• DialTLS with HelloGolang produces a fingerprint that is different from using http.Transport without DialTLS set.
• HelloFirefox_63, HelloChrome_70, and HelloIOS_11_1 all provide a usable connection (but with an incorrect fingerprint), as long as you don't call UConn.Handshake before returning from DialTLS.
• HelloFirefox_63, HelloChrome_70, and HelloIOS_11_1 all give the correct fingerprint, but fail with an HTTP version mismatch, when UConn.Handshake is called inside DialTLS.

Client Hello ID	call Handshake?	client ALPN	result
none	N/A	[h2, http/1.1]	ok HTTP/2
-utls HelloGolang		none	ok HTTP/1.1
-utls HelloGolang	-callhandshake	none	ok HTTP/1.1
-utls HelloFirefox_63		none	ok HTTP/1.1
-utls HelloFirefox_63	-callhandshake	[h2, http/1.1]	malformed HTTP response (HTTP/1.1 client, HTTP/2 server)
-utls HelloChrome_70		none	ok HTTP/1.1
-utls HelloChrome_70	-callhandshake	[h2, http/1.1]	local error: tls: unexpected message
-utls HelloIOS_11_1		none	ok HTTP/1.1
-utls HelloIOS_11_1	-callhandshake	[h2, h2-16, h2-15, h2-14, spdy/3.1, spdy/3, http/1.1]	readLoopPeekFailLocked: <nil> (HTTP/1.1 client, HTTP/2 server)

Is there a way to accomplish what I am trying to do?

[AxbB36]

I had some success by switching http2.Transport for http.Transport:

 package main
 
 import (
+	gotls "crypto/tls"
 	"flag"
 	"fmt"
 	"io"
@@ -11,6 +12,7 @@ import (
 	"strings"
 
 	utls "github.com/refraction-networking/utls"
+	"golang.org/x/net/http2"
 )
 
 func main() {
@@ -51,20 +53,15 @@ func main() {
 		os.Exit(1)
 	}
 
-	tr := http.DefaultTransport.(*http.Transport)
+	tr := &http2.Transport{}
 	if utlsClientHelloID != nil {
-		tr.DialContext = nil
-		tr.Dial = func(network, addr string) (net.Conn, error) { panic("Dial should not be called") }
-		tr.DialTLS = func(network, addr string) (net.Conn, error) {
+		tr.DialTLS = func(network, addr string, cfg *gotls.Config) (net.Conn, error) {
 			fmt.Printf("DialTLS(%q, %q)\n", network, addr)
-			if tr.TLSClientConfig != nil {
-				fmt.Printf("warning: ignoring TLSClientConfig %v\n", tr.TLSClientConfig)
-			}
 			conn, err := net.Dial(network, addr)
 			if err != nil {
 				return nil, err
 			}
-			uconn := utls.UClient(conn, nil, *utlsClientHelloID)
+			uconn := utls.UClient(conn, &utls.Config{NextProtos: cfg.NextProtos}, *utlsClientHelloID)
 			colonPos := strings.LastIndex(addr, ":")
 			if colonPos == -1 {
 				colonPos = len(addr)

These are the results using the URL https://golang.org/robots.txt. Notice:

• The output is the same whether UConn.Handshake is called within DialTLS or not.
• The HelloChrome_70 fingerprint doesn't work.

Client Hello ID	call Handshake?	client ALPN	result
none	N/A	[h2]	ok HTTP/2
-utls HelloGolang		[h2]	ok HTTP/2
-utls HelloGolang	-callhandshake	[h2]	ok HTTP/2
-utls HelloFirefox_63		[h2, http/1.1]	ok HTTP/2
-utls HelloFirefox_63	-callhandshake	[h2, http/1.1]	ok HTTP/2
-utls HelloChrome_70		[h2, http/1.1]	local error: tls: unexpected message
-utls HelloChrome_70	-callhandshake	[h2, http/1.1]	local error: tls: unexpected message
-utls HelloIOS_11_1		[h2, h2-16, h2-15, h2-14, spdy/3.1, spdy/3, http/1.1]	ok HTTP/2
-utls HelloIOS_11_1	-callhandshake	[h2, h2-16, h2-15, h2-14, spdy/3.1, spdy/3, http/1.1]	ok HTTP/2

Aside from the HelloChrome_70 failure, this is basically what I want. The only problem is that it doesn't work against non-HTTP/2 servers, whether utls is used or not:

$ test https://apache.org/
 0 err http2: unexpected ALPN protocol "http/1.1"; want "h2"
 1 err http2: unexpected ALPN protocol "http/1.1"; want "h2"
 2 err http2: unexpected ALPN protocol "http/1.1"; want "h2"
 3 err http2: unexpected ALPN protocol "http/1.1"; want "h2"

$ test -utls HelloGolang -callhandshake https://apache.org/
DialTLS("tcp", "apache.org:443")
 0 err unexpected EOF
DialTLS("tcp", "apache.org:443")
 1 err unexpected EOF
DialTLS("tcp", "apache.org:443")
 2 err unexpected EOF
DialTLS("tcp", "apache.org:443")
 3 err unexpected EOF

[AxbB36]

Yawning has a commit adding uTLS to obfs4proxy's meek_lite mode:
https://gitlab.com/yawning/obfs4/commit/4d453dab2120082b00bf6e63ab4aaeeda6b8d8a3
It uses an I idea I didn't think of. Instead of setting DialTLS on an http.Transport and using it directly, it uses a wrapper http.RoundTripper. On the first dial, the wrapper initiates the UClient connection, then inspects conn.ConnectionState().NegotiatedProtocol. Depending on the negotiated protocol, it creates internally either an http.Transport or http2.Transport. Future dials are passed through to the internal transport directly.

[sergeyfrolov]

This is a terrific solution. I'd like to get a wrapper like this into uTLS.

[Yawning]

Note that my implementation makes certain assumptions that may not be valid for a more general wrapper (omits some locking, destination host is assumed to be static), but there's comments where I do such things, and altering the behavior should be trivial.

This is a terrific solution. I'd like to get a wrapper like this into uTLS.

Was there ever a proper soloution to use uTLS with the net/http Client?

[sergeyfrolov]

Comments above describe a proper solution.
This wrapper had not been implemented in uTLS yet.

Note that my implementation makes certain assumptions that may not be valid for a more general wrapper (omits some locking, destination host is assumed to be static), but there's comments where I do such things, and altering the behavior should be trivial.

I'm trying to write a wrapper for net.http Client based on your soloution. Am I right in thinking that I would need to execute getTransport() for every new host and do some kind of connection pooling for effeciency?

EDIT: I was thinking about this incorrectly regarding connection pooling, The only thing required would be to use something like a map[string]net.Conn so we can handle multiple hosts. Everything seems to be working correctly, after some testing ill submit a PR.

[pharaohW]

the DialTLS says it's for non-proxied HTTPS requests，But if I want to make requests with transport and proxy，How to make that？

[AxbB36]

the DialTLS says it's for non-proxied HTTPS requests，But if I want to make requests with transport and proxy，How to make that？

There are some code samples here, from the uTLS integration into meek. Adding uTLS was this commit:

• Integrate uTLS as an option for TLS camouflage.

Adding proxy support was these commits:

• socks5 proxy support for uTLS.
• http proxy support for uTLS.
• https proxy support for uTLS.

[KyleKotowick]

The problem (or at least part of it) appears to be here: https://github.com/golang/go/blob/ea1437a8cdf6bb3c2d2447833a5d06dbd75f7ae4/src/net/http/transport.go#L1496

The http library determines whether to use h2 or not by evaluating the pconn.tlsState field, which gets set here: https://github.com/golang/go/blob/ea1437a8cdf6bb3c2d2447833a5d06dbd75f7ae4/src/net/http/transport.go#L1513

Since it fails to cast the conn to a tls.Conn (since it's a utls.Conn, which implements the same net.Conn interface but cannot be cast to a tls.Conn), it doesn't set the pconn.tlsState field, and therefore doesn't know later on that it's an h2 connection.

What would be ideal is some way to convert a utls.Conn to a tls.Conn (which shouldn't pose any issues once the handshake is already completed, I think?), and return that from the DialTLSContext custom function.

[molon]

the DialTLS says it's for non-proxied HTTPS requests，But if I want to make requests with transport and proxy，How to make that？

There are some code samples here, from the uTLS integration into meek. Adding uTLS was this commit:

• Integrate uTLS as an option for TLS camouflage.

Adding proxy support was these commits:

• socks5 proxy support for uTLS.
• http proxy support for uTLS.
• https proxy support for uTLS.

Very exciting solution.
I try to make some changes to it, because I especially need a react to req.Context.
For example, in httpProxy.Dial transformed into httpProxy.DialContext.
But in the follow-up, I found that by default, http2.Transport is used directly, and it internally initializes http2clientConnPool instead of http2noDialClientConnPool.
So its dial does not use the logic of http.Transport, which causes it to be used in dial Does not support req.Context react.

[1nfility]

Is there any way to use the RoundTripper solution with a Client so you can set Redirect handler and proxy?