Traffic analysis with mitmproxy

When most people hear the term ``proxy,‘’ they probably think of legacy proxy servers that act as gateways connecting local networks to the global network or as go-betweens protecting local clients against external access (e.g., NGINX, Squid, and WinGate come to mind). Given the name, mitmproxy could be assumed to be in the same category. However, the tool takes a different approach by specializing in HTTP(S) traffic analysis. Like Wireshark, the software is more of a sniffer that records the data traffic between the HTTP client and server and enables analysis by doing so.

The mitm part of the name hints at its functionality: mitmproxy acts as a man-in-the-middle (MITM) proxy that intercepts and modifies HTTP and HTTPS data traffic. You can record the HTTP conversation for later analysis, although the tool is limited to the protocol-specific data exchange. Unlike Wireshark and other sniffers, no other data is logged.

Mitmproxy can also act as a reverse proxy and forward data traffic to a specific server. Script-based manipulation of HTTP traffic is also an option, for which you can use simple Python scripts. Interaction with third-party applications for automatic manipulation or visualization is also possible with the Python API. Mitmproxy can generate SSL/TLS certificates for interception, as well.

A basic understanding of how mitmproxy works is useful if you want to work effectively in the environment. For example, take a look at HTTPS-protected access by a client to a web server. The client uses the HTTP command:

CONNECT server.en:443 HTTP/1.1

A legacy proxy server cannot manipulate SSL-/TLS-encrypted data traffic but simply forwards the request to the target system; it thus lives up to its name as an authorized agent. When you use mitmproxy, the HTTPS proxy sits between the client and server with the classic man-in-the-middle approach. For the client, mitmproxy looks like a server, while pretending to be the client for the server, which allows mitmproxy to decrypt the data traffic from both sides.

The challenge for mitmproxy is that the certification authority’s system is designed to prevent precisely this type of attack by allowing a trusted third party to sign a server’s certificates cryptographically to verify its legitimacy. If any discrepancies are noted, the connection is interrupted, which is why it is often difficult to analyze secure connections.

The mitmproxy developers used a trick to solve this problem. The software itself acts as a trustworthy certificate authority (CA). To do so, mitmproxy comes with a complete CA implementation that generates all the required certificates on the fly. For the client to trust these certificates, you need to register mitmproxy manually as a trusted CA.

Mitmproxy then needs to overcome further challenges to inject the environment between the client and server without being noticed. For example, the domain name of the remote party must be determined so that it can be used in the intercept certificate. To do so, mitmproxy uses upstream certificate sniffing. The tool also extracts the common name (CN) from the upstream certificate and the subject alternative names (SANs). It also cleverly works around the server name indication (SNI) handover.

The following processes take place for HTTPS connections with an intermediate mitmproxy:

1. The client establishes a connection to mitmproxy and generates an HTTP CONNECT request.
2. Mitmproxy responds with a 200 Connection Established message, simulating a CONNECT pipe being opened.
3. The client opens an SSL/TLS connection on the assumption that it is talking to the desired server. On doing so, it states the SNI for the hostname.
4. Mitmproxy opens the connection to the server and sets up a secure connection with the SNI hostname specified by the client.
5. The target system responds with a certificate containing the CN and SAN values required to create the interception certificate.
6. Mitmproxy generates the interception certificate and continues the client SSL/TLS handshake interrupted in step 3.
7. The client sends the request through the open connection.
8. Mitmproxy forwards the request to the server through the connection initiated in step 4.

This simplified process sequence shows the sophistication of the actions performed by mitmproxy just to slip into position between the client and the server.

Mitmproxy is available for Linux, macOS, and Windows. Standalone binaries are available for Windows and Linux, with distribution-specific packages for various Linux distributions (e.g., Arch, Debian, Ubuntu, and Kali). After installing on Windows, mitmproxy, mitmdump, and mitmweb are added to PATH and can be called from the command line. In principle, the Linux packages will also run on Windows Subsystem for Linux (WSL).

If you prefer to use a Docker container, use the following command to launch the mitmproxy terminal interface:

docker run --rm -it -v ~/.mitmproxy:/home/mitmproxy/.mitmproxy -p 8080:8080 mitmproxy/mitmproxy

To make sure the client’s web traffic is routed through mitmproxy, you need to adjust the global settings when installing the software locally and enter a proxy address of 127.0.0.1:8080. You could also use a proxy auto-configuration (PAC) file to simplify the global network configuration; the required files are available online.

If you try to access the Internet with your web browser after configuring the proxy, the attempt will fail. Your browser displays a Your connection is not private error message. The reason is simple: The browser does not consider the intermediary proxy to be trustworthy because it considers the mitmproxy certificate invalid. Once the proxy server is activated, you can pick up the required certificate from the mitm.it website. Valid certificates for all supported platforms are available there. On Windows, use the import wizard to deploy the certificate. On Linux, the easiest way to import the certificate is with the command:

sudo security add-trusted-cert -d -p ssl -p basic -k /Library/Keychains/System.keychain mitmproxy-ca-cert.pem

Installation instructions are available onlinefor all other platforms. You can then view and, if needed, edit the details in the certificate manager belonging to your choice of operating system.

You can now track the exact HTTP commands in the mitmproxy console, but analyzing these commands is not particularly user friendly. The mitmproxy web interface can help, which is accessed at http://127.0.0.1:8081 by default. In the web GUI, mitmproxy shows the HTTP command exchange history. To install the required certificate on the client system, go to File | Install Certificates.

The Options tab offers various customization options for the web interface that mainly relate to the display, but you can also work with block lists and add an empty response for specific requests. Mitmproxy supports the following modes in addition to the Regular default mode:

• Transparent
• WireGuard
• Reverse Proxy
• Upstream Proxy
• SOCKS Proxy
• DNS Server

The choice of mode depends on whether you want to monitor the traffic from a client or a single server. If you want to examine the traffic reaching your own web server from the Internet, you need to enable the Reverse Proxy variant. In this case, mitmproxy acts like a ``normal’’ server that fields the requests from the Internet or from an analysis client and passes them on to the target system.

To make it easier to find the goodies, switch to the Start tab and enter a search term, which is a good way of restricting the view to specific sources, services, or file types. The good thing about this is that if the web GUI is open and you access the server in a second browser window, you can trace the commands in mitmproxy in real time.

The individual requests and responses can be examined both at the console level and in the web interface, but the web interface is far more convenient. You can play back the logged exchanges or select Edit to edit requests and forward them to the target server after doing so. It is easy to apply filters at the console level, intercept the traffic, and save the markup locally.

In view of the huge volumes of data that mitmproxy delivers, it is not always easy to keep track of the content in which you are interested, which is where the Highlight function comes in handy. Clicking on the Highlight button opens a selection menu where you can highlight specific content. You can limit the selection to methods or content types or even use a regular expression.

One of mitmproxy’s special features is the Interception function, which lets you intercept and manipulate requests. Intercepting is not normally desirable because it impairs the browsing experience. To keep you from taking down the entire data exchange, mitmproxy uses a different approach and intercepts requests selectively. You can set this up in the web GUI or at the console with the ~u <regex> flow filter, and you can use the ~q option to avoid intercepting the responses. An ampersand (&) lets you combine multiple filter options.

Once you have interrupted a request, the next step is to modify it to suit your needs. If you are working with the web GUI, first enable Edit mode and select the entry with the request that you want to edit (e.g., you could manipulate the Path or user-agent option).

If you want to make the changes at the console level, press E to enable Edit mode; in the web GUI, open the Request tab and navigate to the entry in which you are interested. Make your changes and exit Edit mode. To resume the interrupted data flow, press the A button at the console or click Resume in the GUI.

Another special feature of mitmproxy is that you can replay previous data flows repeatedly. The tool supports two types of replay requests:

• Client-side replay retransmits previous client requests to the server.
• Server-side replay replays the server’s responses to previously recorded requests.

You can use both variants at the console and in the web GUI. To begin, select the desired requests with filter expressions. To repeat these, press the r key at the console or click Replay in the GUI. You can modify the request here, too.

Thanks to its modular architecture, you can extend mitmproxy’s functionality and draw on the development work of an active community. Table 1 summarizes the most interesting extensions. For example, the log-events.py script runs at console level and generates warnings.

When analyzing traffic, it might make sense to ignore certain domains – for two main reasons. First, some of the traffic might be protected by certificate pinning, and second, some of the traffic might simply be beyond your scope of interest.

In the first case, for example, Windows updates or access to Apple’s App Store will not work if you have set up mitmproxy in the middle. If some of the content is not relevant to you, simply hide it with the view_filter option.

To exclude hosts from monitoring, use the ignore_hosts option and specify this with a regular expression in the host:port string format. The command line alias is --ignore-hosts regex, and the complete command is:

mitmproxy -ignore-hosts '^example\.de:443$'

If you want to restrict recording to specific domains, use the --allow-hosts option.

Another great feature of mitmproxy is that the environment has a modular architecture, and you extend the feature set with the help of Python scripts. A major share of the existing feature set is already based on integrated add-ons. The structure and design of these scripts are kept so simple that it is easy to adapt them or develop your own scripts. Almost 30 scripts are currently available for download from the GitHub project site.

Mitmproxy is an exciting tool for recording, analyzing, and manipulating HTTP traffic. Thanks to its specific focus, the HTTP proxy is great for checking web requests and responses, which makes it a valuable tool, not only for administrators, but also for developers and penetration testers.