Accessing IMAP/SMTP via HTTPS Proxy

In some places Internet access is available only via proxy, which in practice means that you are limited to HTTP and HTTPS protocols only.  But if you have  external email accounts like Gmail,  this is bit limiting, because you cannot access your email via IMAP protocol from your  email client.

However there can be a solution –  if proxy support HTTPS protocol, it means also that it supports CONNECT method that  tunnels a connection to remote server unchanged.  This method could be used to tunnel any protocol, so basically  it could tunnel also IMAP ( and SMTP for outgoing email).

The only piece that is needed is either support from client (but Thunderbird unfortunately does not support this) or a tunnel program running on local machine + slight change in email account – to use local connection to tunnel proxy instead of direct connection to server.   But there is still one small glitch – if your computer is notebook and you come home, there is no proxy, and you want to connect to mail server directly again. To handle this scenario local tunnelling  proxy should be smart and fall- back to direct connection to server, if HTTPS proxy is not available.  I have not found any program like this, so I’ve written mine own – called ptunnel. Recently I returned to this problem and created new ptunnel in Rust – it’s much faster and less resource intensive, so I recommend this one.

Now how to do this in practice:

  1. Get ptunnel from links above (old one requires python, new one must be compiled locally first).
  2. Run it with appropriate parameters – for Gmail it should  be something like this:
    Rust program:

    ptunnel -p proxy_host:proxy_port 9993:imap.gmail.com:993 5587:smtp.gmail.com:587

    Python program:

    ptunnel.py -d -p proxy_host:proxy_port 9993:imap.gmail.com:993 5587:smtp.gmail.com:587

    You should assure that ptunnel is  automatically started, when computer starts – the ways how you can do this are various – I prefer to start it with my desktop (Startup Applications in Gnome, Unity, Cinnamon), you can use /etc/rc.local or some other ways – I believe it may work fine even under other platforms like Win or Mac, even I never tried it.

  3. You have to reconfigure your email account – IMAP server should be localhost:9993 and smtp server localhost:5587.
  4. After reconfiguration, when you’ll try to receive or send  email, you will get security warning – that’s because SSL certificates are not matching.  You now connecting to localhost, but certificates are issued to mail server DNS name – you have to add security exceptions for these certificates.

This tunnelling works for me  fine for quite  some time ( a year or so).  Tunnelling is not limited to IMAP /SMTP only so it can be used for SSH etc.

30 thoughts on “Accessing IMAP/SMTP via HTTPS Proxy”

  1. Wow! this looks cool: Unfortunately I can’t make it work with my versions of Python. I’m having Python 2.4.3, and it complains socket doesn’t have a ‘create_connection’ attribute. When trying with Python 3.2 (alter running 2to3), I get an error from sock.sendall(), TypeError: ‘str’ does not support the buffer interface. Do you have any idea what to do to make it work with Python 3.2?

    -Øystein

    1. That would relate to fact that in python 3 str type is basically unicode – so we need to encode to bytes before sending
      try change
      sock.sendall(Tunnel.CONNECT % tuple(self.server.tunnel[1:]))
      to
      sock.sendall(Tunnel.CONNECT % tuple(self.server.tunnel[1:]).encode(‘ascii’))

      Ivan

  2. Thank you!
    I’ve been fiddling about for ages, with all manner of other options.

    Thank you for a good clean and basic – just works – building block.

    Well done!!!

  3. How could I add authentication to the proxy? something that works with

    ptunnel.py -d -p user:passw@proxy_host:proxy_port 9993:imap.gmail.com:993 5587:smtp.gmail.com:587

    Thanks

    1. proxy authentication is not yet supported. I think you can add it easily – look at method connect_remote_via_proxy (lines 111-119), you can add there support for Proxy-Authorise header.

  4. Hi

    does this marvelous tool accept more than 2 parametres (IMAP/SMTP) so that I can tunnel a whole lot of ports in one instance (IMAP/SMTP/SSH/MORE) ?

    greetings
    uthop intra

    1. Yes you can specify many tunnels – each with parameter local_port:remote_host:remote_port. For each tunnel a thread is created that listens on local port and forwards via proxy to remote host:port.
      I.

  5. I am getting following error on MAC, I am using python 2.7 , could you please help.

    I tried encoding change but dose not help.

    hq-office-7:src root# ptunnel -d -p proxy_host:proxy_port 9993:imap.gmail.com:993 5587:smtp.gmail.com:587
    ERROR:root:Invalid arguments
    Traceback (most recent call last):
    File “/usr/bin/ptunnel”, line 174, in main
    opts, args = _parse_options(oparser, args)
    File “/usr/bin/ptunnel”, line 61, in _parse_options
    raise ArgumentError(“Port in proxy definition must be numeric”)
    ArgumentError: Port in proxy definition must be numeric
    Usage: %s [options] port:host:port [port:host:port …]

  6. You made a great work !
    Your python solution works perfectly with current Thunderbird version also (at least for windows)
    first : I use the network proxy discovery in advance thunderbird settings, to be able to use the Thunderbird mail account assistant to work. I define their my personal imap/smtp account. Then I refine with manuel settings to set SSL option and user authentication. at this step imap/smtp don’t work, since not relayed by my corporate proxy to my ISP.

    Second: I installed latest Pyhon2 version and your “ptunnel.py” script file. I rename ptunnel.py into ptunnel.pyw to avoid getting an unnecessary windows.
    I have created a “ptunnel.bat” like this :
    start “C:\Program Files (x86)\Python2\pythonw.exe” “mypath\ptunnel.pyw” -d -p myproxy:myport 9993:imap.myISPserver:993 4465:smtp.myISPserver:465
    and put a shortcut to ptunnel.bat into my “startup” menu.

    Third: into Thunderbird email account settings, I replace the smtp host/port by localhost, port 9993 for imap, and localhost, port 4465 for smtp.

    It works very well!!
    you saved my life !

    1. Hi,
      thanks for link, the main difference is that ptunnel falls back to direct connection if proxy is not available, so it works for me on mobile notebook.
      Otherwise proxytunnel is cool and can be used for tunnelling as for instance for ssh protocol.
      I.

  7. Hi, I get this error:

    The specified Secure Sockets Layer (SSL) port is not allowed. ISA Server is not configured to allow SSL requests from this port. Most Web browsers use port 443 for SSL requests.

  8. I need help with this step (3)
    3. You have to reconfigure your email account – IMAP server should be localhost:9993 and smtp server localhost:5587.

    I am a complete noob – so bear with me .
    Do I need some IMAP and STMP servers running locally? Which server should I install and how do I configure it?
    Because when I tried to install postfix on my Linux (Debian), it asked me system mail name, what should it be?
    thnaks.

    1. No you do not need any servers running locally – this program is tunnelling traffic – so if you start it with:
      ptunnel.py -d -p your-https-proxy.com:80 9993:imap.gmail.com:993 5587:smtp.gmail.com:587
      It listens locally on port 9993 for IMAP and 5587 for SMTP protocols, which are then forwarded though proxy to their final destination – gmail servers.
      So now you need to reconfigure your client to communicate with this tunnel – e.g. IMAP is localhost:9993, SMTP is localhost:5587

  9. Hi,
    I was looking for the same solution using perl. finally after understanding http proxy and how it works, got solution.

    1. Squid Proxy to add 993 port as SSL and restart proxy
    2. Make http request to proxy using authentication if any.
    3. Make same connection as SSL.
    4. You will be able to login imap server.
    5. Tested with gmail and aol etc.

    If any one need help to implement same solution and having trouble, feel to send mail to pankajmoon@gmail.com
    Thank you,
    Pankaj

  10. Great work indeed. Python script tested with Python 2.7 for windows and Thunderbird mail client working behind a squid proxy server. And it worked.
    Thank you very much.

  11. Hi, I’m trying to configure access to GMAIL. I use CNTLM (executed in localhost at port 3128) which configures the parent corporate PROXY (with authentication.) This is the ptunnel call for GMAIL:

    nohup ./ptunnel -p 127.0.0.1:3128 9993:gmail-imap.l.google.com:993 5587:gmail-smtp-msa.l.google.com.:587&

    In /etc/hosts:

    127.0.0.1 imap.gmail.com
    127.0.0.1 smtp.gmail.com

    The email client I use is gnome.evolution.

    Configuration:
    Server: imap.gmail.com
    User: @gmail.com
    Security: TLS on dedicated port
    Authentication: OAuth2

    NOHUP:

    [2019-12-10T10:59:44Z ERROR ptunnel::proxy] Error in tunnel Tunnel { local_port: 9993, remote_port: 993, remote_host: “gmail-imap.l.google.com” }: Invalid status – 403

    The same problem occurs with another TIM MAIL mail account. What am I doing wrong?

    1. I forgot port:

      Configuration:
      Server: imap.gmail.com
      Port: 993
      User: @gmail.com
      Security: TLS on dedicated port
      Authentication: OAuth2

      1. Hard to say, what’s wrong, I do not have experience with CNTLM,
        Error in tunnel Tunnel { local_port: 9993, remote_port: 993, remote_host: “gmail-imap.l.google.com” }: Invalid status – 403
        says that HTTP connection to proxy returned 403 – Forbidden HTTP status, which suggests that authentication with proxy fails.
        You can also try to run debug version with env. variable RUST_LOG=ptunnel=debug to see debug messages, but it will probably not tell much.
        Try Wireshark or similar to see what is actually going through TCP connections?

        1. From Wireshark:

          CONNECT imap.gmail.com:9993 HTTP/1.1
          Proxy-Connection: keep-alive
          Proxy-Authorization: NTLM *******************************
          Content-Length: 0

          HTTP/1.1 403 Tunnel or SSL Forbidden
          Date: Fri, 13 Dec 2019 13:01:15 GMT
          Connection: close
          Via: 1.1 fp-proxy03.*********.com
          Cache-Control: no-store
          Content-Type: text/html
          Content-Language: en
          Content-Length: 666

          Tunnel or SSL Forbidden
          Tunnel or SSL Forbidden

          Description: 9993 is not an allowed port for Tunnel or SSL connections

          <!– defaul

Leave a Reply

Your email address will not be published. Required fields are marked *