Accessing IMAP/SMTP via HTTPS Proxy

In some places Internet access is available only via proxy, which in practice means that you are limited to HTTP and HTTPS protocols only.  But if you have  external email accounts like Gmail,  this is bit limiting, because you cannot access your email via IMAP protocol from your  email client.

However there can be a solution –  if proxy support HTTPS protocol, it means also that it supports CONNECT method that  tunnels a connection to remote server unchanged.  This method could be used to tunnel any protocol, so basically  it could tunnel also IMAP ( and SMTP for outgoing email).

The only piece that is needed is either support from client (but Thunderbird unfortunately does not support this) or a tunnel program running on local machine + slight change in email account – to use local connection to tunnel proxy instead of direct connection to server.   But there is still one small glitch – if your computer is notebook and you come home, there is no proxy, and you want to connect to mail server directly again. To handle this scenario local tunnelling  proxy should be smart and fall- back to direct connection to server, if HTTPS proxy is not available.  I have not found any program like this, so I’ve written mine own – called ptunnel.

Now how to do this in practice:

  1. Get ptunnel from link above (requires python).
  2. Run it with appropriate parameters – for Gmail it should  be something like this:

    You should assure that ptunnel is  automatically started, when computer starts – the ways how you can do this are various – I prefer to start it with my desktop (Startup Applications in Gnome, Unity, Cinnamon), you can use /etc/rc.local or some other ways – I believe it may work fine even under other platforms like Win or Mac, even I never tried it.
  3. You have to reconfigure your email account – IMAP server should be localhost:9993 and smtp server localhost:5587.
  4. After reconfiguration, when you’ll try to receive or send  email, you will get security warning – that’s because SSL certificates are not matching.  You now connecting to localhost, but certificates are issued to mail server DNS name – you have to add security exceptions for these certificates.

This tunnelling works for me  fine for quite  some time ( a year or so).  Tunnelling is not limited to IMAP /SMTP only so it can be used for SSH etc.

25 thoughts on “Accessing IMAP/SMTP via HTTPS Proxy”

  1. Wow! this looks cool: Unfortunately I can’t make it work with my versions of Python. I’m having Python 2.4.3, and it complains socket doesn’t have a ‘create_connection’ attribute. When trying with Python 3.2 (alter running 2to3), I get an error from sock.sendall(), TypeError: ‘str’ does not support the buffer interface. Do you have any idea what to do to make it work with Python 3.2?

    -Øystein

    1. That would relate to fact that in python 3 str type is basically unicode – so we need to encode to bytes before sending
      try change
      sock.sendall(Tunnel.CONNECT % tuple(self.server.tunnel[1:]))
      to
      sock.sendall(Tunnel.CONNECT % tuple(self.server.tunnel[1:]).encode(‘ascii’))

      Ivan

  2. Thank you!
    I’ve been fiddling about for ages, with all manner of other options.

    Thank you for a good clean and basic – just works – building block.

    Well done!!!

  3. How could I add authentication to the proxy? something that works with

    ptunnel.py -d -p user:passw@proxy_host:proxy_port 9993:imap.gmail.com:993 5587:smtp.gmail.com:587

    Thanks

    1. proxy authentication is not yet supported. I think you can add it easily – look at method connect_remote_via_proxy (lines 111-119), you can add there support for Proxy-Authorise header.

  4. Hi

    does this marvelous tool accept more than 2 parametres (IMAP/SMTP) so that I can tunnel a whole lot of ports in one instance (IMAP/SMTP/SSH/MORE) ?

    greetings
    uthop intra

    1. Yes you can specify many tunnels – each with parameter local_port:remote_host:remote_port. For each tunnel a thread is created that listens on local port and forwards via proxy to remote host:port.
      I.

  5. I am getting following error on MAC, I am using python 2.7 , could you please help.

    I tried encoding change but dose not help.

    hq-office-7:src root# ptunnel -d -p proxy_host:proxy_port 9993:imap.gmail.com:993 5587:smtp.gmail.com:587
    ERROR:root:Invalid arguments
    Traceback (most recent call last):
    File “/usr/bin/ptunnel”, line 174, in main
    opts, args = _parse_options(oparser, args)
    File “/usr/bin/ptunnel”, line 61, in _parse_options
    raise ArgumentError(“Port in proxy definition must be numeric”)
    ArgumentError: Port in proxy definition must be numeric
    Usage: %s [options] port:host:port [port:host:port …]

  6. You made a great work !
    Your python solution works perfectly with current Thunderbird version also (at least for windows)
    first : I use the network proxy discovery in advance thunderbird settings, to be able to use the Thunderbird mail account assistant to work. I define their my personal imap/smtp account. Then I refine with manuel settings to set SSL option and user authentication. at this step imap/smtp don’t work, since not relayed by my corporate proxy to my ISP.

    Second: I installed latest Pyhon2 version and your “ptunnel.py” script file. I rename ptunnel.py into ptunnel.pyw to avoid getting an unnecessary windows.
    I have created a “ptunnel.bat” like this :
    start “C:\Program Files (x86)\Python2\pythonw.exe” “mypath\ptunnel.pyw” -d -p myproxy:myport 9993:imap.myISPserver:993 4465:smtp.myISPserver:465
    and put a shortcut to ptunnel.bat into my “startup” menu.

    Third: into Thunderbird email account settings, I replace the smtp host/port by localhost, port 9993 for imap, and localhost, port 4465 for smtp.

    It works very well!!
    you saved my life !

    1. Hi,
      thanks for link, the main difference is that ptunnel falls back to direct connection if proxy is not available, so it works for me on mobile notebook.
      Otherwise proxytunnel is cool and can be used for tunnelling as for instance for ssh protocol.
      I.

  7. Hi, I get this error:

    The specified Secure Sockets Layer (SSL) port is not allowed. ISA Server is not configured to allow SSL requests from this port. Most Web browsers use port 443 for SSL requests.

  8. I need help with this step (3)
    3. You have to reconfigure your email account – IMAP server should be localhost:9993 and smtp server localhost:5587.

    I am a complete noob – so bear with me .
    Do I need some IMAP and STMP servers running locally? Which server should I install and how do I configure it?
    Because when I tried to install postfix on my Linux (Debian), it asked me system mail name, what should it be?
    thnaks.

    1. No you do not need any servers running locally – this program is tunnelling traffic – so if you start it with:
      ptunnel.py -d -p your-https-proxy.com:80 9993:imap.gmail.com:993 5587:smtp.gmail.com:587
      It listens locally on port 9993 for IMAP and 5587 for SMTP protocols, which are then forwarded though proxy to their final destination – gmail servers.
      So now you need to reconfigure your client to communicate with this tunnel – e.g. IMAP is localhost:9993, SMTP is localhost:5587

  9. Hi,
    I was looking for the same solution using perl. finally after understanding http proxy and how it works, got solution.

    1. Squid Proxy to add 993 port as SSL and restart proxy
    2. Make http request to proxy using authentication if any.
    3. Make same connection as SSL.
    4. You will be able to login imap server.
    5. Tested with gmail and aol etc.

    If any one need help to implement same solution and having trouble, feel to send mail to pankajmoon@gmail.com
    Thank you,
    Pankaj

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">