In some places Internet access is available only via proxy, which in practice means that you are limited to HTTP and HTTPS protocols only. But if you have external email accounts like Gmail, this is bit limiting, because you cannot access your email via IMAP protocol from your email client.
However there can be a solution – if proxy support HTTPS protocol, it means also that it supports CONNECT method that tunnels a connection to remote server unchanged. This method could be used to tunnel any protocol, so basically it could tunnel also IMAP ( and SMTP for outgoing email).
The only piece that is needed is either support from client (but Thunderbird unfortunately does not support this) or a tunnel program running on local machine + slight change in email account – to use local connection to tunnel proxy instead of direct connection to server. But there is still one small glitch – if your computer is notebook and you come home, there is no proxy, and you want to connect to mail server directly again. To handle this scenario local tunnelling proxy should be smart and fall- back to direct connection to server, if HTTPS proxy is not available. I have not found any program like this, so I’ve written mine own – called ptunnel. Recently I returned to this problem and created new ptunnel in Rust – it’s much faster and less resource intensive, so I recommend this one.
Now how to do this in practice:
- Get ptunnel from links above (old one requires python, new one must be compiled locally first).
- Run it with appropriate parameters – for Gmail it should be something like this:
Rust program:ptunnel -p proxy_host:proxy_port 9993:imap.gmail.com:993 5587:smtp.gmail.com:587
Python program:
ptunnel.py -d -p proxy_host:proxy_port 9993:imap.gmail.com:993 5587:smtp.gmail.com:587
You should assure that ptunnel is automatically started, when computer starts – the ways how you can do this are various – I prefer to start it with my desktop (Startup Applications in Gnome, Unity, Cinnamon), you can use /etc/rc.local or some other ways – I believe it may work fine even under other platforms like Win or Mac, even I never tried it.
- You have to reconfigure your email account – IMAP server should be localhost:9993 and smtp server localhost:5587.
- After reconfiguration, when you’ll try to receive or send email, you will get security warning – that’s because SSL certificates are not matching. You now connecting to localhost, but certificates are issued to mail server DNS name – you have to add security exceptions for these certificates.
This tunnelling works for me fine for quite some time ( a year or so). Tunnelling is not limited to IMAP /SMTP only so it can be used for SSH etc.
Wow! this looks cool: Unfortunately I can’t make it work with my versions of Python. I’m having Python 2.4.3, and it complains socket doesn’t have a ‘create_connection’ attribute. When trying with Python 3.2 (alter running 2to3), I get an error from sock.sendall(), TypeError: ‘str’ does not support the buffer interface. Do you have any idea what to do to make it work with Python 3.2?
-Øystein
That would relate to fact that in python 3 str type is basically unicode – so we need to encode to bytes before sending
try change
sock.sendall(Tunnel.CONNECT % tuple(self.server.tunnel[1:]))
to
sock.sendall(Tunnel.CONNECT % tuple(self.server.tunnel[1:]).encode(‘ascii’))
Ivan
Thank you!
I’ve been fiddling about for ages, with all manner of other options.
Thank you for a good clean and basic – just works – building block.
Well done!!!
How could I add authentication to the proxy? something that works with
ptunnel.py -d -p user:passw@proxy_host:proxy_port 9993:imap.gmail.com:993 5587:smtp.gmail.com:587
Thanks
proxy authentication is not yet supported. I think you can add it easily – look at method connect_remote_via_proxy (lines 111-119), you can add there support for Proxy-Authorise header.
Hi
does this marvelous tool accept more than 2 parametres (IMAP/SMTP) so that I can tunnel a whole lot of ports in one instance (IMAP/SMTP/SSH/MORE) ?
greetings
uthop intra
Yes you can specify many tunnels – each with parameter local_port:remote_host:remote_port. For each tunnel a thread is created that listens on local port and forwards via proxy to remote host:port.
I.
Hello,
If thunderbirt does not support it, can you please recommend another good client for windows?
TA+BR, M.
Don’t know about any mail client, that uses HTTPS proxy also for IMAP.
I.
I am getting following error on MAC, I am using python 2.7 , could you please help.
I tried encoding change but dose not help.
hq-office-7:src root# ptunnel -d -p proxy_host:proxy_port 9993:imap.gmail.com:993 5587:smtp.gmail.com:587
ERROR:root:Invalid arguments
Traceback (most recent call last):
File “/usr/bin/ptunnel”, line 174, in main
opts, args = _parse_options(oparser, args)
File “/usr/bin/ptunnel”, line 61, in _parse_options
raise ArgumentError(“Port in proxy definition must be numeric”)
ArgumentError: Port in proxy definition must be numeric
Usage: %s [options] port:host:port [port:host:port …]
it worked for me , just confused for the proxy settings..
You made a great work !
Your python solution works perfectly with current Thunderbird version also (at least for windows)
first : I use the network proxy discovery in advance thunderbird settings, to be able to use the Thunderbird mail account assistant to work. I define their my personal imap/smtp account. Then I refine with manuel settings to set SSL option and user authentication. at this step imap/smtp don’t work, since not relayed by my corporate proxy to my ISP.
Second: I installed latest Pyhon2 version and your “ptunnel.py” script file. I rename ptunnel.py into ptunnel.pyw to avoid getting an unnecessary windows.
I have created a “ptunnel.bat” like this :
start “C:\Program Files (x86)\Python2\pythonw.exe” “mypath\ptunnel.pyw” -d -p myproxy:myport 9993:imap.myISPserver:993 4465:smtp.myISPserver:465
and put a shortcut to ptunnel.bat into my “startup” menu.
Third: into Thunderbird email account settings, I replace the smtp host/port by localhost, port 9993 for imap, and localhost, port 4465 for smtp.
It works very well!!
you saved my life !
Hi,
thanks for link, the main difference is that ptunnel falls back to direct connection if proxy is not available, so it works for me on mobile notebook.
Otherwise proxytunnel is cool and can be used for tunnelling as for instance for ssh protocol.
I.
Hi, I get this error:
The specified Secure Sockets Layer (SSL) port is not allowed. ISA Server is not configured to allow SSL requests from this port. Most Web browsers use port 443 for SSL requests.
Obviously problem is at server to which you try to tunnel. What are you trying to tunnel?
I.
I just try to tunnel Gmail’s IMAP/SMTP, the ISA Server at work blocked the access.
I need help with this step (3)
3. You have to reconfigure your email account – IMAP server should be localhost:9993 and smtp server localhost:5587.
I am a complete noob – so bear with me .
Do I need some IMAP and STMP servers running locally? Which server should I install and how do I configure it?
Because when I tried to install postfix on my Linux (Debian), it asked me system mail name, what should it be?
thnaks.
No you do not need any servers running locally – this program is tunnelling traffic – so if you start it with:
ptunnel.py -d -p your-https-proxy.com:80 9993:imap.gmail.com:993 5587:smtp.gmail.com:587It listens locally on port 9993 for IMAP and 5587 for SMTP protocols, which are then forwarded though proxy to their final destination – gmail servers.
So now you need to reconfigure your client to communicate with this tunnel – e.g. IMAP is localhost:9993, SMTP is localhost:5587
Hi,
I was looking for the same solution using perl. finally after understanding http proxy and how it works, got solution.
1. Squid Proxy to add 993 port as SSL and restart proxy
2. Make http request to proxy using authentication if any.
3. Make same connection as SSL.
4. You will be able to login imap server.
5. Tested with gmail and aol etc.
If any one need help to implement same solution and having trouble, feel to send mail to pankajmoon@gmail.com
Thank you,
Pankaj
For perl there is script connect-tunnel (http://search.cpan.org/dist/Net-Proxy/script/connect-tunnel).
Key difference here is that ptunnel.py has the direct connection as a fall-back – which works well on notebook, where connection changes between restricted ( with proxy) and open (without proxy).
You could use socat:
socat tcp-l:4650,fork proxy:$proxy_host:smtp.gmail.com:465,proxyport=$proxy_port
Yes, but I think it will not fall back to direct connection if proxy is not available.
Great share! Works like a charm really. Thanks!
Great work indeed. Python script tested with Python 2.7 for windows and Thunderbird mail client working behind a squid proxy server. And it worked.
Thank you very much.
Hi, I’m trying to configure access to GMAIL. I use CNTLM (executed in localhost at port 3128) which configures the parent corporate PROXY (with authentication.) This is the ptunnel call for GMAIL:
nohup ./ptunnel -p 127.0.0.1:3128 9993:gmail-imap.l.google.com:993 5587:gmail-smtp-msa.l.google.com.:587&
In /etc/hosts:
127.0.0.1 imap.gmail.com
127.0.0.1 smtp.gmail.com
The email client I use is gnome.evolution.
Configuration:
Server: imap.gmail.com
User: @gmail.com
Security: TLS on dedicated port
Authentication: OAuth2
NOHUP:
[2019-12-10T10:59:44Z ERROR ptunnel::proxy] Error in tunnel Tunnel { local_port: 9993, remote_port: 993, remote_host: “gmail-imap.l.google.com” }: Invalid status – 403
The same problem occurs with another TIM MAIL mail account. What am I doing wrong?
I forgot port:
Configuration:
Server: imap.gmail.com
Port: 993
User: @gmail.com
Security: TLS on dedicated port
Authentication: OAuth2
Hard to say, what’s wrong, I do not have experience with CNTLM,
Error in tunnel Tunnel { local_port: 9993, remote_port: 993, remote_host: “gmail-imap.l.google.com” }: Invalid status – 403
says that HTTP connection to proxy returned 403 – Forbidden HTTP status, which suggests that authentication with proxy fails.
You can also try to run debug version with env. variable RUST_LOG=ptunnel=debug to see debug messages, but it will probably not tell much.
Try Wireshark or similar to see what is actually going through TCP connections?
From Wireshark:
CONNECT imap.gmail.com:9993 HTTP/1.1
Proxy-Connection: keep-alive
Proxy-Authorization: NTLM *******************************
Content-Length: 0
HTTP/1.1 403 Tunnel or SSL Forbidden
Date: Fri, 13 Dec 2019 13:01:15 GMT
Connection: close
Via: 1.1 fp-proxy03.*********.com
Cache-Control: no-store
Content-Type: text/html
Content-Language: en
Content-Length: 666
Tunnel or SSL Forbidden
Tunnel or SSL Forbidden
Description: 9993 is not an allowed port for Tunnel or SSL connections
<!– defaul