HTTPS over proxy fails

Bug #94130 reported by Henrique Rocha
340
This bug affects 50 people
Affects Status Importance Assigned to Milestone
Awn-Py-Applets
Incomplete
Undecided
Unassigned
Python
Fix Released
Undecided
Unassigned
Ubuntu One Client
Invalid
Medium
Unassigned
apport (Ubuntu)
Fix Released
Medium
Martin Pitt
python2.6 (Ubuntu)
Fix Released
Medium
Matthias Klose

Bug Description

Binary package hint: apport

My network is behind a proxy.
I ran "ubuntu-bug -p desktop-effects" and I received a "no route to host" error.
The app should try to use the gnome proxy or at least fail notifying the user that he/she might be behind a proxy.

Tags: usability
Revision history for this message
Henrique Rocha (hmrocha-fc) wrote :

apport should try to use http_proxy environment variable or use the gnome environment proxy.
For reading the gnome enviroment proxy we could read /system/http_proxy

If use_http_proxy is checked, use host:port fields for the proxy.

Changed in apport:
importance: Undecided → Medium
Revision history for this message
Henrique Rocha (hmrocha-fc) wrote :

I've investigated the bug a little further.
Since the url is opened with https protocol it tries automatically to use the https_proxy environment variable which is not set on my system, only http_proxy.
I've set the https_proxy variable and tried to run "ubuntu-bug -p apport" again and I've received a different error, "HTTP Error 501: Not Implemented".

Revision history for this message
Ali Sabil (asabil) wrote :

That is because the https_proxy env variable is not set (which is also a bug in gnome-terminal) and apport makes use of urllib, which implements dumb non standard support for HTTP proxies (basicaly it will issue GET https://.... HTTP/1.0 instead of doing a CONNECT followed by a GET)

The solution would be to drop the usage of python's urllib.

Revision history for this message
Martin Pitt (pitti) wrote :

Ali, would urllib2 work?

Revision history for this message
Ali Sabil (asabil) wrote :

no, urllib2 http proxy + https connection is completely broken. actually in my previous report, I was refering to urllib2 and not urllib.

Revision history for this message
Adrian Penisoara (adrian.penisoara) wrote :

Because of this bug I am unable to report any crashes in Gutsy. It would be very important to have this functionality enabled.

Daniel Hahler (blueyed)
Changed in apport:
status: New → Confirmed
Revision history for this message
Martin Pitt (pitti) wrote :

This is actually a bug in Python's urllib2 module, moving package.

Revision history for this message
Martin Pitt (pitti) wrote :

In apport we should just unset https_proxy since it will not ever work with the current Python.

Changed in apport:
assignee: nobody → pitti
status: New → Triaged
Revision history for this message
Brian J. Murrell (brian-interlinx) wrote :

> In apport we should just unset https_proxy since it will not ever work with the current Python.

How would this solve the problem of apport trying to GET/POST https:// URLs through a proxy? The proxy will still be required. apport needs to use CONNECT instead of GET/POST.

Revision history for this message
marcobra (Marco Braida) (marcobra) wrote :

Same issue here...

Thank you

Revision history for this message
Martin Pitt (pitti) wrote :

I don't have a viable solution for apport ATM which works with HTTPS proxies and thus avoids using Python's urllib2. Suggestions appreciated.

Changed in apport:
assignee: pitti → nobody
status: Triaged → Confirmed
Revision history for this message
Brian J. Murrell (brian-interlinx) wrote : Re: [Bug 94130] Re: apport fails to use environment proxy configuration

On Wed, 2008-03-19 at 13:52 +0000, Martin Pitt wrote:
> I don't have a viable solution for apport ATM which works with HTTPS
> proxies and thus avoids using Python's urllib2. Suggestions appreciated.

A "urllib2 https proxy" (without the quotes obviously) google search
seems to return a wealth of information about the problem including a
number of contributions people have made to handling it. One such
contribution is:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/456195

A lot of discussion seems to point back to this one.

Here's a whole thread
(http://www.gossamer-threads.com/lists/python/bugs/628088) about it,
including another solution which was posted in the last couple of days
in fact (http://bugs.python.org/file9753/http-tunnel-urllib)

Here's another solution: http://www.freshports.org/www/py-openssl-proxy/

I would think out of those three solutions or further search of the
above mentioned google search terms one should be able to solve this
quite readily, no?

b.

Revision history for this message
Ram Kumar (ramkumail) wrote : Re: apport fails to use environment proxy configuration

Is there a chance that it will be fixed in future updates? It seems it has not been fixed for a long time and continues as well in to ubuntu hardy which is very bad. At least i experienced it in the hardy beta.

Revision history for this message
xoco (arthur.ivanov) wrote :

agree

urllib2 broke https support through proxy

Revision history for this message
Loïc Minier (lool) wrote :
Matthias Klose (doko)
Changed in python2.5:
status: Confirmed → Triaged
Changed in python:
status: Unknown → New
Revision history for this message
Georges (georgeskesseler) wrote :

still not fixed, intrepid 8.10 with all patches.
I wonder if such an important feature is broken because of bad urllib, apport should have been changed since a long time to an alternative. If you want ubuntu to be taken serious in the enterprise environment, this is a must!

Well now I have a crash in a dpkg dependency conflict which I would like to report. How do i do this manually?

Revision history for this message
xoco (arthur.ivanov) wrote :

Totally agree with Georges.

Only housewifes uses direct connection to internet(w/wo NAT). 99.999% Enterprise computers behind firewall/proxy.

Revision history for this message
Loïc Minier (lool) wrote :

To report manually without proxy use:
https_proxy="" apport-cli

Revision history for this message
Brian J. Murrell (brian-interlinx) wrote : Re: [Bug 94130] Re: HTTPS over proxy fails

On Tue, 2008-12-16 at 13:50 +0000, Loïc Minier wrote:
> To report manually without proxy use:
> https_proxy="" apport-cli

And how does that work when the machine I'm running on has no routed
access to the Internet? i.e. I *have* to use the proxy per corporate
policy and network enforcement rules?

Revision history for this message
Loïc Minier (lool) wrote :

It doesn't work when you need the proxy (hence the "without proxy" in my comment); it works for me because I only use a caching proxy.

Another possible workaround would be to use a generic interceptor such as proxychains (there are others) which will redirect all connections from a process to a proxy.

I don't think urllib2 will be fixed in the near future, but we could look at other means to connect to https for apport (setting up the ssl socket manually I guess).

Revision history for this message
Brian J. Murrell (brian-interlinx) wrote :

On Tue, 2008-12-16 at 14:38 +0000, Loïc Minier wrote:
>
> I don't think urllib2 will be fixed in the near future, but we could
> look at other means to connect to https for apport (setting up the ssl
> socket manually I guess).

Well, if you were going to do that, why not just do that work in urllib2
instead of apport and then have the resulting work useful to many other
applications? Whether you did the HTTPS support in apport directly or
in urllib2 should be six of one, half dozen of the other so why not aim
for the greater good?

Revision history for this message
Loïc Minier (lool) wrote :

urllib2 as an API needs to allow the programmer to decide how he wants his SSL stack to behave and needs to raise SSL specific exception for instance; for instance what if the website uses an unknown certificate.

Revision history for this message
Georges (georgeskesseler) wrote :

maybe I could provide some piece of "do not reinvent the wheel".
The same problem existed in perl LWP 8 years ago.
Implementation was simple but yielded another supplemental problem:
as soon as the proxy needs a user/password authentication there is some more code needed.
So dear python folks, look into LWP and port that to urllib2

On the other hand I wonder why apport doesn't just use wget or curl, which both work fine. Fear of code injection? Well there are cookbooks out there how to pass parameters to scripts without fear of executing the parameters.
Where are the good old times of unix with the philosophy "one tool for one task"?

Revision history for this message
Martin Soto (soto255) wrote :

Of course this is not a fix, but I just managed to report a bug using apport-cli and proxychains, so I thought I'd report it here as a workaround. I configured proxychains to go through the proxy and, after that, it was just a matter of running

  proxychains apport-cli --crash-file=/var/crash/_usr_bin_evolution.1000.crash

In order to test if proxychains is being able to reach launchpad at all, you can do something like

  proxychains wget https://bugs.launchpad.net/ubuntu/+source/apport/+bug/94130

This should leave a file called "94130" in your current working directory with the contents of this bug page. If this works, the apport command above is likely to work as well.

Revision history for this message
Fernando Miguel (fernandomiguel) wrote :

shouldn't this also affect python 2.6 ?

Loïc Minier (lool)
affects: python2.5 (Ubuntu) → python2.6 (Ubuntu)
Revision history for this message
Brian Murray (brian-murray) wrote :

The related upstream bug report, http://bugs.python.org/issue1424152, appears to have some patches in it.

Rick McBride (rmcbride)
Changed in ubuntuone-client:
status: New → Confirmed
Revision history for this message
orsenthil (orsenthil) wrote :

The upstream bug in Python is fixed in Python 2.7. I am checking with other python-dev, should the changes be backported to py2.6.

tags: added: usability
Revision history for this message
orsenthil (orsenthil) wrote :

This bug is fixed in python maintain stream. The changes have been backported too.

Changed in python:
status: New → Confirmed
Martin Pitt (pitti)
Changed in apport (Ubuntu):
status: Confirmed → Triaged
Revision history for this message
Robbie Williamson (robbiew) wrote :

Matthias is on holiday this week, but assigning to him for review once he returns

Changed in python2.6 (Ubuntu):
assignee: nobody → Matthias Klose (doko)
Revision history for this message
Matthias Klose (doko) wrote :

fixed in python2.6 in karmic

Changed in python2.6 (Ubuntu):
status: Triaged → Fix Released
Revision history for this message
Lorant Nemeth (loci) wrote :

After successful upgrade to 9.10 the problem is still present (using gnome's manual proxy config/use the same proxy for all...)
# python -V
Python 2.6.4rc2

Revision history for this message
orsenthil (orsenthil) wrote :

The bug is fixed in upstream python library, but if you are using an application that is not changed to use the new facility of library, you may want to fix that (or ask the developer to fix that).
http://bugs.python.org/issue1424152#msg89611

2. If you are pointing out at the python process, I would suggest you to setup HTTP_PROXY and HTTPS_PROXY Environment variable and use the set_tunnel method to your request object.

Revision history for this message
Soumitra Pal (soumitrakp) wrote :

The patch given at http://bugs.python.org/issue7291 solved this problem on my Ubuntu 9.10 PC.

Revision history for this message
jaybe1966 (jaybe1966) wrote :

ubuntu-bug works when setting the https_proxy environment variable. It seems that it doesn't use the system proxy settings.

Changed in ubuntuone-client:
importance: Undecided → Medium
Changed in ubuntuone-client:
assignee: nobody → Guillermo Gonzalez (verterok)
Changed in ubuntuone-client:
assignee: Guillermo Gonzalez (verterok) → nobody
Revision history for this message
Johan Walles (walles) wrote :

Still broken in Lucid.

Workaround (inspired by comment 24):
* I have both $https_proxy and $HTTPS_PROXY set to proper values.
* ls /var/crash
_usr_lib_update-manager_check-new-release.0.crash yaml-mode.0.crash
* apport-bug --crash-file=/var/crash/_usr_lib_update-manager_check-new-release.0.crash
* ...
* rm /var/crash/_usr_lib_update-manager_check-new-release.0.crash

Optionally you can use apport-cli rather than apport-bug, apport-bug is just the GUI version and apport-cli is the text version.

If apport-bug / apport-cli could use the system proxy settings as configured through GNOME's Network Proxy settings that would of course be a lot better.

Revision history for this message
Chris (cbaines) wrote :

Still broken in mavarick, apport reports:
" Cannot connect to crash database, please check your Internet connection.
<urlopen error [Errno 111] Connection refused> "
When trying to upload an error report.

Revision history for this message
hawran (hawran.diskuse) wrote :

I'm experiencing this problem as well.
I've just tried to report some bug within audacious and got that "Cannot connect to crash database, please check your Internet connection".
Which is ridiculous as I've been using the same internet connection (www, strems, ...) all the time.

I'm in my office, behind a proxy.
I've set up that proxy's settings via "System -> Preferences -> Network Proxy" and all works for me.

For more info, please see the report I'd got by apport-cli -f -p apport-gtk (I hope this is a dorrect package for the ubuntu-bug binary) ...

Revision history for this message
hawran (hawran.diskuse) wrote :
Revision history for this message
papukaija (papukaija) wrote :

Closing the Ubuntuone part as it is tracked at bug 387308

Changed in ubuntuone-client:
status: Confirmed → Invalid
Martin Pitt (pitti)
Changed in apport (Ubuntu):
status: Triaged → Invalid
Revision history for this message
Johan Walles (walles) wrote :

Martin, why was this marked as invalid in apport?

According to Chris in comment 36 this is still broken in Maverick.

Revision history for this message
Martin Pitt (pitti) wrote :

Johan Walles [2011-03-08 15:51 -0000]:
> Martin, why was this marked as invalid in apport?

AFAIUI this was fixed in Python itself?

--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)

Revision history for this message
Johan Walles (walles) wrote :

User orsenthil indicates in comment 32 that apps need to be fixed to use the new Python functionality.

So AFAIU Apport still needs fixing. I know it's broken in Maverick at least.

  Regards /Johan

Martin Pitt (pitti)
Changed in apport (Ubuntu):
status: Invalid → Triaged
importance: Undecided → Medium
assignee: nobody → Martin Pitt (pitti)
Martin Pitt (pitti)
Changed in apport (Ubuntu):
status: Triaged → In Progress
Revision history for this message
Martin Pitt (pitti) wrote :

I just tested this by installing a local squid, then starting a VM and configuring its firewall to only allow connections to my host's squid and nowhere else. If I set a wrong $https_proxy, apport fails, as expected. If I set it correctly (http://10.0.2.2:3128 with a default squid install and default kvm network settings), problem report uploading works fine.

So this is is at least fixed in current natty, presumably also in maverick.

Changed in apport (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Robert Chalmas (robert-chalmas) wrote :

We still have this problem behind our enterprise's proxy with Natty beta 2.

The proxy was set in GNOME with System --> Preferences --> Network proxy and applied system wide.

In a terminal, here is what I can see :
$ env | grep proxy
http_proxy=http://proxyxxx.ch:80/
ftp_proxy=http://proxyxxx.ch:80/
https_proxy=http://proxyxxx.ch:80/
no_proxy=localhost,127.0.0.0/8,*.local

When trying to report a crash using apport, I get after several minutes an error message like "Unable to connect to the crashes database. Please check your Internet connection. <urlopen error [Errno 110] Connection aborted by timeout>".
(note: this text was translated from French and does probable not match exactly the one in an English system).

Revision history for this message
Gianfranco Liporace (dr.kabuto) wrote :

@Robert Chalmas:
if you put this line in /etc/sudoers
Defaults env_keep="http_proxy https_proxy"

it change anything?

Revision history for this message
Robert Chalmas (robert-chalmas) wrote :

Many thanks to Gianfranco for his suggestion.
This seems to work perfectly.

Revision history for this message
Johan Walles (walles) wrote :

Does that mean that apport needs to add that line somewhere under /etc/sudoers.d?

Revision history for this message
Gianfranco Liporace (dr.kabuto) wrote :

@Johan:
This is a workaround that shouldn't be needed since, in teory, sudo should keep preserve these environment variables. I think this is another bug that it was opened several times in the past, i.e. #284867, #181999 and #194238.

Revision history for this message
Brian J. Murrell (brian-interlinx) wrote :

On 11-04-19 03:24 AM, Gianfranco Liporace wrote:
> @Johan:
> This is a workaround that shouldn't be needed since, in teory, sudo should keep preserve these environment variables. I think this is another bug that it was opened several times in the past, i.e. #284867, #181999 and #194238.

@Gianfranco: except that theory is not meeting with reality, obviously,
since Robert is reporting this problem in Natty.

Revision history for this message
Gianfranco Liporace (dr.kabuto) wrote :

@Brian:
I was only pointing the fact that, in reality, Robert's bug is not related to this one but to another already opened (and supposedly fixed) months ago.

Revision history for this message
Barry Warsaw (barry) wrote :

http://bugs.python.org/issue1424152

Upstream bug was fixed, but the LP bug watch doesn't notice it.

Changed in python:
status: Confirmed → Unknown
importance: Unknown → Undecided
status: Unknown → New
status: New → Fix Released
Revision history for this message
Michael M. Garland, III (mmgarland3) wrote :

I'm also behind a proxy, and I'm finding AWN applets (python driven) also fail because of this bug. This is reported as Bug #185206, and I think updating Oneiric's python to include this fix would squash many bugs with one shoe, so to speak.

Revision history for this message
papukaija (papukaija) wrote :

There shoudn't be be anything in AWN to fix. Please confirm.

Changed in awn-py-applets:
status: New → Incomplete
Revision history for this message
Michael M. Garland, III (mmgarland3) wrote :

Specifically, the Weather applet gives error: "Network error in Weather. Could not retrieve weather data. You may be experiencing connectivity issues." It gives this error when I search for a location and when it polls for new weather data.

I have env variables set for my proxy server, as well as having it set in Network Settings in Gnome. No python scripts work that require direct http connections; for instance, add-apt-repository fails, as well. I use a script that calls gpg directly with the proxy as an argument to get ppa keys.

Revision history for this message
Chadwick Caraway (chewy917) wrote :
Download full text (4.9 KiB)

<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/">
<title>support issues</title>
<link href="/tpo/web/support/-/issues.atom?state=opened" rel="self" type="application/atom+xml"/>
<link href="https://gitlab.torproject.org/tpo/web/support/-/issues" rel="alternate" type="text/html"/>
<id>https://gitlab.torproject.org/tpo/web/support/-/issues</id>
<updated>2021-07-07T09:01:18Z</updated>
<entry>
  <id>https://gitlab.torproject.org/tpo/web/support/-/issues/237</id>
  <link href="https://gitlab.torproject.org/tpo/web/support/-/issues/237"/>
  <title>Add snowflake-client.exe to antivirus allowlist</title>
  <updated>2021-07-07T09:01:18Z</updated>
  <media:thumbnail width="40" height="40" url="https://gitlab.torproject.org/uploads/-/system/user/avatar/259/avatar.png"/>
  <author>
    <name>Gus</name>
    <email></email>
  </author>
  <summary>Add snowflake-client.exe to antivirus allowlist</summary>
  <description>Some users are reporting that their [Windows Defender](https://www.reddit.com/r/TOR/comments/of1gu3/windows_defender_latest_tor_browser_bundle/) is removing/messing up with their Tor Browser 10.5 because of `snowflake-client.exe`.

We should update our recommendations here:
https://support.torproject.org/tbb/tbb-10/</description>
  <content>Some users are reporting that their [Windows Defender](https://www.reddit.com/r/TOR/comments/of1gu3/windows_defender_latest_tor_browser_bundle/) is removing/messing up with their Tor Browser 10.5 because of `snowflake-client.exe`.

We should update our recommendations here:
https://support.torproject.org/tbb/tbb-10/</content>
  <milestone>Tor Browser: 10.5</milestone>
  <labels>
    <label>Documentation</label>
  </labels>
</entry>
<entry>
  <id>https://gitlab.torproject.org/tpo/web/support/-/issues/232</id>
  <link href="https://gitlab.torproject.org/tpo/web/support/-/issues/232"/>
  <title>Add &amp;quot;You should send padding so it&amp;#39;s more secure.&amp;quot;</title>
  <updated>2021-06-17T21:49:20Z</updated>
  <media:thumbnail width="40" height="40" url="https://gitlab.torproject.org/uploads/-/system/user/avatar/259/avatar.png"/>
  <author>
    <name>Gus</name>
    <email></email>
  </author>
  <summary>Add "You should send padding so it's more secure."</summary>
  <description>We have some new proposals and discussion, we should update this answer:

 Like all anonymous communication networks that are fast enough for web browsing, Tor is vulnerable to statistical "traffic confirmation" attacks, where the adversary watches traffic at both ends of a circuit and confirms their guess that those endpoints are communicating. It would be really nice if we could use cover traffic to confuse this attack. But there are three problems here:

    Cover traffic is really expensive. And *every* user needs to be doing it. This adds up to a lot of extra bandwidth cost for our volunteer operators, and they're already pushed to the limit.
    You'd need to always be sending traffic, meaning you'd need to always be online. Otherwise, you'd need to be sending end-to-end cover traffic -- not just to the first hop, but all the way to your final ...

Read more...

Revision history for this message
Chadwick Caraway (chewy917) wrote :
Download full text (4.9 KiB)

<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/">
<title>support issues</title>
<link href="/tpo/web/support/-/issues.atom?state=opened" rel="self" type="application/atom+xml"/>
<link href="https://gitlab.torproject.org/tpo/web/support/-/issues" rel="alternate" type="text/html"/>
<id>https://gitlab.torproject.org/tpo/web/support/-/issues</id>
<updated>2021-07-07T09:01:18Z</updated>
<entry>
  <id>https://gitlab.torproject.org/tpo/web/support/-/issues/237</id>
  <link href="https://gitlab.torproject.org/tpo/web/support/-/issues/237"/>
  <title>Add snowflake-client.exe to antivirus allowlist</title>
  <updated>2021-07-07T09:01:18Z</updated>
  <media:thumbnail width="40" height="40" url="https://gitlab.torproject.org/uploads/-/system/user/avatar/259/avatar.png"/>
  <author>
    <name>Gus</name>
    <email></email>
  </author>
  <summary>Add snowflake-client.exe to antivirus allowlist</summary>
  <description>Some users are reporting that their [Windows Defender](https://www.reddit.com/r/TOR/comments/of1gu3/windows_defender_latest_tor_browser_bundle/) is removing/messing up with their Tor Browser 10.5 because of `snowflake-client.exe`.

We should update our recommendations here:
https://support.torproject.org/tbb/tbb-10/</description>
  <content>Some users are reporting that their [Windows Defender](https://www.reddit.com/r/TOR/comments/of1gu3/windows_defender_latest_tor_browser_bundle/) is removing/messing up with their Tor Browser 10.5 because of `snowflake-client.exe`.

We should update our recommendations here:
https://support.torproject.org/tbb/tbb-10/</content>
  <milestone>Tor Browser: 10.5</milestone>
  <labels>
    <label>Documentation</label>
  </labels>
</entry>
<entry>
  <id>https://gitlab.torproject.org/tpo/web/support/-/issues/232</id>
  <link href="https://gitlab.torproject.org/tpo/web/support/-/issues/232"/>
  <title>Add &amp;quot;You should send padding so it&amp;#39;s more secure.&amp;quot;</title>
  <updated>2021-06-17T21:49:20Z</updated>
  <media:thumbnail width="40" height="40" url="https://gitlab.torproject.org/uploads/-/system/user/avatar/259/avatar.png"/>
  <author>
    <name>Gus</name>
    <email></email>
  </author>
  <summary>Add "You should send padding so it's more secure."</summary>
  <description>We have some new proposals and discussion, we should update this answer:

 Like all anonymous communication networks that are fast enough for web browsing, Tor is vulnerable to statistical "traffic confirmation" attacks, where the adversary watches traffic at both ends of a circuit and confirms their guess that those endpoints are communicating. It would be really nice if we could use cover traffic to confuse this attack. But there are three problems here:

    Cover traffic is really expensive. And *every* user needs to be doing it. This adds up to a lot of extra bandwidth cost for our volunteer operators, and they're already pushed to the limit.
    You'd need to always be sending traffic, meaning you'd need to always be online. Otherwise, you'd need to be sending end-to-end cover traffic -- not just to the first hop, but all the way to your final ...

Read more...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.