wget does not use network proxy in some cases

The problem is the configuration in wget. wget is used by flash, MS fonts, broadcom firmwares, atheros firmwares and all this things that are not in the repos and most be donwloaded from other places.

So, maybe the solutionn is to config the proxy for wget, or, better to change temporaly the proxy config form wget when doing some of this actions.

Hi Julián,

If the problem is that synaptic needs to set the proxy environment variables for wget, then I'm not sure why you've reassigned the bug to wget, as there's nothing to do in the wget package. If, instead, you're recommending that wget respect Gnomish proxy settings, then that would be a legitimate thing to target at wget (though it merits discussion, as that makes an enhancement request against wget, and not the bug against Synaptic).

I'd request to consider this bug "serious" and some steps need to be taken urgently. Since most of the new users are not aware of the wget configuration (I did not know about the wgetrc file before going through some duplicates of this! But I have environment variable http_proxy set in .bashrc and wget works fine in terminal), there must be a way in synaptic to pass on the gnome/synaptic proxy settings to wget.

Hello!

I can confirm this bug.

A way to bypass this bug is to set the proxy in /etc/wgetrc manually.

Of course, synaptic should give the proxy as parameter to wget, when calling it.

Friendly regards,
beauman

This bug has a better title, but the longest discussion of this bug (now marked as a duplicate of this one, despite being raised earlier) is surely https://bugs.launchpad.net/ubuntu/+source/flashplugin-nonfree/+bug/45722. That one was raised in early 2006, while this one implies the bug is relatively new (this year, over two years after the original).

And yet, this is still a serious issue - despite modifying my /etc/wgetrc, Synpatic often fails to download when using a proxy - especially when using NTLM authentication. I'm also still not clear why we have this Gnome app, Synaptic, which not only fails to use the Gnome provided proxy applet (preferences/Network Proxy), but also has it's own settings for Proxy, which it ignores.

This is a serious bug which massively hampers the adoption of Ubuntu as a corporate desktop in the vast majority of corporate environments. At least, I don't know of many large organisations that don't try to reduce internet bandwidth (and engage porn filtering - Websense, Bluecoat, Surf Control, etc) by deploying a proxy server, a proxy cluster or a proxy chain.

I don't wish to denigrate the seriousness of this bug; but it may be worth pointing out that large organizations will more frequently deploy "transparent" proxies, rather than configure everyone's browsers to use something (which employees can more easily work around).

Of course, there are still plenty of organizations (especially, I think, smallish-to-medium-sized ones) that will opt for configuration-requiring proxies; and anyway there are plenty of other reasons to need Synaptic to support proxying properly.

However, let's not forget that Ubuntu is an almost entirely volunteer-run organization; there is always tons of work to be done and far too few people to do it. If you're really motivated to see this change, it's much more effective to look into what it would take to implement it, and take the first steps toward doing so yourself.

I hadn't noticed this before, but of course in this case it's not Synaptic downloading a package, but the package itself that downloads a tarball during the "install" phase. In that case, it could well be considered a bug in flashplugin-nonfree.

All this kind of packages, flash, atheros firmware, broadcom firmware, etc. needs to download a file, and they use wget.. But wget don't read proxy parameters of Synaptic. This is why is not flashplugin-nonfree problem.

In all but the largest of organisations, transparent proxies are rare in my (admittedly limited) experience. Transparent proxies are pretty damn expensive because in order to be transparent, they must be a hop on the way to your internet connection - therefore, they see ALL traffic, not just web. Therefore, they must be exceedingly powerful, at least compared to explicit proxies. Plus, if a non-cluster transparent proxy fails, you lose everything until the hardware is fixed, or bypassed. With explicit proxies, you can usually just push a group policy change (in an AD environment anyway) to tell the browsers to go direct (and open your firewall accordingly). Anyway, implementation aside, this is still an issue for anyone using Ubuntu in an environment where port 80 is closed and explicit proxies are your only route to the internet.

And yep - volunteer led. I get it. That's why I'm filing against this report. However, I don't know the first thing about how to recode Synaptic to make use of Gnome-led initiatives such as preferences/Network Proxy when using wget. In fact, I don't program at all - I'm a network and security analyst and Ubuntu enthusiast. If any of my talents become applicable to resolving a bug in Ubuntu, I'll do my best to help.

For now, I'm just happy that I've managed to introduce Ubuntu to my corporatation (albeit in a small way) and if this bug is nailed, it's got more chance of taking off.

I don't know if this is directly related, but I'm currently having the following problem:

After removing firehol, tinyproxy and dansguardian (with --purge), apt-get and wget became unable to download.

$sudo apt-get update
...
W: Failed to fetch http://de.archive.ubuntu.com/ubuntu/dists/intrepid-security/universe/i18n/Translation-en_US.bz2 Could not connect to 127.0.0.1:3128 (127.0.0.1). - connect (111 Connection refused)
...

$wget google.com
--2008-11-03 18:48:55-- http://google.com/
Connecting to 127.0.0.1:3128... failed: Connection refused.

127.0.0.1:3128 is of course where the traffic went through previously when tinyproxy was in use.
Browsers work correctly.
"wget --no-proxy google.com" and "ping de.archive.ubuntu.com" work too.

I tried reinserting my wifi card, changing the network settings in synaptic, flushing iptables, but nothing worked.

I discovered that there was still a /var/lock/firehol file and removed it. apt-get still doesn't work.

Problem solved for me by changing the proxy settings in gnome-network-preferences.
I don't remember changing anything there, but it was set to 127.0.0.1:3128.

Not only synaptic is affected, but apt also in general. In fact, these packages (msttcorefonts, flashplugin-nonfree and others) use wget to download their own special files from the internet. I can consider that these packages are not prepared for handling bad wget/apt communication --- flashplugin-nonfree waits for available connection forever, however msttcorefonts gives up trying after a specific timeout.

My idea for solution is:

Fix ALL .deb packages (postinst part) to set the http_proxy environment variable from apt's setting (if it is defined).

If the proxy setting is configured well in apt (which also means synaptic is configured properly) then all problematic packages will be installed automagically.

Currently I'm unable to check the newest versions of the postinst files, because I'm still using Feisty. Here I found the following postinst for msttcorefonts:

---8X snip here ---
#!/bin/sh
set -e

. /usr/share/debconf/confmodule
#DEBHELPER#

db_get msttcorefonts/dldir
LOCALCOPY=$RET

db_get msttcorefonts/savedir

SAVEDIR=${RET:+-s$RET}

db_get msttcorefonts/dlurl
URLOVERRIDE=${RET:+-u$RET}

db_get msttcorefonts/http_proxy
HTTP_PROXY=${RET:+-p$RET}

if [ -e /usr/share/fonts/truetype/ms-fonts ] ; then
  cp -p /usr/share/fonts/truetype/ms-fonts /var/lib/msttcorefonts/ms-fonts
  rm -f /usr/share/fonts/truetype/ms-fonts
fi

if [ -d /var/state/msttcorefonts ] ; then
 cp -p /var/state/msttcorefonts/* /var/lib/msttcorefonts/ 2>/dev/null || true
 rm -rf /var/state/msttcorefonts/

 # It's not FHS, and it's probably our fault this is here,
 # so delete it if we can.
 rmdir /var/state/ 2> /dev/null || true
fi

# Since we moved the fonts from /usr/share/fonts/truetype to
#/usr/share/fonts/truetype/msttcorefonts, run update-fonts-dir in
#/usr/share/fonts/truetype

update-fonts-dir truetype

update-ms-fonts $URLOVERRIDE $SAVEDIR $LOCALCOPY
---8X snip here ---

It seems that HTTP_PROXY should be substituted by http_proxy because wget uses the lowercased variable. I am not sure if "export http_proxy" is also needed after it is set.

In addition, flashplugin-nonfree also has a buggy postinst file (just a part of it is copy-pasted):

---8X snip here ---
        else # no local file

                db_get flashplugin-nonfree/httpget
                if [ "$RET" != "true" ]; then
                        fp_exit_with_error "download or license refused"
                fi

                # setting wget options
                :> wgetrc
                echo "noclobber = off" >> wgetrc
                echo "dir_prefix = ." >> wgetrc
                echo "dirstruct = off" >> wgetrc
                echo "verbose = on" >> wgetrc
                echo "progress = dot:default" >> wgetrc

                # downloading the plugin
                echo "Downloading..."
                rm -f install_flash_player_9_linux.tar.gz
                if [ -f /home/bart/src/flashplugin-nonfree/bartm_debug ]; then
                        WGETRC=wgetrc wget http://127.0.0.1/bart/install_flash_player_9_linux.tar.gz \
                                || fp_exit_with_error "download failed"
                else
                        WGETRC=wgetrc wget http://fpdownload.macromedia.com/get/flashplayer/current/install_flash_player_9_linux.tar.gz \
                                || fp_exit_with_error "download failed"
                fi
                rm -f wgetrc
                echo "Download done."

        fi # end if local file
---8X snip here ---

Obviously http_proxy is not set here.

 Kovács Zoltán wrote on 2008-11-29: (permalink)
>If the proxy setting is configured well in apt (which also means synaptic is configured properly) then
> all problematic packages will be installed automagically.

Not with my system. I was upgradingu Ubuntu 8.04.1 to 8.04.2 today.

Proxy is configured in /etc/apt/apt.conf.d/10proxy

  Acquire::http::Proxy "http://squid:8000";

apt-get uses this proxy configuratoin successfully but when flashplugin-nonfree is trying to download a file from macromedia.com using wget, it fails with timeouts.

I checked "apt-get install flashplugin-nonfree" using "ps": When installing the package flashplugin-nonfree, a file /var/cache/flashplugin-nonfree/wgetrc is created that does *not* contain proxy configuration from apt. So when wget is called, the environment variable WGETRC is set to the wgetrc file and the file can not be retrieved from macromedia.com. Even worse, root's .wgetrc is overruled.

As discussed, this is not a bug in synaptic, which is two steps removed from the wget invocation. One can argue whether it's the responsibility of wget or flashplugin-nonfree to honor the already-configured proxy settings, but either way, I don't see that this is a synaptic error.

Pardon my ignorance of the process here - I see that you've moved the package to "wget" from "synaptic", which seems to be fair enough, but can I ask why you've also declined this bug for Jaunty, which is still around 3 months from completion?

Where would be the best place to raise the fact that putting a proxy into Synaptic will not work with any package that invokes wget?

Finally, why does Synaptic have a proxy setting at all when it could be using the system wide proxy setting from gnome? Should/could a bug be raised about this? Between system settings, wgetrc, synaptic and firefox, the base Ubuntu build now has four different places to set a proxy.

I would also like the ubuntu developers to reconsider fixing this for Jaunty.

Suggestions:
1. System > Preferences > Network proxy should be able to properly use /etc/wgetrc and/or .wgetrc
2. There should be a check box of some sort to set up the proxy for apt, wget, gnome and the internet browser in Network proxy tool.
3. flashplugin-nonfree should be able to at least check for a proxy using "apt-config dump" or otherwise. flashplugin is a widely used application and as such should have its own internal solutions.

Re-opening for flashplugin-nonfree package

Opened bug against gnome-control-center (gnome-network-properties) based on suggestion #2
https://bugs.launchpad.net/ubuntu/+source/gnome-control-center/+bug/232469/comments/18

Neil: declining a release nomination just means the release team doesn't want to track the bug - it doesn't mean it can't be fixed for that release, especially if somebody submits a patch.

This could be useful for suggestion #2, gnome-network-properties while applying system proxies could use /etc/profile.d/

Creating the following files in /etc/profile.d, and then this will work in *any* shell for *any* user of the system

#proxy.sh
export http_proxy=http://host.com:port/
export ftp_proxy=http://host.com:port/
export no_proxy=.domain.com
export HTTP_PROXY=http://host.com:port/
export FTP_PROXY=http://host.com:port/

#proxy.csh
setenv http_proxy http://host.com:port/
setenv ftp_proxy http://host.com:port/
setenv no_proxy .domain.com
setenv HTTP_PROXY http://host.com:port/
setenv FTP_PROXY http://host.com:port/

Source: www.fedoraforum.org/forum/printthread.php?t=742

Looks like gnome-network-properties' "Apply System-Wide" button adds the proxy variables to /etc/environment

the issue doesn't seem to be a gnome-control-center, it's correctly setting the environment, wget should be using that

But I think that variables in /etc/environment take effect only after
you logout and log back in, not instantly isn't that right?
Whereas variables in /etc/profile.d/ are taken instantly with a
"reset" command or by opening up a new terminal window.

Scratch that last comment, I think I'm wrong :)

Well, it still doesn't work in karmic, despite all the discussion of whose problem it is. Flashplugin-nonfree still does this:

Setting up flashplugin-installer (10.0.42.34ubuntu0.9.10.1) ...
Downloading...
--2009-12-13 12:39:09-- http://archive.canonical.com/pool/partner/a/adobe-flashplugin/adobe-flashplugin_10.0.42.34.orig.tar.gz
Resolving archive.canonical.com... 91.189.90.142
Connecting to archive.canonical.com|91.189.90.142|:80... failed: Connection timed out.
Retrying.

And /etc/apt/apt.conf contains this:

Acquire::http::Proxy "http://192.168.2.1:8113/";

Look, suppose the problem is "really" wget's problem. OK? Then why can't flashplugin-nonfree solve this mess by simply passing a "-p" flag to wget? Then, when (and if) the wget people get off their collective a***es (Hah, British!) it will continue to work, and the '-p' flag could eventually be removed.

Not my problem... it's their problem... sheesh!

There is a well-known idea in designing network protocols: anything you produce should adhere strictly to the standard, but you should accept small unambiguous deviations from the standard. The same should apply to packages. They should do the right thing, but not necessarily depend on other packages being perfect.

This bug may also affect b43-fwcutter and msttcorefonts.
See bug 464422, Bug #56880, Bug #492892, and many others

Hi,
I've got this bug too, I'm running Lucid beta now, but strangely, I wasn't affected with Karmic and previous versions of Ubuntu.

I'm also having this problem in Lucid, but it's no longer in just some cases (as I was experiencing with Karmic) but now with all URLs that require proxy access. They all fail with the message: "failed: Connection timed out." I'm wondering if this bug could be related: https://bugs.launchpad.net/ubuntu/+source/wget/+bug/554068

I think I've found the problem. The culprit is the "no_proxy" environment variable. Once I unset this wget starts honoring http_proxy setting. Perhaps the "no_proxy" format as set by gnome-settings automatically has syntax error.

Actually even more specific, the "no_proxy" variable should be set without trailing comma and then it works

That's it! Thanks Erlan, that completely fixed the problem. I was beginning to despair! This should definitely be fixed for lucid final release, as it's incredibly disruptive for people behind proxies.

Just to clarify, it fixed the problem with wget not working. The problem with apt-get not working is due to not setting proxy details in synaptic settings. I have no idea why synaptic/apt-get needs it's own proxy settings rather than getting them from gnome proxy settings.

I have all proxy variables set and apt-get goes through the proxy successfully.

However, whenever wget is called by apt-get (AFAIK) to download flash-plugin or msttcorefonts, the proxy is *not* used so that downloads timeout, break, etc. There seems to be no way around this.

I am running Lucid behind a non-transparent proxy. I was experiencing the same issue of wget not working in postinst scripts while installing packages, but I was able to get it to work by setting http_proxy in /etc/wgetrc. I would love to fix this bug, but I am confused as to which is the culprit: postinst scripts, wget, apt-get, gnome-terminal, gnome-control-center, or something else.

I guess there are these three lines to uncomment in /etc/wgetrc

------ Original ------
#http_proxy = http://proxy.yoyodyne.com:18023/
#ftp_proxy = http://proxy.yoyodyne.com:18023/

#use_proxy = on
-------------------------

----- Changed-------
http_proxy = http://proxy.yoyodyne.com:18023/
ftp_proxy = http://proxy.yoyodyne.com:18023/

use_proxy = on
-------------------------

Hopefully this should work ?

And I have noticed that to allow apt-get/aptitude to also use the proxy, the proxy settings in Synaptic also need to be changed.
Wonder why, especially because the GNOME system-wide proxy settings have already been set but wget nor Synaptic follow those !

P.S. I currently do not have access to a system behind a proxy so I cannot test it :|

By what is described this bug has nothing to do with Flash but rather proxy+wget.
If it was flash it wouldn't install from anywhere
Closing flash task

Me and a coworker have an ubuntu 10.04 installed and both are, obviously, behind a proxy. his ubuntu version is i386 mine is amd64. My version doesn't work but his version works! I guess bug exist only on 64bit system.

There seems to be two bugs discussed here.
One is related to "no_proxy" environment variable having an extraneous comma at the end.
This bug exists in both 64 and 32 bit versions confirmed (I use 64 bit while two of my colleagues - 32 bit).

The other one has to do with "http_proxy" not being passed on by "sudo". This leads to apt commands failing to connect.
This concerns adding apt repository and repository public key through command line and some other command line package operations. This indeed affects 64 bit edition only.

On a separate note, I installed Fedora 13 and after configuring proxy in gnome proxy settings and a restart both GUI and command line tools worked like a charm. Come on, Ubuntu, you don't have to be more difficult than Fedora in ease of use :)
No offense intended.

WRT to the two bugs, I see BOTH on 32 bit

I hit the no_proxy one after upgrading to 10.04. It was working fine in 9.04 & 9.10 and stopped working when I upgraded to 10.04. I made no change to the env settings but the install script may have.

After deleting the trailing comma, normal wget worked. However sudo wget still did not and I am on a 32 bit machine.

OK: wget url
FAIL: sudo wget url
OK: sudo http_proxy="$http_proxy" wget url

wget 1.12-1.1ubuntu2

Ohh and I am in a company of > 25K people and we do NOT have transparent proxies, unlike a previous poster suggested.

I get at least one question a week from other developers about how to get the proxy setup right. This will just make it harder. I hope this gets fixed before the bulk of them switch to 10.04.

More info & a workaround for sudo

Operation of wget has not changed. wget will fail in 9.04 just the same as 10.04 if trailing comma is added to no_proxy. Wrong no_proxy value appears to be added by gnome-terminal (at least) as it starts. If you run xterm direct from a launcher no special handling of the gnome proxy settings is done. (Which in this case is a good thing.) Best workaround for this is to sed the no_proxy in .bashrc as was suggested elsewhere.

sudo defaults have changed between 9.04 and 10.04. In 9.04 http_proxy (but not no_proxy) was included in the default env_keep list. (You can see built in defaults with "sudo sudo -V" ). Since no_proxy was not included it was not really correct in 9.04 either (but I never noticed). I verified that a "sudo wget localurl" fails in 9.04. (Its not that that paragraph did not have enough not's in it. NOT!)

The fix/workaround for either version is to explicitly configure the env vars you want to pass through sudo.

sudo visudo

Find the existing line that starts with the word "Defaults", below that add this line:
Defaults env_keep+="http_proxy no_proxy"

Warning: only use visudo and don't save the file if it says you have an error, it will lock you out of sudo and you will need to use the recovery console. (Yes I know from experience, but not this time.)

This will surely work, but its for me not the solution to the problem. The problem is not wget, but the sudo wget. If a user does a proxy setting from the menu, and says 'system wide', it should work system wide. For corporate administrators, this might be the reason not to switch to 10.04. So, the default behaviour of sudo should be changed, in my opinion.

Attached is a patch against gnome-terminal to not add the trailing comma to no_proxy.
"It worked for me" tm
Anyone want to mentor me on how to get this recorded & published in bazar?
I know git pretty well but complete noob on bzr & launchpad & ppa's etc

@Bill:
thanks for working on this patch. I think one of the first step will to report the issue upstream and post your patch.
You should open a bug against gnome-terminal package in ubuntu ("Also affects distribution" -> gnome-terminal in ubuntu).
You can then link your bug in Launchpad to this one in clicking on "also affects projects", ensure you have gnome-terminal selected, and copy the url from upstream bug report).
Thanks a lot!

On 06/21/2010 02:52 PM, Didier Roche wrote:
> @Bill:
> thanks for working on this patch. I think one of the first step will to report the issue upstream and post your patch.
>

Actually it is already fixed upstream. I did not think their fix would
be applicable as the code had been re factored upstream. (proxy
functions moved from terminal-screen.c to terminal-util.c). However it
looks like the the fix was done before the refactor.

http://git.gnome.org/browse/gnome-terminal/commit/?id=c3ed02adadafa1a07393f64a6d4065de15149601

http://bazaar.launchpad.net/~vcs-imports/gnome-terminal/trunk/revision/3229
     [same thing both places]

Their fix is prettier than mine. You should be able to cherry-pick this
(or whatever the bzr equivalent is).

I am motivated to get this fix in, is there something else I can do?

-- Bill

@Bill:
Thanks for your comment,

can you try to cherry-pick this upstream patch (cook it in its minimal form), and try to make it applies to the current version in lucid? Then, next step is to post the patch here and a valid test case to ensure it's fixed.

@Didier
Why don't we instead have Ubuntu pull the up-to-date terminal code from upstream and call it fixed?
The patch appeared back in Feb in Gnome Terminal 2.30 which is within Lucid's update frame.
Shouldn't we use Bill's skills and motivation on the other part of the bug, the one involving "sudo"?
Sorry if that sounds too noob, just striving for optimal use of scarce volunteer resources.

Le mercredi 23 juin 2010 à 17:53 +0000, Erlan Sergaziev a écrit :
> @Didier
> Why don't we instead have Ubuntu pull the up-to-date terminal code from upstream and call it fixed?
> The patch appeared back in Feb in Gnome Terminal 2.30 which is within Lucid's update frame.
> Shouldn't we use Bill's skills and motivation on the other part of the bug, the one involving "sudo"?
> Sorry if that sounds too noob, just striving for optimal use of scarce volunteer resources.
>

@Erlan:
No worry I can give you the rational:
an updated gnome-terminal needs a newer version of libvte, which isn't
an option to update as more than 39 packages are depending on it. By
experience, it's more than possible that it will break another since.

This is why we need to be picky on updates (even with stable updates) as
it can contain a lot of regression code.

On 06/23/2010 12:58 PM, Didier Roche wrote:
> can you try to cherry-pick this upstream patch (cook it in its minimal
> form), and try to make it applies to the current version in lucid?

I will. It will be a couple of day till I have the next window to do
this (as I am learning bzr etc as I go.)

> Then,
> next step is to post the patch here and a valid test case to ensure it's
> fixed.
>

To regression test this we need to poke a few test values into gconf,
run the gnome-terminal (possibly with --disable-factory to ensure we are
running the copy we think??) and then in the terminal context run a
script to check the env values. I can play with --command or --execute
to get the script to run in the terminal context and I can lookup how to
poke config values into gconf. However I am unfamiliar with the test
harness that this will go into. Can you point me to and existing test I
can emulate?

Bill

Don't worry about writing an automatic testcase (it would be awesome, but not needed), just write a manual testcase that people will try to reproduce like:
- step1
- step2
- install the new package
- step 4
…

This bug is very annoying, I re-installed Ubuntu (after using another distro for months) and spent almost an hour trying to figure out why the proxy stop working (I kept the same home directory).

TESTCASE:
1. reproduce problem
1.1. run "gnome-network-properties" and set:
1.1.1. (o) manual config
1.1.2. http proxy: "example.com", port: "8000"
1.2. run "gnome-terminal", and execute:
1.2.1. env | grep '^no_proxy=.*,$' && echo "BUG REPRODUCED" || echo "BUG SOLVED (or proxy not set)"

For people who want to build their own fixed package:

sudo apt-get install build-essential # get compiler etc
sudo apt-get build-dep gnome-terminal # get packages needed to build this package
mkdir /tmp/build # make some temp dir for build
cd /tmp/build # go there
apt-get source gnome-terminal # get debian source package (make sure you have source repositories enables)
cd gnome-terminal-*/src # go to package-source directory
wget http://launchpadlibrarian.net/40476531/terminal-screen.c.patch # get patch
patch -p0 < terminal-screen.c.patch # apply the patch
cd .. # go to main package directory
dpkg-buildpackage # build package

now you can for example:
sudo mv /usr/bin/gnome-terminal /usr/bin/gnome-terminal.orig
sudo mv /tmp/build/gnome-terminal-*/src/gnome-terminal /usr/bin/gnome-terminal

This has been solved via bug #534225. Solution is in lucid-proposed.
To get the fix:
1. https://wiki.ubuntu.com/Testing/EnableProposed
2. upgrade gnome-terminal package