[natty] broken IPv6 default route

Bug #704812 reported by Bernhard Schmidt
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
NetworkManager
Fix Released
High
network-manager (Ubuntu)
Fix Released
Medium
Unassigned

Bug Description

Binary package hint: network-manager

when setting IPv6 settings to "automatic" instead of the default ignore (see my eduroam profile) network-manager adds a static default route after the connection is established. Unfortunately this default route has no next-hop and thus breaks connectivity quite badly.

root@schleppi:~# ip -6 route
2001::/32 dev teredo proto kernel metric 256 mtu 1280 advmss 1220 hoplimit 4294967295
2001:4ca0:0:f200::/64 dev wlan0 proto kernel metric 256 expires 0sec mtu 1500 advmss 1440 hoplimit 4294967295
fe80::/64 dev wlan0 proto kernel metric 256 mtu 1500 advmss 1440 hoplimit 4294967295
fe80::/64 dev is0 proto kernel metric 256 mtu 1280 advmss 1220 hoplimit 4294967295
fe80::/64 dev teredo proto kernel metric 256 mtu 1280 advmss 1220 hoplimit 4294967295
default via fe80::2d0:3ff:fe3b:f400 dev wlan0 proto kernel metric 1024 mtu 1500 advmss 1440 hoplimit 64
default dev wlan0 proto static metric 1024 mtu 1500 advmss 1440 hoplimit 4294967295 <--- this is the broken one

One has to repeatedly delete this static route (I guess nm tries to add it again) before you have connectivity again.

root@schleppi:~# ip -6 route del default dev wlan0 proto static
root@schleppi:~# ip -6 route del default dev wlan0 proto static
root@schleppi:~# ip -6 route del default dev wlan0 proto static
root@schleppi:~# ip -6 route del default dev wlan0 proto static
RTNETLINK answers: No such process

IPv6 clients learn their default route from RA or from static configuration, but you cannot just point a default-route to a non-ptp interface without a next-hop. The kernel will look for every destination (including ipv6.google.com) on the local network segment and fail after a 2 second timeout.

ProblemType: Bug
DistroRelease: Ubuntu 11.04
Package: network-manager 0.8.3+git.20101219t181118.e919218-0ubuntu1 [modified: usr/bin/nm-online usr/bin/nm-tool usr/bin/nmcli usr/lib/NetworkManager/nm-avahi-autoipd.action usr/lib/NetworkManager/nm-crash-logger usr/lib/NetworkManager/nm-dhcp-client.action usr/lib/NetworkManager/nm-dispatcher.action usr/sbin/NetworkManager]
ProcVersionSignature: Ubuntu 2.6.37-12.26-generic 2.6.37
Uname: Linux 2.6.37-12-generic x86_64
Architecture: amd64
CRDA:
 country DE:
  (2400 - 2483 @ 40), (N/A, 20)
  (5150 - 5250 @ 40), (N/A, 20), NO-OUTDOOR
  (5250 - 5350 @ 40), (N/A, 20), NO-OUTDOOR, DFS
  (5470 - 5725 @ 40), (N/A, 26), DFS
Date: Wed Jan 19 10:15:57 2011
IfupdownConfig:
 auto lo
 iface lo inet loopback
Keyfiles: Error: [Errno 2] Datei oder Verzeichnis nicht gefunden
ProcEnviron:
 LANGUAGE=de_DE.UTF-8:de:en_GB:en
 PATH=(custom, user)
 LANG=de_DE.UTF-8
 LC_MESSAGES=de_AT.utf8
 SHELL=/usr/bin/zsh
SourcePackage: network-manager
WpaSupplicantLog:

Revision history for this message
Bernhard Schmidt (berni) wrote :
Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

Thank you for taking the time to report this bug and helping to make Ubuntu better. The issue you are reporting is an upstream one and it would be nice if somebody having it could send the bug to the developers of the software by following the instructions at https://wiki.ubuntu.com/Bugs/Upstream/GNOME. If you have done so, please tell us the number of the upstream bug (or the link), so we can add a bugwatch that will inform us about its status. Thanks in advance.

Changed in network-manager (Ubuntu):
status: New → Incomplete
Revision history for this message
Bernhard Schmidt (berni) wrote :

The bug has been reported upstream as https://bugzilla.gnome.org/show_bug.cgi?id=641333

Changed in network-manager:
importance: Unknown → High
status: Unknown → New
Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

I've since done my IPv6 setup at home and I can't reproduce this, with a DHCPv6 server or using autoconfiguration. I notice there are some differences though, you're also setting up a teredo tunnel, so could it be related? I'm not hugely familiar with the setup, but it looks like you would be trying to announce an IPv6 range to the network using teredo, while also having NM maintain an IPv6 address for the interface?

I may well be reading this wrong though, but I'd check if it's possible to turn debugging way up for miredo (if that's indeed what you're using) to make sure it's not what's adding the default route via dev wlan0. Sadly there is too little information in the attached syslog to tell whether it's NM -- so you also might want to stop NM (sudo stop network-manager), then start it again as such:

sudo /usr/sbin/NetworkManager --no-daemon --log-domains=IP6,DHCP6,CORE,DEVICE --log-level=DEBUG

and attach the resulting log (after making sure it doesn't contain info you feel is sensitive).

Revision history for this message
Bernhard Schmidt (berni) wrote :

I can reproduce it (sorry, no log yet, will do that tomorrow). It happens when the first connection is not successful (here at the university that sometimes happens due to a slow DHCPv4 server or broken EAP-PEAP radius) and I connect a second time.

Revision history for this message
Bernhard Schmidt (berni) wrote :

BTW, it also happens when all other network configuration daemons (isatapd, miredo, DHCPv6 client, OpenVPN) are stopped.

Revision history for this message
Bernhard Schmidt (berni) wrote :

Sorry for the delay, here is the promised debug log. I can still easily reproduce this problem.

1) Set IPv6 to auto instead of ignore
2) connect to a (wireless?) network where IPv6 RA works fine (the client gets an autoconfed address and route), but DHCPv4 fails to answer in time (shutdown the server)
3) after DHCPv4 timeout nm disconnects
4) reconnect to the same network using nm-applet
5) this time make sure DHCPv4 works (start the server again)
6) on successful connect you have two IPv6 default routes, one pointing to the LL next-hop from RA (correct) and one pointing to the interface without nexthop (wrong)

It does not happen when
a) the first connection attempt already succeeded
b) IPv6 is set to ignore in the connection profile

Since nm is still not flushing the IPv6 address when the interface goes down I assume it is recording the list of routes present before step 4 (which includes the correct nexthop to the router) and attempts to readd them after step 5, but fails to set the next hop correctly.

Interesting side effect, I have to delete the wrong route three times before it is finally gone. It looks like nm is readding it immediately, but I don't see anything in the debug log. I could not find the debug category where it shows adding routes at all.

Revision history for this message
Bernhard Schmidt (berni) wrote :
Changed in network-manager:
status: New → Fix Released
Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

The commit we want for this fix would be http://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?h=NM_0_8&id=36ef4a2212c19c810d774978cf6d725f92b52e4f

Bernhard, are you able to test that patch on your own to confirm it fixes the issue, or do you need me to build a package?

Changed in network-manager (Ubuntu):
status: Incomplete → Triaged
importance: Undecided → Medium
Revision history for this message
Bernhard Schmidt (berni) wrote :

Looks good to me. I'm not at university right now where I've seen this issue frequently, but I've tried the patch at home (disabling my IPv4 DHCP server at the first try) and the route is set correctly.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package network-manager - 0.8.4~git.20110228t143901.5cdded6-0ubuntu1

---------------
network-manager (0.8.4~git.20110228t143901.5cdded6-0ubuntu1) natty; urgency=low

  * upstream snapshot 2011-02-28 14:39:01 (GMT) (LP: #726820)
    + 5cdded6b66bdc371acb29e76187b174c55c11470
    - docs: explicitly list libnm-glib objects
    - libnm-util: fix parameter name for docs generator
    - api: add missing Settings.System D-Bus interface bits
    - ifcfg-rh: get rid of LAST_CONNECT
    - core/libnm-glib: don't touch files in /etc when updating timestamps
    - release: bump version to 0.8.3.996 (0.8.4-beta2)
    - api: document more VPN connection failure codes (bgo #643324)
    - libnm-util: silently ignore non-secrets when updating connection secrets
    - libnm-util: fix updating secrets (rh #680385)
    - keyfile: fix tests for no-longer-required serial and PPP settings
    - ip6: fail activation if addrconf fails to start
    - core: don't require serial and PPP settings for mobile broadband
    - ppp: fix warning when PPP support is disabled
    - libnm-util: bump version for several recent changes
    - release: bump version to 0.8.3.995 (0.8.4-beta1)
    - release: update NEWS with changes since 0.8.2
    - libnm-util: fix updating secrets
    - core: flush IPv6 routes even if configuration failed (bgo #641333)
      (LP: #704812)
    - libnm-util: remove GObject Introspection annotations
    - gentoo: simplify OpenRC detection
    - libnm-util: add nm_connection_diff()
    - dns: fix up updating resolv.conf (rh #672282)
    - Revert "libnm-glib: return enums for NMAccessPoint flags accessors"
    - build: allow disabling PPP support at build time
    - core: fix dbus-glib crash on shutdown with systemd
    - libnm-glib: return enums for NMAccessPoint flags accessors
    - libnm-util: fix description of bluetooth setting type property
    - test: don't crash libnm-glib if there are no devices for an active
      connection
    - test: don't crash libnm-glib-test if there aren't any APs
  * debian/patches/revert_52f2295.patch: dropped, fixed upstream.
  * debian/libnm-util1.symbols: adding new symbols introduced upstream:
    - nm_connection_diff
    - nm_setting_diff
  * debian/patches/fix_sorting_connections_w_tstamp_3194cb9.patch: fix sorting
    connections following upstream commit 7d707bd, which is necessary for auto
    activations.
  * debian/patches/dnsmasq_conffile_conn_sharing.patch: don't pass a bogus conf
    file to dnsmasq, just pass no file parameter (LP: #725041)
  * debian/rules: enable build-time tests. (LP: #727395)
  * debian/patches/70_lp145653_no_sigaction_for_crashes.patch: add .pc file
    generated by quilt to POTFILES.skip, otherwise the build will fail due to
    the tests we now run detecting a localizable file missing from templates.
 -- Mathieu Trudel-Lapierre <email address hidden> Fri, 04 Mar 2011 10:53:59 -0500

Changed in network-manager (Ubuntu):
status: Triaged → Fix Released
Revision history for this message
DaveAtFraud (dave-davenjudy) wrote :

I'm still seeing this as of the time of this post. Has the fix not made it out or is this a regression? I'm seeing a routing table under natty that looks like:

route -A inet6 -n
Kernel IPv6 routing table
Destination Next Hop Flag Met Ref Use If
2001:5c0:110c:c000::/64 :: U 256 0 0 eth0
fe80::/64 :: U 256 0 0 eth0
::/0 :: U 1024 0 0 eth0
::/0 :: !n -1 1 12 lo
::1/128 :: Un 0 1 5 lo
2001:5c0:110c:c000::4a/128 :: Un 0 1 21 lo
fe80::20c:29ff:feea:26b3/128 :: Un 0 1 3 lo
ff00::/8 :: U 256 0 0 eth0
::/0 :: !n -1 1 12 lo

Note the two entries immediately below the fe80:/64 entry. The first bogus default route can be deleted but the second can't be. Deleting the first bogus default route and then adding a valid default route fixes the problem.

Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

Dave, please use 'ubuntu-bug network-manager' to file a separate, new bug report so that we can look into the issue. Thanks!

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.