I have two IPv6 tunnels with subnets, one from SixXS and one from Hurricane Electric. (Why? Eh, I'm a tinkerer.) I had them both active and responding to tunnel broker pings on my Cisco E2000 with DD-WRT v24-sp2 rev 14929 firmware, but I couldn't route from both my subnets at the same time.

My SixXS tunnel won't accept packets from my HE subnet, and my HE tunnel won't accept packets from my SixXS subnet, but I can only route by destination...with normal routing.

The answer—besides living happily with one subnet and tunnel—is policy routing. With policy routing I can set up more than one routing table and choose which routing table to use based on the source of the packet, so I should be able to send SixXS-subnet-sourced packets to the SixXS tunnel and HE-subnet-sourced packets to the HE tunnel.

Unfortunately my DD-WRT version doesn't support policy routing for IPv6. So I designated my Linux server as the DMZ host under the DMZ tab under the NAT / QoS tab and set up my tunnels and routing on the Linux box.

Instead of setting up a piece at a time and testing it as I go, I brazenly typed up my whole configuration before trying any of it out on the Linux box. That cost me a few hours of troubleshooting, but the funny part is I mostly had it right the first time. The problem was that policy rule flushing works slightly differently with IPv4 and IPv6, at least on Ubuntu Lucid 10.04.2.

The default rules are as shown:

/sbin/ip -6 rule show
# 0: from all lookup local
# 32766: from all lookup main

The "gotcha" is that I decided to flush the rules before adding my own. I did this because I was duplicating rules when I brought the interface down and up again.

/sbin/ip -6 rule flush
/sbin/ip -6 rule show
# 0: from all lookup local

Flush works a little too well as it takes away the rule that uses the main routing table! Flushing the IPv4 rules puts the default rules back, but not so with IPv6. So when flushing IPv6 rules remember to add the main rule back:

/sbin/ip -6 rule flush
/sbin/ip -6 rule add priority 32766 from all table main

Here is my working policy routing setup that routes to the proper tunnel based on the source address. I have chosen to set up the HE tunnel and subnet as normal and make policy routing decisions for the SixXS tunnel. Other things I did to make this work was to enable IPv6 routing in sysctl and add "200 sixxs" to /etc/iproute2/rt_tables so I could have a routing table named "sixxs", but I could have used a numbered table instead. Ubuntu/Debian /etc/network/interfaces file (partial):

auto he-ipv6
iface he-ipv6 inet6 v4tunnel
    endpoint 216.218.224.42
    address 2001:470:1f0e:b56::2
    netmask 64
    ttl 64

    # Null route HE /48 to prevent sending back to internet
    post-up /sbin/ip -6 route add unreachable 2001:470:b967::/48 || true

    # Null route HE /64 to prevent sending back to internet
    post-up /sbin/ip -6 route add unreachable 2001:470:1f0f:b56::/64 || true

    # Global unicast range route (effective default)
    post-up /sbin/ip route add 2000::/3 dev he-ipv6 src 2001:470:1f0e:b56::2 || true

auto sixxs
iface sixxs inet6 v4tunnel
    endpoint 216.14.98.22
    address 2001:4978:f:178::2
    netmask 64
    ttl 64
    # doesn't work: mtu 1280
    post-up /sbin/ip link set $IFACE mtu 1280 || true

    # Null route SixXS /48 to prevent sending back to internet
    post-up /sbin/ip -6 route add unreachable 2001:4978:192::/48 || true

    # Set up SixXS routing table
    post-up /sbin/ip -6 route flush table sixxs || true
    post-up /sbin/ip -6 route add 2000::/3 dev $IFACE src 2001:4978:f:178::2 table sixxs || true

    # Set up routing table rules, he-ipv6 is default, sixxs is sixxs-sourced
    post-up /sbin/ip -6 rule flush || true

    # To local prefixes, use main routing table
    post-up /sbin/ip -6 rule add priority 100 to 2001:4978:192::/48 table main || true
    post-up /sbin/ip -6 rule add priority 200 to 2001:470:b967::/48 table main || true
    post-up /sbin/ip -6 rule add priority 300 to 2001:470:1f0f:b56::/64 table main || true

    # To nonroutable "global" prefixes, use main routing table
    # 6to4. Uncomment this if I implement local 6to4 conversion
    #post-up /sbin/ip -6 rule add priority 400 to 2002::/16 table main || true
    # Teredo. Uncomment this if I implement local Teredo conversion
    #post-up /sbin/ip -6 rule add priority 500 to 2001::/32 table main || true
    # Documentation: range reserved for dummy addy's in documentation
    post-up /sbin/ip -6 rule add priority 600 to 2001:db8::/32 table main || true

    # From SixXS subnet addr to global unicast (last rule), use sixxs routing table
    post-up /sbin/ip -6 rule add priority 32000 from 2001:4978:192::/48 to 2000::/3 table sixxs || true

    # Need the default main; flushing seems to delete the ip6 main rule
    post-up /sbin/ip -6 rule add priority 32766 from all table main || true

    # Flush routing cache to enable new routing info
    post-up /sbin/ip -6 route flush cache || true

    # Flush rules and re-add main table rule
    post-down /sbin/ip -6 rule flush || true
    post-down /sbin/ip -6 rule add priority 32766 from all table main || true

I null-route my assigned prefixes/subnets to avoid sending inappropriate traffic back into the internet which will just get routed right back to me. When I add routes for my /64 prefixes they will override the null route because they more specifically match the destination addresses. I am also routing to 2000::/3 (all currently assigned global unicast addresses) instead of a default route so I don't send out multicasts or other address ranges I don't intend.

If you're wondering about "|| true", if I don't have that and the command generates an error code, the interface set-up would stop. The "|| true" causes the command to return a "success" code to the ifup/ifdown scripts even if the command fails so that the ifup/ifdown script can continue with the rest of the setup.

My plan is that the main routing table will handle all cases except from the SixXS subnet to  the SixXS tunnel, in my case from the subnet to external global unicast addresses. I could just make sure each table has all the routes I need, but in my case I think it's easier this way. Here are the resulting rules from the above configuration:

ip -6 rule
# 0:      from all lookup local
# 100:    from all to 2001:4978:192::/48 lookup main
# 200:    from all to 2001:470:b967::/48 lookup main
# 300:    from all to 2001:470:1f0f:b56::/64 lookup main
# 600:    from all to 2001:db8::/32 lookup main
# 32000:  from 2001:4978:192::/48 to 2000::/3 lookup sixxs
# 32766:  from all lookup main

Rules 100, 200 and 300 are my local prefixes. They don't need to go out the SixXS tunnel, so I divert them to the main routing table. Rule 600 is the reserved documentation prefix. I think I may try to use that address range when making how-to videos, so I want to divert that to the main routing table and null route it there to prevent sending invalid traffic out to the Internet. Rule 32000 catches all remaining packets from my SixXS subnet that are to the global unicast address range and tells Linux to use the routing table I named "sixxs". The last rule is what sends everything else to the main routing table. The last rule should exist by default, but "ip -6 rule flush" deletes it, and that's the "gotcha" that cost me a few hours' troubleshooting.

The sixxs routing table as configured above:

ip -6 route show table sixxs
# 2000::/3 dev sixxs  metric 1024  mtu 1280 advmss 1220 hoplimit 4294967295