White dot for spacing only
The Dice Project


352: IPv6 investigation: Issues

Issues.html,v 1.142 2016/09/29 15:06:32 gdmr Exp

This project is to conduct an initial investigation into the issues and likely effort involved in bringing in IPv6 support. This page lists (some of!) the issues as they were understood at the time of writing. It will be annotated and added to through the life of the project. It is expected that a number of follow-on projects will result to cover specific work items which are identified. The project's index page has some useful additional links, and the final report is here.

Note that it is not being suggested that IPv4 should be phased out any time soon. On the contrary, dual-stack running is the most likely outcome for several years yet.

Note also that as things progress items in this document may be snipped, changed or struck out in the light of experience or decisions taken.

Thanks to Sam for some very useful comments on previous drafts. Usual disclaimers apply!

Why?

Given that we're managing to get along with IPv4, why do we want to add IPv6 support? There are a few reasons:

  1. Given that there are now some IPv6-only ISPs, with the number only likely to increase, if we want to maintain our visibility to these users we need to have at least our servers able to speak IPv6.
  2. Our users might want to contact IPv6-only services elsewhere.
  3. Pragmatically, machines will be increasingly IPv6-enabled, and if we want to control this and apply some policy then we have to have support built in to our network.

Addressing

(See also the separate document on IPv6 unicast addresses.)

This is the most obvious change from IPv4: addresses are now 128-bits, and are written in 4-hexit chunks, separated by colons. Leading zeros are suppressed, and the largest all-zero block can be elided to "::".

Loopback is ::1/128.

The University's IPv6 block is 2001:630:3c1::/48. Sam's current thinking is to split this off by (BCD-encoded) VLAN: 2001:630:3c1:160/64, for example. Non-VLAN IPv4 subnets can be assigned IPv6 network numbers as required.

Link-local addresses: these come from fe80::/10, and are usually based on the MAC address. You can always use these to talk to immediate neighbours. They're also used in neighbor (sic) discovery ("ND", like ARP) and DHCPv6, inter alia. Multicast-DNS services (e.g. bonjour, avahi) might also advertise these, though for managed machines this may not matter.

There are various other classes and scopes of IPv6 addresses. It's proposed to ignore these for now as much as possible, and to take another look at how we might extend our use of IPv6 once the whole thing is up and running and bedded in. See RFC 4291 for the details!

Which machines?

There's no reason not to support it in such a way that potentially every machine could have (and use) a global IPv6 address. Or indeed many IPv6 addresses each. And that's the model we should be aiming for.

However, there's no reason why we would actually have to give an IPv6 global address to everything. Network switches, for example, might very well be able to get along with only link-local IPv6 addresses (alongside their existing IPv4 addresses). Printers might not even need IPv6 addresses at all.

Getting your IPv6 address

(See also the separate document on IPv6 unicast addresses.)

How?

  1. You can hard-wire it in. This is essentially what we do with (most) managed DICE machines, and there's no obvious reason why the same couldn't be done for IPv6. It should in principle just be a case of extending the network component slightly to look up and set up the IPv6 addresses as necessary.
  2. You can use DHCPv6. This might be what we would want to do on the "self-managed" subnets.
  3. SLAAC ("StateLess Address AutoConfiguration"). This is based on the prefix and policy that we would have our routers transmit in their RA packets. It's not obvious why we might want to do this, and it might well not work nicely with the switches. It seems to work quite happily with Macs and recent Linux.

However you get your address, it'll be validated by DAD ("Duplicate Address Detection") before you're allowed to use it.

Question: how do we track the MAC-IPv6 mapping?

We need this so that we can audit traffic and tie addresses to the corresponding switch ports.

For IPv4 we use arpwatch. There's ndwatch, perhaps known as NDPMon, in sourceforge, which might be worth investigating. It appears to have been packaged for Ubuntu, though not for RedHat. We have packaged up addrwatch to provide some similar functionality.

Alternatively we might harvest the core switches' tables, though this gives a bit less in the way of immediacy of notification as well as potentially missing short-lived address mappings.

Question: how do we handle our existing RFC1918 blocks?

Some of these are this way because we don't need them to be globally routed. Some of them are this way because we deliberately don't want them to be routed for some reason. We'll have to find a way to deal with the latter group. Filtering at the edge is likely to be the answer.

IPsec

IPv6 incorporates IPsec: specifically AH and ESP are part of the base spec. This is mostly a host issue (but see below re OSPFv3), where we should eventually look at IKE.

How do we want to assign the "host" part of the address?

(Note: we're considering static CO-assigned addresses here. For dynamic addresses handed out by DHCP we might want to make different choices.)

This is a fundamental decision that we will have to take fairly early on, as the more machines with (global) IPv6 addresses that we have the harder it will be to renumber later. The very first thing we have to decide is whether we want to try to link the IPv4 and IPv6 addresses in some way, or whether we would be happy to have them be completely independent.

Linked

The development meeting held on 2nd September 2015 decided there was no obvious need to link addresses, and some good reasons not to. With hindsight and some experience, it really wouldn't have been a good idea to have tried to link them.

Since the IPv4 and IPv6 stacks run totally independently (apart from address selection when connecting) the only reason we might want to maintain a link would be for our own convenience, both when assigning addresses through dns/inf and when interpreting them ourselves.

If they are to be linked then we would need a scheme whereby the IPv4 address could be translated to a corresponding IPv6 address, and back again, in an obvious way.

Unlinked

This was the approach chosen at the Development Meeting held on 2nd September 2015. With experience, it turns out to have been eminently sensible!

On the other hand, if we decide we don't need to link the IPv4 and IPv6 addresses then we can just number arbitrarily within each /64 subnet. This approach has its attractions too.

And given that IPv6 address space is plentiful, there's scope for creative use!

DNS

We'll need to have some way to assign fixed IPv6 addresses to hosts. Several ways come to mind:

  1. (If the IPv4 and IPv6 addresses are linked) have makeDNS automatically assign IPv6 addresses based on listed IPv4 addresses and some heuristics. The advantage of this is that it would be straightforward to implement. The disadvantage is that we might not actually want every host to have an IPv6 address, as that might be taken to imply that they would have it enabled for use.

    (In any case, makeDNS will have to know how to implement our preferred IPv4-to-IPv6 mapping scheme. See above.)

    We would also need a way to add IPv6-only entries, ideally in a way better than simply adding "#verbatim" AAAA RRs.

  2. (Again, if the addresses are linked) implement some indicator in makeDNS to request an IPv6 address "based on" the corresponding IPv4 address, as above. Sub-choice: a flag per machine which would result in all its IPv4 addresses being shadowed, or per-individual-address? Advantage: explicit. Disadvantage: arguably clumsy. This might be the best option.
  3. Implement IPv6 addresses explicitly in makeDNS. Advantage: explicit again. Disadvantage: perhaps more error-prone, as names will appear in duplicate (triplicate...) much more often than now, and slips will be made entering addresses and conforming to our addressing plan. However, if we do decide to treat the IPv4 and IPv6 addresses as completely independent then our tools can be written in such a way that errors can be checked for and minimised, particularly if we move away from hostfile format.
  4. Implement or obtain a completely new IP address provisioning tool. This was the approach adopted (makeDNSv6). It doesn't replace makeDNS for IPv4, but rather works in tandem.

Once we have the addresses in our zones, our current versions of bind doesn't have any problem with serving them out.

Should we bother to continue to generate reverse mappings for dynamic DHCP-leased addresses? That's a question that we can defer to the DHCP sub-project.

lcfg-dns is in serious need of an IPv6 makeover!

getaddrinfo() and /etc/gai.conf

At the moment we have lcfg-dns set up /etc/gai.conf to suit our IPv4-only setup. There might be changes needed here in the light of experience with a live parallel IPv6 stack. It turns out that this just works for IPv6 using the default rules, as augmented by us for IPv4, and no change is necessary. In any case RFC 6724 has added some additional entries to the default policy table which we should track in lcfg-dns.

Switch-level support

The following of our switches support IPv6:

The following of our switches DO NOT support IPv6:

We wouldn't expect to assign IPv6 addresses to the edge switches, unless it turned out that some feature didn't work without. However, there are some IPv4 features that we would want to have duplicated for IPv6:

We don't actually do this for IPv4 at the moment, but it would be really good if we could restrict RA to trusted systems. It looks as though ra-guard does this.

If we set any IPv6 address on a switch then it becomes a target for "management" connections. We therefore have to restrict these, as we already do for IPv4.

Routing-level support

Advertise the default route using RA from the core, and block from everything else: use ra-guard. Don't appear to be able to set preferences (fixed in later firmware versions), so that the first RA received sets the default route. This is actually better for the self-managed subnets than what they have with IPv4, as there would be router redundancy for them, as opposed to just one default route handed to them by DHCP. However, it does mean we wouldn't want to use RA on most of the Linux edge routers.

Question: what do we want to do about RA on E42 and E160? We could perhaps just punt this to the EdLAN routers?? Alternatively, BIRD (at least) can do RA so that's what we've done.

OSPFv3:

Note that the security model for OSPFv3 for IPv6 is rather different from OSPFv2 for IPv4. This will need a some thought and development.

OSPFv3 or BGP to speak to EdLAN?? To be decided. OSPFv3 would probably be simpler, while BGP would offer more control. We're using OSPFv3. It's up and running!

Filtering-level support

This might well become more important, with the absence of RFC1918-type addresses. There is already some basic IPv6 support in lcfg-iptables, though, and in principle it would "just" be a case of modifying the existing scripts and fragments to cover the IPv6 cases. Done: turned out to be straightforward.

DHCP

We would definitely need this before we could put IPv6 on the self-managed and managed-Windows subnets. DICE machines mostly use DHCP only at installation time, and could be installed over IPv4 as at present. Deferred to a separate project. SLAAC seems to work, mostly.

OpenVPN

OpenVPN's support for IPv6 is coming, but isn't really mature yet. It might be that it would offer enough for our target use pattern (road-warriors connecting back home), but some more detailed investigation will be required at some point.

We may also need a way to assign addresses on non-VLAN subnets. This case arises where we have split an IPv4 /24 into /25 or even /26. As these are routed directly by the endpoints there's no need to assign a VLAN tag for any of them. However, for IPv6 we would need a chunk of global address space to use in place of a "tag" block.

Host systems

(See also the separate document on IPv6 unicast addresses.)

Managed DICE machines will need to know how to set their static IPv6 address(es). This may just be a case of extending lcfg-network to be able to acquire the correct values to drop into the configurations. SLAAC seems to work pretty well except where a CO-defined address is really required.

In addition to any statically-configured IPv6 address, (SL6 at least) DICE machines will autoconfigure a global one with the same interface number as they use for their autoconfigured link-local address. (If we don't have this turned on then it looks as though they won't pick up a default route through RA, so we'll just have to live with it.) It doesn't appear to be problematic (so far). There's an ifcfg-file switch which seems to control this, should we want to.

Windows and Macs will probably just acquire and use an IPv6 address through DHCP, once we have it set up. It should Just Work.

tardis?

The tardis people have asked on and off for a while about IPv6. They currently have 193.62.81/24 which is routed directly by EdLAN for them. As we're proposing moving them behind our edge anyway, it would make more sense for their IPv6 allocation to be done through us, most likely by them getting a handful of "non-VLAN" blocks of EdLAN space.

Sequencing it all

As soon as we turn on IPv6 on at least some of our subnets, the machines on them will likely immediately start to use it. So we need to sequence the way we test and enable things (strike out as done):

  1. Decide on an addressing scheme.
  2. Write a blog article. It's here.
  3. Enable IPv6 on S32 and S33 on the Forum core switches, AT1 on the AT core switches, and B (64, transit) on all. All the machines on those subnet are managed, so nothing should notice or try to use it. With any luck link-local addresses should suffice. Done, and also temporarily on march on S33. ping6 between them works.
  4. Take and peruse some MIB-walks.
  5. Check and set as necessary the allowed-managers lists. We might prefer management to come only from IPv4 addresses. Testing strongly suggests that these lists are independent on at least some of our switches, so this is definitely something that we'll have to address. As an interim measure, ensure that we always set up at least one IPv6 manager address on any switch which has an IPv6 address. Instructions have been added to our local IPv6 documentation. NOTE: pcScan still needs taught how to set IPv6 manager addresses. It will now ignore any that it finds have been set by hand.
  6. Decide on OSPFv3 or BGP to speak to EdLAN. We don't actually want to quite yet, but what we do will affect the OSPFv3 settings. We're using OSPFv3 to speak to EdLAN.
  7. Decode what we want to do about OSPFv3 security. We would want to be at least as secure as for OSPFv2 for IPv4, so should probably use AH. It's not clear that confidentiality is required, so perhaps not ESP. We do need to coordinate this with IS. NOTE that we might not be able to do this (see above) due to the apparent lack of AH or ESP on the core switches. Meantime we'll carry on without authentication, on the basis that we only speak OSPFv3 on a small number of reasonably-managed subnets.
  8. Enable OSPFv3 on S33, AT1 and B on the Forum and AT core. Routes (intra-area and ASBR-injected "connected") should propagate, and we should see these appear. Clone an OSPFv3 version of the netman-scripts tool to show these.
  9. Extend our configuration tools so that they know how to deal with IPv6 MIB objects, and can enable and disable things as required.
  10. Extend or rewrite makeDNS so that we can assign static IPv6 addresses in a controlled way, in accordance with our addressing scheme decision.
       rfe dns/inf6
    
    Forward and reverse zones are now carried on all our nameservers, and have been delegated from above.
  11. Extend lcfg-network so that machines can pick up and use IPv6 addresses. Default routes should appear through RA. NOTE though that until we are routing external traffic any machines which do have an IPv6 address may end up experiencing timeouts and delays as a result of actually trying to use it. (MPU: bugzilla#897) As a workaround in the meantime, there's a macro in live/netinf-macros.h which can set up a static address for a VLAN/subnet.
  12. Extend the filtering scripts so that blocks and holes are added as required, as for IPv4.
  13. Bring up OSPFv3 on our Linux routers, using either BIRD or quagga.
  14. Enable IPv6 on E42 and E160 aka AT2 and A.
  15. Have IS speak OSPFv3 (or perhaps BGP) to us. OSPFv3 was enabled on VLANs 160 and 42 on 2nd February 2016. This makes us globally visible, so anything with an IPv6 address must now be in a position to answer appropriately. At this point it becomes reasonable to attempt to use IPv6 for anything other than testing.
  16. Write another blog article. It's here.
  17. Enable IPv6 on remaining "managed" subnets at all sites (though note the caveat above concerning the AT labs). Done!!
  18. Implement audit tools equivalent to arpwatch. (We can defer this to here because all addresses will either have been assigned statically or will be "global dynamic" based on the MAC address.)
  19. Summary blog article. Done.
  20. Project signoff!

The following were originally bundled into this project, but it now appears that it would make more sense for them to be done separately, either as their own individual projects or as operational or miscellaneous small-scale development. They are retained here for the record.

  1. Bring DHCP for IPv6 into service. (project)
  2. Enable IPv6 on all remaining subnets, and announce its general availability through another blog article. (project)
  3. Sort out the inevitable teething problems!
  4. tardis??
  5. IPsec and IKE? (project)

Niggles

This is a list of niggles that we have spotted along the way, in no particular order:


 : Units : Infrastructure : Projects : 352-IPv6 

Mini Informatics Logo - Link to Main Informatics Page
Please contact us with any comments or corrections.
Unless explicitly stated otherwise, all material is copyright The University of Edinburgh
Spacing Line