17th October 2019
JEN LINKOVA: Good morning. So, if you are interested in anti‑abuse you are in the wrong room. If you are interested in ‑‑ in IPv6, you are in the right room.
So, slides. Can we get chairs' slides? So meantime, it's me, Jen, my co‑chair is Raymond and Benedikt in the first row. Actually, this is an agenda, quietly quite busy, at least I hope, we are going to have some fun in the last 20 minutes in the session. Any additions to the agendaor suggestions?
I can't remember what I'm supposed to say without my slides.
I think I remember. So first of all, thanks for our magicians, I mean stenographers. And we have a Jabber scribe, right, Nathalie, scribing? Okay. Sorry. Yes. We have scribes. And slides. So this is agenda, we went through it. Microphone etiquette, if you are going to the microphone, and I hope you will, please say your name and affiliation, slowly and clearly.
Meeting minutes. I think we did send meeting minutes from RIPE 78 to the list. How many people read it? Any comments? You are happy? Approved.
So, another agenda item, we will go very quickly through it, Ray, apparently has been with us for a while and his term as a Working Group co‑chair came to its end, Ray kindly volunteered to continue. We sent it to the list, I might be wrong but I have not seen anyone else volunteering. No objections to this statement? Good. So Ray, welcome back.
RAYMOND JETTEN: Thank you.
JEN LINKOVA: Please continue sending minutes to the list because I keep forgetting. And you know what you are all supposed to do, right? You are supposed to rate the talks, because if you don't ‑‑ if you don't, we have no idea what you'd like to hear next time, so don't complain. Also, you can put some free TXT feedback there and it's very useful so that's why I changed the colours, it's not green any more because someone complained I am excluding colorblind people. Now, please enjoy black and orange.
AUDIENCE SPEAKER: That's green.
JEN LINKOVA: So you can see right, so you have been cheating? Okay. The first talk of today, Veronika will tell us some people think is impossible, IPv6 only corporate networks.
VERONIKA McKILLOP: Thank you, Jen. Good morning everybody, I am IPv6 network architecture in Microsoft IT, and I really have to emphasise this because when people think about Microsoft they just see this big company, yes, it is a big company but there is ‑‑ there is the public and product facing part and then there is the corporate network, and I'm from the organisation. Behind me you can see Microsoft CSEO, call services and operations in other organisations, this is called IT. My team is Cloud and Connectivity Engineering, we look after the whole network, corporate, guest, corporate backbone, Internet peering, etc.
We work with ‑‑ any complaints about as our I can take them and feed them back to our partner organisation but can't solve them for you. I am the customer of Microsoft, as you guys are.
There was a talk about IPv6 efforts in Microsoft three years ago at RIPE 73. Hands up, how many people have seen that talk? That is probably 30% of the room.
My second: How many of you are from an enterprise rather than service provider, hand up? That is probably 20% of the room. And how many of you guys are actually deploying IPv6? Excellent. But there was probably still 35 pest of the room. Lets talk about this myth I can beast, IPv6 only.
Before I start I am going to give you an overview of our network. So, we are big company, people kind of know that, Microsoft is used by about 85% laptop users, the network that ‑‑ we split into four regions: The main region is in Seattle, state of Washington, where we have got about 100 buildings that support about 100,000 employees. Then the remaining regions, the largest one really still is North America, Europe, Middle East and Africa and ‑‑ we are somewhere between 750, 80100 office, constantly keep building and renovating and moving people around.
Our services are provided to our employees from autonomous data centres and also from as our Azure. We have gone through data centre rationalisation over the last five years, so right now, we have got about five of them in total, but obviously everything else as usual.
Our branch offices are connected through MPLS and we have got about 11 dedicated Internet edge locations that we use for connecting to the outside world.
What does that mean in terms of number? In total about 250,000 end users, 150,000 permanent employees. From IT perspective we support 2,200 line of business applications and you can see the network carries quite a lot of traffic and connects lots of different devices.
What has changed since three years ago when you heard the first talk? This is a new strategic initiative where we are working on network segmentation, not just corporate or guest network, we actually want to have the network much more secure than it is today from our perspective so this is something is ‑‑ that also influences other work, including IPv6, and basically it has got some impact, which then leads also to IPv6 only.
So, what we have done, and I included this time chart for you to see that the work on IPv6 in Microsoft started many years ago, it's now 18 years ago, because in 2001 Microsoft research started looking into this new thing, this IPv6 because obviously Microsoft OS needed to support this protocol. At the time the research centres were connected through ISATAP, first on the window service and then through also some dedicated hardware infrastructure. In 2006 the team deployed IPv6 on the backbone network but there were still no end user segments, if you can enable, if you don't connect the users see no traffic.
That has actually changed in 2016 when the team completed global roll out of IPv6 in the whole corporate network. It was very significant and but because we still don't have all Internet services enabled with IPv6, we see about 22% of user traffic on IPv6 and I think that's kind of in line with the top 1,000 statistics, right, that's where we are.
The work has obviously continued and while in early 2000s IT had one IPv6 prefix from ARIN, by 2015 we have obtained three prefixes, so we have got one from ARIN, one from RIPE and one from APNIC. And I would say why, do you really need that, you don't have that many locations, etc.? Well, I can tell you it's operation pretty easy, I can look at the address or any of our engineers and we know which region that address is from or the prefix is from. So, that actually then helps us to have nice addressing architecture.
The work on dual stack continued, in 2018 we have launched pilot for our guest network dual stack. I think three years ago, my colleague at the time he was talking about the effort of turning our guest network to IPv6 only, but then we found out that some VPN clients don't work on IPv6 only and because we have quite a lot of people who come to work in Microsoft, they are external people and need to connect back, we unfortunately had to conclude that we could not pursue this path for the guest network and we had to stop this effort, that was early 2017. So, basically from that moment we really focused lets just do dual stack and we will do stream tests and in probably in next year or two when we feel that we are not going to impact our users as much, the visitors I mean.
In 2018 we also rolled out next generation VPN solution and that was dual stack from the very beginning and the work simply continues for dual stack on guest we want to have all the infrastructure, v6 enabled, including the authorisation portal, we are using clear pass so that work happened and we are right now in process of deploying globally. So that's going to be another significant network segment to be completely dual stack.
I mentioned network works segmentation efforts, that's why currently we are undergoing change or update of our IPv6 addressing architecture, because we need to make sure that all the new network segments have IPv6 and basically the existing architecture from 2006 wasn't really ‑‑ didn't take into account such a potential evolution over the network.
But it's actually still pretty easy to update that.
And as I say, there is one thing that those of you who have not deployed IPv6 or have not started yet, the biggest challenge here is always the mindset, it's the people, not the technology. Come on, we are in 2019, all the OS is supported, the network equipment, it's kind of there, you have got bugs and stuff like that but the features are there, is the people who still, they didn't think this was necessary to happen so in 2018, thanks to all the work my team was doing our organisation recognised this is not just problem of my team but it's a problem of the whole organisation that we all really need to work to make IPv6 happen.
So, what is the result, what we have on the corporate network: I know people are interested in this, what type of address assignment mechanism we are using, so we are using SLAAC and Stateless DHPCv6, since Windows supports RDNSS and I can tell you it works quite well. We are going to retrofit and have on the corporate network and also on guest and the simple reason is that our ‑‑ guest network is often used by different devices which not all support RDNSS or Stateless, so we as a network provider have to do both, but it works, you know and that's fine.
For the guest infrastructure, we have come across an interesting issue for the authorisation portals we found out that the protocol which is used for database synchronisation in the cluster, they are using a different port so it was interesting, we deployed it and couldn't work out why our cluster didn't want to syncronise and the and didn't want to propagate and in the end working with the vendor found out there was a different port we needed to open on our firewalls and it took sometime uncover that and we get through that and we have that very important piece of infrastructure, v6 enabled so keep on going.
Because the Internet resources are still v4 only we don't see as much v6 traffic as we would like to considering everybody has IPv6 network connection. And some of you who might be using Azure or working with Express Route, this is something still not there so hopefully we will have some good news before the end of this year. V6 supporting V nets was made public I think early this calendar year, there is more coming and I think something is in private preview and peering on express route should be announced in the foreseeable feature.
So, this is where we are, when it comes to dual stack on the normal network and just a few details about remote access VPN; that currently, based on the latest statistics I got from the team yesterday we support about 250,000 users so you see it's a little bit busy. The good news is this VPN solution works through NAT 64, I am using it here successfully and have used it for a long time. Our challenge with VPN it still requires v4, all the dual stack so that's why we are saying dual stack is not v6 done completely, it's an intermittent step, that helps people to get accustomed to IPv6 in the network but not something that gets us out of the biggest problem we got and that is not having enough v4 addresses. We want to run v6 only inside the tunnel and we started the test RIR two years ago, eventually we discovered that the vendor had some dependency on IPv4 and they didn't allow us to configure on the VPN to have v6 only client pools, so again that took some effort, engagement with the vendor. We got early code for testing about a year ago and then production in January. After that, we have discovered that the native Windows VPN client didn't support v6 only so we had to work again with the Windows team but they were super good to work with, you know, and understood the business need and I can till it works with v6 only so we have got both options, internally we are using nature VPN client and the vendors client and they both work with with IPv6 only on IPv6 only ‑‑ inside the tunnel IPv6 only so I think there will be others who will benefit from this in the future. Right now we are at a point where we have got user pilot set‑up and ready to go. The team that is doing the testing is right now providing really good feedback so we need to just enable and get few 100 or maybe few thousand users on this pilot to verify the user experience and mostly figure out what doesn't work for NAT 64, because all the corporate resources which are not v6 enabled we will have the send traffic through NAT 64.
We don't have this problem some others have where they send all the traffic through VPN. We perform corporate goes through VPN for the Internet traffic it's send out the normal interface.
What is the problem with dual stack, though? It hides bugs. And I can assure you that until you really run v6 only network you will not see stuff and you think your v6 is working, and then you realise it isn't working and we just found out something yesterday. So it happens all the time.
So, that's the reason why we also need to go to v6 only, to make sure that our production code is good, you know, we have stable software, we tonight have any bugs with v6. In general, let me summarise with our other drivers for v6 only. There is the industry pressure, thanks to Apple and this is probably the greatest thing that happened for IPv6, Apple in 2015 announced that all apps submitted to App Store as of 2016, had to work through NAT 64, and for some people this might be surprise, Microsoft is a multi platform company, we live on everything, Linux, Windows, IOS, Mac OS, Android, so we need to make sure all of our software works on these platforms and where there is the danger if the stuff doesn't work it gets removed from App Store and start to pay attention, that means we as a network infrastructure team we have to support them in this effort.
The second biggest problem really is overlapping and depleting private IPv6 space. We are using private on the inside, probably only 3% of the public v4 which Microsoft has, that is what we use on our network edges for NAT 44, all other public space used for Azure, for ‑‑ because everybody else using RFC 1918, whatever you can think of the past and is happening today, everybody is using the same space and then we are in a situation when we are really scrambling is there at least one /24 which we could actually peer on or how is this going to work? And I would say, we might be in a situation where all our systems will be v6 enabled so this acquisition potentially could connect to an IPv6 but I can tell you that's not going to be the case. We have seen the number of hands, it's still low numbers for the IPv6 enablement. And yes, I know about ‑‑ don't worry, I'm talking with them as well about them enabling IPv6, so let's see what's going to happen.
The other thing here is for the VPN, our partners use it to connect back to Microsoft, use the same space again, we have got a problem here because we could sometimes override the routing.
So, things happen.
And exhaustion is imminent. Thanks to the network segmentation I have heard we are practically out.
Operation complexity, I don't need to talk about this any more and obviously it does impact us.
So for v6 only to work, even when we have everything internally v6 enabled, just dual stack for the services, I am not being too hard to people, I am saying please at least enable it alongside v4 and user segments will be v6 only. So you can see here behind me the map with all the edge locations that we have got, and right now, we have got ‑‑ we have got the ‑‑ this very green to the ‑‑ we have got the delivered Europe aPAC in Singapore and India and the other locations are currently work in progress to be completed by summer next year. From that perspective we will be in a position where we can go to any office around the world and actually deploy production IPv6 only network.
So, what have we done to support our product groups? We have actually given them v6 only Internet only network. This has been actually working quite well for over two years and it's a pure Internet connectivity which kind of simulates that user experience where users have on the Internet connection only IPv6, obviously this all sits behind NAT 64 because still about 70% of the Internet is IPv4. It works rather well and we have solved quite a few issues where the app people were telling some service provider there is a problem in your network because of the way you have deployed NAT 64, you bad network provider. We tested and Apple using some old APIs and the team was actually able to go and fix it.
Besides App Store there is actually increased level of government interest, especially in the US. I don't know if you guys heard of the state of Washington, which is the home state of Seattle, they have announced past 2017 and started working on it quite actively recently, that by end of 2024 all their services and everything they run will be v6 only, which is quite interesting. They are our big customer as well, so another thing that helps us to make the case.
On this network, as I mentioned, we test for all the platforms, Android, I O is
A. Mac OS, Windows, whatever you have got, you can text Linux test Linux. Right now in 12 locations ‑‑ this is demand driven.
The more interesting one, which has got really impact on our business and v4 run‑out, is that ‑‑ our IPv6 only corporate network. We have been running pilot since April last year and we have decided to run it as on ‑‑ same security mechanisms for joining so the devices to be managed and have the right security certificates and be member of a particular security group so they are able to join this network. I say here that there is a tidier device mix, simply because we are more able to control what connects wireless than wires, that is why we have chosen the wireless network. We are supporting, as I mentioned, all different operating systems, which is really handy because earlier this week some of our Windows users were telling us we are not able to connect, and it's like okay, is it something again in the stack in Windows or is it something else? We confer with the Mac users they have the same issue so now we know it is clearly the network again.
At the beginning, we were ‑‑ we were quite confident that this is going to work well because on the test network for product groups, we didn't really receive any complaints about stuff not working and bad user experience. Well, the network is used slightly differently and because of time I will probably have to talk about it with you off‑line, but on this we found immediately three issues. We use both Cisco and Aruba for our wireless, so they are in different locations and different regions but we have come across three quite big issues which cause the users using Internet connectivity. They were basically not getting DNS information or they were simply getting completely disconnected from the network and recently we found Stateless state issue, controls dropping the packets, so, again, the client didn't have any DNS information. You can't see this on dual stack, you see this only on IPv6.
As I said, this can actually help you clear out the code. So my question to everybody is: What is your meantime to innocence? Right? MTTI. So you have got got three aspects and hopefully we will be able to not worry about the client stuck in the future. Right now, the question is who has done what on the network? Second, there is something funky with all the IPv6 features in the various iOSes, so what is going on? Luckily we have got good collaboration with our Windows OS team and they understand the need to fix things. And when it comes to applications, I call on the usual suspects and what do I mean? Because, really, to be completely honest, v6 only is about applications. The network, yeah, you are going to hit bugs but you're going to sort it pretty quickly. Applications, that's a big problem. Here I am sharing what we have discovered so far. There might be more but basically, it's a user behaviour when people are used to connecting to v4, like SSH, developers like to do it, we will have to work with them to prep end the NAT 64 prefix or put the host name in so that works.
Obviously hard coding v4 addresses, that really exists. IPv4 on API and function calls. This is something we found thanks to that failing Windows VPN client because it leverages universal Windows framework and we found there are actually lots of v4 only APIs in Windows. Well, that's obvious. But there are obvious ‑‑ they are better cousins which are IP version agnostic. There is this inertia in the industry and they reuse what they wrote 20 years ago. So we are working with the Windows team to kind of deprecate v4 only PI codes, this is going to have interesting impact, but it is going to be effort, which is going to run for next couple of years. But you might hear something early next year, I will make sure that you are aware of.
The back‑end systems, so can perform redirect so you connect to some location and it works through v6 only and NAT 64, everything is fine and you try to get something from the location, something funky happens at the back where the system tries to connectivity to another host but because that happens in the back‑end and the client doesn't initiate that connection, the DNS 64 in the way doesn't kick in and basically, nothing happens, right? For example you can't download an app. So, you can see we have got quite a lot of applications. We care about the stuff which is business critical and we run and manage to run the business but there's tens of thousands of third party applications, so this is a good fun, it's ‑‑ and we do find stuff which doesn't work. Here there are a couple of examples. Spotify, steam, postman, people can't actually connect to URL in postman. These apps written for Windows and I have got real challenge finding the team that manages that environment to actually start putting there some rules for people who right applications on U W P. I found some payment portals don't always work so you end up getting charged twice or not getting your order through. That is external, and as I mentioned, for the internal stuff we have got so many dependency, people don't always appreciate anything, enable it on the server, that connects to some ‑‑ to some CDN or it needs the v6 peering, etc., right? So, the most frustrating part is when stuff starts to work that didn't work for last 12 months and we are trying to work out, what happened when all of a sudden what was broken, now it's working when we, network team haven't done anything.
The conclusion really is the network is big, it's complex, and there are loads of of dependency, which means lots of work but if the whole organisation actually embraces it it's impossible to hatch. We know this is not something happened overnight and basically have to work on it for the next few years but because everybody is involved it is really possible. And also some of the new network segments will have to remain dual stack, we can't make that v6 only happen everywhere because we don't have complete control over what connects to that. Imagine all these iOS devices, printers, security cameras, we will have to have v4 and 6 on those segments, which is fine, and what we are really focusing on is we want to have IPv6 enabled end‑to‑end. As I say, dual stack is business as usual so you guys get on with it and stop thinking about should I do it or not. And unless you do v6 only you will really not be sure if your v6 implementation is fully working.
So I ask the Windows guys and people come to me and say they have got these issues, the ‑‑ how do I get this to you, right? Oh, they say they should submit a feedback, in Windows 10 you type feedback, this thing pops up and you have to sign in and you can actually send feedback. The best advice is you have to say that you represent few thousand users ideally, so it actually gets the right priority. But this is the way how you can actually communicate without trying to chase a poor Microsoft person at a conference.
I would like people would really like to have all the answers now. I can tell you you really need to have imagination and just have a vision and something that you want to do. There are lots of obstacles you are going to meet along the way, it's doable, 2019 so let's just do IPv6. Thank you.
JEN LINKOVA: Questions?
MIKAEL ABRAHAMSSON: Great that you are sharing this, I often refer IPv6 sceptics to your talks, Microsoft is doing this, please do it as well. I approached one of your colleagues in the Windows somewhere, I don't know, about porting the Windows mobile C lath code. I was not very successful in convincing him that this was a good idea. Has there been any talk and would you helped if Windows had a C lath?
VERONIKA McKILLOP: No, we don't care; we want everything v6.
MIKAEL ABRAHAMSSON: But for ‑‑ moving this entire ecosystem of older apps and so on, you would prefer to do the Apple way and prefer try to get the programmers to change the set ‑‑
VERONIKA McKILLOP: Yes.
MIKAEL ABRAHAMSSON: Thanks for the clarity.
JAN ZORZ: They have Android now.
VERONIKA McKILLOP: Jan is making a good point. Since about two weeks ago we have announced working with Samsung now, have got an android device.
MIKAEL ABRAHAMSSON: Anyone who wants to do that has to redo their application anyway‑‑
VERONIKA McKILLOP: Let's have it clean. Why not?
GEOFF HUSTON: APNIC. Look, I hesitate to put the poor Microsoft person at the conference on the spot. There are two things I wanted to observe and the first one is, your assertion that DNS 64 is actually crucial to this dual stack strategy and the question why is it okay to lie in the DNS?
VERONIKA McKILLOP: What?
GEOFF HUSTON: To lie in the DNS?
VERONIKA McKILLOP: You mean because the ‑‑
GEOFF HUSTON: It's not what was actually publish it's what you synth size. What we are finding in the DNS there is more push to put validation closer to the edge, so it's not just the recursive resolvers but end systems that are going this is not truth. This worries me a lot the intermediate strategy relies so heavily on structural lying in the DNS and over in the DNS world there is this push to go, lie is bad, and that is kind of bothers me. I don't expect an answer from you, I am not putting you on the spot, I am observing.
VERONIKA McKILLOP: I have got a view on this. I am network operator and for me what matters that stuff works for my users, so I am going to do whatever it takes to make things work.
GEOFF HUSTON: That brings me to my second point.
VERONIKA McKILLOP: It's not for dual stack, for v6 only and my ultimate goal is to avoid it as much as possible, to have native v6 everywhere. What are we going to do about the Internet?
GEOFF HUSTON: That is problem, I am not saying I have a solution but I am observing. But you are saying you want things to work. So right now, out there on the big Internet, if I put extension headers in a v6 packet the drop rate is over 40%, this is terrible. What does your company's software do about extension headers in v6? Are you happen I hadly trying to avoid it at all costs or blithely think this network is going to carry this stuff when in fact it drops it?
VERONIKA McKILLOP: Never thought about it and I am not Microsoft Windows
GEOFF HUSTON: Please to, it's a fantastically difficult problem because we can't fragment in v6 in particular because those extension headers have a really unacceptable level of delivery probability and if you want things to work in v6, and we all do, none of us feel we can clean up every cheap router on the planet. The only other way to do this is not use extension headers and that requires a degree of discipline and engineering back in your stack factory and everyone else's stack factory, not to do it.
VERONIKA McKILLOP: I am going to ask the stack guys what are they doing because I don't know.
JEN LINKOVA: I would like to comment on your DNS 64, Geoff. People who care about DNS second signing their zones should deploy v6, 4 problem disappears in destination has an AAAA.
GEOFF HUSTON: Thanks, Jen.
AUDIENCE SPEAKER: I also want to comment on what Geoff said about DNS 64 as one of the co‑authors of the DNS 64 RFCs. We had an extensive look at the living together of DNS 64 and DNSSEC and the only way you get into trouble is if you do validation on the client, which most people don't do and the client is unaware of the existence of DNS 64. If the clients understands that DNS 64 is a thing it can do DNSSEC validation no problem.
AUDIENCE SPEAKER: I wanted to ask have you tried at all any software mechanisms like MAPT, MAPE, stuff like that?
VERONIKA McKILLOP: No.
AUDIENCE SPEAKER: You just went for DNS
VERONIKA McKILLOP: It's an enterprise network. But it's one I think that that kind of in the back of our mind, that if we go to v4 as a service on the network, we could potentially leaverage, no.
AUDIENCE SPEAKER: Do you have any preference among the alternatives there, have you considered at all?
VERONIKA McKILLOP: Not really.
JEN LINKOVA: Thank you, Veronika.
The next talk is about IP addresses, right? The fundamentals in IPv6.
ENNO REY: There will be some IPv4.
JEN LINKOVA: A bit of history, okay.
ENNO REY: Good morning. I am going to talk about IP addresses from different dimensions, v4 and v6.
It's a good thing to see the room so full of of people, and the interesting thing is that this is the second presentation on enterprise space, which might be an indicator of things going on there as well.
Very quickly, who am I? NRA IPv6 enthusiast for a long time. I have a background in, what was called in the '90s, large scale networking. I have been doing lot of security stuff in between. I do IPv6 in my day job. I run the IPv6 programme in a large tech company. But this talk is on private capacity and based on observations in enterprise environments where I was consulting to bring in IPv6 in the last years.
The agenda is quite simple: We will look at or I will look at, you will have to listen, at the ‑‑ how the IPv4 address space has developed and which type of IPv4 addresses are there, and what are their operational implications. I will have a strong focus on operational processes and how those are affected by properties of IPv4 and later IPv6 addresses. So we will have a look at the IPv6 address space and I will talk about the implication of changes between v4 and v6 when it comes to addresses and the properties.
From a very simplistic perspective, looking at where IP addresses, be them v4 or 6, play a role in an IT environment, let's assume there were systems, systems running applications, there was a network connecting systems, this network might be composed of different entities, route ‑‑ packets and middle boxes performing security functions or other types of functions and there was users in applications which the users access. And when one thinks about where do IP addresses come into play here, I think there is three main dimensions.
One is obviously IP addresses identify systems or their place in the network, but those systems to be able to communicate in an IP based network in the Internet, they need to have IP addresses. Then, there is systems performing functions based on the IP addresses of other systems, forewarning decision or a security decision of dropping packets, those are performed based on IP addresses but not on my own IP address but on those of the packets which I am carrying or forwarding and there might be, say, applications which process IP addresses, which take an IP address as an entity to put it in the database or look at it and to store it or to use it as an identifier of an entity in the network, so there is say application level perspective for processing IP addresses, and I would say from what I observed, this is ‑‑ can be a very difficult part of say an IPv4 to IPv6 transition. Getting systems equipped with addresses, especially in a dual stack world, that's doable, but having these upper layer functions where IP address are processed, doing the right thing in an IPv6 network, are being the same thing they have done in an IPv4 network, that might be actually quite difficult. Looking at those systems, or looking at the life cycle of the system there is many, say, parts where IP addresses are involved, there is provisioning, system has to be brought up. There is usually say an inventory function for stuff done in enterprise networks with what is called the CMDB, usually it's many of them. There is like a function which looks at systems, what state are they in, the monitoring or given I have a security background, vulnerability management, which systems are there have to have you will havibilities and have to be cured in a certain amount of time. IP addresses might play a role there. There is the function of the application, whatever it does. And there might be help functions, the DNS SNTP, authentication. IP addresses might play a role there, too. So there is, looking at the individual systems, many places actually during the lifecycle of IP addresses play a role and affect the way how operational processes are done. This is what we have ‑‑ I am going to look at in a second, the way how operational processes are affected by properties of IP addresses.
So, very quickly, over the view of how the IPv4 address space has developed over the years. In the beginning, IPv4 uses 32 bits and in the very early days, those 32 bits of an address were supposed to be 8 bits for network part, there is actual papers from the '70s which literally say, oh we don't have many networks so 260 ‑‑ 2568 bits should be sufficient to identify all networks which we are going to have in this future TCP IP things. But very quickly, a year later, in 1981, declares A, B and C scheme was introduced which then in '93 was replaced ‑‑ enhanced by the CIDR thing going classless. These are some statistics from '92. At the time, about, say, 40% of the IPv4 class A networks were assigned already. And the ‑‑ these assignments happened in the early days, in a somewhat improvised way. There is a rumour that John Possle, who by the way died on this day 21 years ago, had a paper notebook where he kept, say, notes on which network addresses had been assigned to which organisations.
And from the very early days on, there was a notion of so‑called special addresses for specific purposes, like zero, zero, or the zero /8 network or 1271. These were addresses supposed to use for specific purposes but not for being assigned to individual systems, and to be routed in the Internet. That's how the IPv4 address space roughly looked in 1999. One can already see there is some networks which are still in use today familiar names. Heavily driving IPv6 in the last, say, 15 years. There are 25 and 26, which many organisations squatted on. There is being assigned to Apple. Some of this is still visible today. And today's, say, space of what is called IPv4 special addresses, is visible on this one. So there is quite a few, and I will discuss some of this in a second.
In '92, there was an RFC, a 1380, entitled IESG deliberations on routing and addressing, and this is an interesting one, it discusses IPv4 address exhaustion in '92 and it proposes, there is different strategies to get over this. Well, transitioning to another protocol with a larger address space would be one of them, which is why we are here today, somewhat. Addresses which are not globally unique, a partitioned Internet might be a strategy but to me this sounds like an oxymoron, or reclaiming network numbers, which even at the time was considered not a viable long‑term strategy. Looking at this, and this was written 27 years ago, it turns out that many of these ideas were actually, say, materialised in different RFCs and their specifications. There is this one on, say, a two tier address structure of the Internet, why not use say a public Internet and not so public Internet? There is CIDR was invented as a way to use the existing address space in a more efficient way. There is this one, which is predecessor of RFC 1918 which essentially states there is private addresses and the following ones and there is this one, on NAT.
All these are in the '90s and were kind of born out of the insight IPv4 address space is running out.
So looking at how the IPv4 address space looked at the late '90s, we can classify addresses into three main categories. There is public address space, but the public address space in itself is composed of different, say, groups of IP addresses, those are are actually which are globally routed and which were assigned before the RIRs were formed; the stuff that as of today is called legacy addresses. There is blocks which were assigned early but which are not publicly routed, the stuff like DOD space, which people later ‑ on and the stuff handed over by IANA to the RIRs to be distributed and at the time there was quite of that still available. So public space in different flavours, private address space, RFC 1918 space used in enterprise organisations and usually used for home networks. That needs NATS to get to the Internet. And these special addresses for specific purposes.
Talking about legacy address space, if your organisation holds legacy address space and this I have stated a bit of an enterprise background and I have been working in quite a few large enterprise organisations actually holding /16s or even larger spaces from the legacy days and obviously those have a price and the price tag which nowadays which is still rising, the idea is we could sell this space at some point. At the same time, in many organisations, especially from the traditional enterprise space, manufacturing, automotive, even chemicals, there is this talk about we want to back digital company. If your company has, say, anything like that, there is C level talk on we want to be a digital player, don't sell your space. It's a critical resource. I know that these discussions are ongoing in a number of organisations and while ‑‑ I am a v6 guy, you could say get rid of it, do IPv6. Still, being say a practitioner, my strong advice is don't done do that, just keep it, you will need it at some point in the very near future and still do IPv6.
That said, just a quick look at RFC 1918 space, not globally routed. There is people who consider this a security benefit from some angle, an enterprise spaces might be debatable for home networks, this is certainly the case, having like pointers in your home networks or types of IT devices which have 192, 168 address and not being able to be reached from the Internet, might be actually a good thing.
There is, looking at RFC 1918 space, and I am very happy Veronika already stated this, in enterprise space that is a major pain point, at least one merger has happened, it means some overlapping address space. One‑and‑a‑half years ago I worked in a sufficiently large insurance company and they had an initiative going on, the XY global network, putting all the European subsidiaries, putting them all on one unified network and many had overlapping or same IPv4 address space and I am talking about 2018, was a major pain and we spent months and weeks just discussing how to deal with that. And I was always raising my hand, just do IPv6 and get rid of this, say for the centralised services use IPv6, don't try to fix the overlapping address space. From a security perspective, overlapping address space is a major pain point. One has ‑‑ if you have an organisation vulnerability scanning process and have to identify, oh this ten address belongs to that subsidiary and to another one, and we have to deploy different scanning and stuff like this, that is a real pain.
There is a third category. I might not have mentioned this six months ago, if I had given the same talk, the special addresses, the zero, 127, 240, these addresses, but there is some interesting debates recently, so I will quickly comment on this. Some of you might know there is this IPv4 Unicast extensions initiative which tries to bring up like well, why not enable on the Linux kernel level that these types of addresses can actually be used for production purposes. From a practitioner's point of view, to be honest, and not even wearing my IPv6 hat I am quite sceptical about this, for a simple reason, from an operations perspective: The, say, current non‑handling of these types of addresses, and they are often called in the respective filters are called going bogon filters, so not carrying this in production traffic, can be done by non‑routing, ACLs, many organisations have uplinks part of their ACLs, it's a bogon list, so the actual drop thing can happen in different ways. It can happen on different, say, places in a network, it can happen on routers or middle box, it might happen on local systems; there is people who put this into their IP tables rules. It might happen on a kernel level, kind of built in. It might be configureable or not, you might be aware it happens or not. It might be a default or not. So, it's quite probably difficult in complex environment to understand where this actually happens. So, there is means ‑ it means one wants to use this address space one would have an understanding first, can it be used end‑to‑end within a network? And I doubt this can be done in any sufficiently complex networks. So the short version of this thing is, I don't expect, and I am not against this, I am not an apologist, an IPv6 apologist here, my position is whatever works and brings IP connectivity, that is beneficial, I don't expect this one to work.
That said, let's get back to the ‑‑ I am still with v4 and I know this is ‑‑ say, you might ask, okay what's the point here just talking about IPv4? Let's look at this IPv4 address space from a security perspective. As stated there is public addresses. Those usually get some filtering on network intersection points. They don't need special handling when it comes to auxiliary security functions like filtering ‑‑ like vulnerability management or like incident response. You can usually identify, there is this system with an IP address and I can easily find it within my complex network or more or less easily. There is private addresses usually from a filtering perspective, not much happens on the Internet uplinks you just ‑‑ they might be part of the bogon, but you don't like 10, 10, 10 /24 is filtered where 10, 10, 20 /24 is allowed, this doesn't happen on the filtering level. And there is bet ‑‑ special addresses and those are usually just in some way ignored dropped.
Looking at this from a formalised perspective, it turns out, operationally, filtering requires operational effort, and for private addresses, uniquely identifying them within a complex networks requires operational effort so the green one here just means we have to take care of this. As for filtering, for private addresses, you don't have to take care of this usually. But for the auxiliary security functions, you will need some extra effort but you won't for the public addresses.
Now, looking at IPv6 and the IPv6 address space, I tried to identify, okay are these ‑‑ what are the main blocks within IPv6 and what are the properties. So there is this RFC 4291 version 6 addressing architecture. It's a complex RFC, so a bit complicated. It's a mess, so to say. Some of you might recall that at RIPE 74 in Budapest I gave presentation and I was referring to IETF 6 man and I had this picture of Albert Hoffman driving to home after he took the first dose of LSD, and I was tempted to apply this at this point again, but enough of that. In practice and in reality, IPv6 address, those projects which I have done, there is usually pretty much everybody goes with global addresses and that's it. Some handling may be for bogon or IPv6 special addresses but those are just a few and they have ‑‑ don't have the role like in IPv4 networks.
And maybe some special things being done with link local addresses, but usually it's just global addresses, which means from the ‑‑ from the table I just used, we only have one column that is global address, which have to be taken care of on the network borders, but which, given they are unique within a network, are ‑‑ don't need special treatment when it comes to vulnerability scanning or, say, incident response related functions.
Handling need for the filtering, no ‑‑ but no handling needed in the auxiliary security function space which means, it makes people's lives easier, when talking to security people or groups, that's one benefit of IPv6, as you get rid of all the extra effort you need for overlapping IPv4 space in your organisation.
So let's look at the implications once a shift from IPv4 to IPv6 happens from that angle.
IPv4, private address space, the reality in many enterprise organisations, doesn't need special handling when it comes to filtering, needs special ‑‑ operational efforter when it documents auxiliary security functions. This is not a case for v6. So the operational model, where to put the operation al energy nto from a security perspective, has to change. I save some operational cycles here, but I have to spend them in another place. That's a shift of, say, architecture and a shift of processes of where to ‑‑ what to look at.
In a dual stack environment, this would look like this and if one wanted to summarise, I think, not because the slide is created by myself, but if one wanted to summarise why dual stack is a pain, just look at this. It it means one has to run two different ‑‑ in most enterprise organisations, here assuming that private IPv4 space is used, one has to run two addresses on a single system which have different properties when it comes to their security handling. So it's not ‑‑ well, it's kind of double operational effort and different operational effort. So looking at this it becomes clear I think why dual stack from a security operations perspective is not a good idea.
Once you are in an organisation which already has a large amount of public IPv4 address space and you are like, well, I am used to this, I am already working with whatever type of global IPv4 addresses and once I bring an IPv6, not much does change, it's the same operational model. To some extent this is true, unfortunately, again, wearing my security hat, you have to think hard and deep, can I perform my security functions in the same way I did in IPv4? Vulnerability scanning comes to mind, this changes between v4 and 6, even in many organisations it changes, even with public spaces used before and after. Anything that works on black‑listing or reputation‑based basis might change. ACLs, once you runs a network which has a large number of ACLs in IPv4 just transferring this to IPv6 from IP might bring your memory to its limits. So it's into the easy but still from an overall operations perspective not much does change which is a good thing.
While we are at it, very quickly, just in one slide, getting back to this IPv4 Unicast extension thing and why I strongly doubt this is going to work, in v4 networks, this space is considered poisonous. If one wanted to use this in production one had either to go to this model, like consider zero /8 as it was kind of allowed or enabled in the latest Linux kernel, consider this private space which you use for your container networks, one would go to this model, consider this public space as some people have in mind like at some point IANA should release 24 ‑‑ 240, for public use, then it would go to like this operations model, right now nobody knows, so the operational model changes and you don't even know in which direction, again don't do this. This is not going to work on large scale in any complex network.
Coming to the end. Now, after what I have stated, I strongly suggest, and this is my experience from IPv6 projects, that you look at ‑‑ keep especially this one in mind when it comes, what does the advent of IPv6 mean, what does this mean for all the places where IP addresses are used within a systems lifecycle and for the associated operational processes? Enabling v6, like just bringing it to a system, that's the hardest part. This is, from my experience, that's the hard part and that's what you should focus on when you perform the transition from v4 to 6.
IP addresses have different properties, different properties which have an impact on the way how operations works, especially security operations. And that's what you have to keep in mind once, say, this transition happens. That's one where I can state where real, say, energy and intellectual energy is going to be used in to find out in which way does it affect all these operational processes and namely the security ones. Thank you for your attention. And it's time for IPv6.
(Applause) Questions or comments?
AUDIENCE SPEAKER: Firstly, I am really uncomfortable with you saying that dual stack is a problem because we all know if people believe that, then the answer that they are going to come up with is, don't do dual stack, don't do IPv6 only, just stick with IPv4 only. That's not what we want people to do. Can you bring up a slide with four quadrants, the IPv4 versus 6.
ENNO REY: Which one?
AUDIENCE SPEAKER: With the two, IPv4, IPv6, two red, two green.
ENNO REY: This?
AUDIENCE SPEAKER: Not this. So, the thing is, I think you are going a bit too quickly if you say private means special handling and public means no special handling. I think these are different things. There is private that requires special handling and private that doesn't need. You are also for getting about ULAs, so if you are comparing here the private IPv4 versus public IPv6, then of course if I'm on a corporate network, for instance, if I am wi‑fi with a device, then I get an IPv4 private address and IPv6 public address but the IPv4 private address is not a special use address, at least not any more than IPv4 ‑‑ IPv6 public address, it's just it needs to be NATTed because we don't have enough IPv4 give everyone on the wi‑fi a public IPv4 address, does mean that it has some magic security properties. Of course if you are deep inside your storage network or whatever, regardless of the type of address, even if it's good global Unicast IPv6 it still needs special handling even though it's public so I don't think this is correct.
JEN LINKOVA: Take this off‑line probably, just for sake of time. Last question. Quickly, if possible, please.
AUDIENCE SPEAKER: Short notice on your unwanted advice, which was somewhere in ‑‑ can you also extend that to, if you don't have a lot of IPv4 addresses and your company wants to become a digital company and you don't have an IPv6 strategy, start buying!
JEN LINKOVA: Your name, please, microphone etiquette. Sorry.
AUDIENCE SPEAKER: Free sew ‑‑
JEN LINKOVA: Thank you very much, and let's continue ‑‑ you might have noticed we got unusual level of activity on the mailing list, recently. Which is kind of good thing I believe, right. Some remember will remember they subscribed. So Jens volunteered to give us a short talk on why we should gall home. I am looking forward, I don't know what it's going to be about, actually.
JENS LINK: So, people, repent, you are all wrong, IPv6 is a failure, will not work. Okay, I am freelance consulting doing Linux, IPv6, networking stuff, automation and maybe you have seen this cartoon, I am the guy being thrown out of the window for telling to do IPv6 to people. They don't want to listen.
I finished my first big IPv6 project on June 5th 2012, in the afternoon, June 6th was World IPv6 Day.
JENS LINK: Launch. I helped launch for German larger IPv6 websites, same engineering team but different, we took about three months, not full‑time, so when the developers changed what they needed to change for IPv6, this changed something that looked at this was IPv6 already, so my problem, I don't understand why it's so hard to implement IPv6.
On October 3rd I sent an e‑mail. As you can see, there is not much traffic, as Jen mentioned, here and in the last month there is a little, yes. I was not alone. Wolfgang Zenker, the bearded guy, sent another mail starting a threat on the mailing list, which led to some large discussions. Erik asked: What did you drink? Just coffee and mineral water, at least on the morning when I wrote the mail. I had some ideas also, I wrote some ideas down, how the successor to IPv4 should look like. It must have classes, people always telling me that they have class A, B, C network. It must have DHCP and must have ARP, the ICMP thing is way too complicated. You easily can drop ICMP, it has no impact in IPv4. You must have numbers instead of dots, and absolutely no brackets around URLs, not my ideas, just what people keep telling me about IPv6.
So someone asked is there technical reasons for this? No. Just people who don't want it.
There are other problems. Vendor training. Well you only need to know about IPv6 if you want to get more than 90% of the test score. Security checklist, you have to turn IPv6 off. I ask one of the security guys they said true for Microsoft systems as well, he said yes aye ask him if he knew what Microsoft was saying about turning off IPv6. I sent him a link and he panicked and didn't talk to me again.
And there is some more ideas why people don't do IPv6. You can read them, stack overflow. This is from a Berlin university, the T stack stands for technology. They have lots of nice reasons why they don't do IPv6, just read it. Always good before you ask someone, what are your plans about IPv6? I found out it's easier to click on IPv6 excuses and get an answer.
So, back in 2012, we sat together for beers and asked the question: When will IPv6 dominant protocol on the Internet and 2020, 2520, 2030 were among the answers. I think the better question today is when will be IP addresses at €100 per address? I am selling at 150, so...
A couple of years ago, at a workshop, so whatever you do, make sure your hardware supports IPv6. We sought the software for a couple of 100 thousands of euros and it does not support IPv6 and we can't do it. Yes. Just listen, don't listen to me.
Sometime later we headed for a project the discussion with SE from vendor, you are using IPv6 in production? I sent this to non‑public mailing list and the moment I hit send my phone rings and someone was shouting at me "names, I need names."
Today, people are still building/designing IPv4 only software, hardware, networks, whatever. If they do support IPv6 v6, it often looks like they are not testing it, so I forgot the product but I think the lesson learned from, failed IPv6, the lesson we should have learned was, with IPv6, the lesson that was learned was oh, no, IPv6 doesn't work, so ‑‑
VERONIKA McKILLOP: It's not product, products, all of them.
JENS LINK: I am working to one special product, I don't remember which one. Veronika asked who of you is deploying IPv6, only 25%. So it is probably you. At least you are not being thrown out of the window for at least several times.
This costs me a beer. And one very last thing, one very last thing: Veronika already mentioned them, before the last RIPE meeting GitHub wrote, if you want to know about IPv6 just come to us and ask. Maybe they didn't deploy IPv6 then because nobody asked. If someone from GitHub is in the room could you please press big red button to enable IPv6. Thank you.
JEN LINKOVA: You know what, may I suggest, I will give my five slides which probably will be partially answered to this and then we go to open microphone discussion which will cover to both this presentation and the next agenda item. But Jan ‑‑
JAN ZORZ: For couple of moments I thought that was secret Working Group, it's not.
JEN LINKOVA: Secret IPv6 Working Group, I love this. It was perfect timing because I put this on agenda item before all these discussions on the mailing list started, so because we actually discussed what Working Groups are supposed to do at the last RIPE and Working Group chairs lunch and then we start getting as Ray counted more traffic on the mailing list in two weeks than six years. And it's been ‑‑ claim has been made we failed as a Working Group. So, and also, I still think people confuse us, right?
AUDIENCE SPEAKER: You look the same!
JEN LINKOVA: It wasn't intentional and I don't know how that happened. Look at us more carefully. Don't just ‑‑ so I think we need to clarify a few things. So, if we are talking about failure, let's look what we are doing now. I look in the charter and I was pointed out, I sent an e‑mail we need to update the charter, and fix the full sentence because it's not the next generation, it's current generation, and it's my action item, we will do that. So I look what we are actually doing. The charter is saying we do outreach, education, sharing experience and discussing stuff, and I said this group has been a nice place for people to come and talk about what they have done and I think it's actually probably a good idea, right? So if we mention in failure or success it would be interesting to know what success criteria should be. I think when I became a chair I suggested our end game plan should be get rid of this Working Group because we do not have IPv4 Working Group so why would we need a special IPv6 Working Group it should be default, it looks like there is a kind of confusion on the mailing list on actually multiple mailing list, what is our end goal is? I do not think it would be reasonable to expect we are going to get 100% IPv6 adoption everywhere and IPv4 turned off, because honestly, since 7 years ago I got complaints my router is sending some DECNET packets on some internet exchange. It was a default.
So, it definitely will be long tale, when I started it was 0.5% on v6, nobody cared. It will probably be 0.5 percent for a long time. I got a suggestion in my mailbox send some thresholds, like 80% websites support v6, I am not sure it's realistic goal because we don't have any control, at least I don't. So hats off, I actually think that we can consider just keep coming here, have a nice interesting talks and when people stop coming because they have deployed IPv6 we can consider it being done.
So, about measurements our success, there is actually better picture but you could not find the copyright, so we at Google see 30% of users coming over v6. People who deployed to all customers claim, and I talk to some other ISPs, about 70 or 80% of traffic shifts to v6. At the same time enterprises say they don't have any use cases for v6 and yesterday it was mentioned that internet exchanges see very, very little v6 traffic. There is an explanation for this because most large go from large ISPs to content providers, it's all private peering and maybe transit. It depends which end you are looking at this and measure, you are going to get completely different numbers, so I don't think we should be using 3% on Amsterdam exchange as discouraging number. I looked at the last version of survey, for why people do not deploy IPv6 and what this Working Group can do. So I kind of have some suggestions, right. Obviously we cannot give people time; if they don't have time to do, nothing we can do. No requirements, okay, I don't think we can convince people who do not want to be. However, sharing the knowledge and fixing the bugs or at least reporting the bugs is probably what this Working Group can do. And I try to summarise some ideas thanks to everyone who send me some Unicast e‑mails about we probably need to build a list of software which does not support v6, any volunteers for hackathons to get that fixed? Gap analysis. We have this or document we published for supports, we probably should be continue doing this. I really would like to hear ideas about what we can measure or do we need any additional measurements to help people with IPv6 deployments. If communities think they need education, well I know RIPE is doing great job on this and I am quite sure we can discuss here what particular topics those should cover. And we had the discussion on the mailing list about lead the industry and setting an example and doing IPv6 only wi‑fi, I think we should continue talking about this and finding the best way to do this.
However, please don't that I Working Group chairs are going to do all that stuff. I'm not. I am happy to help, I am happy to do some of it but now, I am looking for suggestion, ideas and, most importantly, volunteers. Now. Open mic.
ANNA WILSON: HEAnet and chair of the Net news Working Group. Net news didn't stop meeting because everyone stopped using use net, that happened after. We stopped meeting because the coordination requirements that originally required the group to meet at RIPE meetings twice a year at the time, went away. We suddenly had enough transit or enough peering capacity and we just didn't really a coordinating in the way we had to previously. Sometimes it's okay to go the coordination work we can do as a Working Group, is no longer as pressing as it was and now it's become a wider thing, the industry is going to do its thing and there are things that we can do in the RIPE meeting and RIPE as a whole, in the RIPE community as a whole, that might not require a Working Group any more.
JEN LINKOVA: Thank you.
AUDIENCE SPEAKER: Fernando ‑‑ from Spain. I want to say three things. One is that I don't think RIPE itself trusts us very much. There were two Working Groups, anti‑abuse and IPv6, and they put us in the small room.
Second one, and more important, I don't think it's ‑‑ I don't think it's a ‑ problem, I think it's a ‑ problem, I am not ‑‑ survey valid we trust and we know it, if IPv6 is not deployed and we end in a wall of NAT over NAT over NAT, then the Internet as we know now will disappear and we will beto bigger corporation of Amazon and Google and everyone ‑‑ every one corporation that wants to sell this product and we will finish in some kind of ‑‑ Geoff Huston some years ago, wild garden. We will be, in the end, customers of wild gardens and the Internet as we know will disappear.
And last thing is what the proposal of this group, if we don't success or we don't survive, we will be a group of anonymous colleagues because we didn't survive, and if we have success, it will be the same but we will have a lot of problems trying to all of customers to be on IPv6, we thought routers and applications, in the end we will need a lot of psychological report. Thank you.
AUDIENCE SPEAKER: Hello, from the pretty mountains of Switzerland. I have two things to say, and in general, when it comes to IPv6 deployment I have very strong opinions. There are only two things to fix: One of them is connectivity point and the other one is content point, just fix those two and we are done.
JEN LINKOVA: Are you volunteering?
AUDIENCE SPEAKER: Absolutely.
JEN LINKOVA: Good.
AUDIENCE SPEAKER: The point is, connectivity by some of us are actually working network operators so the question is what can we do. One of the answers is you can influence your work every day. When it comes to the content side, I have a bit of like radical opinion there, and that is every one of us here is a private person, we are not ‑‑ we are all working maybe for somebody, some of them might not be working, it doesn't matter, we all have something that is very common, Open Source world actually, we have some free time or spare. What we can actually do, like every one of us here, we can build IPv6 only services, products or games. Think about it, like there is this really nice game out there, or being built, I heard, that IPv6 penguin jump or run, runs only over IPv6, and all those small things is something that every one of us can do and so that's something I would recommend.
And the last point if you want to have an IPv6 sticker ‑‑
LEE HOWARD: I am from IPv4 dot global. It seems unusual to say that. I like many of the suggestion you make, Jen, those are really brilliant because they are mostly mine. And there are a couple of those I think I can help volunteer with, for instance actually the one that you didn't have an arrow on saying making the business case you don't think we can do that. I have been doing that at NOGs for ten years so happy to ‑‑
JEN LINKOVA: How do you measure success of this, that's a question I have?
LEE HOWARD: That is my business, isn't it? Which I suppose brings me to if Jen's selling addresses for €150 I will sell you for half of that.
JEN LINKOVA: No marketing.
LEE HOWARD: On the hackathon ideas, collecting Open Source projects and even maybe the suggestion about creating new programmes, new apps that are v6 only I think I can at least put together a Google sheet saying here is a list of things that need contributions and if we can get some logistic support to do a hackathon here on some of those projects I can at least help if somebody will help me out.
JEN LINKOVA: Thank you. It's nice we have a meeting minutes, it probably will be the most interesting meeting minutes I ever read at RIPE because there is always some action items.
BENEDIKT STOCKEBRAND: Speaking for myself. A couple of things. First, we have a killer app for IPv6, we do it's called IPv4, and in Germany it's basically on land lines we have at least 20% of the users in Germany connected through DS‑Lite, so if you turned off IPv6 there, their IPv4 would die immediately. It's something to keep in mind. So IPv6 is there, no matter what it looks like.
Another thing, Jen. Could you go back on that slide with the excuse one more, I think. That one. You see the bottom line there, I find it hard to believe that the ‑‑ being ‑‑ that problem of having to convince the decision‑makers throughout V ‑‑ to have v6 is actually the least of our problem ‑‑ the least of our problems, these days. Today, the biggest problem with IPv6 is actually us and our colleagues, and convincing all that whole bunch and that's something we should actually work on.
JEN LINKOVA: It's actually top line, people do not see the case. I think we run out of time. Thank you very much, don't forget to rate the talks and see you in Berlin. We will continue mailing list. I actually think I might ‑‑ when the meeting minutes out, I might summarise the result and might send survey to the mailing list later. Thank you very much, thank you to everyone for participating. Rate the talks.
LIVE CAPTIONING BY AOIFE DOWNES, RPR