Open Source Working Group
RIPE 79
16 October 2019
2 p.m.
MARTIN WINTER: Good afternoon, everyone, or good morning for the ones who just got up. This is the Open Source Working Group. I welcome everyone. We have quite a full programme again for lots of fun and really interesting talks. So, let's start.
Before we start, first one thank over there we have Robert who is the chat monitor. We have Vesna who is like doing the fast typing and doing the scribe. Thank you and good luck.
Now, before we have ‑‑ you probably have seen the agenda, there any request, any agenda changes, anything we need to add? Okay, I don't see anything. The previous minutes are posted for at least a few weeks on the RIPE website, I hope some of you had a chance to read it. And before we start, we also, if you may remember once a year, we basically have the re‑elections for the Working Group Chairs. That's always before the fall meeting, two months before this meeting, we send out the requests for any nominations. We did not receive anything, and Ondrej and myself we are happy to continue it, so, thank you if you want to become a Chair in the futility, we have next year before the fall meeting.
And with that, I'll introduce myself. I am Martin Winter, I am one of the co‑chairs.
ONDREJ FILIP: I'm Ondrej Filip, I am also one of the co‑chairs. Let me add to what Martin has said, something about the agenda. You know, there is a reason why they are putting us just after the lunch, because you know that usually if that would be a normal thing people would sleep, you'd be bothered and checking e‑mails. With our Working Group is the best, so the agenda is the best to we will be awake and we will be listening to the great presentations.
So we have four presentations today. We wanted to have some lightning talks at the end but there were nobody who requested that, so, no lightning talks today.
The first presentation will be from Leslie Daigle about the collaborative Open Source MANRS validator. Then I wanted to go through that in a moment. The second presentation will be a recent development of DRB validator from the RIPE NCC. Then we will a presentation about the next generation initiative. And the last one will be automating network using Salt without running proxy minions. That's the agenda, and I think I can start. Leslie, please come ‑‑
LESLIE DAIGLE: Thank you. So, I was told that I should probably introduce myself a little bit and I didn't really know what to say. So I should say that thinking cat enterprises is me, and all you really need to know about that is if you do a WHOIS look up of thinking cat dotcom it was registered sometime in 1996‑ish. And that's about all I'm going to say about me.
So, what I wanted to talk about today is collaborative Open Source pilot project that we're doing. Apparently validators are really cool today and maybe it should have been so many various so little time. But chief what I wanted to talk about is this notion of collaborative Open Source software. Then a little bit about the MANRS pilot project that we're working on, and then a point or two I hope some further discussion. And really the discussion is what I'm here for today, so I hope we will be able to do something useful there.
So, collaboration in an Open Source software. So I mean, Open Source software is collaboration, isn't it? And the reality is, as I'm sure everybody here is keenly aware, I need to go back ‑‑ it can be very collaborative or at least involve a the low of people. If you are working on a large project framework that has an established platform and a governance system and many people participating, and contributing and so on and so forth.
It's not always so successful with little projects. You know, I'm sure at least half the people in this room have had a great idea for a little project that could help improve the Internet. And maybe you have gone and done some work on it at a hackathon here or elsewhere, but often times, the kind of input and feedback you get is kind of like crickets, right. It turns out that if it's just you working on something, sometimes it's hard to get a bit of an attention on what you're doing and sometimes it's hard to find the people that really should be workinging with and giving contributions with some notable exceptions of people who are very good at doing that in this community. I'm not talking about Job.
So, the hypothesis is for some of these good but small ideas, they are more likely to happen if there was a framework, something a little smaller than the Linux foundation, a framework in which they could sit. As a test of that hepatitis sis, we are doing this pilot project for the Neill assured norms for security routing effort. Which you all know about, right? Well you will by the end of the week. And so part of what we are looking at is through helpful coordination getting more people involved, going from an idea to concrete and tractable proposal and hopefully attracting some resources to implement and generalise and expand. This is app generic sort of arc of the optimistic outcome.
So, then moving on specifically to the MANRS validator pilot project. So, the MANRS project at this point has I think over 200 participating networks. So a large group of engineers from many different networks across the globe who are committed to making sure that their networks are managed well in a routing security context. And to promote the same for others. Great!
MANRS provides app manifesto, saying here are the specific points that you have to do in order to ensure that your network is, you know, properly maintained from the standpoint of routing security. But it's difficult to measure compliance objectively. There are projects out there that can sort of retro actively identify whether or not networks have complied with the manifesto requirements, but there is an idea of how about having a tool that could validate router configurations to in advance before you actually turn on your, update your routers to validate that they actually would be compliant with the MANRS requirements. So this is one of these situations where there was somebody, Richard come tonne, who had a really good idea and proposed it in a context where you know everyone else is like minded in thinking that routing security is important. And crickets is exactly what he got. Silence. Nothing, you know, weeks go by in the mailing list. Nothing.
So, with a little help, a little bit of project coordination, okay, now here is a vision statement and some concrete tests and now we have a preliminary implementation which in full disclosure witch did with his colleagues but he did it once there was sort of encouragement of people might want this, and there is a framework in which we can have a discussion about it.
So, the tool, the pilot project tool, is implemented with robot framework and it has specific tests for each router vendor to test for MANRS compliance. And here is some sample output. It exists. It runs. It was Rich Compton which provided this screenshot which is almost legible. But as you can see, that the tool works with the router configuration and performance the stated tests against the configuration files to see if it works or not. It passes or fails.
And this one that you probably cannot read, but hopefully you can read on the online version, is a screen‑shot of you know what are these tests actually look like and what are you actually testing in order to see that your configuration is compliant?
And I think I'll leave this here for a minute so you can ponder what sorts of things you might agree with or disagree with here. If you are really paying attention, you might or might not agree with am so of the ways in which the specific tests are set up. You may also be thinking about things like well, that's great if you know, if all the tests need to be performed on one router. But if you actually have different routers at your edge performing different functions necessary to comply with MANRS is this going to work? And I really hope that you are thinking those kinds of questions because in a little bit I'm going to ask for some input.
So what we have in in this project is a first implementation. It's currently set up for JUNOS and it's also set up assuming a single router reflects all the configuration necessary, which is limited, but it's a start and so what we're looking for at this point is some sense of is this kind of tool useful to operators? I mean, you probably don't need it in your network because you're already on top of it and you have got all your configurations sorted so that you're compliant with MANRS, but you can probably think of a few networks in the world that could maybe make use of this sort of a tool and maybe you would like to help ensure that the tool is useful and works well for some of those other networks that may or may not attach to yours.
So, that's again, that's a hypothesis, that's a question, definitely would like input on that front.
Other areas in which this project could use some input is what are the other vendorers that could be useful to support? Is MIkroTik a person to be helped with. How do we support this project to ensure that it works in different network scenarios.
So there is all kinds of opportunities in terms of moving this project forward.
I don't yet have a signup sheet and that's largely deliberate, but I'm here through tomorrow, and there is even app MANRS table today. So, if you talk to one of us about interest in this particular project that would be great, we can get you connected, and I certainly hope that there will be comments and questions at the mic.
But I'm not done yet.
What I have tried to create here is sort of an arc of if we collaborate we can do better things together particularly in the interests of moving the Internet forward. And clearly there are useful tools to be built that maybe are going to suffer from that, you know, somebody has a good idea but they don't know who to talk to to work with, to help move it forward. I mean, I know a lot of that happens here in this Working Group, in other Working Groups within RIPE, even just in the hallways. But a little bit of coordination can go from, yeah, I'll work on in a really cool idea some day to an actual project.
And there are a lot of ideas, good or otherwise, that are being implemented and built today by people who are willing to put their hands on key boards and write code. And they may or may not be operators. So, another theory is that it will be better if operator, network operator input was actually included as part of a design and build process to ensure that requirements are understood, and to ensure that it's usable in the end and to prevent the wrong sort of expertise dominating in the technical work.
Because you really don't want the wrong sort of expertise dominating in the technical work.
So, that was the pilot project. And looking a little more about across industry collaboration on projects for operational issues. But I'm also interested in understanding better what does and does not work for operators with Open Source software in general. In terms of using it, in terms of helping to build it, in terms of supporting it financially. There are various theories and rumours abounding about whether or not operators actually engage in Open Source, use it, revile it, can contribute to it, don't have the time for it. So, instead of just letting those theories abound, I have questions. I really shouldn't have called it a survey. I have questions, I hope that some of you have some answers, because I'd really appreciate some input on that information gathering exercise at the link here.
And I think that's actually all that I wanted to say unless there are questions?
ONDREJ FILIP: Are there any questions? We have plenty of time so don't be shy.
AUDIENCE SPEAKER: Hi. Blake. Thanks for putting this together. Just as a comment, which means I would normally go to the back the line, but since there is no line I will make it now. As a DB we may be a tier 1 carrier but our operational team is fairly small and something that would definitely facilitate stuff for operational organisations such as ours is kind of a buy a feature type thing. If there is a really easy way for us to participate in Open Source projects by just having like ‑‑ maybe if it's not necessarily public but at least say you know, what is your hourly rate or daily rate or whatever if I want to add my features into this project, make it as easy as possible for manager to do so so I can just fill out a PO and get it done. Then that really helps.
LESLIE DAIGLE: That's a really interesting point. When I put together a list of questions, I did try to suggest would this help and that help? I would say that that's not one of the things that's on the list. However, there is an "other" button, so, if you had the time, I would certain appreciate if you could fill it out and add that because then it won't just be that I'll remember it but when I publish the results from this, it could be clear to people that that would be us useful.
And I should perhaps make that clear too, that this particular set of questions that I have, it is data gathering. It's not just data gathering for me. It does say on the website that I planned to put together a report so that we all can hear what do different people think about Open Source and operations.
ONDREJ FILIP: Okay, any other questions or comments? So. Thank you very much Leslie.
(Applause)
So please, try to help Leslie with the survey. And next speaker is Mikhail.
MICHAIL PUZANOV: Good afternoon everyone. This used to be a presentation for sort of internally, inside of the company. It's rather technical. It's quite catchy named. In principle, it's sort of for software engineers and system engineers and at some point we decided that it's quite probable that it will be interesting to the broader community, not only inside of the company. So, that's basically ‑‑ and a lot of people from the community was involved in the testing, the software, so they probably want to know what happened to it actually.
Some or RPKI validator is part of the suite that RIPE NCC provides since 2009 probably or so. The specifics about it is that it's basically the only software we have that runs not at our premise. So it runs somewhere next to users routers, so that's why we have to be pretty diligent about taking into account all possible cases, thinking about resources, some range of platforms, we had people running it on open BSD for example, we had to think about that as well. It also supposed to be sort of easy in terms of configuration. We don't want people to, I don't know, install a database next to it so that ‑‑ or a proxy or stuff like that, so basically that's the specific of the project from the software point of view.
So, what it actually has to do? What does it have to do? Not so much. It doesn't look like some big data or things like that. It's about 60, maybe 65,000 objects now. Not so big ‑‑ a little above 100 megabits in size. But this is a rolling data, so it's constantly being updated by five trust anchors at the moment. And in a week you can 2 to 3 big gigabits of data which are stored locally in the validator, so that it can be sort of smart about about what it's doing in case some repository are not available at the moment.
So software, we're having about 900,000 BGP announcements floating around. And approimately 10% of them are signed. They have corresponding objects. So in principle it can up to ten times more in theory. Of course it's not going to happen in practice, but that's the theoretical limit.
And performance‑wise, currently historically it's been using a Java Crypto lib RIS, which is bouncy castle in this case, industry standard. It takes about one‑and‑a‑half, two minutes to do the whole Crypto validation on modelen CPU core, one core. But it's work that can easily be parallelised so it's not really so sequential.
So, the current version is validator 3. Which it's been around for about a couple of years I think. And it was a replacement of validator Version 2, and somewhere long ago there was validator Version 1. And the reason to replace it was basically some stability issues, out of memory problems, we had complaints even up to the most ‑‑ the latest version. The data corruption, very rarely, but it still happened. High memory consumption. So it stores quite a lot of data in the memory. So I think for the current version recollect it's about three and a half, four gigabytes to just run it, so in Sweden in Scala, which is a quite exotic language for people I think so we didn't expect much of the pool requests or contributions from the community.
So the decision was let's rewrite it and make it more accessible so it will Java 8.
Fine. So, we picked up pretty default very, very classical technology stack which is usually associated with any Java project more or less. The most important here is that basically we picked up the H2 with the database with the old object relational mapping on top of it, and all this kind of stuff. A lot of data from the memory went to the database so that we can make it the consumption more acceptable for the users.
The whole project was about nine or ten months. And of course, there wasn't perfect. Suddenly. We started to get problems. Started to get first bug reports. Even though we managed to get the memory in the limits of 1 gigabyte, which was much better than the Version 2, we still got regular out of memory problems. We got quite a lot of complaints and bug reports about it sort of doing nothing, starting and doing nothing, or crashing and then not doing anything after restart and things like that.
We had one bug report from someone who had enough patience to wait until the database grew up to 50 GB in size before actually thinking that there was something wrong about it. So, this sort of behaviour issues. And of course we started to fix it all, and basically we were busy with this the beginning of the year I guess, yeah, beginning of the whole string more or less. We found a lot of specific technological problems which were basically the hibernate, this operational realisation mapping. It uses memory in quite an arbitrary fashion, which you don't have much control over. It makes a lot of things very slow. It's pretty hard to get right the transactional ‑‑ trust anchors management or concurring the concurrent work inside of the application which we had a lot actually there, because of running a lot of parallel processes inside.
The H2 database. That happen wasn't probably the best choice, because it wasn't very good at recovering after crashes or just restarts. And probably the worst on the feature level, it doesn't have online compaction. So if you restart the application it will shrink, but otherwise it only grows, or maybe not, but it's not clear from the documentational behaviour or things like that.
Basically, that was the situation. A growing amount of people starting to use it but, or at least try it. Noticing bugs. We had a few months ‑‑ well maybe three months, four months, something like that, when we almost every work had to fix some bugs in the validator. So, at some point the situation was we needed to change something fundamentally and apparently we needed to change this embedded database.
That seems to be the core of all troubles and the performance issues and other things.
Okay, so, what can be the opposite of the fancy object mapping on top of something? It can be something very simple. So in this case, LMDB, I think a lot of you know what it is, it's a very simple key value storage, it's been there for quite a while, and it's proven to be very reliable in performance. So there are some Java BINDings for it, pretty low level, but it has all the features related to transactional guarantees. It's also very efficient, there is basically no caching, no writer head logs, no separate compaction, garbage collection, things like that. Very easy crash recovery. Basically, very ‑‑ it's ‑‑ a library without maintenance pretty much, and it's also very fast. That was the initial choice.
And indeed after about a month of work I guess, somewhere about, somewhere around IETF in March or April, when it was, March, it was implemented. So literally every database operation became faster. Some of them ridiculously faster from, I don't know, 200 times, 500 times faster, something like that. This is a literal quote from the log which looks pretty impressive so we sort of insert 35,000 objects in the database in this millisecond times. Very good latency. So, CPU is used quite extensively by the validator because it does a lot of cryptographic things. So, even in this case, pretty responsive smaller database, quicker, no situations of crashing and not being able to come back up or something like that.
We were able to actually shrink the amount of memory it uses from 1 gigabyte to ‑‑ initially to half of a gigabyte I think but it was a little too small, so, it stopped the 640. So basically a lot of problems disappeared at once. And of course this kind of things don't come for free. LMDB is very simple, it's just, very low level. So the database basically doesn't know what's stored in it. So it doesn't have any met database, no schema, no schema migrations, nothing at all. So we had to implement this all ourselves.
Serialization of data, indexing, some sort of high level API or something like that, which basically resulted in quite a growth of the code base, so about 35%, if you look at the source code lines. And we also had to carry around the native library and the dependencies which is not a big deal but introduces a little bit of inconvenience in terms of packaging and distribution.
And then this. So, we managed ‑‑ we somehow got into this bug with the data corruption. Very rare, after a few days of work, three days of work of the validator, we get one of the objects in cache, one or two objects out of 200,000, wiped with zeros, and that was pretty hard to reproduce. It happened on Linux more often, Mac less often, we had FreeBSD users and it never happened there.
So basically we sent the whole month trying to figure out what's going on and where exactly is the problem. We pretty ‑‑ we were pretty sure that it's not LMDB itself of course, it's very well bottle tested thing, so no.
We believe it's somewhere between the GVM memory and this memory mapped files. Somewhere there. Some misunderstanding between these two things. But in any case, we didn't try to investigate this further.
The only positive effect was that we basically, while trying to fix the thing, we minimised the amount of modifications we do to the database, to the bare sort of necessary minimum,
But in any case we had to move further and go for some other storage implementation.
Which was found in this case. The thing called Xodus, written by jet brains, purely Java implementation, so slight positive thing that we don't have to carry around a native dependency. Backup there is also key value store a little bit more sophisticated, but we didn't need that essentially. The same transactional semantics more or less. The performance is reasonable. It's slower than LMDB. We had some measurements, two, three, maybe five times on average but not a big deal because by that moment, it wasn't already a bottleneck.
Somehow we managed to have even smaller database. And actually, rotating a lot of smaller files and it makes better usage of the space. So, right now, I think we navigate about 2 gigabytes of space or something like that.
The only problem we found is that we had to increase the Hib size because for big writing transactions it uses memory proportional to the amount of data in this trust anchors. So, basically, the current version distributed with one‑and‑a‑half gigabytes of Hib by default but it's sort of just in case, most of it is never used, it's there for some cases of memory usage.
So we haven't found, noticed anything exceptional about it, no bugs, no corruptions, no problems, nothing really, not great, not terrible, and after testing for about a month, it ended up in version 3.1, which was released I believe in May or June.
Some lessons from that.
Do not use hibernate. Better, never use it at all, or don't use it if you have to think about performance in principle or memory usage, even though it's still very common when people right something in Java, it's not, just don't. H2, probably not for this purpose. It's kind of nice, but not really reliable for such kind of use cases. Even LMDB initially was very nice, there is apparently somewhere, some bugs when you use it with J VM. We managed to get it. Other people didn't. It's sort of not very reliable situation, so, in the future, we probably will not try to do it again.
We had to do a lot of performance measurements, really a lot of different performance measurements. Most of the time we thought oh this is where it's low. It was wrong, it's slow somewhere else. So, that took quite a chunk of work actually.
The other issue was probably that we relied on sort of big chunky frameworks too much, and ended up in a situation when we were not really ‑‑ when we don't really know what's the system is doing at the specific moment. So, more research, less sticking to the default, even though it's supposed to be good.
And finally, so, we have RPKI validator 3.1, it's pretty stable. We haven't seen any stability bug reports from anyone so far.
There is not so much work planned on it at the moment. We're going to work on packaging and usability things here and there, improving documentations and other small things. Maybe compiling it to a native binary, that's also depending on how much time we have this year. But in the future, yeah, it's still the thing to be defined. About the future of the project.
That's it.
CHAIR: Okay. Thank you very much. Are there any questions?
AUDIENCE SPEAKER: Anton Prado. Quite happy user of RPKI validator Version 2. So my question is: Did you ever consider the idea to leave, to escape from Java?
MICHAIL PUZANOV: That's the idea we are considering the last half a year or something, going back and forth about it. Maybe. It's pretty big project. It's complex, it needs for consensus from the community what we are spending our effort on and money and all that. So, yes, of course, we can play around with validators in different languages and you know, we can do that, but it's more of a matter of priority.
AUDIENCE SPEAKER: Alex Band. I work at NLnet Labs. I want to say thank you for all of this work that you have done over the years. This was new. Technology was new. A lot of work had to be put in to get this off the ground, and sure, this presentation is a lot about stumbling, but I just want to commend you and all of the RIPE NCC for sticking with it, because if you look at RPKI now, the up take the maturity, the interest that there is, it's all because of that work that you put in. And I think that needs to be said to you.
(Applause)
AUDIENCE SPEAKER: JD Pojet from an IXP in France. We are using your validators for years now, and I was wondering what is the best channel to get announcement about the new release, new bugs or whatever.
MICHAIL PUZANOV: Good question. What's the best way to get the announcement?
AUDIENCE SPEAKER: Yeah.
MICHAIL PUZANOV: I don't think we have a very reliable channel for that now. So ‑‑ we tried to keep the change log basically, so probably the best would be to look at GitHub, which, yeah, it doesn't sound very convenient way to do that but if you look at GitHub, most happens there I guess.
AUDIENCE SPEAKER: Is there one of the RIPE mailing list suitable for having news about the process.
MICHAIL PUZANOV: There will be, I think big changes will be on the mailing list, yes. But small bug fixes, if there is somebody has an issue and then we basically fix the issue and then we rely to this person, we close the issue, we put the line on the change log and we roll out the update. If you manage to have RPM based system you just update it and you get everything automatically. If it's not a break in change, which is up to my knowledge never is really.
So, that's the smoothest way I guess. But if it's a big change, it will be on the mailing list, yes. But we have to improve that. That's sort of communication, yeah, that's indeed not very centralised.
MARTIN WINTER: I see no more questions. Okay. Thank you very much for your talk.
Next up we have Mikhail from NLnet and talking about some fancy new initiative there I think.
MICHIEL LEENARS: So, thanks for having me. So, I want to quickly take you back for the people that don't know. NLnet, back in '82, some nice people started doing some networking here in Amsterdam and basically an organisation grew out of that and just to clarify things, there are two different organisations. There is NLnet stitching and NLnet labs and we both changed logos. Meanwhile one is from 1989 and one is 1999. Alex who just spoke is from the other strand of the organisation, they are of course as you see some historical similarities, and liaisons but it's no longer the case.
So, 30 years ago, this year, at this moment Tim Berners‑Lee put out this proposal where somebody from management wrote out a fake but interesting, that's the same year NLnet was founded after seven years of informal operations. And basically, well, everything worked out and now we are here and this is what at this moment Berners‑Lee said when he got his award is that the web has failed to serve humanity and that it's become a large scale immerging phenomenon, which is anti‑human. This is a quote from last year, so I had to adopt the quotes slightly.
So that's this, that's this thing, probably people recall this from the old maps from 2001.
And of course, if somebody like at this moment Berners‑Lee says that, then we take on our optimist hat and say well today we create the Internet for tomorrow.
So, we got contacted by the European Commission in 2016, hilarious story, I won't tell it, but ultimately, they took up their plans to basically do things differently. They had this future Internet programme for decades and really not a lot had happened, so they wanted to completely reFlash the whole thing, and we got involved and we helped them write together with the team from Gertner, we helped them to write a new vision for what they should be doing and it's quite an ambitious high level idea basically that's what happens when the European Commission has to do something, because if it gets too concrete and they start funding block chain projects.
And we realised that this next generation Internet initiative, if anything, such a thing could ever take place is orders of magnitude bigger than travelling to the moon. Because the Internet is the biggest thing that we have ever made, and IP mean you can fill up many rooms of people like this with people that are doing on a daily basis, keeping it up and running, but there is also 3.8 plus billion people using it, and some of the biggest economic actors, some of the biggest political stakes from China, Russia, US, everybody is trying to do stuff with T it's completely overloaded expectations. And so, well if want to do something of course there is no time to lose.
A nice historical analogy to take is, that guy is the guy who took the Americans to the moon 50 years ago. He is also the guy that bombed that city, or is it that guy which is Robert Openheimer which was the guy quoted by Tim Berners‑Lee when he talked about the dystopia that he had created. Actually, I had forgot and it was non intentional, I forgot who created the mess in the middle but cleaning stuff up is the mission that we have as app community.
Something like the European Commission starts moving, it's like a tanker. They draw up huge plans and they have lots of different tasks that they want to fill out. And obviously, it ends up like this. Because it's just fragmented and there is no point in doing it that way.
So, that's why we constructed this whole high level vision, because it gives kind of an easy thing to look into and we split up the goals for the programme into three different things with the high level thing it being beneficial to mankind, it may sound cheesy, but it is something that is not a given in the system that is currently in place. We're currently not heading that way.
And of course we want to make the Internet more resilient because ultimately I know people that have heart values that have linked to the Internet via Wi‑Fi is literally going to the bodies, going into space, going into the road, so we need to be more resilient. We need to be also transparent and trustworthy, and for that, there is a whole work that needs to be done. Because there has been, after Snowden and so on, we are quite a lot of problems all the way down to the lowest layers. And then of course it has to remain open. We can't have a single, or an oligopoly ‑‑ a very large actors taking over control of the rest. And that is if we want to create diversity. If we want top maintain the global diversity we have had in Europe, and we want to promote well‑being, this is a high level goal for this programme.
We got involved and basically this is our vision of how things should be. So, there was earlier when Leslie gave her talk she said somebody has a cool small idea and he is out on his own and all the cricket are chirping and there is nothing happening. This is our instant team. So people can come to us and ask us for money and then we will give them an instant legal team, a community building team, we will give them a documentation team, a localisation team, give them a standards, a route to the standards community, we give them a security quick scan so we do all these things to help people get up to speed.
And it's, by design we want dog food on our own promises. So if we say the security is going to be secure and for everybody, then every project we produce we have it tested by the Dutch official organisation thattest its for accessibility, it's going to be certified so that blind people had obviously use it. Because blind people want to be secure and safe and use the Internet too. So we are very geared towards deployment. We package all the software in a reproduceable way. We have this whole mechanism to support you.
And that goes for two of the funding schemes that are inside the NGI scheme. The search and discovery and the privacy technologies. And the whole idea is it should be as easy as writing to a mailing list to apply. The whole idea is to be lightweight and flexible because you can't, most of these funding schemes you have to do a pitch perfect pitch and then it gets through the man re or it doesn't and if it's too focused on achieving practical results it can't, and our whole funding scheme is open standards, he were Open Source, open hardware driven and we want to get as low as possible cost for the people that apply so at least when we don't give you money and cannot give you the support, you wouldn't have wasted much time.
Because we notice that people were spending literally cumulatively they were spending years in order to apply for other grant mechanisms and we didn't want that.
And we help people to make a good idea. So if they come in and we know that other people are doing something very similar. We inform them and they can prepare the proposal, which is kind of unique this this kind of funding scheme. And like I said, we have this whole team of really good non profits ready to help out, and also, for ourselves to be consistent, because that's a critical thing is, you can't make claims and not follow‑up on them. So you can't say the Internet is going to be sewer and then you give somebody money and they come back and nobody has even looked at it.
So, because that's just not going to fly in the real world.
And well we do lots of, we have lots of things, I'll not bother you with these ‑‑ the whole idea is you get an instant team around the idea that you have. We are able to divide 5.6 million euros in both of these schemes, search and discovery is about making stuff discoveriable and this can be from alternative naming schemes all the way up to activity hub and higher level standards up in the web stack. Privacy and trusting technologies can handle anything that's fundamental. This goes from the silicone phase, all the way up to protocol verification, and alternative browsers, it's a huge load of projects. We currently have 47 search and discovery projects and 77 projects inside privacy and trust analysis technologies. We are quite proud that we are able to get people that are not part of the funding beast. So 90% of the people have not received money from the EC. Before they are just normal people like yourselves that want to do stuff and we are not really interested in people that play innovation, we just want people that come and fix the Internet.
Because this is to us, the missing ‑‑ the sort of the missing puzzle pieces inside this whole ecosystem. These are people that don't per se need money. They want support. They want to achieve stuff. They want to be amplified, but they don't want to beg for money and do this rat race with the block chain mobile analyst I can people who who gets to write the sexiest proposal.
That's basically the offer that we have to make, is ‑‑ and I know it's kind of blasphemous because you already kind of work for the Internet. But if you are not happy where you are or if you know people that are not happy, it's a full‑time, part time paid work from wherever you want to be. We'll give you the money. You get to collaborate with some awesome people around the world. And the calls are open.
So, the next call is actually December 1, and this is a quick glimpse into, well, you saw the dead tree that was falling down. This is the live tree, these are actually live data ‑‑ it's live ‑‑ it's a screen‑shot from our live data. So we try to map everything on the existing map that we have and see whether stuff is happening or whether stuff isn't happening and we hope that people will pick up on on this and say hey, there is a bit of a dead branch around somewhere, the legacy middleware stuff isn't taken care of in the R&D, so let me pick up on that.
So that's the whole story. If you want to be part of this, all you need to do is go on to these wonderful short URLs, and then you can read all about it. We can hand out amounts between €5 and 50,000. So it's a fair amount of money. The whole procedure takes roughly seven weeks, seven, eight weeks to complete from doing an application if you submit from the deadline. And well, we're looking forward to any applications. That's it.
(Applause)
I actually forgot to put something important. So, apart from giving away money to really cool people that you could be such a person, but there is also over 100 projects that are doing really really cutting edge awesome stuff. For instance, the Verifpal is a new project doing symbol I can verification, this is a project that allows students to do this in two hours. So it's really geared towards engineers and practical people that want to do symbolic verifications and see if the protocols they are proposes make any sense and have any security flaws. And there is a a gazillion of other protocols from Wireguard to the Risk 5 Open hardware project. So if you have any spare time, or an interest or a professional interest in this, do have a look, it's on the same place, the website and we have projects being added all the time, so it's a great way to spend your holidays for the next years to come.
And that's really it.
ONDREJ FILIP: Thank you very much. Are there any questions?
AUDIENCE SPEAKER: Robert, from the NCC I have app comment from Jabber.
From an anonymous person, and the comment is not a single word about being environment friendly, sustainable for example, choosing technology that reduces energy consumption.
MICHIEL LEENARS: It is part of the sustainability. That's our third goal. We have some projects on eco friendly CPUs. The problem is, and that is the issue that we have with the EC funds, everything is chunked up. So, our chunk is privacy and trusting house technology and search and discovery, if we don't have ‑‑ we can apply these principles like ecological awareness, so we have, for instance, one search engine that project that will be able to show you whether or not the website that you are about to visit in the search results is eco friendly or not. So, we are very aware of this need and we try to accommodate as much as possible. So, ethical search engines, improved efficiency hardware, it is definitely something that is on our place. If people have any ideas to improve global routing and make it or more energy efficient, please propose and we can see if we can either handle it ourselves or if we can reach for another sister project of ours to fund it.
ONDREJ FILIP: Thank you. And I'm trying to find the last speaker. Here he is. Automating networks using Salt without running proxy minions.
MIRIEA ULINIC: Hello. Hi. I am very happy to be here, thanks for accepting my proposal, it's always a pleasure to be here in front of you. It's a bit humbling and we had very good speakers and interesting topics, I hope I will keep the level up.
And I'm representing this company Digital Ocean, we are a pretty young company funded in 2012, only seven years old and we are already the third Cloud provider. But what is interesting from a network perspective is this number. 12. We have 12 data centres around the world. It's not so much spread out geographically, but all these are pretty well concentrated and about a few hundreds of racks in each data centre and all of these also interconnected through a global backbone and what this means, translates to a few thousands of devices we have to manage to make this Cloud work.
And of course, we don't do this manually. The one reason I'm here in front of you is speaking about how we do this and also, how you can do this yourself, plus some new features that might be of your interest.
We are using this automation tool called Salt; The reason being that it needs to scale to our needs to be able to manage thousands of devices distributed all over the world. And Salt presents some advantages for us. Historically we have been looking into other tools and eventually we migrated to Salt because it offered all the advantages we needed for our environment.
And a quick look at this tool. Salt. It has typical architecture. We have a master, and one or how many minions you want to have. One master generally can scale up to 30,000 minions, and what is app minion is actually a software package that rounds the server, you typically want to manage on the machine. (A)
Also, it's not only one master, you can have multiple masters, you know, which the minion can interconnect to these masters, complete or incomplete closed topology.
From typically install this.minion on network devices, most of the network platforms, the old ones, especially can't really install custom software on them, so this is why this ‑‑ it has been introduced so another package called proxy minion that allows you to manage I don't recollect devices, and instead of having to install this package directly on the device, it's a software product can run anywhere.
And the idea is that the interface between this master and the network device and the communication between the proxy minion and device can be over whatever channel, it can be over net could have, http or anything that the network device supports.
In brief this proxy minion has the advantage that it's just a process that can run anywhere, but it needs only two things. It needs to be able to connect to the Salt Master, and also to be able to establish the connection with the new device using the API of choice.
However, saying this, there are many types of topologies due to the flexibility that it can run anywhere. I have seen deployments on a single server or distributed on multiple servers where we have, so we have this proxy minions running on multiple servers, we have thousands of network devices, you are going to need to manage 1,000 system services.
Or, Docker containers or anything else, for example managed by Kubernettes. As an aside this is how we do it. And another option I have seen in the wild is running into a Cloud. So you have this proxy minion services run in a Cloud provider so you don't need to manage that architecture.
That also means that you have to manage all of these processes, it gives you some flexibility and also has this degree of speed, meaning that you want to run a command or deploy something, you just have it instantly available, you don't magically needs to wait, it's just there whenever you want to have something, you just go ahead and run it.
It also means that you have an overhead of managing so many processes, which is not always beneficial. It can be beneficial when you, let's say, deploy configuration changes very often, but as we also have, let's say about 10% of the network, and also other topologies like when you manage Salt servers or CPEs, don't need to do this everyday, you know, typically you do, you interact with your net advice let's say once a year, and running ‑‑ having a process, keeping the memory allocated for one year just for you to run one command is not particularly beneficial.
So, this is why I have developed this package called Salt‑sproxy which is a plug in that comes on top of Salt and allows you to manage your network devices without running this proxy minion services.
And you can have all the advantages, you typically have with Salt, just that you just don't have for this proxy minions.
So, going back to the slide number 7, I was mentions that we have this master, this proxy minions which are interfacing with the network device. Using Salt‑sproxy, the proxy minions simply go away and instead of the master you have this Salt‑sproxy, which talks directly to the network device. The difference here being that you have this dashed line meaning that the connection with the network device is not always up, it's only when you need it, it's going to establish a connection through the channel of choice. So, you don't have always running processes.
And this Salt‑sproxy can run on any machine, your own computer or server, anywhere. It's not even an always running process. It's just on demand when you want to run something, you just execute it.
And getting started is very easy. Salt inst installation typically can be quite complex sometimes, Salt‑sproxy is just that H you just PI P install Salt‑sproxy and there is that. I have a recording here, if you are going to click on the link I have on the slide, you can see it live how goes the instillation.
As an example, just to present it and then if you want to follow it off line or at home or in a few weeks, you can follow these steps here.
Let's say you have a database of devices, for simplicity let's say we have all these devices you want to manage in a file but can be anything else for now, let's say we have a router, which is a Junos, another one which is IOS XR and so on, we also have a firewall, and all listed in a file. Then you researches this file into this special top dot SLS file and say here are my devices into the devices file. Pretty simple so far.
There is another one which, where you can provide the credentials, where we say I want to connect to my devices using these this user name and password and other parameters depending on the API of choice or the proxy minions you want to manage the devices with.
Also, another example here, more specific let's say if you want to manage your devices using NAPALM, a third‑party library, this would be a typical setup. You have here under the proxy key you have proxy type. You say that use NAPALM to manage my devices. And user name, password and the host here. This host looks a little bit interesting. What it means actually, it says that render this field here is rendered independently for each device. This file enforcement, SLS, means this standard file format used by Salt which is by default a combination of Jinja and YAML, if familiar with languages for temporary rendering or serialising data, Which is firstly this data is rendered using Jinja and then the data is loaded as a Python object.
To this is the part that is going to be rendered as Jinja, that's the ops ID that points to the name was device. This is as complicated as it can get, so you just say for each device you are going to connect to this host name. For example, if, for the device name is router 1, the host name is going to be router 1 dot AS 123.net and so on. It's just an example, you don't need to follow this. The logic can be as complex as specific as you need for your own business logic.
Going further, another example is actually if you want to manage the devices and want to connect them to, these devices through your own user name, so, not as in the previous example, I can go back ‑‑ in this example, the user name is the same for all your devices in Salt. If you want to connect through your own user name, so, you know exactly who executed what. You can follow this example here and say manage my devices and connect through using this user name. You can also put other options to use the SS key and so on.
And at the end, this is the last step. You include this file I was speaking about previously, proxy dot SLS and you researches it in the same file I presented earlier, top dot SLS and say researches here the proxy dot SLS file.
And there is one extra detail to notice, is that this roster, pillar says here, which means that it tells the Salt proxy that hey, I have my data into the pillar and this is where I have the database of devices you should be looking at.
That doesn't mean it's only there. It can be other, many other options. You can see this detail is in the documentation. This is just an example I have used here for simplicity.
Once we have threes three files, so it's the database of devices which can be a pillar or anything else, then is the file providing the credentials, and then the configuration file where there is the roster. And then you can go ahead and start executing commands, for example to extract the Arp table to execute a command tool to do a configuration change and so on.
Here is a detail when applying the configuration change, for example, to add an NTP server, something very small but hopefully sufficiently clear and self explanatory to see what is going ‑‑ what is this doing. So notice here the command is Salt‑sproxy, and then is the target the device you are aiming to apply the configuration change, and then say load this configuration change on device, and at the end, optionally, you have test remains a dry run don't actually commit the change on the device, just apply into a dry run and return the DIF.
I was mentioning, you don't actually need to have the database in the way I have shown. It can be ‑‑ you can have that also in a file I have shown, but also you can put in Postgres databases and many other places.
I actually have another example here. Let's say you have all of your devices put into a data centre of database of such as net box, if you are using this, and the only difference compared to my previous example that here instead of saying roster pillar, you say roster net box saying that go and collect the list of devices from net box, and then other details and options, so it knows where is this net box application running. In this case it's net box dot live, it's actually works, net box life application what you want. If you want to play with net box and see if you like it, and token and other details.
In the end, I was saying that with the Salt‑sproxy you can use, the advantage Salt can give you, one of this is the Salt rest API, you can still use it through Salt‑sproxy, just in a slightly different way. I mean just a level of detail, but the configuration is exactly the same. You have ‑‑ you say that, you start the Salt rest based on port 8008 and hopefully you want to run it with HTTPS and provide the certificate in the key for HTTPS. And then just go ahead and execute the requests and execution, you can execute commands via HTTP through the Salt‑sproxy without again I was saying, without running proxy minions.
Here are, here is an example request where you pass in, your user name and password to be authenticated and to make sure that only you and authenticated users are allowed to execute these sorts of commands. As previously here, we have a target, a function, you want to use and this is the returned thing from Salt API.
Pretty much everything that is available in Salt is available through Salt‑sproxy, it's much easier to get started with and you don't have the pain and the headache of managing thousands and thousands of proxy minion services.
It is Open Source. Pool requests are open and because I'm working for this company, we are actually having this month Octoberfest, you still have two weeks if you want to submit requests, you are going to get some goodies. So thanks for your attention and if you have any questions.
(Applause)
AUDIENCE SPEAKER: Hello. Costos. Thank you very much for your work. I personally intend to use it, so I would like to have a talk afterwards, if possible.
Two questions: One is, will this work be incorporated into mainstream Salt distribution from SaltStack?
MIRIEA ULINIC: Probably not.
AUDIENCE SPEAKER: You have already been in contact with them and ‑‑
MIRIEA ULINIC: Yes, yes. That was our initial intention, but SaltStack has a different policy now. They are trying to break out the big project into something, some smaller pieces to be easier to be tracked for example they want to take out, if you are familiar with Salt SSH, they to put it into a separate repository, I think also Salt Cloud so they want to break it into multiple projects to be easier to be maintained because now the main project is quite beefey and becomes harder and harder.
AUDIENCE SPEAKER: And they do not consider having this as one of the separate projects?
MIRIEA ULINIC: We'll see how much ‑‑ if they can help, and also the feedback from the community. It's too early. It's been released only three months ago, it's still in its infancy, hard to say.
AUDIENCE SPEAKER: The other question is a bit provocative. So, this turns Salt into something very much like Ansible, so what would you think personally are the top two, three reasons to go with this approach instead of using Ansible?
MIRIEA ULINIC: Tough question. That's very fair to ask that, and personally flexibility because you don't have any constraints, you can ‑‑ personally I like it because you just write some Python code without having any specifics, like I have in Ansible where you just have some guidelines and you have to follow them otherwise it won't work. It's a very easy to develop a module and extend it in your own environment and then it's just picks the code up and runs it. Also, the thing I was mentioning, the rest API is also very important. And something I didn't have time to expand on, but you can find this in the documentation, if you go to SaltSproxy.docs.IL, you can go to the section that mentions you can still have the event driven automation, you can still react to a different Internet or external events. So ‑‑ which is out of the scope of Ansible generally. So pretty much this would be the top three ‑‑
AUDIENCE SPEAKER: Thank you very much.
MARTIN WINTER: Okay. I think no more questions?
Okay. Then thank you very much for your presentation.
(Applause)
Now, we have Robert who wanted to give a quick announcement.
Robert. Not as Jabber scribe but as a representative of the Atlas tame he guess, some of you may know we have Open Sourced the probe measurement code before, so if wanted to you could get to it before. Now we have basically made available on GitHub the full source code for the RIPE Atlas probe with the intention that we wanted to do what we call software probe, this would be software packages installable on various Linux‑ish machines that act as RIPE Atlas probes. If you want to hear more about this, then please come to the MAT Working Group tomorrow, I'll be speaking for this a bit in more detail.
MARTIN WINTER: Okay. Thank you.
That completes our session. Sorry...
VESNA MANJOLOVIC: Hi, this is Vesna from the RIPE NCC. Last weekend we had IoT hackathon and the by default the code developed there is Open Source, and so the report from the hackathon will happen in the IoT Working Group so if you are interested please come to that Working Group. Thank you.
MARTIN WINTER: Okay. That gets us to the end of it. So, before we close, I just want to abuse this audience for a quick moment because I know many years ago I talked to Debbion developer here and I have no idea who that was any more. If you are one of these folks I would appreciate if you could contact me somewhere afterwards. And that's the end of the session. Thank you everyone for coming. And enjoy the rest of the RIPE meeting.
(Applause)
LIVE CAPTIONING BY
MARY McKEON, RMR, CRR, CBC
DUBLIN, IRELAND.
