GOTO - The Brightest Minds in Tech

Container Security • Liz Rice & Eoin Woods

• Liz Rice, Eoin Woods & GOTO • Season 1 • Episode 5

Share episode

This interview was recorded for the GOTO Book Club.
http://gotopia.tech/bookclub

Liz Rice - Author of "Container Security"
Eoin Woods - CTO of Endava

DESCRIPTION
What should you do to secure your containers?
Liz Rice, author of the book  Container Security: Fundamental Technology Concepts that Protect Containerized Applications & VP of open source engineering at Aqua Security, and Eoin Woods, CTO at Endava, explore what containers are, what are the implications of a shared kernel and how to assess potential security risks that could affect your deployments. Learn best practices and understand how containers work in this Book Club interview.
The interview is based on Liz's book "Container Security": https://amzn.to/3oU4iJe

Read the full transcription of the interview here:
https://gotopia.tech/bookclub/episodes/secure-your-containers-liz-rice

RECOMMENDED BOOKS
Liz Rice • Container Security • https://amzn.to/3oU4iJe
Liz Rice • Kubernetes Security • https://www.oreilly.com/library/view/kubernetes-security/9781492039075

https://twitter.com/GOTOcon
https://www.linkedin.com/company/goto-
https://www.facebook.com/GOTOConferences

Looking for a unique learning experience?
Attend the next GOTO conference near you! Get your ticket at https://gotopia.tech

SUBSCRIBE TO OUR YOUTUBE CHANNEL - new videos posted almost daily.
https://www.youtube.com/GotoConferences

Bluesky
Twitter
Instagram
LinkedIn
Facebook

CHANNEL MEMBERSHIP BONUS
Join this channel to get early access to videos & other perks:
https://www.youtube.com/channel/UCs_tLP3AiwYKwdUHpltJPuA/join

Looking for a unique learning experience?
Attend the next GOTO conference near you! Get your ticket: gotopia.tech

SUBSCRIBE TO OUR YOUTUBE CHANNEL - new videos posted daily!

Intro

Preben Thoro: Thank you for joining us today.

Eoin Woods: I’m Eoin Woods, CTO at the software engineering services firm Endava and I’m very pleased to be joined by Liz Rice, VP of Open Source at Aqua, the cloud security tools security company, who is author of the excellent book “Container Security”.  Welcome, Liz!

Liz Rice: A pleasure to be here Eoin.

How did you come to write a book on container security?

Eoin Woods: So, Liz, I thought we'd just start with perhaps the most obvious question, which is, how did you come to write a book on container security? It's a really important topic, but it's quite a specialist one.

Liz Rice: Yes. So I've been working with containers for quite a few years now and obviously, they've exploded in popularity over the last four or five years. I think we've gone from like, "What even is this Docker thing," to lots of developers, there's a huge proportion, using containers. And I've been working in container security now for about four years. Yeah, I've come from the developer angle, and I've always been interested in kind of how things work, what's happening under the covers. With my sort of security perspective, it's like, "The way that a container works, how do that present issues for security? How does it create attack surfaces?"

I've been doing talks and writing blog posts and creating content about containers and container security for quite a while. Then when O'Reilly approached and asked if I had an idea for a book, I kind of knew that well, container security is what I've been working in and what I know about, and it was nice to have the opportunity to dig in more detail than you can necessarily cover in a 40-minute conference talk. In a book, you can dive into the details.

How does your position at CNCF Technical Oversight Committee relate to your work on security?

Eoin Woods: Which you do. There's a lot of detail in the book to help people understand it, which I mean, it's a great aspect of the book. And I noticed, of course, your day job's at Aqua, in the area of cloud security, but you've also got a CNCF affiliation, how does that relate to container security?

Liz Rice: Yes. So, I am currently the chair of the technical oversight committee for the CNCF, which in case anyone doesn't know, it's the organization — part of the Linux Foundation — that owns Kubernetes, Prometheus, a whole host of cloud native projects. With the TOC, we're the group that assesses whether projects are cloud native and should join the foundation and at what level of like sandbox incubation or graduation. So that gives me a really interesting vantage point over the world of cloud native in general, and I get to meet and talk to some interesting people.

Security is just one part of the cloud native landscape, but it is a very important part. So it's a privilege to be getting that view across the broader landscape, not just security.

Eoin Woods: Yes, there's a lot of hard work probably in your spare time I suspect.

Liz Rice: Spare time? What's that?

What is the essence of a container?

Eoin Woods: It's rather nice to have somebody leading a kind of technical committee who's focused on security. That's really good. I was wondering because clearly, you've got quite a broad view of containers. One of the nice things in the book, I just love the way that you sort of build a container at the shell prompt. I think there are lots of us who have come to containers, and we got Docker and we thought this is this magical packaging technology and seems to keep everything safe, and then we weren't quite sure how safe it was, and then you wrote a book on that. Can you just explain for someone who's just used containers, what the essence of a container really is?

Liz Rice: I think if your experience of containers is running them, you know, Docker run and maybe executing into a container, there is this sort of tendency to think of them as virtual machines. Like, you used to hear this phrase, lightweight virtual machine. It's so misleading because all the containers on a given machine share the same kernel, which is the fundamental difference between a container and a virtual machine. Virtual machines have their own kernel.

And really what a container is, it's a process or maybe several processes running under Linux, and that process is isolated from the other processes. And it uses mechanisms called namespaces and cgroups and it has its own view of the file system. But really, from the host perspective, it's just another process. It's really nothing special. And that has huge implications for security because the isolation is, it's only so much, it's not the same strength of isolation as you would get from virtual machines, largely due to that shared kernel.

Eoin Woods: That's interesting. So, the VMs kind of thing with an operating system kernel in it, the container isn't, you sort of hinted a bit there in that question: they are different from a security perspective. Well, from the perspective of people who use them as opposed to people who are deeply embedded in building them, how would you compare and contrast them specifically, from a security perspective? What does that shared kernel mean in terms of security things you need to be aware of?

What does that shared kernel mean in terms of security things you need to be aware of?

Liz Rice: So the main consequence of that is what happens when things go wrong. If you do manage to escape a container, you're a bad actor, and you've managed to compromise a container. If there is a way of escaping onto the host, you can potentially escape through the kernel, if there's a vulnerability in the kernel, and we have seen vulnerabilities in the kernel. Or if there's a vulnerability in the container runtime, it's possible to escape from the container onto the host.

I mean, it's not like this isn't ever possible in virtual machines. There have been security issues in virtual machines as well. But the fact that you have this shared kernel makes it just so much more likely that if something goes wrong, you're gonna get from the container onto that host machine where all of the containers are running. And once you're on the host, particularly if you're root, you've got access to everything that’s there.

Eoin Woods: Most containers, container systems I've worked with, they are root, right? I mean, that's one of the questions.

Liz Rice: This is one of the kind of really fundamental things that I really want people to improve about their container security: you're running as root by default today, at least. There are exceptions. If you're using Podman, that doesn't run as root by default. If you're using OpenShift, that doesn't run as root by default. But for most people's experience running with Docker, running containers on Kubernetes, unless they've specified a user by default, they're root. And as I've shown in the book, and as I've shown in a few talks before, that root identity is exactly the same in the container as it is on the host. So if you do somehow escape from the container, you're root on the host.

And I guess another thing that's quite a big contrast between containers and virtual machines is you can escape the container through just misconfiguration. It doesn't necessarily require an actual vulnerability in the kernel or the runtime. You can, for example — if you mount the whole host file system, there's nothing to stop you from doing this. — you can mount root on the host into a container. Running as root inside a container with access to the full host file system, the world is your oyster as an attacker.

Software supply chain security with containers

Eoin Woods: That can only sound like a bad thing. Interesting. Thanks. And, I mean, one thing you talk about a bit in the book, which I think has got a lot of... You were just talking about parallels with other previous technologies, which also is a concern of the places, but strikes me as really important with containers, is software supply chain security. So, that's the posh name for pulling in all your dependencies. This is where it is with things like Maven and Node, and you import one package and you seem to end up with half the open source universe, and you don't know much about it.

Containers have always worried me a little bit more. I don't even really know why. Maybe it's emotional or maybe I don't know enough about them. But it seems like you're pulling down really complicated bits of software, which you probably aren't very familiar with and they're all being stacked in this kind of integrated stack, and you need all of it to make it work. I mean, how do you even go about thinking about software supply chain security with containers?

Liz Rice: I think the reason why it feels more complicated in the world of containers is because every single one of your containers is running with its own set of dependencies. One of the wonderful features of containers is you can package up an application and all its dependencies into one image and chip that image around, so every image has its full set of dependencies. So every image could be different. And you've got to check every one of those images, because they all might have... You've got this forest of images and somewhere in there maybe there's a vulnerability and it can seem quite daunting, particularly if you're sort of used to a traditional security process where you SSH into a machine and apply patches. You can't really do that in a containerized environment, particularly if those containers are coming up and down pretty frequently. The average lifetime of a container in production is around about a day, so imagine having to patch 1,000s of containers, you just can't do it manually.

So automation is really your friend. Like everything else in the cloud native world, the more we can automate things, the less human interaction you have, the less chance there is that somebody's going to make a mistake. A lot of security issues are not people behaving badly deliberately, they're just slip ups. People leaving things open inadvertently. So, automation is your friend. Vulnerability scanning is particularly important for the supply chain.

Eoin Woods: That point about people who don't do daft things deliberately is important. I think that's a really important point, isn't it? That there seems to be so much technology that from a security perspective, it's just a minefield for people to kind of wade through. I always describe in my security talks that if you had to create an attack vector, you'd start with a web browser. It's such a rich environment to build attacks in. Actually, as soon as you look at this, the security person goes, "Yeah, that's a huge security problem." How do you think about containers? I mean, do you think there are things that are in configuration today that maybe should be hard to change, or do you think there are particular vulnerabilities in the way containers are configured that could be improved, I mean, especially from a CNCF perspective?

Security concerns with containers

Liz Rice: There are all kinds of aspects of how you run containers and how you run your overall orchestration system and these days, it's probably Kubernetes. Over time, the settings for Kubernetes have gotten better by default. So, gone are the days when by default, you would leave an administrative dashboard open to the internet by default, which was the thing. And fortunately, if you take the standard install tools now, you will get things like secure API ports rather than just allowing unauthorized access, and so the default settings have improved a lot over the past couple of years or so.

There's still a lot of room for improvement. One thing that I'm quite excited about right now is rootless containers, which is, I mean, as we were talking before about how by default, you typically run as root. Rootless containers mean that although from the containers' perspective, you appear to be root, it's not actually root on the host, and that's a huge improvement. And, in fact, yesterday, as the date of recording, yesterday, Docker released their latest update, which has moved support for rootless out of experimental.

There's still some work to go in the Kubernetes world to bring rootless. There are some experimental demonstrations of running rootless, but it's not really the production sort of GA right now. But I'm hopeful that we're not too far away from a future where, by default, we don't run it as root, and that will be a big step forward.

Eoin Woods: Yes, that sounds like great progress. Good to be able to influence that kind of thing. Because one of the technologies you've mentioned a couple of times is Kubernetes. And I think in many people's minds, you've talked to many software developers, containers and Kubernetes, Kubernetes and containers, that they're really kind of almost the same thing. And in the book, you don't spend a lot of time talking about the orchestration technology. What complications, do you think, Kubernetes brings to the container environment from a security perspective, because you alluded to the fact that it certainly doesn't make things easier?

Liz Rice: So one of the reasons why I didn't talk about Kubernetes security in the book is partly because it wouldn't have been two or three times the size, and partly because it's changing more quickly. I think that the fundamentals around how containers work are pretty well understood, whereas I think there are lots of aspects of Kubernetes and Kubernetes security and the complementary products that sit around Kubernetes like registries where there are still lots of improvements and lots of great innovation going on, not that there isn't innovation going on in the container space. But I wanted the book to not be out of date in six months or a year's time, and I think that Kubernetes security is moving sufficiently quickly that there would be a danger of things being a little bit aged after a year or two.

Eoin Woods: Where would you recommend people go to find out more about Kubernetes security aspects and how to track that because it is moving quite quickly, isn't it?

Liz Rice: It is moving quickly. In fact, I co-authored an O'Reilly report with Michael Hausenblas who's now at Amazon, and we did that a couple of years ago. I definitely feel that that report is kind of due a refresh. It's got some sensible things in it, but it's there are things I would like to update in that book. So in the absence of that, I would say the Kubernetes documentation itself has got better and there are people who are writing sections in Kubernetes docs about security aspects. So things like if you want to make sure you've got secure connections between your different Kubernetes components, that's all documented now. There's quite a lot of best practices documented there. There's also a Slack channel on the Kubernetes Slack dedicated to security.

I think one of the really nice things about Kubernetes in general, the CNCF in general, the whole community is very welcoming. If you do have a question, whether it's about security or anything else, coming into the Slack channel there will be people to help. Asking on a mailing list, there will be people to help. So, I definitely encourage people to kind of embrace the community. It's one thing to read a book, but you can also find people who know this stuff, and they're happy to help.

Eoin Woods: That's terrific to hear, yes, because it makes such a difference when you connect with a human being who can actually point out the bit of the book you've misunderstood. So that's really important. Thank you.

Who is responsible for container security?

Eoin Woods: A lot of these technologies are often sort of bundled into a broad area of people called DevOps, which is terribly undefined word. It drives me mad. But if we just take it as meaning smoothing the path from development into operations, there are, even if I'm a great fan of integrated teams, we haven't got a silo.

And a former colleague of mine had a terrific phrase for it. He said, "The desert of responsibility in between Dev and Ops, and you're trying to take that away, you do still have specialists in the teams, container security." I know the simple answer. The answer I always give my teams is security is everybody's problem. But when it comes to container security, where are the key responsibilities? What kinds of people do you need involved in your project that you're gonna get the right attention on it?

Liz Rice: Yes. You've used the phrase DevOps and people also use the word DevSecOps. I think that just goes to illustrate that actually, there are lots and lots of skills involved. And people like to pretend we can just sort of mash them together and it all be magic. But actually, that's a big cultural change.

Eoin Woods: Which won't change anyway.

Liz Rice: So I think it reflects the fact that you're going to need people who understand how to operate your cluster, your containers, how they're dealing with deployments and rollouts and scaling and all that side of things.

You've got your developers who understand the application, and you've got security specialists who understand things like, well, if we do have a particular vulnerability, what's the impact of that? There's lots of kind of jargon in the security world that can be a bit impenetrable, and you need people to know how to navigate that and really understand if we find a particular security flaw, maybe we're scanning our images, we find, "Here's this terrible, critical vulnerability. What should we do?" And you need people who understand how to investigate what that really means and can ask the right questions of the developers and the operators to figure out what the overall response should be to a particular vulnerability or an incident even.

You need all those skills and how those are organized in different organizations and how they're brought together into different teams. I think that's something that our whole industry is working on and figuring out how to deal with that effectively.

Eoin Woods: I agree. I don't think anyone's got an easy answer to that. But it's important we all, at least, agree where the goal is and sort of move in that direction.

Liz Rice: There's definitely an aspect of tooling where security tooling in particular, where I would say most security tooling is really at the level of telling people that issues exist. If we think about things like runtime security, the technology exists, there are some really great tools. I mean, I'm working for a vendor, so I would probably think that. But there are some great runtime security tooling that can prevent your containers from doing untoward actions.

But in reality, most people are not using that yet. They're mostly in the world of kind of even trying to identify what the issues are, let alone how to react to them, how to manage them, how to prioritize different security concerns. So, I think that the security industry and the container security industry is trying to level up people's understanding of security tooling, and things like how do you deal with these issues still require quite a lot of human expertise right now.

Eoin Woods: Yes, I agree. Absolutely. In fact, I think one of the big security concerns is overreliance on tooling, isn't it? That we've bought a lot of tools, or, I mean, people are trying to keep DevOps the same way, "I bought the tools. Why didn't I get DevOps?" 

Liz Rice: Different tools can report different results for completely legitimate reasons. I was just recording a talk earlier this week about different vulnerability scanners: one really great example is there are two different ways of scoring vulnerabilities. It's called CVSS, a common vulnerability scoring system. But there are two versions in common use today, version two and version three. If you use two different scanners, one of which is using version two and one of which is using version three, they're going to show you completely different scores. They'll probably show you the same set of vulnerabilities. But under one scoring system, something could be critical, but only medium in the other. I think as a user, unless you kind of know about the scoring system, you just look at those results and think, "Well, what do I do with that? How can I make sense of this?" So, there's quite a lot of education and improving how that is communicated.

Security tools

Eoin Woods: There's so much jargon in security that a lot of the tools still talk in jargon. The other thing I've observed with security tools, because we use quite a few. At Endava, we're constantly looking for the holy grail. An awful lot of them, dare I say, with all due respect to the vendors, warmed-over versions of the batch ones that used to be run at the end of the week or the end of the splintering even before release. And, of course, teams now wanna be running them every time there's a check-in, and actually, lots of the tools are not very evolved in that direction.

Eoin Woods: For integrating security tools into your dev environment, which I think for containers is, as we were saying earlier is so much more important than it used to be. Have you found that teams have approached that problem?

Liz Rice: One thing that I've seen, particularly in that kind of CNCF community world, is that developers are more and more aware that, "Oh, security is a thing, and we can't just sort of develop an application and then somebody else will deal with the security later." Because they're supposed to be shipping function every week or every day or however frequently. We've called that shift left across the industry, that idea of moving security earlier into the development process.

I think developers are increasingly aware that there's probably something we need to do here. So there are things like building scanning into IDs, building scanning into build pipelines. Things that GitHub have done to start exposing vulnerability information through the dependable alerts. This is all really good for raising awareness to developers that, "Okay, you may actually need to update this particular library because it's got a problem."

Main takeaways from the book

Eoin Woods: It's just as much your problem as everyone else. Here are your alerts. Yes, that's great. So we've covered quite a lot of ground there. I mean, I'm really interested, did you just have all this background knowledge and it kind of collided with an idea for a book and you just wrote the book? Or did you kind of start off thinking, "I know a lot in this area." But as you wrote the book, you kind of thought, "My goodness, what a lot of other interesting areas." Did you have to learn quite a lot as you went or was it just putting things together?

Liz Rice: There were definitely areas of this where I could almost use the book as an excuse to dig in more detail or kind of refresh my memory. So, one chapter in particular where I talk about networking. Back in the day at my first kind of professional computing job was writing network protocols. I knew all that stuff, but it had been a while and I had to go and have a chat with a few friends and just kind of remind myself. So, "What actually happens? Remind me how that works?"

Eoin Woods: I knew you'd remember that. I hear something magic happens.

Liz Rice: Yes. And actually, it was quite interesting. My mental model was a little bit not quite right, and it was really good to have that refreshment and then to update my thinking around that.

Eoin Woods: It was quite nice having spent the time, isn't it?

Liz Rice: Yes.

Eoin Woods: Yes, that's really interesting. Thanks. So, it's not a big book. In fact, one of the things I was impressed with is you cover ground quickly and you don't hang around, and you really make the point in a very structured, rapid manner, which I was very grateful for. It's less than 200 pages. But what from that, if you had to boil that down, what are the two, three, four messages that you think people building, operating, using containers today really need to take away?

Liz Rice: So I've got sort of like a couple of favorite bits of what I find really fascinating that I think are interesting and important. But maybe I will answer the question in another way first. So the other way I'll answer it is, although I'm not a big fan of checklists because a lot of security does involve people kind of trying to tick boxes on the checklist without necessarily understanding what those items mean. So, I'm not normally a fan of checklists. But nevertheless, I wrote an appendix in the book that I call "The Checklist," but it's really more of a set of questions to try and help people think through what security issues are important to them. Do they understand? Are they scanning their containers for vulnerability?

Eoin Woods: Yes, the thing I really liked about your checklist was you say, "There are no check boxes," and it's not five words. It's quite a long sentence that really makes you think, "Oh, actually, I've gotta stop and think about that." Rather than, "Yes, yes, yes. That's five."

Liz Rice: Yes, because there's no one-size-fits-all solution. Everybody's security, everybody's deployment is different. The data they're looking after is different and varies in value, the likelihood of attack varies, the complexity of the applications varies, so I don't think there's like a one size fits all. But I hope those questions will help people kind of think about what's important. But the other way I'd answer that question of like what are the important messages, I think if you're using containers and you want to understand the security implications, just really understanding that they are Linux processes makes a huge difference to how you think about attack surfaces and the sort of foundational mental model that you need. So, I definitely recommend the chapters where I talk about container isolation and how a container is built up, because everything else kind of comes from there, I think.

Eoin Woods: That's great. That is a really easy message with a lot of implications for someone to think about once they've actually got some containers running out there in the wild. Thank you, Liz. It's just been an absolute pleasure to talk to you. I think we could talk for another hour quite easily.

Liz Rice: We could.

Eoin Woods: Of course, we won't be allowed to. So, thanks ever so much for taking the time. I've certainly learned a lot. I think there's a huge amount there for people to learn and take away and just improve security in the operation of their containers. Thank you.

Liz Rice: Oh, it's my pleasure. Thank you.

Preben Thoro: Well, both of you, thanks a lot. It was really, really inspiring.