Fixing CI/CD security issues you didn’t know you had

Pelotech Founder Joachim Hill-Grannec goes deep on CI/CD Security.

CI/CD systems like Jenkins/Github Actions/Gitlab CI need permissions to access other systems they need to act on such as cloud providers, k8s, and others. A popular and straightforward solution for this problem is using long lived credentials. While this approach is convenient, it also presents a security risk. Most teams set and forget these credentials without ever rotating them. If a secret is leaked, an attacker can gain access to your production infrastructure.

At first glance you could say “Let’s just run our own runners in our cloud provider”. This does allow you to switch to using instance roles on the runners and getting away from the long lived credentials. “Woohoo, problem solved”! However that’s just scenario 1, now let’s look at larger organizations where we might have various groups with various cloud accounts. In this case we’ll just say each team has one non-prod account and one prod account. How do you prevent teams from accessing other accounts which that host might have access to via an assume-role or the likes without adding a huge amount of complexity within the build system?

One option would be to use runner groups which would be limited to specific teams/repos. However it’s a pretty poor optimization of resources. It means you need to create specific runners for every team making maintenance a pain as well as increase the operational costs which will push organizations to limit runner availability which will drive developer productivity down.

So how do you get shared hosted or self-hosted runners in a way that still satisfies the security needs, a simpler maintenance/scaling model,, and no long lasting keys?

(Continued at link)