DevSecOps – Challenges at Scale
Over the past few years, we have seen organizations of every size and industry trying to adopt, scale, and mature in their DevOps practices. While DevOps adoption is still on a rise, malicious attacks on application are also growing, and every other organization is facing data breach1. An American multinational hospitality company faced a major breach exposing data of up to 50 million guests to the hackers. The breach costed $123 million in costs between fines and court-related expenses with loss of brand reputation.2
As per the 5th annual DevSecOps community survey 2018, 33% organizations suffered with breaches from open source vulnerabilities or web application in last 12 months, a rise of 121 percent since the survey started in 2014.3 Another report by 2019 Verizon on Data Breach Investigations, there was a noticeable shift in financially motivated crime (80 percent), with 35 percent of all breaches occurring as a result of human error.4 In another two years cybercrime will near approx. $6 trillion of damages.
The numbers are large and can any lapse in security can result into major revenue losses or brands going out of business. The risks of relaxed security practices are real, immediate, and very costly. This puts application and technology leaders under pressure to implement security practices and scale them for enterprise-wide adoption. This blog post will cover three major challenges in adopting and scaling DevSecOps and how to overcome those challenges during the shift to DevSecOps.
Challenges at Scale
1. Security at the pace of Development – With DevOps, development cycles have reduced, and features are released frequently for delivering customer-value. With the traditional waterfall releases, security was considered late in the cycle. But with the rise of breaches and speed of releases, security needs to match the pace of development.
Speed is the core tenet of DevSecOps, and Automation is the way to achieve it. In Continuous Integration/Continuous Deployment (CI-CD) environment, the speed of pushing code matters above all. For this, organizations have to make security part of the CI-CD flow. CI-CD pipelines must include the automation of all security tests and integration of static application security testing (SAST) on nightly builds. Teams can customize SAST scans to scan the high-priority code changes to keep up with the daily code changes. Also, teams must include dynamic application security testing (DAST) into development lifecycle. Automation of security testing in the software development decreases the chances of pushing bad code in the production.
The second thing is to manage automating security processes for open-source vulnerabilities. As per a security audit report, 84% of breaches occur on the application layer. And, open source components comprise 60-80% of the code base, this makes managing open-source vulnerabilities crucial for DevSecOps teams.
Teams can avoid open-source vulnerabilities with structured open source adoption process. An open-source policy with detailed steps with approved licenses to use and vendors to adopt will help developers. Automated open-source code approval workflow will eliminate the chances of potential security vulnerability. Using right open-source vulnerabilities aggregators help in final build analysis before it gets pushed into production.
2. Shifting security left in the SDLC – Security principles apply to overall organization and not just to security engineering teams. Organizations must adopt it as a shared responsibility.
Shifting security left means considering security in the DevOps pipeline to ensure faster resolution of issues. Before you build the DevSecOps pipeline, conducting a threat risk analysis to determine the type and level of threats, impact of an attacker gaining highest privileges on the application/system running the application, determining the type of data that the application stores is one of the most important considerations. So how does it look like in practice?
Product Owners must decompose higher-level user stories into specific features with security requirements and tasks. Developers to follow The Open Web Application Security Project (OWASP) security principles. The comprehensive list of OWASP principles reduces the chances of breaches and cyber-attacks. Mature teams can use STRIDE/DREAS threat risk modelling ways to come up with ways of defending potential ways of cyber-threats. IT Managers play an important role in application security from an operations perspective of assessing current and future technology architecture for vulnerabilities, weaknesses for updates and improvements. Security engineering teams must focus on overall architecture and development for security-first design, code hygiene, security test integrations and configuration.
3. Managing False Positives –Managing false positives is a trifold problem and there’s no defined approach of tackling them. Identifying false positives, pin-pointing individual false positive and scaling the process throughout the enterprise. This takes a lot of time for limited number of security engineers working for all the applications.
False positives come in large numbers; even large-sized teams face difficulty in managing them. Security engineering teams must address false positives for highly critical applications impacting customer experience. For optimal management of false positives, narrow the base rules in the tool sets, and identify patterns with fixed and new set rules. This will help teams in identifying which rule is giving more false positives and find out the right combination to minimize false positives.
Teams must adopt multi-tool approach to look at the number of false positives generated by them. Because every tool has its own technique of determining false positive such as signature patterns, behavioural detections, etc. Multi-tool approach helps in detecting individual false negatives that gets buried in the noise of false positives. Scaling management of false positives to an enterprise-level will take time. As teams become familiar with the application, they can create new rules to reduce the number of false positives and might end up with minimum false positives.
Organizations must understand the importance of adopting security practices for convenient and personalized experiences, they are curating for customers. C-suite must champion the culture of tighter communication between security engineering and development teams to drive security requirements in DevOps processes. Enable development teams with skills and tools to develop automated deployment pipeline with tight security checkpoints and robust feedback loop. Invest in new tools and technologies to create combination of multiple tools that can assess all the aspects of application. Contact us to improve cyber security practices throughout the enterprise.