SY0-701 Study Notes

Case: The Golden Image Failure — Hardening at Launch Time

6 min read

The Scenario

TechCore Systems was in the final sprint to launch a critical customer-facing product. The engineering team had decided to deploy 200 new virtual machines to AWS for the application tier, database tier, and supporting services. The operations team had prepared a “golden image” AMI (Amazon Machine Image) weeks ago and had been using it for the initial development and staging environments.

Two weeks before the launch, the security team was finally asked to audit the golden image as part of final security sign-off. The CISO assigned it to Marcus, who ran a standard cis-benchmarks scan using a CIS-provided scanning tool against the golden image. The results were devastating: 34% compliance with CIS Benchmarks Level 2.

The issues Marcus discovered:

  • SSH was configured to allow root login and password authentication (CIS requires key-based auth and root login disabled)
  • The firewall-ruleset was disabled on all instances
  • Unnecessary services were running: X11 display server, printing daemons, IPv6 if unused
  • file-system-permissions on critical files (/etc/passwd, /etc/shadow, /etc/sudoers) were permissive
  • No logging agent was installed for centralized log-management
  • The root password was hardcoded in the base image documentation (shared in a team wiki)
  • Default accounts hadn’t been renamed or disabled
  • firmware-updates hadn’t been applied to the underlying EC2 host hypervisor
  • No application-allowlisting policy existed; any binary could execute

The golden image was also massive: 47 GB, bloated with development tools (gcc, git, Docker in Docker), debugging utilities (tcpdump, strace, netcat), and testing frameworks that had no place in production. The bloat increased boot time, created a larger attack surface, and violated the least-functionality-principle.

Marcus immediately escalated the findings to the product team leadership. The response was panic. The launch was in 14 days. Remediating the golden image would require hours of work. Testing the remediated image would require days. Redeploying 200 instances and validating the new configuration would require more time than was available.

The product team pushed back: “Can’t we just deploy as-is and fix it after launch?” The CISO said no. But the product team went to the CTO, who was under enormous pressure from the CEO to hit the launch date. The compromise was rushed: Marcus had 48 hours to produce a remediated golden image, the engineering team would validate it in a single 8-hour staging test, and it would deploy to production.

In those 48 hours, Marcus:

But Marcus couldn’t do everything. He didn’t have time to implement application-allowlisting (which would require extensive testing of which binaries each application legitimately used). He didn’t have time to implement full stig-security-technical-implementation-guide compliance (the NSA hardening standard went much deeper than CIS). The staging test validated that the new image worked and passed a basic CIS scan, but it didn’t stress-test the firewall rules or validate that all legitimate application features were available.

The 200 instances deployed successfully with the remediated image. Compliance jumped from 34% to 78%. But within 48 hours of production launch, the security team detected that firewall rules Marcus had implemented were too aggressive and were blocking legitimate application traffic. The product team, frustrated by the launch delays and now operational issues, pressured the ops team to disable the firewall. Within a week, the instances were nearly as permissive as the original golden image.

The product launched successfully, but the security posture was compromised by the rushed remediation and lack of support for hardening from the product team.

What Went Right

  • Late-stage security audit: Running a CIS benchmark scan before production deployment caught the issues before they affected customer-facing systems at scale.
  • Partial remediation: The hardened image, even without full stig-security-technical-implementation-guide implementation, did improve from 34% to 78% compliance.
  • Centralized logging: Installing the log-management agent enabled later forensic analysis if breaches occurred.
  • Build-time scanning: Integrating CIS scanning into the AMI build pipeline meant future images would automatically be checked.

What Could Go Wrong

  • No security review during golden image creation: The image was built by the operations team without security team involvement. If security had been consulted during the design phase, many issues could have been prevented.
  • Application-allowlisting not implemented: Without whitelisting which processes can execute, the hardened firewall was the only defense against malware execution.
  • Rushed hardening under pressure: The 48-hour timeline meant Marcus had to make tradeoffs. Some rules were overly aggressive, others incomplete.
  • No long-term enforcement: After launch, when the firewall rules caused issues, there was no governance preventing the team from reverting to an insecure configuration.
  • Insufficient secure-baseline-images testing: The 8-hour staging test was inadequate for validating complex firewall rules across diverse application use cases.

Key Takeaways

  • Involve security in base image design from day one: secure-baseline-images should be built by a joint team of security, operations, and engineering. Security review should happen before the image goes into production use.
  • Cis-benchmarks are minimum baselines, not comprehensive hardening: CIS provides good low-hanging fruit (SSH config, firewall, file permissions), but stig-security-technical-implementation-guide compliance requires additional hardening for critical systems.
  • Least-functionality-principle should be enforced at the image level: Remove development tools, debugging utilities, and unnecessary services at build time, not as an afterthought.
  • Remove-default-accounts-and-passwords must be automatic: Golden images should ship with root login disabled, default accounts removed, and SSH key-only authentication, not as optional hardening.
  • Build-time scanning enables quality gates: Integrate cis-benchmarks scanning into the AMI build pipeline so non-compliant images never reach production. Fail the build, don’t warn and hope.
  • Registry-and-gpo-hardening should be declarative: Use infrastructure-as-code-iac to codify hardening rules so they’re repeatable, version-controlled, and auditable.
  • Hardening governance must include compliance enforcement: Once hardening is deployed, implement ongoing compliance monitoring to prevent security teams from removing rules under operational pressure.

Graph View

Backlinks