There has been quite a bit written about Code Spaces and how unauthorized access to its ITaaS console granted enough permissions to delete everything out of Amazon, including backups. There are lessons here not only for tenants, but also for those vendors who create ITaaS consoles, such as VMware (vCHS, vCD, vCAC, vCenter, Orchestrator, etc.), Virtustream (xStream), OpenStack, and many others. These consoles need better controls and security so that such behavior is prevented, logged, and monitored, and the proper authorities are informed. Now, we may think this is a cloud-only attack, but we use these tools within our own environments day in and day out. For anyone using virtualization, private, or hybrid cloud consoles and automation tools, it is time to take a good long look at role-based access controls (RBAC). The steps we discussed at the end of my other lessons article still apply.
Articles Tagged with lessons learned
Recently, we experienced a fairly catastrophic SAN failure: we lost two drives of a RAID-5 array. Needless to say, recovery was time-consuming, but it also pointed out some general issues with many disaster recovery, business continuity, and general architectures involved with virtual environments. Luckily, we were able to start one of the drives, let the hot-spare take over for the second failure, and recover the vast majority of our data. Yes, there was corruption, so that is where our backups came in and the ultimate dependencies for restoration. How do you recover from a catastrophic failure? Do you fail over automatically to a hot-site or cloud environment? Even if you fail over, how do you recover from a catastrophic failure?
Several years ago I was working at a company that had a ton of legacy silo applications that collectively represented the entire process flow that supported the core business. The process flow was made up of years and years of legacy technologies and legacy business processes.