Quick and Dirty ESXi 5.1 Operations Quick Start Guide

At a previous job, I had to configure an ESXi server for the first time. It was the first time I had ever built and configured one. Additionally, it was the first time my former employer was using ESXi, so I had to design and engineer the solution, then implement it.

In order to allow future employees to build and operate the server, I wrote an informal operations guide. I just edited that document to make it more generic and I posted it in the File Collection portion of this site. It's a very rough guide, but it also includes a lot of step-by-step pictures.

The one piece that I took out of the guide was the part about networking. While no ESXi operations guide is complete without a networking component since it's integral to the operation of the guest VMs, I simply did not have time to customize that part of the document. ESXi networking can be complicated and the manner in which it needed to be setup for my former employer was extremely complicated, but it was very interesting and I believe others will find the setup useful as well. Therefore I'm splitting that out into a separate blog post, or potentially an additional operations guide, which will be coming.

Informal Discussion for Beginners: Why DDoS and Stuxnet are Very Different

Who didn't own one of these Micro Cruzers?

I wrote an email tonight to a friend of mine who had a question about potential Russian cyberattacks and the differences between a regular DDoS attack and something like Stuxnet. Obviously, an email to a friend is pretty informal and the details about Stuxnet here are completely from memory, but this is accurate to the best of my recollection. For context, my friend had said he thought DDoS was like a medieval siege attack and Stuxnet was more like a targeted terrorist attack. I thought some others might find it interesting at least:

I’ve never heard a DDoS attack be described that way, but that’s actually a pretty accurate characterization. As for Stuxnet being like a targeted terrorist strike, I would say it was more of a targeted special operations surgical strike (terrorist strikes are specifically designed to cause mass terror, not necessarily disable/kill infrastructure/capability/personnel of the enemy force as surgical special operations are).

DDoS stands for Distributed Denial of Service. The point is to flood the servers with so much traffic that they are incapable of accepting new connections or responding to any connections that do occur. Either all of the resources of the server are consumed or the pipe leading to the servers is completely filled up, meaning there is not enough bandwidth to handle the incoming traffic. These used to just be DoS attacks, but the “distributed” part came about when pipes got wider (more bandwidth for connections) and servers got beefier (they now have more resources to handle many connections at once). Now, attackers use various methods to recruit a large number of source machines to send traffic toward the victim. As for being just a hacker protest tactic, that’s not completely true. That’s probably its most well-known usage, but DDoS attacks are widely used in digital “protection” rackets, just like the old world Mafia had. Sites that absolutely need to be available at certain times but who also don’t want a lot of attention from authorities, such as gambling websites during the Super Bowl, will be threatened by hackers such that, if the website does not pay a certain amount of protection money, it would be really horrible if a DDoS attack were to happen when you really need your site to be up…

Stuxnet actually didn’t require boots on the ground, which was actually the most impressive part of its design given that the Iranian centrifuges were air-gapped (not connected to a computer network which had Internet access). The only way to spread malware to the centrifuges was to have someone connect an infected removable storage drive (most likely USB) to one of the systems connected to the centrifuge network. Despite this limitation, the creators of the virus were able to design it to automatically infect removable storage devices, then spread silently whenever it was connected to different machines. The virus was released in very targeted areas where intelligence reports indicated those with access to the Iranian nuclear program would be. The real genius is that the virus was specifically designed to infect the Iranian programmable logic controllers (PLCs) that control the centrifuges’ operation. These PLCs were fairly standard off-the-shelf components, meaning that they are in many industrial sites across the world. This is where the boots on the ground came in. Someone, most likely Israeli intelligence, knew the exact configuration of the PLCs, so the Stuxnet virus was designed to only activate itself if it detected the configuration known to belong to the Iranian PLCs. The virus then played havoc with the speed of the centrifuges and continuously destroyed them by rapidly changing speeds or running too quickly in general. The Iranians couldn’t figure out for months why the centrifuges were breaking, though, because the virus was also designed to infect the monitoring systems and alter the output to falsely show that the centrifuges were operating within normal parameters. Given that Stuxnet was released into the wild to find its own way to the Iranian systems, it’s not surprising that the virus was also found in various other industrial control systems around the world once people knew it existed. The virus just never activated because it did not detect the Iranian PLC configuration.

Using Virtual Machines at Home for Fun, Learning, and Ridiculousness

Technology has certainly progressed beyond the point of requiring a single physical server for every task. Using virtualization, I can easily have as many systems as I would like. Virtualization of servers allows for the opportunity to build and use systems for single intended purposes with minimal physical overhead. This is particularly useful for technology enthusiasts or budding professionals who need an environment to "play around with" in order to learn about various systems and services. For example my personal virtualization server, which I will describe in detail below, hosts three CentOS 6 servers, an Ubuntu 12.04 server, and three Windows Server instances, all running different services and purpose-built to run those services or applications efficiently. Additionally, it is easy to build a new server, or even an instance of a desktop OS, whenever I need to test something. I probably don't need all of these systems or all of the ones I'll build in the future, but it's fun and I like scaling my infrastructure to ludicrous proportions despite the fact that I am the only one that uses my network.

Virtualization Types and Options

All virtual systems, called "guests," have to run on a base system. This base system is called the hypervisor and there are two major types of hypervisors:

• Type 1: Bare-metal hypervisors. These hypervisors are operating systems in themselves and run directly on physical hardware. The only task that can be done with a bare-metal hypervisor is configuring and running a guest OS. The major players in this market, at least the ones I'm most familiar with, are:
  • Microsoft Hyper-V Server. This is a stripped down version of Windows Server used only for running virtual machines. This is a very high quality and stable product, although with a little more resource overhead and support for fewer guest operating system types than other solutions.
  • VMware ESXi. My hypervisor of choice since there is a free version which supports systems with up to 32GB of memory. This is a rock-solid product made for use in production environments with high availability requirements.
  • Xen. I do not have much experience with this hypervisor, but it's certainly on my list. It has a reputation for stability, although it is a bit complex to configure at first.
• Type 2: Hosted hypervisors. These hypervisors run on top of pre-existing operating systems. For instance, there are software hypervisors that run on Windows which enable a guest OS to be run inside of the host OS. Some major type 1 hypervisors are:
  • VMware Workstation. Versions are available for Windows and Mac. This is a paid product, but it provides high quality, stable virtualization.
  • Windows Hyper-V. Hyper-V is a feature built into certain versions of Windows, including Windows 8 Pro and Windows Server 2012, which allows users to run guest operating systems inside of Windows.
  • VirtualBox. A free product available for Windows, Mac, and Linux with a good track record of support and stability, but not something that should be used in a high availability production environment.

Many people who simply want to play around with alternative operating systems will use a type 2 hypervisor to just run a virtual machine on their existing computer. However, those who want have these alternate virtual operating systems to run all the time or want to run a lot of virtual machines have the option of building a separate machine and using it only to run virtual machines.

ESXi -- Stringent Hardware Requirements, but Worth the Effort

ESXi is my chosen virtualization solution due to its combination of stability, ease of setup and management, and free licensing up to 32GB of memory. However, since ESXi is meant to run on server-class hardware in large production environments, there are some stringent hardware requirements. I did not want to spend the money on server hardware, so I had to build a custom machine and carefully select my components. I made sure the components were supported by the version of ESXi I would be running (ESXi 5.5) by comparing the components with the VMware hardware compatibility list. The hardest thing to find was a non-server motherboard that had a compatible networking chipset. I ended up building the system with the following hardware:

• Motherboard: ASUS P8Z77 WS -- this motherboard has dual server-class NICs. Some dual NIC motherboards run each NIC off of a different chipset, but both NICs on this board run on the same model chipset so both ports can be used by ESXi.
• Processor: Intel Core i7-3770K -- I chose the Core i7 to take advantage of the extra cores and threads as much as possible. Component purchasing tip: MicroCenter always has the lowest prices on processors and they have great warranty coverage.
• Memory: Corsair 32GB DDR3 1333 -- just make sure the memory is compatible with the motherboard you select.
• Hard Drive: 2TB Seagate -- If you notice I only bought one hard drive, you'll realize my storage in this system is not setup in a RAID. That's a mistake I'll be rectifying soon.
• Case -- It's a case with sufficient ventilation...I don't understand why people get so worked up over cases.
• 500W PSU -- Once again, the boring component, although I probably could have spent a little more money on a power supply to guarantee clean, steady power.
• DVD-ROM Drive -- No one's gotten excited about a DVD drive since 2004.

ESXi Tips

Here are a few things I've learned about running an ESXi server that will hopefully help some others along the way.

• Attach the "Client Device" to the virtual CD/DVD Drive before exporting virtual machines into OVA format packages. I have not tested in ESXi 5.5, but in at least ESXi 5.1, you must select the Client Device option instead of an image file before exporting to OVA. If you don't and you try to import an OVA with the image file setting, the import will fail.

• The desktop vSphere Client has been deprecated by VMware. The desktop client used for managing the ESXi host will no longer be updated. VMware is pushing for people to use the web client to manage ESXi hosts, but you need to pay for a license to run the web client.
• Use VMware Workstation to manage settings on VM Versions 9 and 10. VMware has been in the virtualization business for a long time and they have extended their virtual machine format continuously to support newer technologies and operating systems. They keep track of which features different virtual machine containers support by using version numbers. Using the vSphere Client, you can only create VMs up through version 8. VMware Workstation can connect to ESXi and create newer VMs though. A free trial of VMware Workstation is available if you only need to do this once.
• ESXi does not natively support NAT. Any virtual machine with a physical network connection will connect directly to the network and must have its own IP address. ESXi will not perform NAT for hosts. However, you can go through the trouble of setting up a software firewall system to route your traffic through. I will likely do a post on this in the future but the summary is that the firewall system would have two virtual NICs, one connected to the network and one connected to an ESXi internal virtual switch. All of the guest machines that you want to perform NAT on would then connect to the internal virtual switch and use the software firewall as the gateway.

Conclusion

In the future, I'll go into detail about my personal network setup and what I use all of my virtual servers for. In the meantime, know that ESXi offers an excellent, high quality, high stability solution for the perfect price: free. If you need to build a virtualization lab at home for playing around or professional development, then my hardware selections will provide you with a high powered system at a much lower cost than a true server.

Essentials: Importance of Backups

Welcome to the Essentials Series   

As this is the first post in the Essentials  series, let me explain this series' purpose. The Essentials  series is meant to give information technology beginners an understanding of the basic building blocks of a secure IT environment. This means essential steps for securing data, workstations, and networks. Overall, the technical detail of these articles will be limited so that beginners can understand the content and the concepts laid out within. Other articles and series on this blog will be geared toward more advanced individuals and will expound greatly on the concepts from the Essentials  series.

Importance of Backups 

Imagine having a child and over the course of their first years of life taking thousands of photographs and hundreds of videos. Those may seem like outrageous numbers, but it's easy to be overzealous when you carry a camera in your pocket all day. Now imagine starting your own business and creating lists of business contacts and clients, invoices, and all of the accounting information that goes along with a business. Most likely, all of this information will be stored on your computer and, unfortunately, every computer stops working at some point.

If you have been diligent in backing up all of these precious files, then losing a computer is not a big deal since you only need to replace that computer. However if you have not been diligent, you have now lost priceless memories of your child and your entire financial livelihood. There are countless stories of people losing critical information due to a computer crash. Fortunately, all of this trouble can be avoided with a little bit of work.

Golden Rule of Backups

Before choosing a backup method, there is one rule about backups of which you should be aware. You should have at least three copies of all important data. This means the original plus two backups of truly important files. If you only backup your information in one place, such as on an external hard drive which you keep stored with your laptop, you could easily lose both if you leave your bag somewhere or there is a fire in your home. This is why one copy of all of your important data should always be kept off-site. Many people will not have the diligence to maintain three copies of information, so below I will provide some options for automating the entire backup process.

Easy Backup Options: On-Site  

Mac

If you own a Mac, buy a large external hard drive and use Time Machine. When you first plug-in the external hard drive, your Mac will ask if you would like to use that drive for Time Machine backups. It's literally that simple.

PC

I feel comfortable ending the Mac conversation that quickly because Time Machine is a well advertised feature. Windows has a similar feature that few know about called "File History." If you own a Windows 7 or newer PC, you can easily take advantage of this feature by connecting an external hard drive then going to control panel and opening the File History dialog box. Then it's as easy as pressing the "Turn On" button. This will automatically save all files from your libraries, desktop, contacts, and favorites to the selected external drive. For advanced users, you can also add a network location for File History to store backups.

Easy Backup Options: Off-Site

In order for a backup option to be considered easy, it needs to be completely automated and require little to no maintenance. That means you're going to be backing up to the cloud to store your important data off-site. Many people already use online storage services like Dropbox, Google Drive, or OneDrive to store some of their data and sync it between computers. This is useful and keeps your data safe and available, but it is not necessarily secure. If this is your chosen method of off-site storage, it is better than nothing, but you can do better. 

A recent survey by the University of Kent found that 1 in 30 people have had their data affected by CryptoLocker (although there was likely some significant sampling bias), which is a virus that encrypts your data and requires you to pay a ransom to get it back. If those affected had a backup of their data, it would not have mattered because they could have restored their information without paying the ransom. However, CryptoLocker is capable of encrypting your Dropbox, Google Drive, or OneDrive files if you have them syncing to your computer.

Solution: Use Carbonite

Carbonite is an automated cloud backup service which backups up your entire computer to its servers continuously, allowing you to restore any lost data at any time. Since Carbonite provides file versioning support, CryptoLocker would not have been able to cause a loss of data. Additionally, Carbonite is more secure than the other cloud storage services mentioned above because all of the data is encrypted with an encryption key which you control. This means Carbonite cannot look at your data, all they can do is store and restore it for you. This service is relatively cheap considering the aggravation it will save you.

UPDATED 6/2/2014: The following three automated online backup services are considered some of the best currently available, based on various consumer reviews I have read. All of the services below offer the ability to create your own password to encrypt your backup, meaning that the company will not be able to access your data.

1. Carbonite: currently priced at $59.99 for one year of unlimited backups for a single computer.

2. CrashPlan: also priced at $59.99 per year for unlimited backups of a single computer.

3. Backblaze: this relatively new competitor offers a number of advantages over the other services. First, the price for a year of unlimited backups is $50. Additionally, Backblaze allows you to also backup your external hard drives, which other services do not allow.

Conclusion

Backups used to be difficult. However, there are many features built-in to modern operating systems which will save you the hassle of remembering to backup. Cloud backup and storage services also save you the hassle having to remember to backup. At this point, there is honestly no excuse for not ensuring your data is safe and secure.

Use LastPass to Manage Your Passwords and More

A couple of people have asked me about LastPass recently so I figured I would write a little about it generally and, more specifically, about how I use it.

No one actually likes using more than one password. We all have enough things to remember, we don't need to clutter our brains with remembering which password we used for which website. That is where password manager apps come into the picture. With such apps you only need to remember one password and the rest of your passwords will be stored, hopefully securely, in the app.

As far as I am concerned, LastPass is the best-of-breed app in the password manager category. In the infancy of its popularity, LastPass was vetted and approved by Steve Gibson. He provides a lengthy review of LastPass' technology at his site and, fortunately, LastPass has continued to be open about its technology. It has also continued to add features over the past few years, both to improve functionality and security.

Why is LastPass Secure?

The company has done everything right in terms of securely encrypting and storing your sensitive data.

• All encryption is done locally on the client, so all the LastPass company ever sees is a pseudorandom blob of data.
• Encryption is performed with AES-256.
• Password hashing is performed using PBKDF2 implemented with SHA-256.
• Users can select the number of hash iterations their password is put through.
• Many multi-factor authentication options are supported.
• Mobile app access to your LastPass vault is restricted. Users have to explicitly allow access from each new mobile app that tries to access their vault.
• The company is open and responsive about their technology, as evidenced by Steve Gibson's interactions with them.

LastPass Has Extensive Functionality

In addition to being secure, LastPass provides an enormous number of features.

• Ability to store and automatically fill forms on websites with many types of personal data, including usernames, passwords, credit card, and address information.
• "Secure Notes" allow users to store other information of their choosing, including text, documents, and images.
• Supported on Windows, Mac, and Linux using Internet Explorer, Safari, Chrome, Firefox, and Opera.
• Extensive mobile support. Apps are available for iOS, Android, Windows Phone, and Blackberry. LastPass Premium is required for this, but it is only $12 per year.

How to Keep Your LastPass Vault Secure

While LastPass is natively secure, there are a number of configurable options which can make LastPass even safer against attackers. All of these options can be configured in the settings page of your LastPass vault.

• In the General tab, make sure your password hash is being iterated at least 5000 times, you only allow logins from countries to which you travel, and you disallow logins from the Tor network.

• In the Security tab, enable as many circumstances as possible for which you will be prompted for your password.
  

• Enable some form of multi-factor authentication.

Conclusion

LastPass is the perfect tool for managing your many online identities, as well as storing sensitive personal information, both securely and conveniently. It's easy to use and available on every platform. Plus, the price is right.

Beginnings...

This post originally stated that this was going to be a blog about information security but I decided to expand the topic of this blog to anything technology related, so you can see the edited version below.

As this is the first post on this blog, I feel like it is important to discuss what you will be able to find here as time goes on. While the length, detail, and formality of entries here will vary greatly, all of the content here will focus on technology. Obviously this is a broad topic, but my intent is for the topic to be broad. I will be writing about anything that catches my interest in the realm of technology.

I want to welcome people of all skill levels to read this blog and discuss the posts. We all have our own areas of expertise and it's always best to learn from each other whenever possible.

I have also begun to maintain a collection of files on this site related to technology. This collection will also range in content, but as the collection grows I will make sure to change the organization as necessary. For now, please take a look at the presentation I posted about the basics of information security. I recently gave this presentation to a small group of lawyers who needed more information about security principles in order to understand HIPAA compliance matters and the presentation was very well received.