The Storage Guy

I often get inquiries from IT folks (usually systems or network administrators) that are interested in breaking into storage management.  They generally want to know what kind of work the job entails, how it meshes with other IT disciplines, and what kind of skills they need to acquire to have a shot at a storage analysis position.  In this post, I'm going to take a stab at the last question and address the skills and experience I'm looking for when it comes to hiring. Obviously, this is a topic that can be highly subjective and I'm limited to my own anecdotal experience, but hopefully this will be beneficial to those of you that are looking at moving into (or moving up within) a storage role.

 

It goes without saying that soft skills are important in IT (or any job, for that matter).  Although we rarely interact with end users, we spend a substantial portion of our time interacting with our peers in storage and other IT disciplines, vendors, contractors, etc.  As such, the manner in which you present yourself via your résumé, during your interview, and through other methods of communication will play a major role in your success.  Polish your résumé and have someone else look it over.  Prepare for your interview ahead of time and make sure you're knowledgeable about the core subject areas involved.  If you communicate via e-mail or another "informal" method with your interviewers, make sure you use the same level of discretion used on your cover letter.

 

When it comes to technical skills, I always look for people with that have a solid understanding of centralized storage concepts. I don't really expect storage-specific certifications unless I'm hiring an engineer or architect, but I expect storage analyst candidates to know the basics of enterprise storage.  I've provided a handful of questions below that I routinely ask during interviews.  They're all very basic questions that are intended to meet two purposes.  First, what is the candidate's knowledge level with basic storage concepts?  Second, and perhaps more importantly, is the candidate comfortable enough with these technologies to come up with a reasonable application of those technologies when faced with a real-world scenario?

 

Here are a couple of questions I routinely ask, along with sample answers:

 

1) Describe the difference between block-level (SAN) and file-level (NAS) storage. How does each type of storage operate? Compare and contrast the two storage archetypes and give me examples of scenarios in which you would choose one over the other.

Block-level storage utilizes the storage appliance as a simple data repository.  The host (e.g. an ESXi host) connected to that storage appliance is responsible for managing the file system (e.g. VMFS).  Since the host is managing the file system, the storage appliance has very little insight into what has been written.  This can be a very efficient storage method and has a lower impact on the storage appliance resources, but it also limits the ability of the storage appliance to interact with the data.  For instance, the storage appliance has no idea where one file starts and the next begins, there is no way to do file-level snapshots at the storage appliance.  You also can't have end-to-end deduplication with these technologies.  To continue with the ESXi example, you can deduplicate datastores at the VMFS level and you can deduplicate again at the LUN level on the storage appliance, but there is no way to perform end-to-end deduplication.  Fibre Channel (FC), iSCSI, and Fibre Channel over Ethernet (FCoE) are good examples of centralized block-level storage.

File-level storage is just the opposite.  The storage appliance not only holds the data but is also responsible for maintaining the file system.  Since the storage appliance owns the file system, it knows where files begin and end.  This permits file-level snapshots, meaning you can restore a single file or directory from a snapshot instead of restoring the entire LUN.  This can be a huge time-saver.  Moreover, you can do end-to-end deduplication, meaning you can perform a single deduplication operation on the LUN from the storage appliance and realize your entire space savings at the LUN level.  CIFS (Windows) and NFS (UNIX) are the two predominant file-level storage standards.

 

2) Give me an overview of a Fibre Channel storage environment. Tell me about the components of a Fibre Channel infrastructure and tell me what each component does.

Fibre Channel has long been the dominant centralized enterprise storage medium.  Newer, cheaper technologies like iSCSI and FCoE have stolen a good portion of the market, but fibre channel remains the dominant standard for datacenter deployments for the moment.  Fibre channel networks generally consist of at least one storage appliance, one fibre channel switch, and one or more host bus adapters (HBAs, effectively a NIC and a SCSI adapter in one).  Of course most large-scale deployments utilize multiple FC switches and storage appliances in a mesh configuration.

 

3) Tell me about the pros and cons of server virtualization. What are some ideal candidates for virtualization? What types of systems would you avoid virtualizing?

Server virtualization is pretty much a standard approach to managing servers these days, and desktop virtualization is also making great strides.  Virtualization is a method of hosting multiple autonomous systems on a single piece of hardware, with a "hypervisor" providing a common platform on top of that hardware.  VMware is by far the biggest player in the market, with Microsoft and Citrix also having sizable shares.  Any production environment is going to have a minimum of two host servers, with an N+1 methodology being a common starting point.  Essentially, if you're putting some or all of your eggs in one basket, you want to make sure you've got a very reliable basket.  You want to make sure you've got enough overhead to permit at least one server to fail or be taking down for maintenance/patching without running out of capacity to support your guests.

One of the most obvious benefits to virtualization is the abstraction of the guest OS from the physical hardware.  With the appropriate licensing, you have ability to seamlessly move a VM from host to host without impacting end users.  The VMware-branded name for this technology is vMotion.  This can be done manually if you need to shut down a host for maintenance, or it can be done automatically to meet a variety of needs.  VMware offers a technology known as Dynamic Resource Scheduling, or DRS, that will automatically balance your virtual machines across a cluster of hosts based on factors such as CPU consumption, memory utilization, network IO, etc.  A newer technology known as Storage DRS performs a similar function for datastores, automatically migrating VMs from datastore to datastore with a goal of balancing datastore capacity and storage IO.  Finally, a technology called High Availability, or HA, will monitor your host servers for failures.  If a failure is detected, your VMs are automatically booted up on other hosts within the cluster.

Choosing ideal virtualization candidates can be somewhat tricky.  When moving existing physical servers to a virtual environment (known as physical-to-virtual, or P2V), it's always easiest to start with the low-hanging fruit.  Choose the one-off application servers that aren't heavily taxed.  Systems running on legacy hardware are another valuable commodity, as you can eliminate risks to your infrastructure by getting rid of crufty systems.  Save your more complex systems for last, including systems with heavy resource utilization and IO loads (e.g. SQL servers).  Some of these systems may not be ideal candidates for virtualization.  Also, watch out for systems that require special hardware such as modems, USB license dongles, and sound cards.  Some of these will need special care when virtualizing, and others may not be convertable at all.

In short, the goal of this question is to get an idea of the candidate's familiarity with virtualization technologies and VMware in particular.  Depending upon the candidate's response, I may dig deeper into storage-specific questions in a VMware environment.  For senior-level candidates, I might ask about raw-device mapping (RDM), storage IO control (SIOC), multipathing in ESX, or other similar concepts.  Again, the goal is to determine just how familiar the candidate is with enteprise storage in a virtualized environment.

 

4) Our environment is incredibly heterogeneous. We have a wide variety of storage appliances from virtually every manufacturer under the sun fulfilling nearly every imaginable role. How would you consolidate the management and monitoring of these disparate devices?

For this question, I'm really looking for answers that either enumerate the candidate's knowledge of the field, or creative answers that show me a candidate can think quickly on their feet.  For senior-level storage analysts, engineers, and architects, I would expect a response about specific products the candidate has utilized in the past.  If they're coming from a heterogeneous environment, I would expect vendor-specific answers such as EMC Unisphere Server, NetApp OnCommand Unified Operations Manager, etc.  However, I would press the candidate for a vendor-agnostic solution such as SolarWinds Storage Manager or EMC's upcoming Project Bourne.  We have a fair amount of legacy hardware along with some oddball products that can't be centrally managed by much of anything, so I might follow up by asking the candidate to describe how they would manage the hardware that doesn't fit into their overarching management application.

 

5) One of our major goals moving forward is to pull IT assets out of our 40+ facilities and consolidate them into dedicated, centralized data centers. Tell me how you would manage this effort. Which assets would you consolidate first, and how would you accomplish it? (Looking for commentary on virtualization, centralized file systems/stubbing, centralized backup/archiving, etc).

This is a big one.  This isn't really a common interview question so it tends to catch candidates off-guard, which is really ideal for me as an interviewer.  It gives me an opportunity to see if the candidate can think and react quickly.  This is intentionally a very open-ended question and I've received some very thought-provoking answers.  There are several different angles the candidate could take.

Virtualization is an obvious route.  Internal candidates are aware of our big push toward virtualization at all of our facilities, so they may discuss the foreward-looking plan to move as many VMs as possible out of the facilities and into our centralized datacenters.  I would expect a candidate to discuss the improved security and reliability of the infrastructure at those facilities, as well as the simplified management that would come from this form of consolidation.  In short, it's a cheap and relatively simple way to move critical infrastructure out of the facilities.

Many candidates also discuss migration of file shares.  We're in the process of replacing traditional Windows-based file servers (some clustered and some not, some DAS and some shared storage, etc.) at the facilities with CIFS-based shares hosted directly on dedicated storage appliances.  Many candidates discuss distributed file system (DFS) and similar technologies that can be implemented seamlessly today and leveraged later to migrate users to remotely hosted shares.  Many candidates also discuss the implementation of storage-level replication technologies to migrate this data.  Finally, some candidates discuss stubbing technologies that allow infrequently-accessed data to be moved to a remote repository and replaced with a file stub (think "shortcut") that points at the new location.  All of these are excellent strategies, and I always appreciate hearing a candidate suggest a combination of them to come up with a plan that will meet the diverse needs of our customers.