Hyper-V & SMB Direct (RDMA)

Windows Server 2012 brings an update to SMB which has so many new features in it Microsoft have bump it up an entire version and call it SMB 3.0. Many of the new features included in are there to directly improve your experience with Hyper-V.

Hyper-V now supports your using SMB storage for your virtual machines which opens up whole new possibility for deployment scenarios, which will benefit not only large corporates but will also make highly available virtual infrastructures available for the smaller customer without the costs of dedicated SANs and complexity of fibre and iSCSI LUNs.

One of these new SMB 3.0 features is SMB Direct which makes use of RDMA (Remote Direct Memory Access). RDMA  allows for computers on the network to send and receive data without having to use processor time, interrupt the OS or cache. This obviously aids with VM density – you will be able to have more VMs on your host machine as the processor won’t be so tied up with network operations but also allows for data transfer with very high throughput with ultra low latency.

RDMA works by using a protocol on the NIC (you need to make sure you purchase an RDMA NIC – both servers will need an RDMA compatible NIC) if this hardware is in place is makes it possible for one computer to directly read data and write data to another computers memory.

As mentioned above you need to have the correct hardware in place and that involves having the right NIC which re sometimes known as R-NIC. There are currently three different types available from various different manufacturers. The three types are: iWARP, RoCE an Infiniband.

Setting up your server infrastructure to support this could not be simpler – you don’t need to do anything! When two computers start to talk they make a standard connection via TCP, once the connection is established they share information about what they are capable of doing (data transfer also beings at the same time so there is no overhead or latency) once both computers have decided they are both capable of running SMB 3.0 and have RDMA capable hardware they will seamlessly switch.

Using some of the new NICs that are available from vendors such as Mellanox (the ConnectX-3) you are able to get incredible speeds up to 56Gb/s!!! when using Infiniband. These are some amazing speeds but when you start to pair this with SMB 3.0’s new multi channel feature and Windows Server 2012 network teaming capabilities the speeds possible really are incredible. Jose Barreto who works on the file server team at Microsoft has a blog post on how to configure the Mellanox.

Microsoft presented some stats on how using SMB data storage with RDMA performs which are defiantly worth having a look at.

I would be very interested to hear from people who have started to play with technology and see how you are finding it in a real world environment.

RDMA compatible NICs are defiantly something to add to your shopping list next time you are purchasing server infrastructure.