The Bandwidth on the wire is efficiently used, means all the bw available is used at any time needed. For example, if a NIC partition is assigned certain bw say 2G and if that partition needs more that 2G at certain time when it ends up with burst traffic such as vMotion, the dynamic bw feature will allow it to use more than the limit if the all 10G bw on the wire is not fully utilized by other partitions. This is efficient way of using all the bw available on the wire vs HP assigns BW per flexNIC and it will not share with any other partition even if needs it…that means very static partitioning…minimizing effective use of all the wire speed when needed. Admin as to fine tune configuration per need….which could change day to day. Huge differentiation for our NPAR.

Npar provides makes things economical….as it replaces multiple adaptors including storage and Ethernet if it supports FCoE and iSCSI. Now you can partition one to do Storage function with appropriate bw like 5G or 6G as needed.

Npar supports capability to have Jumbo frames on one partition while other partitions use normal frame size….this eliminates the need to have a separate adaptor. That is a unique differentiation which lets customer to have only one adaptor for iSCSI storage with Jumbo frames enabled and normal LAN traffic on the same wire. Less adaptors, cables, and switches….

NPAR is OS agnostic, since it is enabled at bootup it is presented to OS as though they are normal physical NICs…so OS don’t have to change any behavior. So working with all the standard and advanced functions adaptors support today. Very much deployable in to existing infrastructures bringing these new technological and economic advantages in to existing data center.

NPAR can address NIC/adaptor sprawl, while making it easy to transition to 10G, even your application fully don’t need all the 10G bw. It brings you with speeds that are in between 1-10G to fine tune to your app needs…

To summarize the Dell Npar differentiation:

• Switch agnostic implementation
• Dynamic BW use and flexible bw size allocation
• Make things more economical- less adaptors, less switch ports, less cables!!
• Supports Jumbo frames for iSCSI
• OS agnostic
• Solution to NIC sprawl

Let me know your thoughts…

Cheers

Sri

That’s totally not a really smart idea!

Read this from UCS configuration guide:

Pinning Server Traffic to Server Ports

All server traffic travels through the I/O module to server ports on the fabric interconnect. The number of links for which the chassis is configured determines how this traffic is pinned.

The pinning determines which server traffic goes to which server port on the fabric interconnect. This pinning is fixed. You cannot modify it.

As a result, you must consider the server location when you determine the appropriate allocation of bandwidth for a chassis.

You must review the allocation of ports to links before you allocate servers to slots. The cabled ports are

not necessarily port 1 and port 2 on the I/O module.

If you change the number of links between the fabric interconnect and the I/O module, you must reacknowledge the chassis to have the traffic rerouted.

Note

All port numbers refer to the fabric interconnect-side ports on the I/O module.

Chassis with One I/O Module (FEX)

So you can have two servers per uplink leading to 2:1 subscription if all the four links are used, but the traffic flows are fixed. For example, traffic from Server 1 and 5 share uplink1. You don’t have a choice to change that, remember pinning is fixed and you cannot modify, but your choice is to decide if  two servers A and B has higher bandwidth requirements, then you don’t want them in Slot 1&5 or 2&6 or 3&7 or 4&8 together, to avoid bandwidth starvation. So you may want to plug Server A with Server C because server C needs little bandwidth. If all the servers need higher bandwidth, then all of them will starve now and then…..

What happens if your uplink bandwidth needs change?

For example, let say you had 2 uplinks connected and you made sure servers are plug in to appropriate slots such that none of them starve for uplink bandwidth. Finally you optimized that.

Then later you decided that you connect more uplinks to support your new bandwidth needs, let’s say 4. Since the server slots pinned to a specific uplink changes as you connected 4 uplinks, you have to figure out again which slot specific server should go so that they don’t compete for uplink bandwidth and avoid starvation. 

I think Cisco “Pinned” themselves wrong with UCS!! Don’t you??

Cheers

Sri

Here is John’s post:

“The slogan “The Power to do More” has never been more appropriate when discussing Dell’s latest PowerEdge 2 Socket servers combined with the latest 4 and 6 core Intel Xeon 5600 Westmere-EP processors.

On February 14, 2011 Dell announced a very significant update to the entire lineup of two-socket PowerEdge servers, including the M610, M710, M610X and M710HD blades, R710, R610, R510, R410, C1100, C2100, C6100 rack servers, and T710, T610, T410 towers with new Intel Xeon 5600 series Westmere-EP processors.

It is reasonable to assume that a great deal of attention was directed at Dell by the tech world considering the extensive depth of the refresh, especially when it offered the latest Intel 4 and 6 core processors at previous dual core price points. Yes, that is right, up to six cores at previous price parity.

However, one day after the Westmere-EP announcement Dell released financial results that were so far above financial analysts expectation that the buzz throughout the world about Dell’s financial success overshadowed what may just be one of the most significant processor upgrades in Dell’s product line since the release of Dell 11th Generation Power Edge servers in 2009.

Dell Power Edge Blade, Rack and Tower servers are widely popular with a variety of customers for their ease of management, virtually tool less serviceability, power and thermal efficiency and customer inspired designs, but when it comes to performance they have once again leap-frogged the competition by offering significant gains in performance with the latest Intel Xeon 5600 series refresh.

As an example, testing conducted by Dell’s internal laboratories using a 1U 2Socket R610 rack server equipped with the new X5687, 3.60 GHz Intel Xeon processor, delivered previously unachieved performance benchmarks above any other competitors 8, 12, and 32 core product in both SPECint_rate2006 and SPECfp_base2006 as detailed on www.spec.org. When reviewing the benchmark list the competitors which Dell testing results blew past is extensive and includes products from IBM, Fujitsu, Hitachi, NEC, and Oracle/Sun, just to name a few.

Dell’s use of the Intel Xeon 5600 Westmere-EP processor to refresh its lineup of two socket servers offers customers the opportunity to realize outstanding integer-computing throughput and floating-point computation performance at a level previously not available on Dell Power Edge 2 socket blade, rack and tower servers.

Get past the technical buzzwords and what it means to Power Edge customers in real-terms is that they can expect to see improved performance as they share, access and update the most commonly used files across their network including word documents, spreadsheets, graphics presentations, 3D applications as well as various other financial, analytical and highly algorithm dependent data shares.

Every newly released Intel 5600 processor regardless of where it appears in the stack, offers AES-NI and TXT technology specifically designed to reduce malware risks so virtualized mission critical applications are more secure making previously dedicated servers available for upgrades, repurposing, or replaced as business needs change.

For those waiting to perform a refresh so they could dive deeply into the virtualization era or IT managers struggling to keep their servers online with duct tape, bubble gum and a prayer, the time has never been better to upgrade to Dell Power Edge 2 socket servers with the latest Intel Xeon 5600 series processors. To find out how you and your organization can get “the Power to do More” click http://www.dell.com/poweredge”

He shed light on various aspects of these servers and why he thinks these Dell Blades are best enabled to let to with the power to do more….I agree with him.

What do you think? Glad to hear from you…

Cheers

Sri

Since the bandwidth is a constraint in 1Gig world as today’s applications grew out of those bandwidths available, we needed to manage traffic congestion, and all these technologies that talk about dedicated bandwidth per traffic type made much sense…

But when you move on to 10GE connectivity…That is a different story!!…Since you got more bandwidth than you need for today’s traffics and while that guarantees congestion free traffic flows; traffic management tools like NIC partitioning techniques like HP Flex-10, VMware Rate Limiting, Network IO control (NetIOC)….are not needed at all.

At this point, all these bandwidth management schemes are unnecessary and should not be used to avoid the complexities that they bring along with feature functions they support..

10GE brings much needed port consolidation (especially in compact form factors like blades) and also a bandwidth capacity that is more than you need for today’s applications and traffics..

We have to make that point clear to the technology consumers and shift them from old school thinking of dedicated physical ports with specific bandwidths, etc…for meeting traffic isolation and bandwidth needs; to new school thinking of sharing the same wire and using VLANs and other techniques.

To sell their stuff…what some companies (selling proprietary techniques like Flex-10) are doing is, fanning the existing paranoia or fear of critical traffic not having enough bandwidth on the wire…while not highlighting that 10Gig actually solves bandwidths needs by itself, on it’s own, & no need for special techniques, period!!

If your core infrastructure is not ready for 10Gig connectivity, then you should look for high port density solutions like Dell 1GE Quad Port NICs with PowerConnect M6348 high port density Switches for M1000e Blade Chassis…to serve the need to have more ports to support virtualization.

Right industry standard technologies are evolving to support the need to improve efficiency in IO virtualization like Single Root IO Virtualization (SR-IOV)…which is much needed, to bring the benefits of Virtualization to every type of computing including HPC.

SR-IOV is not about carving bandwidth, in fact it does not do that at all. It virtualizes a physical NIC and presents as multiple devices to hypervisor…hypervisor does that today. So SR-IOV technique is offloading that work to a NIC and leaving more cycles for hypervisor to do important things like supporting more VMs and running them more efficiently. Also it supports direct access to these devices from guest VMs/OSs, bypassing hypervisor…drastically improving efficiency while still able to do vMotion as hypervisor still have access to control plane of these devices. All these devices share the same wire…so there is no dedicated bandwidth per device here…remember 10Gig is enough!!

When in future, if Users run into bandwidth contention, even with 10Gig NICs because apps grew out and started generating larger traffics, then they can add more, like they do today with 1Gig NICs…

In a “worst case”, where they still run in to bandwidth contention as they ran out of space to add more 10NICs, then we still have better tools like NetIOC or Rate Limiting for traffic shapping….and I don’t see that need in the near future.

Hope this makes sense…and provides the bigger picture on 10GE for Virtualization..

Let’s get it right, the correct perception of using these 10GE super highways…where you don’t need traffic management, when there is no congestion….in my words “simply don’t add complexity when not needed”. 

In the mean time, we are working hard to make those technologies that are much needed like SR-IOV, happen in real products…as it makes better sense.

That’s my take on this topic…what’s yours?

Here are few papers out there which dwells deep on this subject,

1. “Simplify VMware vSphere* 4 Networking with10 Gigabit Server Adapters”

2. “Optimizing QoS for 10GE in Virtual Networks”

3. 
   

   “Virtual Switches Demand Rethinking Connectivity for Servers”

   Signing off for now…..Sri

During the conversation regarding UCS scalability and bandwidth management, he mentioned about how easy it is to get UCS chassis added to data center. I thought I share that conversation. So read on….what Brad said and then what I said….

——————————————————————————————————————————-

Here’s how many steps it takes to get a Cisco UCS chassis added to the data center:

1) Connect the power cables

2) Connect the Ethernet cables

3) Acknowledge the chassis in UCSM

–DONE–

 How many configuration steps are required to add a Dell chassis to the data center? That’s my challenge to you.

 Cheers,

Brad

——————————————————————————————————————————-

Keep reading…

——————————————————————————————————————————-

Brad,

Another big claim from Cisco and now you, is how easy it is to get UCS added to data center!! How easy it is to get those computing nodes working in three steps…and help customer business just like that I wish it is true…

 For a second, I will take your word on it. I will complete those three steps like you said and say DONE! Do you think, now I got my Cisco servers running and handling my business?

 In fact, I know I will end up with just more HW powered up and do nothing but creating electric bills You failed me Brad!

 Now don’t tell me, I have to do more. Like before I do those 3 steps, I have to have or if I don’t, then setup the infrastructure to deploy servers (let’s start with image deployment, application deployment, Virtualizing servers etc…).

 I don’t think UCS does all that, it needs third party tools. So, I have to make sure all those tools which I use are integrated in to this new UCS Manager API. (That definitely needs a lot more steps not just three!!)

 If only 3 steps like you said are needed, Cisco don’t need to create new company ‘Acadia’ with the help of partners to provide services to help customers with setting this UCS system in data center. So they know it is complex enough so customers will be willing to pay for it. Cisco sees this as another business, so they started a venture capital to sell Vblocks…

 In fact, Cisco just delivers pieces of HW with an optimized device manager to manage them, taking advantage of still evolving technologies such as DCB, FCoE, SR-IOV…etc, based upon pre-standard proposals/drafts or proprietary ones, remember VN-Tag…

 I don’t think Server Customers are that naive, to believe that Cisco came up with this revolutionary platform that will auto magically in three steps make their data centers as a dynamic IT infrastructure. Years of providing server technologies by companies like Dell, provided some deeper understanding of what it takes to deliver an optimized data center to support business needs, I don’t think Cisco lack’s that…wait a minute! Or Does it? So, why are they asking customers to ‘Nuke & Pave’ their existing environments  and go with new architecture with UCS, in order to get those efficiencies. Definitely they know networking side of the business needs and services, but are we sure if Cisco knows about Server side of it…

In my humble opinion, I think Cisco failed to push Nexus into data centers from last 3 years, (as customers are not in favor of adapting Nexus OS with subset of  features as compared to Catalyst IOS). So back to selling vision game plan and they are selling this as new vision ‘Unified Computing System’, to sell more Nexus gear. Didn’t Cisco do some things like this before; remember Unified Communications vision to sell more Catalyst gear….

So, selling another vision!! Great new vision or just a polished version of original data center vision, server and virtualization vendors had 5 years back? Kudos to Cisco marketing…for painting that rosy picture . I agree they really brew good cool-aid..

 I am sure you will not moderate my response and will post this one as is! And dissect it with all honesty. 

Yes with all honesty Dell Chassis needs more steps and does lot more! By the way, you know, we tell customers that there is no need to ‘Nuke & Pave’ their environments, in order to get those new data center efficiencies. Just leverage all those technologies you trusted your business with all these years, while embracing newer complimentary technologies like 10GE, Converged Enhanced Ethernet (per DCB), FCoE, iSCSI, and IO Virtualization, to name a few..

 Cheers again!

Sri

——————————————————————————————————————————-

Wondering how many really think this new flavor of old Data Center cool-aid which Cisco selling aggressively is different and revolutionary from what we envisioned, even before Cisco learned, “What is a server?” ….

Please share your thoughts…

–        By default, PowerConnect Switches are in “Normal” mode

–        New series of SimpleConnect LAN IO Modules exclusively work in Simple mode.

Providing ability to aggregate traffic from multiple downlinks to a single or fewer uplinks and provide port consolidation at the edge of Blade Chassis.

The Port Aggregator feature minimizes the administration required for managing the Blade-based switches.

Benefits:

1. Ease of deployment/management for I/O Modules for Blade Server connectivity –Is simple to configure; just map the internal ports to the external ports, assign a VLAN to the group (if required). This feature automatically configures multiple external ports into an LACP trunk group.
2. Reduce involvement of network admin in blade deployments –Eliminates the need to understand STP, VLANs, & LACP groups.
3. SimpleConnect is completely interoperable. Full integration of PowerConnect blade switches with 3rd party I/O H/W (Cisco, etc.)
4. Provides cable aggregation benefit offered by integrated blade switches.
5. SimpleConnect feature provides loop-free operation without using STP.
6. It works across a stack of switches (M6220 and M6348) so that you can now manage switches as one via the easy-to-use interface. 

Check out this technology overview video series on Simple Connect LAN modules for M1000e Blade Chassis.

Video1: Shows how simple it is, to deploy blade servers with SimpleConnect, into existing DataCenter LANs.

Video2: This one shows how to deploy blade servers with SimpleConnect, into existing VLANs in customer DataCenters.

Video3: Shows you how easy it is, to manage Blade Server traffic flows with SimpleConnect connecting them to your existing LANs or VLANs in your Data Center.

Technique behind it:

–        The downlink and uplink ports are grouped with a reserved VLAN, so that traffic can only be flooded within that group, and cannot go across.

–        Uplink ports are enabled as members of LAGs (ie: form a Link Aggregation Control Protocol (LACP) trunk group) to connect to uplink switches.

–        Enabling multiple downlink traffic aggregation to fewer uplinks with load balancing capabilities.

Configuring Simple Mode when connecting to Cisco switches

1. Boot the M6220/M6348/M8024 switch with the default configuration.
2. Configure four ports (port 1 to port 4) to be a member of a dynamic LAG on the Cisco switch.
3. Connect a link from port 1 on the Cisco switch to any 1Gig external port on the M6220/M6348/M8024.
4. Send traffic from both sides. Traffic is switched.
5. Add another link from port 2 on the Cisco switch to any 1Gig external port present on an M6220/M6348/M8024 switch. The newly added link would automatically be added to the LAG that is being used by the Port Aggregator group and data would be load-balanced.

SimpleConnect Blade Servers with PowerConnect 6220 in Dell M1000e

SimpleConnect Blade Servers with PowerConnect 6348 in Dell M1000e

SimpleConnect Blade Servers with PowerConnect 8024 in Dell M1000e

SimpleConnect provides flexibility to configure multiple port aggregation groups, to meet customer needs to group specific blade servers to manage traffic flows as needed.

SimpleConnect provides flexibility to support traffic flows per customer needs

Also this is designed to simply connect virtualized blade server avoiding complex configuration and management of standard switches.

SimpleConnect simply connecting virtualized Blade servers

Have any of you implemented this in your DataCenters today? Let me know…

HP sponsored Tolly report saying UCS bandwidth sucks as compared to HP BladeSystem…

http://www.tolly.com/Docdetail.aspx?Docnumber=210109

Brad Hedlund, Data Center Solutions Architect at Cisco Systems, calling it Folly:

http://www.internetworkexpert.org/2010/03/02/the-folly-in-hp-vs-ucs-tolly/comment-page-1/#comment-789

Tolly’s paper, it actually exposes bandwidth bottle necks with FEX “pinning” feature and it’s diminishing effect on UCS scalability. 

Here is my analysis on it:
This surprised me as I understood it! As I understood this ”pinning” feature of UCS Fabric Extender (IO Module)!!! It has fixed configurations depending on how many uplinks are used to connect FEX to 6100 FI.

From UCS Manager GUI config guide:
Pinning Server Traffic to Server Ports
All server traffic travels through the I/O module to server ports on the fabric interconnect. The number of links for which the chassis is configured determines how this traffic is pinned.

The pinning determines which server traffic goes to which server port on the fabric interconnect. This pinning is fixed. You cannot modify it. As a result, you must consider the server location when you determine the appropriate allocation of bandwidth for a chassis.

You must review the allocation of ports to links before you allocate servers to slots. The cabled ports are not necessarily port 1 and port 2 on the I/O module. If you change the number of links between the fabric interconnect and the I/O module, you must reacknowledge the chassis to have the traffic rerouted.

For example: if you use 2 uplinks today and tomorrow you decide to use 2 more, then to optimize bw use of servers, you got to physically shuffle servers, so that two busy server don’t pair up. In a virtualized evironment this means atleast move VMs around right physical server pair for proper bw optimization!! Otherwise they may have to fight for BW

So you can have two servers per uplink leading to 2:1 subscription if all the four links are used, but the traffic flow are fixed.

For example: Traffic from Server 1 and 5 share uplink1. You don’t have a choice to change that, but your choice is if server A and B has higher bw requirements, then you don’t want them in Slot 1&5 or 2&6 or 3&7 or 4&8 together, to avoid BW starvation. So you may want to plug Server A with Server C because server C needs little bw.

This paper exposes this fundamental design problem and highlights it as limited BW aggregation capability. In the enthusiasm of doing that, they forgot that Cisco uses two FEX modules and both can be active at the same time. So effectively aggregate uplink bw will be 9.1 x 2 = 18.2 Gbps per two servers if all uplinks are used.

So from server pair point of view, if it has 2×10gig CNAs, then 40gig downlink traffic should share 20Gig uplink bw. Means 2:1 oversubcription….

If scaled to 320 servers as Cisco UCS claims, then the oversubscription will be 8:1, in other words, if customer has apps running on these blades that need high bw, then scalability story runs short quickly..

I have posted a comment on Brad’s blog to his response to Tolly with his Folly, I wonder if he is going to publish it and is up to my challege to explain the bandwidth and scalability story truly….

So this is my TRULY for that TOLLY and FOLLY….

What is yours?

Now it’s more than that! …

Phase 2 adoption of 10GE is with Enhancements to Ethernet to support FC traffic, leveraging widely implemented FC storage infrastructures in data centers. Since new technologies are added and being evaluated, this phase 2 adoption needs to go thru maturity period. These new convergence technologies CEE/DCB (Converged Enhanced Ethernet based of Data Center Bridging IEEE standards) and FCoE (Fibre Channel over Ethernet) get implemented in the Server/Network edge first, then Core of the network, and later at Storage End with FCoE Arrays etc… 

‒      Provides Single Fabric: Reduces management and deployment costs, as Fabric of Choice

‒      Optimized for virtualization: More VMs means more server and network traffic so an efficient network is required. Multi-core processor-based platforms handle larger workloads so need  faster network throughput to reduce latency to storage devices

 ‒      Drives storage over Ethernet: Consolidate LAN and SAN (w/ FCoE) FC infrastructure/traffic on a single 10Gig Enhanced Ethernet based of DCB standards, with full traffic isolation & robust Quality of Service with Congestion Management.

Single wire with 10GE

10GE Turbo-Boosts Infrastructure Consolidation in Data Center

–      Reduces the number of cables and server adapters

–      Lowers capital expenditures and administrative costs

–      Single fabric type simplifies setup and VM migration  

–      Reduces server power and cooling costs

–      Blade Servers and Virtualization drive consolidated bandwidth

So much going on with 10Gig Ethernet, as we see the hockey stick happening with it’s adoption…2010 for 10Gig…coincidence… not really…it’s just happening!!

10GE Adaption

It may take few years for complete infrastructure to upgrade to 10GE technology – ALL 10Gig !! 

Cost of 10GE still needs to come down for further adaption, while performance & throughput of 10GE technology has some challenges in virtualized environments inhibiting further adoption at server level…Should we still wait??

I would say, it’s time to adapt:

10GE, it just works!

What do you think it is? Share your thoughts…

My Data Center Cystal Ball, What's Your's?

 

CY2010: Data centers will have either pure iSCSI or FC implementation, from storage point of view

• 10Gigabit Standard Ethernet adaptation for consolidating 1Gigabit Ethernet infrastructures to support virtualized data centers accelerating data center consolidation.
• SAN traffic continues to converge on to Standard Ethernet with iSCSI implementations (mostly Greenfield deployments) and 10GE infrastructures. 10GE iSCSI arrays become common in storage deployments furthering iSCSI adaptation.
• FC networks remain separate, while 10Gigabit Enhanced Ethernet and FCoE is evaluated by early adaptors of these technologies, as products get released in the later in 2010.

CY2011-2014: Data centers will have mixed iSCSI & FC/FCoE.

• As Enhanced Ethernet standardizes in the middle of 2010, and early standard based products become available by the end of 2010, evaluation of 10GEE starts.
• During mid 2011, existing FC infrastructures start to merge with early deployments of 10Gigabit Enhanced Ethernet networks with FCoE capabilities promoted by investment protection strategy.
• At the same time iSCSI deployments leverage benefits of Enhanced Ethernet, furthering it’s adoption. New storage deployments will be either 10G iSCSI or 8G FCoE arrays.
• This pattern will continue till 40G Enhanced Ethernet products start shipping some time 2012-2013 accelerating the adoption of iSCSI further. 16G FCoE arrays start showing up during that time as an alternative to iSCSI.
• 40GEE evaluations and early adoptions at the core network throughout 2014 in to 2015.

CY2015 and beyond: Data centers will have converged storage (predominantly iSCSI and to some extent FCoE to attach older storage infrastructure).

• Converged Enhanced Ethernet networks become mainstream as 40Gigabit Ethernet deployments in the core networks and 10Gigabit Ethernet at the edge becomes common.
• iSCSI based Storage arrays becomes predominant due to high speeds and performance as compared to FC/FCoE arrays.
• 100G Enhanced Ethernet!!

 Share your’s…let’s compare