Let the flame wars commence! Okay, first read the post, THEN flame, capice? That's better. I wrote this up ages ago, and just never bothered to post it, because I didn't feel like having people scream at me for a week straight. Well hey, I'm in a self-abusing mood.
So yeah. I hate Macs. As in absolutely loathe and refuse to own a current Mac, period. Why? I've got a whole list of reasons, but let's step back a moment. Remember when Macs shipped with SCSI disk and 68k processors, then later PowerPC 603's and 604's? I loved Macs back then. Owned two. Because they actually differentiated themselves from Windows PCs, offered value for the dollar, and were superior for the tasks I needed to do on them.
These days? The Mac population consists largely of rabid zealots who think you're the anti-christ if you don't like Mac for any reason. That's a big reason I hate Macs and refuse to own one. The other part of it is the same zealots who refuse to accept or acknowledge that Macs and OS X are not the superior platform for every single task imagined. Let's put this all into a nice neat list though.
OS X cannot run software I need daily. Do not give me bootcamp or Parallels excuses. If it can't run it natively, then it can't run the damn software. Here's a list. Software that is totally unavailable or unworkable is in red, with severe limitations in yellow. Severe limitations includes lack of plugin availability or cross-platform compatibility issues.
Mozilla Firefox, Microsoft Outlook, Microsoft Word, Microsoft Excel, Microsoft Visio, TweetDeck, vim/Gvim, NFS, CIFS, Pad2Pad, Pidgin, Zune, Steam, VMWare Workstation, Ableton Live, Propellerheads Reason 4, City of Heroes
Ableton Live is in orange because it's there. But half my needed VSTs do not have a Mac version available, or do not work at all on Mac, making it nigh unusable. And a complete lack of Visio means it is completely unusable, because I abuse Visio in depth, daily.
"It's the software, stupid" arguments from Mac zealots. Hello, see immediately previous point. The very existence of Fusion, Parallels and Bootcamp all destroy that argument in nothing flat. If it was the magical OS X, then why do you have to run Windows on your Mac with virtualization software so you can access applications that aren't available on it? Yeah. It's totally everyone else's fault entirely and they're idiots for not maintaining multiple code trees and development teams just for Mac. Sure. And you know what? If I'm just going to reboot the damn thing into Windows anyways, then why the hell would I buy OS X at all?
The hardware is out of date and overpriced. No, you are not getting a "superior" product, you are just paying a ridiculous brand premium for a fashionable chassis. To meet my requirements requires a Mac Pro which costs well over $7,000.00 USD before the 3 year warranty (required thanks to Apple's continuing string of QC and QA problems.) For this $7,000.00 pricetag I get a SINGLE last generation Xeon Quad Core 3.33GHz based on Intel's abandoned 5000-series chipset with only 16GB of nigh unavailable Fully Buffered DIMMs, a $700 non-hardware non-BBU RAID card which I can buy elsewhere for around $100, an ATI Radeon HD4870 512MB at $200 which is not only a generation old but available in the proper 1GB configuration at a little over half that price, a pair of middle of the road DVD drives, iWork, about half of Microsoft Office, and a mediocre warranty with support where I still have to buy the OS upgrades out of pocket with no discount. THIS IS NOT A GOOD VALUE BY ANY MEASURE. For under $7,000 I can build a dual socket 12/24 core system which is not only watercooled but packing two Radeon HD5870's in CrossFire, an SSD and two 500GB/32MB drives in RAID0, and 24GB of DDR3. And there's not one bit of difference except that A) I'm using CURRENT generation hardware B) I don't have the brushed aluminum chassis. Oh wait, I do, except it's black and has hotswap drive bays and is quieter. Hell, I charge under $7K for that built and shipped to your door with half the parts packing 5 year manufacturer warranties!
Apple loves to lock in their vendor lock-out. Nobody gets to just write applications for OS X. You have to pay your special Apple taxes. It's not just an iPhone thing, either. You must pay Apple for documentation, for licenses, and so on. And if they decide they aren't happy with you, they can cut off any developer with no notice, leaving the users in the lurch. Want to use anything non-Apple that isn't an external thumb drive? If they didn't pay their Apple taxes, it's a crap shoot if it'll work or be supported. When was the last time you plugged in a keyboard or mouse to your PC and found that it was completely and utterly incompatible because of the software?
I play games. Get over it, Mac heads. Apple is not a gaming platform. If it was a viable, popular gaming platform, then popular games would be released for it. Instead, you have a handful of MMOs with Mac clients as an option, and some older games. Mass Effect 2 is nowhere to be found, nor is Steam, much less Modern Warfare 2 or Bioshock 2. ME2, MW2 and Bioshock are 3 of the absolute most popular, best selling games out there. They have massive, incredibly large budgets. Yet there is no OS X version of any of these games. Take the hint; if you enjoy playing a variety of games to relax, then Apple is definitely not for you. And World of Warcraft doesn't count, when the vast majority of LUA-based addons don't work on Macs.
The attitude of the company and the "community" just SUCKS. "Look at us in our faded jeans and black turtlenecks and OHMYGODNEWIPHONE MUST HAVE NOW!" Yeah, because a guy with 20 years of system design and building experience is the kind of guy who goes out and buys the latest 1.0 because somebody put a shiny Apple logo on it. But gods help you if you should rationally explain why you don't like iTunes or the iPod or the iPad to most Apple users. Seriously, I've had one actually take a swing at me for suggesting that an old G4 Xserve with significant performance problems was not the best answer, because of the performance problems. I have found on dozens of occasions that Apple users have absolutely zero interest or focus on whether or not the job is done right, or even if the job is done - their only concern is rabid support of a shiny white logo. And the only "appropriate" response to any question of the usefulness or functionality of any Apple product is to shout the person down and insult them.
Please note, this isn't a blanket statement about all Apple users. I know several that are perfectly reasonable and rational, and completely understand and support my refusal to use a product that doesn't work for me. ("Why should you, if it doesn't do what you need? That's silly.") However, as I said, I have had Apple fanatics actually swing at me for questioning Apple. And more often than not, that is exactly the reaction I get when I point out that an Apple product doesn't do what I need or want - screaming, insulting, and swinging. I want no part of any organization or community that considers shouting down acceptable behavior.
The presumption that because I'm a Unix guy I must love OS X. Look. It's an operating system. Get over it. AIX is an operating system. Solaris is an operating system. Windows is an operating system. The notion that any one of these does not suck is a complete fallacy. All operating systems suck, just in different ways. It's a fact of computing. It shouldn't be, but it is, and I deal with it every day. Just because I'm a Unix guy does not mean I prefer or even necessarily want Unix on my desktop. My operating system does not give me any sort of self-gratification or pleasure whatsoever, no matter what it is. I don't flipping care if it's Unix-like! It's an operating system, not a damn religion.
If I wanted Unix on my desktop, then I would use FreeBSD or AIX. But see, the thing is that I don't. I want to get things done and I do not want to waste time beating them into working through emulators like WINE or rebooting to a different OS so I can play a game. I want to turn on my computer, get work done, close it out and go play some games, then shut it down at the end of the day. That's it.
So, I figure I've explained sufficiently why I hate Macs personally. Calmly and rationally. This of course means, that I can expect irrational screaming and death threats from a select few who frankly have no business giving me advice or making demands about what hardware and software I use. Those of you who want to scream and yell at me, go hang out with the people who agree with me that an iMac completely fails to meet my disk requirements and a Mac Pro is horrifyingly overpriced and learn from them. Please.
Wednesday, February 17, 2010
Thursday, February 11, 2010
Big ESX in a Tiny Box - What's up with the delay here?!
Okay, I owe everyone an explanation, so here goes.
I've run into a problem that was not present on the prior build. This is an extremely severe problem that makes the system completely unusable. Understand this is through no fault of design or implementation here, but rather, due to a very severe bug in ESX/ESXi's Intel EtherExpress driver, specifically in the MSIX Vector section of the e1000 driver. Please understand, this bug did not present previously in any situations I tested. Remember, every component in the system is on the HCL. The problem here is with an Intel i82574L ethernet controller; you can find it under IO Devices, Networking, partner name Intel.
At this point, I'm trying to get a 12x5 Basic contract via VMware so that this bug can be escalated properly. The exact issue, going into the technical side of things, has to deal with how ESX/ESXi handles the Interrupt Vector Address Routing or IVAR for PCI-Express MSIX. If this part goes way over your head, don't worry; it's supposed to. This requires that you have prior experience developing drivers and doing kernel programming to understand. It also requires knowledge of the Intel EtherExpress family and PCI-Express bus.
So, the i82574L/LA has a 5 entry IVAR. Typical drivers will use only the first three IVARs and ignore activities on entries 4 and 5. (Technically, 3 and 4, as it starts at 0. So I'm going to be starting from 0 here.) ESX/ESXi uses or touches all of the IVAR table, 0-4. The i82574 can operate in a number of modes, which are identified by the Function registry entry. In normal operation on most systems, Function will be 0, which indicates the following list of items:
- Operating as LAN0
- Operating as LAN1
- Operating as LAN0 shared with IPMI/BMC
- Operating as LAN1 shared with IPMI/BMC
Yes. A single Function mode indicator, indicates FOUR functions. So how do we control whether we're doing operations strictly for the host, or we're doing operations for the IPMI? By asserting via MSIX and the IVAR table.
Here's where ESX/ESXi's e1000 driver breaks in a predictable and reproducible fashion. I can't explain why it's breaking, only exactly HOW it's breaking.
When operating normally, the e1000 driver will LOSE the MSIX vector completely. This results in the Interrupt Status Register being lost, causing the driver to lose awareness of the controller state as well as halting all network traffic. This is not enough to crash ESX/ESXi, and the driver continues operation without asserting an error, even though interrupts are doing nothing. This also means that the driver is unaware of link state changes, so any HA/FT features will be rendered useless as far as that host is concerned. (Remote hosts will have to assert failure condition on network going unreachable and yank control.) If you attempt to use ethtool to diagnose the i82574L/LA at any point during operation in either online or offline mode, you will get this failure:
PCPU 0 locked up. Failed to ack TLB invalidate (0 others locked
up).
cr2=0x0 cr3=0x400ed000 cr4=0x16c
0:8353/ethtool 1:6231/sfcbd *2:4109/helper1-0 3:5287/sfcbd
4:5101/openwsman 5:5001/hostd 6:5100/openwsman 7:8450/prop_of_i
--
Saved backtrace from: pcpu 0 TLB NMI
Sanitizing and rewriting to make it make sense, here's the actual code path of the failure.
FindIRQInfo+0x69
RemoveIRQInfo+0x41
vmklnx_request_irq+0x32f
e1000_diag_test+0xc5f
ethtool_self_test+0xfc
__ethtool_ioctl+0xe62
vmklnx_ethtool_ioctl+0x7a
netdev_ioctl+0x101
NicCharOpsIoctl+0x65
VmkApiCharDevIoctl+0xe6
DevFSIoctl+0x3e5
FSS_Ioctl+0x17d
UserFile_PassthroughIoctl+0x44
LinuxFileDesc_Ioct+0x7e
User_LinuxSyscallHandler+0xa3
So what's the exact problem? When it allocates the interrupt vector, it promptly loses it. It appears to be IVAR entry 5, as ESXi reports looking for 0x5a5a5a5a and instead getting 0xffffffff.
Exposure of the problem appears to be tied to a change in board settings which most users will make, meaning it's very easy to trigger. (The initial build was slightly different.) Until I get this issue escalated within VMware to the point where they're actually guaranteeing further investigation as well as a fix for this problem, I can't tell you what parts to buy because obviously, they no longer work right now, even though they are on the VMware HCL.
Sorry folks. I'm working my tail off here to get VMware to look at this. I have the crash dumps available on request. I just don't have a support contract.
I've run into a problem that was not present on the prior build. This is an extremely severe problem that makes the system completely unusable. Understand this is through no fault of design or implementation here, but rather, due to a very severe bug in ESX/ESXi's Intel EtherExpress driver, specifically in the MSIX Vector section of the e1000 driver. Please understand, this bug did not present previously in any situations I tested. Remember, every component in the system is on the HCL. The problem here is with an Intel i82574L ethernet controller; you can find it under IO Devices, Networking, partner name Intel.
At this point, I'm trying to get a 12x5 Basic contract via VMware so that this bug can be escalated properly. The exact issue, going into the technical side of things, has to deal with how ESX/ESXi handles the Interrupt Vector Address Routing or IVAR for PCI-Express MSIX. If this part goes way over your head, don't worry; it's supposed to. This requires that you have prior experience developing drivers and doing kernel programming to understand. It also requires knowledge of the Intel EtherExpress family and PCI-Express bus.
So, the i82574L/LA has a 5 entry IVAR. Typical drivers will use only the first three IVARs and ignore activities on entries 4 and 5. (Technically, 3 and 4, as it starts at 0. So I'm going to be starting from 0 here.) ESX/ESXi uses or touches all of the IVAR table, 0-4. The i82574 can operate in a number of modes, which are identified by the Function registry entry. In normal operation on most systems, Function will be 0, which indicates the following list of items:
- Operating as LAN0
- Operating as LAN1
- Operating as LAN0 shared with IPMI/BMC
- Operating as LAN1 shared with IPMI/BMC
Yes. A single Function mode indicator, indicates FOUR functions. So how do we control whether we're doing operations strictly for the host, or we're doing operations for the IPMI? By asserting via MSIX and the IVAR table.
Here's where ESX/ESXi's e1000 driver breaks in a predictable and reproducible fashion. I can't explain why it's breaking, only exactly HOW it's breaking.
When operating normally, the e1000 driver will LOSE the MSIX vector completely. This results in the Interrupt Status Register being lost, causing the driver to lose awareness of the controller state as well as halting all network traffic. This is not enough to crash ESX/ESXi, and the driver continues operation without asserting an error, even though interrupts are doing nothing. This also means that the driver is unaware of link state changes, so any HA/FT features will be rendered useless as far as that host is concerned. (Remote hosts will have to assert failure condition on network going unreachable and yank control.) If you attempt to use ethtool to diagnose the i82574L/LA at any point during operation in either online or offline mode, you will get this failure:
PCPU 0 locked up. Failed to ack TLB invalidate (0 others locked
up).
cr2=0x0 cr3=0x400ed000 cr4=0x16c
0:8353/ethtool 1:6231/sfcbd *2:4109/helper1-0 3:5287/sfcbd
4:5101/openwsman 5:5001/hostd 6:5100/openwsman 7:8450/prop_of_i
--
Saved backtrace from: pcpu 0 TLB NMI
Sanitizing and rewriting to make it make sense, here's the actual code path of the failure.
FindIRQInfo+0x69
RemoveIRQInfo+0x41
vmklnx_request_irq+0x32f
e1000_diag_test+0xc5f
ethtool_self_test+0xfc
__ethtool_ioctl+0xe62
vmklnx_ethtool_ioctl+0x7a
netdev_ioctl+0x101
NicCharOpsIoctl+0x65
VmkApiCharDevIoctl+0xe6
DevFSIoctl+0x3e5
FSS_Ioctl+0x17d
UserFile_PassthroughIoctl+0x44
LinuxFileDesc_Ioct+0x7e
User_LinuxSyscallHandler+0xa3
So what's the exact problem? When it allocates the interrupt vector, it promptly loses it. It appears to be IVAR entry 5, as ESXi reports looking for 0x5a5a5a5a and instead getting 0xffffffff.
Exposure of the problem appears to be tied to a change in board settings which most users will make, meaning it's very easy to trigger. (The initial build was slightly different.) Until I get this issue escalated within VMware to the point where they're actually guaranteeing further investigation as well as a fix for this problem, I can't tell you what parts to buy because obviously, they no longer work right now, even though they are on the VMware HCL.
Sorry folks. I'm working my tail off here to get VMware to look at this. I have the crash dumps available on request. I just don't have a support contract.
Wednesday, January 27, 2010
Big ESX in a Tiny Box, Part 1 – My Home Environment Of Doom.
I have a pretty big home environment. Of course I do – I’m a Unix guy. It’s what we do. What people don’t understand just how big, ugly, and loud my home environment can be. Let’s start with the “smaller” boxes. I sit at my primary workstation, which is an aging Intel E6750 with 4GB and a Radeon HD4870, since I play a lot of high requirement games. It’s still a fairly respectable box, running Windows 7 Ultimate for the NFS client. I have VMware Workstation as well, which I use for testing some things.
Then comes the actual Unix environment. This is where things turn from “mild” to downright horrifying. Starting with the oldest systems first, I have a code-control system which is used by multiple people – it’s an IBM Netfinity 5500 with dual Pentium III 733s and 768MB, forcibly crammed into the desk. I do my damndest to not turn it on ever because it’s loud. I’ve even moved most of its workload elsewhere. It’s big, it’s power hungry and it’s loud. Everything I don’t like in a home server.
Adding to this is my IBM Netfinity 4500R, sitting atop my desk. This system is extremely, extremely loud at times. Packing dual Pentium III 733’s and a whopping 2GB of PC133 ECC Registered, this is one of my test machines. It was originally purchased for PCI-HP development testing and Ethernet driver testing. It’s not so power hungry, dual 280W power supplies, but it’s long past its usable lifespan.
The current “core infrastructure” server is an embarrassment, to be honest. Handling routing, NAT, mail serving, database, and webserving duties is an AMD Athlon XP 1GHz. That is not a typo, folks. It has a whopping 256MB of DDR. Not DDR2, and not even PC2100 – PC1600 original DDR. It no longer has working video output or keyboard input, the serial console stopped working last month, and the four 20GB IDE disks are probably at double their rated MTBF. One of them has begun having spin-down issues intermittently.
To solve the “core” server, I slapped together a replacement with spares, and began the slow process of a total rebuild and migration. About mid-way through, Abit stopped making motherboards and ran out of warranty replacements. So a low-hour E6550 with 4GB and a pair of new 250GB SATA disks instead became the “limp along” server, currently handling database duties, internal web serving, some proxying load, and a bit of NFS serving.
Utterly and absolutely destroying all of these boxes is “Big Bertha.” I do not ever turn Big Bertha on except when it is absolutely necessary. That doesn’t happen often. That’s because it’s a special box: an Iwill H8502 with eight Opteron 2.0GHz processors, 64GB of memory, dual LSI MegaRAID 320-2X’s, and more. Oh. And it has 4 1650W power supplies which require 2x20A circuits to start up in sequence. Don’t even ask what it takes to start up in parallel; I’ve never even attempted it, and I don’t think it’s possible.
I also have other machines scattered about on top of this. Another low-end workstation, three laptops, an IBM POWER4, a Sun Ultra 2, you pretty much get the idea. Not to mention my Xbox360, my PS2, a Wii and yeah... the list goes on. Repeat after me, everyone: THIS DOES NOT WORK. AT ALL.
I am beyond out of space, my power bills are through the roof, and most of the equipment is well past it’s usable life. Let’s do some math here. These numbers are based on measured or calculated power draw. 745W + 690W + 420W + 322W + 470W + 505W + … you know what? Let’s just stop. You’ve already figured out that it’s over 2KW if I turn everything on, before Big Bertha. Big Bertha does draw heavily – 4x 8.5A @ 110V = 3.7KW while running. (“Do NOT turn on the hair dry-DAMNIT!”) Seriously, if your home network has ceased to be measured in watts and moved on to “number of breakers required,” you have got serious, serious problems.
So, how do I fix this monstrosity? Again, I’m a Unix guy, and this is my production environment! I receive 90% of my email on that ancient server, and it provides services to more than just me. Tossing everything and just getting some crappy Linksys router isn’t even remotely an option. Enter VMware ESXi 4.0 Update 1. FreeBSD works beautifully in it, overhead is pretty damn low, resulting in pretty good performance. I don’t hammer on the boxes obsessively, and given the existing hardware, I think newer coffee makers have more processing power than the individual systems. Sure, I could just do it with FreeBSD jails on a big box, but that box will still end up having to be pretty damn big. And honestly, jails are a management nightmare, and don’t solve the firewall problem. I still need a separate physical firewall then. Not so with VMware – I can put everything (except the Sun and IBM) onto a single physical system provided I have enough NICs, and get appropriate memory isolation.
So, now that you’ve got a grasp on the environment, the fix starts to make sense. I'm consolidating a bunch of complex systems on ancient hardware into a single ESX server. But I really don’t want a big box. So how do I solve that problem? How do I really cut my power bills? And how do I make this something more than just a fix for me, but an actual setup that can be expanded into a true Enterprise grade ESX setup? (Look, when I say Enterprise grade, trust me, I actually do mean Enterprise grade.) So I put my 20 years of x86 system building and engineering to work along with my web browser, and found an answer. This answer was briefly discussed in VMTN Community Podcast #79 – I’m @RootWyrm for those of you on Twitter.
The final product has been ordered, and parts will be arriving later this week. Along with parts will come more on this topic, including the hardware details and the ESXi build. Stay tuned, it’s pretty darn awesome.
Sunday, December 13, 2009
IBM SVC - upgrading the hardware to 2145-CF8 (for people who know SVC.)
Hey! Hey you! There's an update at the bottom!
Let's start this one off with a disclaimer:
If you aren't comfortable doing medium to major maintenance on your IBM SVC, this post is so not for you. Seriously. I'm skipping a lot of steps that fall under "things you just do as part of repairs." Instead, you should start over here:
Then consult here:
IBM SVC Reference: Replacing nodes nondisruptively
Okay. Disclaimer done. Let's get down to business. You FINALLY got approval to upgrade your SVC hardware, and it's six kinds of awesome. I cannot stop raving about the improvements in the 2145-8A4 / 2145-8A4 hardware. Seriously. If you can upgrade only one part of your environment in a quarter, this is it. And because it's SVC, you can spread the upgrade out if you have to.
First, let's talk caveats. Like all SVC clusters, hardware intermix is supported within limits. Original hardware (2145-4F2) intermix is not supported with the 2145-8A4. 2145-4F2 clusters can be upgraded to 2145-CF8/8A4 hardware non-disruptively, but prolonged intermix is not supported. All other models can mix with the CF8/8A4 hardware in the same cluster on a short term basis.
If you have 2145-4F2 nodes, you need to contact IBM support prior to upgrading. The documentation for non-disruptive 2145-4F2 to 2145-8A4 upgrades is impossible to locate (again). The procedure is different!
Customers with x305m and x306m Master Consoles (lift the SVC label to check your MT-M) will need to go to the SVC support website and read the advisory regarding system boot problems with these systems. This will only affect the Master Console and not the SVC. Frankly, I recommend buying a pair of small x Series and making the Master Console a MS Cluster. The default MC ship sucks.
First, prerequisites. One, you should already be on 4.3.1.10 or better. You have the latest PTF. All I/O groups have two node members. Normal cluster maintenance has been done and all errors cleared as "FIXED." No vdisks or mdisks are offline or degraded. You have the latest software upgrade test utility and you have run it with no errors returned.
Now let's talk about the 8A4 versus the CF8. I'll make it super, super simple for everyone to save you a ton of headaches.
8A4 is virtually identical to the 8F4, except using SATA simple-swap disks. 8GB of memory, single 3.0GHz dual core 6MB CPU, PCI-Express 4 port FC HBA, and that's about it.
CF8 is the one you want. Seriously. CF8 steps up the hardware big time by packing in the latest generation 2.4GHz quad core 12MB Xeon. Then it cranks it up with an LSI SAS RAID controller and 6 SAS bays taking both SAS and Solid State disk for cache. Adding to the awesome is dual redundant power supplies.
Breaking it down: all you care about is SSD or no SSD. Even if you don't use SSD, you want the CF8. Seriously, it's a monster. (I don't recommend blindly using SSDs, I'll get to that later on. Buy them anyways. Just read on further, okay?)
So you've got your new hardware. You've got your existing nodes upgraded to 5.1. Are we ready to rock? You betcha!
Now the "aw crap" part of our program: things you aren't going to like about 5.1 and the CF8. The CF8 has a VERY nasty caveat if you use SSD and internal RAID functions. In any CF8 node failure, you must move the RAID controller, cables, and disks to the replacement node. Failure to do so will result in data loss on the SSD array. You must do this for any and every CF8 using SSD and RAID. I don't recommend using SSD in RAID, but I do recommend putting a pair of 146GB SSDs in any CF8 you order. You'll find some way to make use of them sooner rather than later.
UPDATE: The wonderful Barry Whyte of IBM posted his own blog entry about the CF8 with some information on the node upgrades. This resulted in a discussion on Twitter (big surprise there) wherein he corrected me on SSD behaviors and the actual RAID card. (I had to go off photos, so cut me a bit of slack, please.) ;)
One, the SSDs can mirror between nodes. So your worst case is data loss isolated to a single node, requiring a recopy of SSD data between nodes. I should have known IBM wouldn't miss that little point! Minus one point for me, definitely. So yes, you also do need to match your SSDs between nodes within each IO Group. That said, you still need to take your standard data protection precautions - RAID is not a backup, tapes are your friend, and so on.
Two, there's two SAS controllers installed in the new CF8 nodes. One is an LSI as I mentioned, but the LSI is actually the boot disk controller and not the SSD controller. The SSD controller is a custom designed solution from the fine folks at IBM Hursley and is just an HBA. Actual SSD configuration is stored within the cluster, meaning SSD configuration won't be lost in a node failure.
There's still some technical caveats in node repair situations, of course, but it's going to depend on the recommended course of action from support and the CE. (e.g. system planar replacement will require moving the LSI in place as normal, but there may be some trouble when the LSI itself needs replaced.) As ever, failures and repairs depend primarily on field experience, so time will give us all better knowledge there.
Three, he pointed out a use case that I like enough to share here. You can use the SSDs as a VDisk RAID1 mirror of MDisk RAID5's to get some screaming read performance. I can definitely get behind this little trick, especially since it's being done at the VDisk level. Meaning you don't have to play match the array, and can do it for individual VDisks that need more read performance. Write performance is mostly unaffected because you're still dependent on the slowest disk in the pairing.
Ultimately, there's not a single reason to not put SSDs in the new SVC nodes. Like I said; even if you don't use them immediately, you will find a use case for them that works for you. Don't be afraid to experiment with them to find what works for your situation. Just be aware of the restrictions and limitations, the same as ever. SSD is not a cure-all for performance problems - it's another tool to add to your arsenal.
Let's start this one off with a disclaimer:
If you aren't comfortable doing medium to major maintenance on your IBM SVC, this post is so not for you. Seriously. I'm skipping a lot of steps that fall under "things you just do as part of repairs." Instead, you should start over here:
Then consult here:
IBM SVC Reference: Replacing nodes nondisruptively
Okay. Disclaimer done. Let's get down to business. You FINALLY got approval to upgrade your SVC hardware, and it's six kinds of awesome. I cannot stop raving about the improvements in the 2145-8A4 / 2145-8A4 hardware. Seriously. If you can upgrade only one part of your environment in a quarter, this is it. And because it's SVC, you can spread the upgrade out if you have to.
First, let's talk caveats. Like all SVC clusters, hardware intermix is supported within limits. Original hardware (2145-4F2) intermix is not supported with the 2145-8A4. 2145-4F2 clusters can be upgraded to 2145-CF8/8A4 hardware non-disruptively, but prolonged intermix is not supported. All other models can mix with the CF8/8A4 hardware in the same cluster on a short term basis.
If you have 2145-4F2 nodes, you need to contact IBM support prior to upgrading. The documentation for non-disruptive 2145-4F2 to 2145-8A4 upgrades is impossible to locate (again). The procedure is different!
Customers with x305m and x306m Master Consoles (lift the SVC label to check your MT-M) will need to go to the SVC support website and read the advisory regarding system boot problems with these systems. This will only affect the Master Console and not the SVC. Frankly, I recommend buying a pair of small x Series and making the Master Console a MS Cluster. The default MC ship sucks.
First, prerequisites. One, you should already be on 4.3.1.10 or better. You have the latest PTF. All I/O groups have two node members. Normal cluster maintenance has been done and all errors cleared as "FIXED." No vdisks or mdisks are offline or degraded. You have the latest software upgrade test utility and you have run it with no errors returned.
Now let's talk about the 8A4 versus the CF8. I'll make it super, super simple for everyone to save you a ton of headaches.
8A4 is virtually identical to the 8F4, except using SATA simple-swap disks. 8GB of memory, single 3.0GHz dual core 6MB CPU, PCI-Express 4 port FC HBA, and that's about it.
CF8 is the one you want. Seriously. CF8 steps up the hardware big time by packing in the latest generation 2.4GHz quad core 12MB Xeon. Then it cranks it up with an LSI SAS RAID controller and 6 SAS bays taking both SAS and Solid State disk for cache. Adding to the awesome is dual redundant power supplies.
Breaking it down: all you care about is SSD or no SSD. Even if you don't use SSD, you want the CF8. Seriously, it's a monster. (I don't recommend blindly using SSDs, I'll get to that later on. Buy them anyways. Just read on further, okay?)
So you've got your new hardware. You've got your existing nodes upgraded to 5.1. Are we ready to rock? You betcha!
- Install and cable your new CF8 nodes. Don't connect the ethernet or FC cables yet!
- Did I say turn your CF8 nodes on? No! Don't!
- Go to your existing cluster, and locate your configuration node.
- After you've found your configuration node comes the ugly. You need a list of dependent vdisks via "svcinfo lsnodedependentvdisk" - oh, and any dependent quorum disks.
- IBM says you should stop I/O to dependent vdisks. If you aren't using SDD, probably you should. If you are using SDD at the hosts, just be extremely careful. Test with non-production first, obviously. ESX users, sorry, you're probably boned here if you aren't on 4.
- Quorum disks! This is beyond important. Relocate quorum disks BEFORE shutting down a node. I mean it.
- Actually write down the WWPN and WWNN (and iSCSI name) of the node you're about to replace.
- Ready? NOW you can stop the node!
- Node's stopped? Remove it from the cluster!
- Power on the removed node, and change it's WWPN and WWNN to all F's. How? While the panel is displaying "Node WWNN:" press and hold down, press select, then release down. You should see "Edit WWNN" on line 1, and the WWNN on line 2. Use up and down and left and right to change it to F's. Press select to save your changes.
- IBM says this is when you install and cable. NO. You should have already done that! Makes things go quicker, trust me.
- Power on your glorious shiny new node! (From the UPS, dangit! Not the front panel of the node!)
- Hey, did you actually connect those FC and Ethernet cables? No? Good.
- Write down the WWNN and/or iSCSI name of your shiny new CF8. You won't need it unless you're reusing nodes, but write it down anyways.
- Remember how we made the other node FFFFF in step 10? We're going to do those same steps, except we're going to give our replacement node the WWNN of the node we replaced.
- Wait about a minute. The new node panel should display "Cluster:" - if it doesn't, call IBM support. If it does, you're ready to add it to your cluster.
- Very carefully take your lovingly labeled (you DO label, right?) cables from your old node, and relocate them to your new node. Every port must match exactly by the Q names.
- Use lsnodecandidate to verify that you applied the WWNN correctly. If it's not there in lsnodecandidate, fix the WWNN.
- Use "svctask addnode -wwnodename WWNN -iogrp IOGroupName" to add your new node into the cluster as a replacement. If the node is behind on software or a different version, it may take up to 20 minutes. Relax. Grab a coffee. You're just about done anyways.
- Verify the new node is online in the cluster. Verify your hosts see the node as restored; if they see a new path rather than an existing path having come back online, something went wrong and you should probably call IBM.
- Lather, rinse, repeat for all other nodes. Remember to do the Configuration Node last.
UPDATE: The wonderful Barry Whyte of IBM posted his own blog entry about the CF8 with some information on the node upgrades. This resulted in a discussion on Twitter (big surprise there) wherein he corrected me on SSD behaviors and the actual RAID card. (I had to go off photos, so cut me a bit of slack, please.) ;)
One, the SSDs can mirror between nodes. So your worst case is data loss isolated to a single node, requiring a recopy of SSD data between nodes. I should have known IBM wouldn't miss that little point! Minus one point for me, definitely. So yes, you also do need to match your SSDs between nodes within each IO Group. That said, you still need to take your standard data protection precautions - RAID is not a backup, tapes are your friend, and so on.
Two, there's two SAS controllers installed in the new CF8 nodes. One is an LSI as I mentioned, but the LSI is actually the boot disk controller and not the SSD controller. The SSD controller is a custom designed solution from the fine folks at IBM Hursley and is just an HBA. Actual SSD configuration is stored within the cluster, meaning SSD configuration won't be lost in a node failure.
There's still some technical caveats in node repair situations, of course, but it's going to depend on the recommended course of action from support and the CE. (e.g. system planar replacement will require moving the LSI in place as normal, but there may be some trouble when the LSI itself needs replaced.) As ever, failures and repairs depend primarily on field experience, so time will give us all better knowledge there.
Three, he pointed out a use case that I like enough to share here. You can use the SSDs as a VDisk RAID1 mirror of MDisk RAID5's to get some screaming read performance. I can definitely get behind this little trick, especially since it's being done at the VDisk level. Meaning you don't have to play match the array, and can do it for individual VDisks that need more read performance. Write performance is mostly unaffected because you're still dependent on the slowest disk in the pairing.
Ultimately, there's not a single reason to not put SSDs in the new SVC nodes. Like I said; even if you don't use them immediately, you will find a use case for them that works for you. Don't be afraid to experiment with them to find what works for your situation. Just be aware of the restrictions and limitations, the same as ever. SSD is not a cure-all for performance problems - it's another tool to add to your arsenal.
Sunday, November 29, 2009
Flat Tiering - it doesn't suck, it's just your configuration that sucks.
I hear so much anti-flat-tiering noise lately, it's past time I spit out my thoughts on it.
First, what is flat tiering? That's any storage system where you have one or two types of disk, and that's it. Some examples would be the Compellent, Pillar Data Systems, Data Domain Networks, and 3par T and F series. All of these systems use one or two disk types in the system, and that's it. Why is this a good thing? One, it reduces servicing complexity - you don't have to identify which type of disk or controller needs replaced, because they're all the same. Point two goes back to the subject above - configured right, there's no reason to mechanically tier everything.
Flat tiering is not one-size-fits-all. It is not right for every configuration. It is not right for every application mix. But it fits most setups well - when it's configured correctly.
Certainly, this is far from specific to flat tiering. Do not ask how many badly configured mechanically tiered setups I've seen. Bad configurations are more common than people want to admit. Usually it's holdover configurations from initial deployments done in a rush, where there's too much loading or not enough spare resources to perform a reconfiguration on. Either way, bad configurations are bad configurations wherever they are.
Here's the thing with flat tiering though - when your configuration sucks, you REALLY know it. It's not like a DS5k where your bad configuration only pushes your response times up to around 20ms, or an SVC where you reduce theoretical peak throughput by 50% (which still gives you about 4.5GB/sec.) A bad configuration on a flat tiered system can push your response times over 50ms and drop your IOPS through the floor. And it's much easier to badly configure a flat tiered system than a mechanically tiered system.
The first thing you have to do, and I really do mean have to do when you work with a flat tiering system is throw everything you know about configuring mechanically tiered systems away. Forget all of it. It's only going to wreck your configuration. The second thing you have to do is document and test more first. I don't mean figure out what needs high priority and how many terabytes it needs - I mean really document and test. What's your typical IOPS for this application? What's your maximum acceptable response time? Is this random disk seeks or sequential loading?
Let's look at mechanical tiering. Basically, it goes something like this: if it has to be fast, put it on fast disk. If it can be slow, put it on slower disk. If it can be really slow and it's mostly sequential, put it on SATA. Flat tiering does away with all of that. Every last bit. There is no fast and slow disk, there is just disk.
This is where the biggest mistakes happen. People assume that this means they simply shift their entire design philosophy to capacity and nothing else. Capacity becomes the commodity and all performance is equal.
Can I get a show of hands from 3par, Compellent, DDN, and Xiotech who agree that flat tiering means capacity is the sole concern of their product, and all performance for all LUNs presented to hosts will always be 100% equal at all times?
Huh. I didn't see any hands there. Maybe I didn't ask loud enough? Or maybe that's because it's just not true. While performance across all LUNs on most of these systems will tend towards equal that does not mean it is equal, or should be equal on all LUNs.
Think about it. Do you want your PeopleSoft database to have the same access priority as your customer service file shares? Of course not. That's just silly. You need that PeopleSoft database to be on the fastest possible disk you can get. But you're on a flat tiered system, so you don't have faster disk. And there's only one product where 'flat disk' means 'flat performance' - and I won't even mention which, because it's just a bad storage system period. Ask any of the vendors I've mentioned above if flat disk means flat performance, and they'll tell you bluntly "no." It does tend toward flatter performance, but that isn't the
This comes back to the point of better documentation and understanding of storage requirements. If you attempt to treat all storage as equal, you are much more likely to get burned. A performance hit to your Windows shares becomes a performance hit to your ERP systems. Obviously this is the last thing you want to have happen in your production environments. That's why there needs to be greater focus on the storage, and a greater attention to detail. Storage complexity doesn't reduce, it just moves. You need to have a greater attention to detail than you're used to.
Nor can you just apply traditional best practices. They don't apply here. With different RAID modes, different technologies, and different methodologies of achieving storage performance, simply going with what you're used to isn't an option. You can't just throw together 4+1 RAID5's and call it done. You need to look at how each system does RAID, and how it applies to performance from the host perspective. You can't just throw more spindles at a slow database - it may even further hurt performance. You may need to increase controller priority, or adjust how the arrays are caching.
The other thing I see is a stiff resistance to SATA in traditionally FC spaces. This is right and wrong. Whenever any vendor is trying to sell you SATA instead of FC, you should be extra critical. That doesn't mean throw it out. That means you test, test, and test again. You test on real hardware with loads as close to real as you can get. You don't buy a thing until they prove every single claim. The fact is that SATA sucks for random, period. Every single vendor I've named knows and acknowledges this - that's why they all offer FC or SAS drives and SATA drives. If the salesperson is trying to push you to go ALL SATA, chances are they don't know what the hell they're talking about, or they see something in your workloads that you missed. Understanding your workloads is something you need to do before you even start talking to sales.
Flat doesn't necessarily mean really flat. Sometimes it means replacing 6 controllers and 32 arrays of FC disks with one or two controllers with far less arrays, achieving equal or greater performance. It does not need to mean limiting yourself to a single disk type or speed.
And let's go back to making them prove their claims. 3par can brag about some of the best SPC-1 results around, the F400 pulled off 93K IOPS with a respectable response time. That's a great demonstration of what their technology can do, using 146GB 15K FC disks. It is not a demonstration of what it will do in the configuration they're selling you. Your workloads might push their controllers harder than they expect. Your workloads may be poorly suited to the disks they're suggesting. Test, test, and test again. Make them back up their promises in writing. That's true of all storage, sure, but doubly so in this space.
The thing is, I know for an absolute fact that at least two vendors I've named in here (no, I can't say who) have backed up their claims of offering better performance and reliability in writing, in full. I know this because one, I was involved directly or peripherally in it, and two, they made good on those promises. I seriously can't say who, because it is NDA'd and proprietary information regarding customer agreements.
But I will tell you right here, right now, that if you ask any one of the vendors listed above to back up their performance and reliability claims in writing, with a guarantee to cover costs of removing them or going back to your existing storage there are two who can and will look you right in the eye, say they can do it, and will put it down in writing. And the requirement that you test, and work with support to achieve it? If that isn't standard operating procedure for you on traditional arrays, you need to reexamine your SOPs. (Again, test test test! You cannot test enough!) Anybody who won't back up their claims in writing, either shouldn't be making claims, or should be axed from your shopping list fast.
Oh, and that flat array I won't mention? IBM XIV. My recommendation? Don't even touch that thing. It is dangerous, and not ONE of the claims I have heard from sales in the past is true. IBM's own presentations show that XIV can barely beat the DS3400 in random IOPS and can't even match the DS3400's response times. XIV is complete crap for anything that isn't purely sequential read with <40% write and <20% random. If that isn't true, why hasn't IBM backed their claims on XIV with SPC results? DS3400 has 'em, DS5300 has 'em, XIV still doesn't over two years after it's introduction.
First, what is flat tiering? That's any storage system where you have one or two types of disk, and that's it. Some examples would be the Compellent, Pillar Data Systems, Data Domain Networks, and 3par T and F series. All of these systems use one or two disk types in the system, and that's it. Why is this a good thing? One, it reduces servicing complexity - you don't have to identify which type of disk or controller needs replaced, because they're all the same. Point two goes back to the subject above - configured right, there's no reason to mechanically tier everything.
Flat tiering is not one-size-fits-all. It is not right for every configuration. It is not right for every application mix. But it fits most setups well - when it's configured correctly.
Certainly, this is far from specific to flat tiering. Do not ask how many badly configured mechanically tiered setups I've seen. Bad configurations are more common than people want to admit. Usually it's holdover configurations from initial deployments done in a rush, where there's too much loading or not enough spare resources to perform a reconfiguration on. Either way, bad configurations are bad configurations wherever they are.
Here's the thing with flat tiering though - when your configuration sucks, you REALLY know it. It's not like a DS5k where your bad configuration only pushes your response times up to around 20ms, or an SVC where you reduce theoretical peak throughput by 50% (which still gives you about 4.5GB/sec.) A bad configuration on a flat tiered system can push your response times over 50ms and drop your IOPS through the floor. And it's much easier to badly configure a flat tiered system than a mechanically tiered system.
The first thing you have to do, and I really do mean have to do when you work with a flat tiering system is throw everything you know about configuring mechanically tiered systems away. Forget all of it. It's only going to wreck your configuration. The second thing you have to do is document and test more first. I don't mean figure out what needs high priority and how many terabytes it needs - I mean really document and test. What's your typical IOPS for this application? What's your maximum acceptable response time? Is this random disk seeks or sequential loading?
Let's look at mechanical tiering. Basically, it goes something like this: if it has to be fast, put it on fast disk. If it can be slow, put it on slower disk. If it can be really slow and it's mostly sequential, put it on SATA. Flat tiering does away with all of that. Every last bit. There is no fast and slow disk, there is just disk.
This is where the biggest mistakes happen. People assume that this means they simply shift their entire design philosophy to capacity and nothing else. Capacity becomes the commodity and all performance is equal.
Can I get a show of hands from 3par, Compellent, DDN, and Xiotech who agree that flat tiering means capacity is the sole concern of their product, and all performance for all LUNs presented to hosts will always be 100% equal at all times?
Huh. I didn't see any hands there. Maybe I didn't ask loud enough? Or maybe that's because it's just not true. While performance across all LUNs on most of these systems will tend towards equal that does not mean it is equal, or should be equal on all LUNs.
Think about it. Do you want your PeopleSoft database to have the same access priority as your customer service file shares? Of course not. That's just silly. You need that PeopleSoft database to be on the fastest possible disk you can get. But you're on a flat tiered system, so you don't have faster disk. And there's only one product where 'flat disk' means 'flat performance' - and I won't even mention which, because it's just a bad storage system period. Ask any of the vendors I've mentioned above if flat disk means flat performance, and they'll tell you bluntly "no." It does tend toward flatter performance, but that isn't the
This comes back to the point of better documentation and understanding of storage requirements. If you attempt to treat all storage as equal, you are much more likely to get burned. A performance hit to your Windows shares becomes a performance hit to your ERP systems. Obviously this is the last thing you want to have happen in your production environments. That's why there needs to be greater focus on the storage, and a greater attention to detail. Storage complexity doesn't reduce, it just moves. You need to have a greater attention to detail than you're used to.
Nor can you just apply traditional best practices. They don't apply here. With different RAID modes, different technologies, and different methodologies of achieving storage performance, simply going with what you're used to isn't an option. You can't just throw together 4+1 RAID5's and call it done. You need to look at how each system does RAID, and how it applies to performance from the host perspective. You can't just throw more spindles at a slow database - it may even further hurt performance. You may need to increase controller priority, or adjust how the arrays are caching.
The other thing I see is a stiff resistance to SATA in traditionally FC spaces. This is right and wrong. Whenever any vendor is trying to sell you SATA instead of FC, you should be extra critical. That doesn't mean throw it out. That means you test, test, and test again. You test on real hardware with loads as close to real as you can get. You don't buy a thing until they prove every single claim. The fact is that SATA sucks for random, period. Every single vendor I've named knows and acknowledges this - that's why they all offer FC or SAS drives and SATA drives. If the salesperson is trying to push you to go ALL SATA, chances are they don't know what the hell they're talking about, or they see something in your workloads that you missed. Understanding your workloads is something you need to do before you even start talking to sales.
Flat doesn't necessarily mean really flat. Sometimes it means replacing 6 controllers and 32 arrays of FC disks with one or two controllers with far less arrays, achieving equal or greater performance. It does not need to mean limiting yourself to a single disk type or speed.
And let's go back to making them prove their claims. 3par can brag about some of the best SPC-1 results around, the F400 pulled off 93K IOPS with a respectable response time. That's a great demonstration of what their technology can do, using 146GB 15K FC disks. It is not a demonstration of what it will do in the configuration they're selling you. Your workloads might push their controllers harder than they expect. Your workloads may be poorly suited to the disks they're suggesting. Test, test, and test again. Make them back up their promises in writing. That's true of all storage, sure, but doubly so in this space.
The thing is, I know for an absolute fact that at least two vendors I've named in here (no, I can't say who) have backed up their claims of offering better performance and reliability in writing, in full. I know this because one, I was involved directly or peripherally in it, and two, they made good on those promises. I seriously can't say who, because it is NDA'd and proprietary information regarding customer agreements.
But I will tell you right here, right now, that if you ask any one of the vendors listed above to back up their performance and reliability claims in writing, with a guarantee to cover costs of removing them or going back to your existing storage there are two who can and will look you right in the eye, say they can do it, and will put it down in writing. And the requirement that you test, and work with support to achieve it? If that isn't standard operating procedure for you on traditional arrays, you need to reexamine your SOPs. (Again, test test test! You cannot test enough!) Anybody who won't back up their claims in writing, either shouldn't be making claims, or should be axed from your shopping list fast.
Oh, and that flat array I won't mention? IBM XIV. My recommendation? Don't even touch that thing. It is dangerous, and not ONE of the claims I have heard from sales in the past is true. IBM's own presentations show that XIV can barely beat the DS3400 in random IOPS and can't even match the DS3400's response times. XIV is complete crap for anything that isn't purely sequential read with <40% write and <20% random. If that isn't true, why hasn't IBM backed their claims on XIV with SPC results? DS3400 has 'em, DS5300 has 'em, XIV still doesn't over two years after it's introduction.
Monday, November 23, 2009
My FTC Mandated Excessive Disclosure
So, I figure I should put this out here. That way nobody can say I'm partying with some company's reps in Las Vegas during lulls.
First and foremost, this blog represents my opinions. People are free to agree with them, but they're still my opinions. Not any employer's, or any other person or thing. Mine. Nothing I say here represents the opinions of any other person or company, and I never speak for anyone else.
I also may post or publish in other places and other blogs, one way or another. That doesn't change the fact that my words are my own, my opinions are my own, and they don't represent anyone other than me as a person. If that ever changes, I'll be sure and mention it clearly.
I never have and never will accept payment in cash, hardware, or anything else in return for favorable opinions or statements. While I may do consulting, my opinions are not for sale. Sorry.
If you've got something neat or cool you want me to look over, then let me know! I may or may not talk about it here. And if I forgot something you know about in my writing, well, hey. I'm an IT guy, but I don't know every product or technology on the market. Sorry for the omission, but I can't write about products I don't know, and I won't write at length about products I'm not comfortable working with.
If you really, really want me to write on something, then my time is for sale. (Hey, I do consulting. Got to pay the bills somehow.) However, just because you're paying me for my time doesn't mean you get to pick what I say. You get to pick the topic and the deadline, and even where it gets published; you just don't get to pick the outcome.
I have or have had access to information which is covered under NDA from several companies, as part of my prior employment. This information is frequently incorporated in my thought process and writing, without being disclosed. Because of the nature of the NDAs, I cannot disclose who they are with or what they cover. Some of these agreements did or do incorporate cross-marketing or case study agreements, but they never came to me for quotes (because I would say something sucked if it sucked, and they don't like that for some reason.)
I'm typically pretty blunt in my writing here. Please don't confuse this for who I am, or how I usually write. This is my blog. That means when I have a bad day courtesy some vendor making a stupid design decision, it just may end up here. (Actually, it probably will end up here.) By the same token, if something makes my day, I'll probably write about it here. I'm going to point again to the first statement - these are my opinions. Insert obligatory 'opinions are like' here.
I lurk around in various places, and generally I don't advertise that I'm watching or listening. I may also post or comment under aliases. All this means is that one, I value my privacy, and two, I prefer not to be harassed for having an opinion of my own. (Yes, I have had people harass me at length. I have better things to do than put up with it.) People who know, know. Those who don't, don't expect me to change that any time in this lifetime.
Hm, yep. That covers everything.
First and foremost, this blog represents my opinions. People are free to agree with them, but they're still my opinions. Not any employer's, or any other person or thing. Mine. Nothing I say here represents the opinions of any other person or company, and I never speak for anyone else.
I also may post or publish in other places and other blogs, one way or another. That doesn't change the fact that my words are my own, my opinions are my own, and they don't represent anyone other than me as a person. If that ever changes, I'll be sure and mention it clearly.
I never have and never will accept payment in cash, hardware, or anything else in return for favorable opinions or statements. While I may do consulting, my opinions are not for sale. Sorry.
If you've got something neat or cool you want me to look over, then let me know! I may or may not talk about it here. And if I forgot something you know about in my writing, well, hey. I'm an IT guy, but I don't know every product or technology on the market. Sorry for the omission, but I can't write about products I don't know, and I won't write at length about products I'm not comfortable working with.
If you really, really want me to write on something, then my time is for sale. (Hey, I do consulting. Got to pay the bills somehow.) However, just because you're paying me for my time doesn't mean you get to pick what I say. You get to pick the topic and the deadline, and even where it gets published; you just don't get to pick the outcome.
I have or have had access to information which is covered under NDA from several companies, as part of my prior employment. This information is frequently incorporated in my thought process and writing, without being disclosed. Because of the nature of the NDAs, I cannot disclose who they are with or what they cover. Some of these agreements did or do incorporate cross-marketing or case study agreements, but they never came to me for quotes (because I would say something sucked if it sucked, and they don't like that for some reason.)
I'm typically pretty blunt in my writing here. Please don't confuse this for who I am, or how I usually write. This is my blog. That means when I have a bad day courtesy some vendor making a stupid design decision, it just may end up here. (Actually, it probably will end up here.) By the same token, if something makes my day, I'll probably write about it here. I'm going to point again to the first statement - these are my opinions. Insert obligatory 'opinions are like' here.
I lurk around in various places, and generally I don't advertise that I'm watching or listening. I may also post or comment under aliases. All this means is that one, I value my privacy, and two, I prefer not to be harassed for having an opinion of my own. (Yes, I have had people harass me at length. I have better things to do than put up with it.) People who know, know. Those who don't, don't expect me to change that any time in this lifetime.
Hm, yep. That covers everything.
Thursday, November 12, 2009
The Protocol Wars Continue.
http://viewyonder.com/2009/11/12/the-end-is-nigh-for-protocol-passionistas/
Hey guys. My turn.
A lot of the Protocol Passionistas are arguing for all the wrong reasons, absolutely agreed. Some folks probably lump me in there, because I am resolutely against FCoE at this point in time. But not because I like having to run MTP and pinch my fingers in high density cassettes. And not because I hate Ethernet. Oh, and also not because I'm against FCoE either.
The problem I have with FCoE, is the problem I had with NDMP, with iSCSI, with NFS, with any number of protocols. People like to jump the gun and don't pay enough attention to best practices or the limitations of the protocol. FCoE has great potential for good, but far more potential for bad. Think about it for a minute. Can you honestly say you have never seen a poorly configured SAN or Ethernet switch, not once in your entire career? If you said no, you probably haven't been looking. I actively seek those problems, to solve them before they become real problems.
Now let's apply this to converged networking, where we now have FC (disk and tape,) iSCSI (slower disk,) and Ethernet (networking and NAS,) all carried on the same fiber through the same adapter. The Reliable Ethernet part of the stack was only very recently declared tech stable, and is without question the most critical element for FCoE. Buying into things before you know you won't have to forklift for the final standard, not wise. It's important to point out that none of the protocols involved are spring chickens. Even iSCSI is years old. And compared to it's age, real stability is very recent.
And now, we're shifting all of these onto a single converged point. What this means for business is the Elephant in the Room that nobody, and I mean absolutely nobody, wants to admit to. Networking failures and problems, just became everything problems. This isn't a Protocol Passionista point - it's a point period. And a very good and valid one. Lumping all your eggs into one basket, even with redundancy, can be very dangerous. Remember what I said about configuration problems before?
Jumping into the new challenge of converged networking, without meeting the challenge of current networking is such a bad idea, I can't recommend it. And it's still relatively young, giving me additional pause. There is tremendous risk of making mistakes, and mistakes cost real money. Especially when a failed Ethernet port now means failed disk as well.
No question, converged is the future. Like it or not though, the reason isn't what people want it to be. It's not because it's technically superior, it's not because the old division of protocols is a bad idea.
Converged is the future because it's cheaper.
Yes, that's really what's going to drive adoption. The folks controlling budgets saying "why do you need two $1200 HBAs and two $400 multiport GigE cards when the salesman says we can do it with a single $1100 converged adapter?" Ask any budget decision maker - will they take a proven reliable method when there's something significantly cheaper that's 'good enough'?
And once again, it will fall to us to turn 'good enough' into "we can't ever be down unexpectedly ever."
Hey guys. My turn.
A lot of the Protocol Passionistas are arguing for all the wrong reasons, absolutely agreed. Some folks probably lump me in there, because I am resolutely against FCoE at this point in time. But not because I like having to run MTP and pinch my fingers in high density cassettes. And not because I hate Ethernet. Oh, and also not because I'm against FCoE either.
The problem I have with FCoE, is the problem I had with NDMP, with iSCSI, with NFS, with any number of protocols. People like to jump the gun and don't pay enough attention to best practices or the limitations of the protocol. FCoE has great potential for good, but far more potential for bad. Think about it for a minute. Can you honestly say you have never seen a poorly configured SAN or Ethernet switch, not once in your entire career? If you said no, you probably haven't been looking. I actively seek those problems, to solve them before they become real problems.
Now let's apply this to converged networking, where we now have FC (disk and tape,) iSCSI (slower disk,) and Ethernet (networking and NAS,) all carried on the same fiber through the same adapter. The Reliable Ethernet part of the stack was only very recently declared tech stable, and is without question the most critical element for FCoE. Buying into things before you know you won't have to forklift for the final standard, not wise. It's important to point out that none of the protocols involved are spring chickens. Even iSCSI is years old. And compared to it's age, real stability is very recent.
And now, we're shifting all of these onto a single converged point. What this means for business is the Elephant in the Room that nobody, and I mean absolutely nobody, wants to admit to. Networking failures and problems, just became everything problems. This isn't a Protocol Passionista point - it's a point period. And a very good and valid one. Lumping all your eggs into one basket, even with redundancy, can be very dangerous. Remember what I said about configuration problems before?
Jumping into the new challenge of converged networking, without meeting the challenge of current networking is such a bad idea, I can't recommend it. And it's still relatively young, giving me additional pause. There is tremendous risk of making mistakes, and mistakes cost real money. Especially when a failed Ethernet port now means failed disk as well.
No question, converged is the future. Like it or not though, the reason isn't what people want it to be. It's not because it's technically superior, it's not because the old division of protocols is a bad idea.
Converged is the future because it's cheaper.
Yes, that's really what's going to drive adoption. The folks controlling budgets saying "why do you need two $1200 HBAs and two $400 multiport GigE cards when the salesman says we can do it with a single $1100 converged adapter?" Ask any budget decision maker - will they take a proven reliable method when there's something significantly cheaper that's 'good enough'?
And once again, it will fall to us to turn 'good enough' into "we can't ever be down unexpectedly ever."
Subscribe to:
Posts (Atom)
