Update: Rob Enderle responds over here! Thanks Rob! :)
Disclaimer; I have no idea if it's preferred as "Xbox360" or "Xbox 360" or whatever. I use a space.
So, some backgrounder. Rob Enderle says that Microsoft shouldn't be in Console Gaming. Greg Knieriemen agrees with him, Microsoft should not be selling hardware. John Obeto says that Microsoft HAD to enter console gaming. Jay Livens shares his own Thoughts on Microsoft and the Xbox as well. So of course, I have to chip in my two cents - especially given that one, I'm an Xbox 360 owner. Two, I'm a Zune owner. Three, I run Windows on the desktop and have for nearly 20 years.
First, let's look at what started this whole discussion: a graph showing where Microsoft gets their profits. You can find it right over here. People drew the conclusion that Microsoft's entertainment business is a drag on their profits, which looking at just that chart, would give that impression. But it's also not true, in my opinion.
Microsoft needed to jump in, because PC gaming isn't going anywhere - by both definitions.
I'm a PC gamer, yes. And let's be blunt, and let's be honest. It's not going anywhere, and it's doing it very quickly. And I very much mean in all senses of it. When was the last time you saw a truly innovative game only available for PC? PC gaming outside of MMOs is a pretty stagnant market for a variety of reasons, a few of which I'll cover in this post. But it's not dying either, contrary to what others might say. But with DRM that is increasingly customer-hostile (hello Ubisoft, looking at you! Hi SecuROM, you too.) and the plethora of problems that come from the fact that PCs are not a stable platform. In my personal experience, more users are turning to consoles for non-exclusive titles.
They want the game, they just want to buy and play, rather than wait for a "beta" patch with no permanent fix ever offered. See most specifically, the ordeal with The Saboteur from EA and developed by Pandemic Studios. PC users have been plagued by severe problems from day one and have yet to receive any support whatsoever. Players who purchased the Xbox 360 version have raved about the game and reported few problems.
The Xbox 360 gives Microsoft their one - and only - stable platform.
Ever looked at an EMC Interoperability chart? People talk a good game, but the fact is that PCs are just as bad as that, if not worse. This sound card may or may not work with this motherboard. These drivers aren't compatible with this game. To say nothing of game and OS interoperability issues, ranging from the "just isn't supported" to "this one impossible to find setting buried deep in the registry causes this game to crash constantly." To say nothing of the myriad ways one can merrily screw up settings on a Windows system - and most gamers usually do. Windows Mobile isn't even close to a stable platform - look at how many Windows Mobile devices there are, on how many completely different processors and architectures.
Xbox 360 has none of these problems. It's a true stable platform. Microsoft controls the OS, drivers, and UI software. User settings are limited. There's a bunch of accessories - all tested and known compatible. By maintaining this tight control and limiting adjustment, Microsoft gives themselves a way to control the overall experience, which brings me to my next point...
The Xbox 360 gives Microsoft a way to control the quality of the experience.
Face facts; Microsoft has no control over the systems people install Windows on. They can control the logo stickers, but that's it. If Joe Blow ships a poorly built system with junk parts running misconfigured Windows 7, users are going to complain about Windows 7 being utter crap - even though it's entirely Joe's fault.
Xbox doesn't have this problem, and probably never will. Microsoft isn't just preventing Joe Blow from making things miserable for end users, they're taking responsibility and ownership of any issues. This extends into the games themselves - Microsoft is the final arbiter of whether or not your game gets released on their system. If they have concerns about the game crashing every two hours, they can prevent release. If they have concerns about the quality of the game, they can prevent the release. And they've done this to games before.
This extends to supplemental products - like Windows Extender and Zune - and brings me to my next point...
The Xbox 360 gives Microsoft a core for their integration strategies.
You can argue Windows is at the core - but it's not. Windows "sort of" is at the core; Windows Extender, Zune, these depend on Windows but get driven to your Xbox 360. But here's the thing - which is more likely to already be in your media center, the Xbox 360 or an HTPC? The Xbox 360 becomes the core because everything is being driven to it, regardless of what it runs on. Rather than fighting the limitations of it, Microsoft has chosen to embrace the limitations and use those limitations to drive other products.
You buy an Xbox 360, you want to watch movies on it - so you now upgrade to Windows 7 so you can use Windows Extender. You like music and have your good stereo downstairs; Zune lets you play your collection on your Xbox 360 that's already there. This is what damages the argument that Xbox 360 is a losing proposition - this integrate and extend mentality allows Microsoft to upsell other products alongside the Xbox 360. Don't forget about the continually evolving Games for Windows LIVE interface as well.
Ultimately, it all ties together around what the Xbox started with the Live Marketplace; everything goes back to Live Marketplace.
You Can't Do That - Yet, But Soon.
Microsoft first released the Xbox in 2001. It was huge. The controllers were too big. Initial sales were disappointing and in some ways, downright depressing. They had to create a smaller controller to replace the almost unusable controllers. It was widely considered a flop and a failure. Microsoft quickly set to work on fixing the various issues, but they had one thing going for them - their online play system, Live. They also had a winning launch title - Halo. Project Gotham Racing also was a well received title at launch, but the system was repeatedly delayed and criticized for difficulties in developing for it.
If there is nothing else Microsoft is good at, they learn from their mistakes.
The Xbox 360 has had it's own mistakes. Most widely known is the Red Ring of Death. Microsoft owned up to the problem and handled that very poorly. They resolved it by performing a major revision of the hardware - which virtually ended the RRoD plague in one swoop. The Xbox interface has been marked much like the Xbox itself, by a process of continual improvement. From the original Xbox to the Xbox 360 to the Xbox 360 "NGE" interface, Microsoft has continually added features and fixed complaints. They listen to the users, and they act on it.
The Xbox 360 extends and embraces Microsoft's attempts to be the go-to place for developers, and succeeds.
Bet you didn't think I'd use that line ever in my life. SURPRISE! A good friend of mine is an independent game developer, and has console development knowledge. I've personally done research and some programming for the Cell BE which powers the PS3. Let's be very blunt and to the point: programming for Cell BE is a special level of hell. Seriously. It's abysmal, and pure misery. Developing for the PS3 is incredibly difficult, complicated, and expensive. You need the full console developer kit, which costs thousands of dollars, and if you want to publish? Well, you need to do the same dance the major publishers do with Sony, pretty much. It is an incredibly expensive and difficult proposition to develop for PS3, and almost none of your code will be reusable.
Now let's say you want to develop an Xbox 360 game.What resources do you need to buy and invest in? An Xbox 360 Elite and an XNA Content Creator account, oh, and Visual Studio - but that's actually optional. Congratulations, you are now fully equipped to develop for the Xbox 360. But what about publishing your game? Through Live, Microsoft has a solid reliable and well marketed distribution channel for independent developers. Independents can put up their latest creation, and start making money in short order. Microsoft is very active about promoting quality titles from XNA.
You want an example? No problem! Check out Xbox Live Indie Games. How easy is it to get your game up and making you money on Live Marketplace? Here's my friend J's Pendoku on Xbox Live - developed over a few weeks - and here's the free Flash version. Putting it on Marketplace was as easy as uploading and going through a few menus to publish, then waiting for approval, which usually takes around two weeks I believe. It's very easy. Which is what independent developers not only want, but need - lower barriers.
The Xbox 360 is not meant to be a profit powerhouse.
This is probably the most important point here. Since when has Microsoft ever thrown good money after bad for so long? The answer is; never. Microsoft has an internal and very secret set of goals for Xbox 360, and externally they seem to be rather pleased with them.
I'm going to wager a guess on what one of them is; be more popular than PS3, or at the minimum be perceived as such. In that respect, they're doing just fine. Take a look at this ArsTechnica Look at 2009 Console Sales - the Xbox 360 runs ahead of the PS3 almost all of 2009. 4.77 million units last year, while Nintento absolutely dominates the more casual market. But that's not the entire picture, oh no. Remember what I said about it being so easy to publish on Live Marketplace? Here's some sales statistics on Indie Games from GamerBytes. See for yourself; 160K copies sold of "A GAM3 W1TH Z0MB1ES" developed by James Silva. That's huge and more than some major publisher releases! This increases visibility, increases popularity, and encourages more independent developers.
It all translates into one thing; Microsoft has never really planned for Xbox 360 to make them billions of dollars in profit. Game consoles just don't. Instead, they planned for a specific goal and seem to be very happy with their results this far. They know as well as I do that there is no chance of Game or Entertainment segments becoming replacements for Windows or Office. Think about it rationally - how many offices do you know that put an Xbox 360 on everyone's desk? Now how many put a PC on everyone's desk. That's why Xbox 360 is tiny compared to Windows and Office. But that "tiny" number as perceived by others, appears to be making Microsoft very happy overall. If it wasn't, they'd change their strategy, and there's no sign of that. Instead they're continuing on their combination of extending and improving with things like Project Natal, going for a differentiation from the PS3 and maintaining a strong lead in Online.
So ultimately, I'm entirely behind Microsoft's entry into the gaming console market. Let's be blunt here - if you play any sort of "violent" game, Wii is not for you, period. Nintendo has the strictest content policies out there; so bad that they've blocked some games, and forced Dead Rising to be almost totally rewritten to meet Nintendo's content guidelines. Independents want nothing to do with Nintendo - getting a game approved is a nightmare process, and also requires a huge investment in hardware.
Sony has lead in innovation, and could be said to still have a substantial lead there in some areas. The problem remains that the PS3's initial release was a disaster in all ways, and Sony chose not to correct it aggressively. In fact, they sat on it and did nothing to deal with it. Their innovation has cost them titles and the cost of development has cost them exclusivity agreements. It's hard for publishers to make money on PS3 games, and indie developers haven't steered clear as much as been tossed over the high cost fence with a 'kick me' sign on their back.
Microsoft is the only company out there right now that is bringing multiple areas together, and doing a damn fine job of executing on it. Think about it - it has franchises like Halo, a violent FPS that caters to the 12-24 male segment. But they also have games like Lego Rock Band, which is decidedly a family title - especially given that Lego has strict content guidelines like Nintendo. Add to this, they have a whole raft of not just hardcore but casual and family friendly games from independent developers. They're winning the hearts and minds battle, because when those developers can (and do) make money from putting their games on Live Marketplace, they tend to become vocal supporters. The Xbox 360 provides them with a stable platform to ensure a quality experience that users may not be able to get on their PC for any number of reasons.
Microsoft absolutely needed to enter the console market, and now that they've been here a while, it's blatantly obvious that they not only intend to continue but deserve to continue. Say what you will, but the fact is that Microsoft isn't winning the hearts and minds battles in the console space purely on marketing muscle or dollars spent. They're winning it because they came to the game with an attitude that they were going to change the game, and deliver a seamless, solid experience. They've stuck to that aggressively positive attitude, and it's delivered not just for Microsoft, but for users as well.
MANDATED DISCLAIMER: I own a Microsoft Xbox 360, two Microsoft Zunes, a Zune Marketplace subscription, a couple Windows XP and 7 licenses - all at my own expense. I also will be buying a PS3 the day Gran Turismo 5 is released - sorry, Microsoft. Forza doesn't do it for me like Gran Turismo does. Also, a good friend is an XNA CC member, as mentioned above.
That said, these people STILL won't give me free stuff! What gives? ;)
Tuesday, February 23, 2010
Wednesday, February 17, 2010
Why I Hate Macs
Let the flame wars commence! Okay, first read the post, THEN flame, capice? That's better. I wrote this up ages ago, and just never bothered to post it, because I didn't feel like having people scream at me for a week straight. Well hey, I'm in a self-abusing mood.
So yeah. I hate Macs. As in absolutely loathe and refuse to own a current Mac, period. Why? I've got a whole list of reasons, but let's step back a moment. Remember when Macs shipped with SCSI disk and 68k processors, then later PowerPC 603's and 604's? I loved Macs back then. Owned two. Because they actually differentiated themselves from Windows PCs, offered value for the dollar, and were superior for the tasks I needed to do on them.
These days? The Mac population consists largely of rabid zealots who think you're the anti-christ if you don't like Mac for any reason. That's a big reason I hate Macs and refuse to own one. The other part of it is the same zealots who refuse to accept or acknowledge that Macs and OS X are not the superior platform for every single task imagined. Let's put this all into a nice neat list though.
OS X cannot run software I need daily. Do not give me bootcamp or Parallels excuses. If it can't run it natively, then it can't run the damn software. Here's a list. Software that is totally unavailable or unworkable is in red, with severe limitations in yellow. Severe limitations includes lack of plugin availability or cross-platform compatibility issues.
Mozilla Firefox, Microsoft Outlook, Microsoft Word, Microsoft Excel, Microsoft Visio, TweetDeck, vim/Gvim, NFS, CIFS, Pad2Pad, Pidgin, Zune, Steam, VMWare Workstation, Ableton Live, Propellerheads Reason 4, City of Heroes
Ableton Live is in orange because it's there. But half my needed VSTs do not have a Mac version available, or do not work at all on Mac, making it nigh unusable. And a complete lack of Visio means it is completely unusable, because I abuse Visio in depth, daily.
"It's the software, stupid" arguments from Mac zealots. Hello, see immediately previous point. The very existence of Fusion, Parallels and Bootcamp all destroy that argument in nothing flat. If it was the magical OS X, then why do you have to run Windows on your Mac with virtualization software so you can access applications that aren't available on it? Yeah. It's totally everyone else's fault entirely and they're idiots for not maintaining multiple code trees and development teams just for Mac. Sure. And you know what? If I'm just going to reboot the damn thing into Windows anyways, then why the hell would I buy OS X at all?
The hardware is out of date and overpriced. No, you are not getting a "superior" product, you are just paying a ridiculous brand premium for a fashionable chassis. To meet my requirements requires a Mac Pro which costs well over $7,000.00 USD before the 3 year warranty (required thanks to Apple's continuing string of QC and QA problems.) For this $7,000.00 pricetag I get a SINGLE last generation Xeon Quad Core 3.33GHz based on Intel's abandoned 5000-series chipset with only 16GB of nigh unavailable Fully Buffered DIMMs, a $700 non-hardware non-BBU RAID card which I can buy elsewhere for around $100, an ATI Radeon HD4870 512MB at $200 which is not only a generation old but available in the proper 1GB configuration at a little over half that price, a pair of middle of the road DVD drives, iWork, about half of Microsoft Office, and a mediocre warranty with support where I still have to buy the OS upgrades out of pocket with no discount. THIS IS NOT A GOOD VALUE BY ANY MEASURE. For under $7,000 I can build a dual socket 12/24 core system which is not only watercooled but packing two Radeon HD5870's in CrossFire, an SSD and two 500GB/32MB drives in RAID0, and 24GB of DDR3. And there's not one bit of difference except that A) I'm using CURRENT generation hardware B) I don't have the brushed aluminum chassis. Oh wait, I do, except it's black and has hotswap drive bays and is quieter. Hell, I charge under $7K for that built and shipped to your door with half the parts packing 5 year manufacturer warranties!
Apple loves to lock in their vendor lock-out. Nobody gets to just write applications for OS X. You have to pay your special Apple taxes. It's not just an iPhone thing, either. You must pay Apple for documentation, for licenses, and so on. And if they decide they aren't happy with you, they can cut off any developer with no notice, leaving the users in the lurch. Want to use anything non-Apple that isn't an external thumb drive? If they didn't pay their Apple taxes, it's a crap shoot if it'll work or be supported. When was the last time you plugged in a keyboard or mouse to your PC and found that it was completely and utterly incompatible because of the software?
I play games. Get over it, Mac heads. Apple is not a gaming platform. If it was a viable, popular gaming platform, then popular games would be released for it. Instead, you have a handful of MMOs with Mac clients as an option, and some older games. Mass Effect 2 is nowhere to be found, nor is Steam, much less Modern Warfare 2 or Bioshock 2. ME2, MW2 and Bioshock are 3 of the absolute most popular, best selling games out there. They have massive, incredibly large budgets. Yet there is no OS X version of any of these games. Take the hint; if you enjoy playing a variety of games to relax, then Apple is definitely not for you. And World of Warcraft doesn't count, when the vast majority of LUA-based addons don't work on Macs.
The attitude of the company and the "community" just SUCKS. "Look at us in our faded jeans and black turtlenecks and OHMYGODNEWIPHONE MUST HAVE NOW!" Yeah, because a guy with 20 years of system design and building experience is the kind of guy who goes out and buys the latest 1.0 because somebody put a shiny Apple logo on it. But gods help you if you should rationally explain why you don't like iTunes or the iPod or the iPad to most Apple users. Seriously, I've had one actually take a swing at me for suggesting that an old G4 Xserve with significant performance problems was not the best answer, because of the performance problems. I have found on dozens of occasions that Apple users have absolutely zero interest or focus on whether or not the job is done right, or even if the job is done - their only concern is rabid support of a shiny white logo. And the only "appropriate" response to any question of the usefulness or functionality of any Apple product is to shout the person down and insult them.
Please note, this isn't a blanket statement about all Apple users. I know several that are perfectly reasonable and rational, and completely understand and support my refusal to use a product that doesn't work for me. ("Why should you, if it doesn't do what you need? That's silly.") However, as I said, I have had Apple fanatics actually swing at me for questioning Apple. And more often than not, that is exactly the reaction I get when I point out that an Apple product doesn't do what I need or want - screaming, insulting, and swinging. I want no part of any organization or community that considers shouting down acceptable behavior.
The presumption that because I'm a Unix guy I must love OS X. Look. It's an operating system. Get over it. AIX is an operating system. Solaris is an operating system. Windows is an operating system. The notion that any one of these does not suck is a complete fallacy. All operating systems suck, just in different ways. It's a fact of computing. It shouldn't be, but it is, and I deal with it every day. Just because I'm a Unix guy does not mean I prefer or even necessarily want Unix on my desktop. My operating system does not give me any sort of self-gratification or pleasure whatsoever, no matter what it is. I don't flipping care if it's Unix-like! It's an operating system, not a damn religion.
If I wanted Unix on my desktop, then I would use FreeBSD or AIX. But see, the thing is that I don't. I want to get things done and I do not want to waste time beating them into working through emulators like WINE or rebooting to a different OS so I can play a game. I want to turn on my computer, get work done, close it out and go play some games, then shut it down at the end of the day. That's it.
So, I figure I've explained sufficiently why I hate Macs personally. Calmly and rationally. This of course means, that I can expect irrational screaming and death threats from a select few who frankly have no business giving me advice or making demands about what hardware and software I use. Those of you who want to scream and yell at me, go hang out with the people who agree with me that an iMac completely fails to meet my disk requirements and a Mac Pro is horrifyingly overpriced and learn from them. Please.
So yeah. I hate Macs. As in absolutely loathe and refuse to own a current Mac, period. Why? I've got a whole list of reasons, but let's step back a moment. Remember when Macs shipped with SCSI disk and 68k processors, then later PowerPC 603's and 604's? I loved Macs back then. Owned two. Because they actually differentiated themselves from Windows PCs, offered value for the dollar, and were superior for the tasks I needed to do on them.
These days? The Mac population consists largely of rabid zealots who think you're the anti-christ if you don't like Mac for any reason. That's a big reason I hate Macs and refuse to own one. The other part of it is the same zealots who refuse to accept or acknowledge that Macs and OS X are not the superior platform for every single task imagined. Let's put this all into a nice neat list though.
OS X cannot run software I need daily. Do not give me bootcamp or Parallels excuses. If it can't run it natively, then it can't run the damn software. Here's a list. Software that is totally unavailable or unworkable is in red, with severe limitations in yellow. Severe limitations includes lack of plugin availability or cross-platform compatibility issues.
Mozilla Firefox, Microsoft Outlook, Microsoft Word, Microsoft Excel, Microsoft Visio, TweetDeck, vim/Gvim, NFS, CIFS, Pad2Pad, Pidgin, Zune, Steam, VMWare Workstation, Ableton Live, Propellerheads Reason 4, City of Heroes
Ableton Live is in orange because it's there. But half my needed VSTs do not have a Mac version available, or do not work at all on Mac, making it nigh unusable. And a complete lack of Visio means it is completely unusable, because I abuse Visio in depth, daily.
"It's the software, stupid" arguments from Mac zealots. Hello, see immediately previous point. The very existence of Fusion, Parallels and Bootcamp all destroy that argument in nothing flat. If it was the magical OS X, then why do you have to run Windows on your Mac with virtualization software so you can access applications that aren't available on it? Yeah. It's totally everyone else's fault entirely and they're idiots for not maintaining multiple code trees and development teams just for Mac. Sure. And you know what? If I'm just going to reboot the damn thing into Windows anyways, then why the hell would I buy OS X at all?
The hardware is out of date and overpriced. No, you are not getting a "superior" product, you are just paying a ridiculous brand premium for a fashionable chassis. To meet my requirements requires a Mac Pro which costs well over $7,000.00 USD before the 3 year warranty (required thanks to Apple's continuing string of QC and QA problems.) For this $7,000.00 pricetag I get a SINGLE last generation Xeon Quad Core 3.33GHz based on Intel's abandoned 5000-series chipset with only 16GB of nigh unavailable Fully Buffered DIMMs, a $700 non-hardware non-BBU RAID card which I can buy elsewhere for around $100, an ATI Radeon HD4870 512MB at $200 which is not only a generation old but available in the proper 1GB configuration at a little over half that price, a pair of middle of the road DVD drives, iWork, about half of Microsoft Office, and a mediocre warranty with support where I still have to buy the OS upgrades out of pocket with no discount. THIS IS NOT A GOOD VALUE BY ANY MEASURE. For under $7,000 I can build a dual socket 12/24 core system which is not only watercooled but packing two Radeon HD5870's in CrossFire, an SSD and two 500GB/32MB drives in RAID0, and 24GB of DDR3. And there's not one bit of difference except that A) I'm using CURRENT generation hardware B) I don't have the brushed aluminum chassis. Oh wait, I do, except it's black and has hotswap drive bays and is quieter. Hell, I charge under $7K for that built and shipped to your door with half the parts packing 5 year manufacturer warranties!
Apple loves to lock in their vendor lock-out. Nobody gets to just write applications for OS X. You have to pay your special Apple taxes. It's not just an iPhone thing, either. You must pay Apple for documentation, for licenses, and so on. And if they decide they aren't happy with you, they can cut off any developer with no notice, leaving the users in the lurch. Want to use anything non-Apple that isn't an external thumb drive? If they didn't pay their Apple taxes, it's a crap shoot if it'll work or be supported. When was the last time you plugged in a keyboard or mouse to your PC and found that it was completely and utterly incompatible because of the software?
I play games. Get over it, Mac heads. Apple is not a gaming platform. If it was a viable, popular gaming platform, then popular games would be released for it. Instead, you have a handful of MMOs with Mac clients as an option, and some older games. Mass Effect 2 is nowhere to be found, nor is Steam, much less Modern Warfare 2 or Bioshock 2. ME2, MW2 and Bioshock are 3 of the absolute most popular, best selling games out there. They have massive, incredibly large budgets. Yet there is no OS X version of any of these games. Take the hint; if you enjoy playing a variety of games to relax, then Apple is definitely not for you. And World of Warcraft doesn't count, when the vast majority of LUA-based addons don't work on Macs.
The attitude of the company and the "community" just SUCKS. "Look at us in our faded jeans and black turtlenecks and OHMYGODNEWIPHONE MUST HAVE NOW!" Yeah, because a guy with 20 years of system design and building experience is the kind of guy who goes out and buys the latest 1.0 because somebody put a shiny Apple logo on it. But gods help you if you should rationally explain why you don't like iTunes or the iPod or the iPad to most Apple users. Seriously, I've had one actually take a swing at me for suggesting that an old G4 Xserve with significant performance problems was not the best answer, because of the performance problems. I have found on dozens of occasions that Apple users have absolutely zero interest or focus on whether or not the job is done right, or even if the job is done - their only concern is rabid support of a shiny white logo. And the only "appropriate" response to any question of the usefulness or functionality of any Apple product is to shout the person down and insult them.
Please note, this isn't a blanket statement about all Apple users. I know several that are perfectly reasonable and rational, and completely understand and support my refusal to use a product that doesn't work for me. ("Why should you, if it doesn't do what you need? That's silly.") However, as I said, I have had Apple fanatics actually swing at me for questioning Apple. And more often than not, that is exactly the reaction I get when I point out that an Apple product doesn't do what I need or want - screaming, insulting, and swinging. I want no part of any organization or community that considers shouting down acceptable behavior.
The presumption that because I'm a Unix guy I must love OS X. Look. It's an operating system. Get over it. AIX is an operating system. Solaris is an operating system. Windows is an operating system. The notion that any one of these does not suck is a complete fallacy. All operating systems suck, just in different ways. It's a fact of computing. It shouldn't be, but it is, and I deal with it every day. Just because I'm a Unix guy does not mean I prefer or even necessarily want Unix on my desktop. My operating system does not give me any sort of self-gratification or pleasure whatsoever, no matter what it is. I don't flipping care if it's Unix-like! It's an operating system, not a damn religion.
If I wanted Unix on my desktop, then I would use FreeBSD or AIX. But see, the thing is that I don't. I want to get things done and I do not want to waste time beating them into working through emulators like WINE or rebooting to a different OS so I can play a game. I want to turn on my computer, get work done, close it out and go play some games, then shut it down at the end of the day. That's it.
So, I figure I've explained sufficiently why I hate Macs personally. Calmly and rationally. This of course means, that I can expect irrational screaming and death threats from a select few who frankly have no business giving me advice or making demands about what hardware and software I use. Those of you who want to scream and yell at me, go hang out with the people who agree with me that an iMac completely fails to meet my disk requirements and a Mac Pro is horrifyingly overpriced and learn from them. Please.
Thursday, February 11, 2010
Big ESX in a Tiny Box - What's up with the delay here?!
Okay, I owe everyone an explanation, so here goes.
I've run into a problem that was not present on the prior build. This is an extremely severe problem that makes the system completely unusable. Understand this is through no fault of design or implementation here, but rather, due to a very severe bug in ESX/ESXi's Intel EtherExpress driver, specifically in the MSIX Vector section of the e1000 driver. Please understand, this bug did not present previously in any situations I tested. Remember, every component in the system is on the HCL. The problem here is with an Intel i82574L ethernet controller; you can find it under IO Devices, Networking, partner name Intel.
At this point, I'm trying to get a 12x5 Basic contract via VMware so that this bug can be escalated properly. The exact issue, going into the technical side of things, has to deal with how ESX/ESXi handles the Interrupt Vector Address Routing or IVAR for PCI-Express MSIX. If this part goes way over your head, don't worry; it's supposed to. This requires that you have prior experience developing drivers and doing kernel programming to understand. It also requires knowledge of the Intel EtherExpress family and PCI-Express bus.
So, the i82574L/LA has a 5 entry IVAR. Typical drivers will use only the first three IVARs and ignore activities on entries 4 and 5. (Technically, 3 and 4, as it starts at 0. So I'm going to be starting from 0 here.) ESX/ESXi uses or touches all of the IVAR table, 0-4. The i82574 can operate in a number of modes, which are identified by the Function registry entry. In normal operation on most systems, Function will be 0, which indicates the following list of items:
- Operating as LAN0
- Operating as LAN1
- Operating as LAN0 shared with IPMI/BMC
- Operating as LAN1 shared with IPMI/BMC
Yes. A single Function mode indicator, indicates FOUR functions. So how do we control whether we're doing operations strictly for the host, or we're doing operations for the IPMI? By asserting via MSIX and the IVAR table.
Here's where ESX/ESXi's e1000 driver breaks in a predictable and reproducible fashion. I can't explain why it's breaking, only exactly HOW it's breaking.
When operating normally, the e1000 driver will LOSE the MSIX vector completely. This results in the Interrupt Status Register being lost, causing the driver to lose awareness of the controller state as well as halting all network traffic. This is not enough to crash ESX/ESXi, and the driver continues operation without asserting an error, even though interrupts are doing nothing. This also means that the driver is unaware of link state changes, so any HA/FT features will be rendered useless as far as that host is concerned. (Remote hosts will have to assert failure condition on network going unreachable and yank control.) If you attempt to use ethtool to diagnose the i82574L/LA at any point during operation in either online or offline mode, you will get this failure:
PCPU 0 locked up. Failed to ack TLB invalidate (0 others locked
up).
cr2=0x0 cr3=0x400ed000 cr4=0x16c
0:8353/ethtool 1:6231/sfcbd *2:4109/helper1-0 3:5287/sfcbd
4:5101/openwsman 5:5001/hostd 6:5100/openwsman 7:8450/prop_of_i
--
Saved backtrace from: pcpu 0 TLB NMI
Sanitizing and rewriting to make it make sense, here's the actual code path of the failure.
FindIRQInfo+0x69
RemoveIRQInfo+0x41
vmklnx_request_irq+0x32f
e1000_diag_test+0xc5f
ethtool_self_test+0xfc
__ethtool_ioctl+0xe62
vmklnx_ethtool_ioctl+0x7a
netdev_ioctl+0x101
NicCharOpsIoctl+0x65
VmkApiCharDevIoctl+0xe6
DevFSIoctl+0x3e5
FSS_Ioctl+0x17d
UserFile_PassthroughIoctl+0x44
LinuxFileDesc_Ioct+0x7e
User_LinuxSyscallHandler+0xa3
So what's the exact problem? When it allocates the interrupt vector, it promptly loses it. It appears to be IVAR entry 5, as ESXi reports looking for 0x5a5a5a5a and instead getting 0xffffffff.
Exposure of the problem appears to be tied to a change in board settings which most users will make, meaning it's very easy to trigger. (The initial build was slightly different.) Until I get this issue escalated within VMware to the point where they're actually guaranteeing further investigation as well as a fix for this problem, I can't tell you what parts to buy because obviously, they no longer work right now, even though they are on the VMware HCL.
Sorry folks. I'm working my tail off here to get VMware to look at this. I have the crash dumps available on request. I just don't have a support contract.
I've run into a problem that was not present on the prior build. This is an extremely severe problem that makes the system completely unusable. Understand this is through no fault of design or implementation here, but rather, due to a very severe bug in ESX/ESXi's Intel EtherExpress driver, specifically in the MSIX Vector section of the e1000 driver. Please understand, this bug did not present previously in any situations I tested. Remember, every component in the system is on the HCL. The problem here is with an Intel i82574L ethernet controller; you can find it under IO Devices, Networking, partner name Intel.
At this point, I'm trying to get a 12x5 Basic contract via VMware so that this bug can be escalated properly. The exact issue, going into the technical side of things, has to deal with how ESX/ESXi handles the Interrupt Vector Address Routing or IVAR for PCI-Express MSIX. If this part goes way over your head, don't worry; it's supposed to. This requires that you have prior experience developing drivers and doing kernel programming to understand. It also requires knowledge of the Intel EtherExpress family and PCI-Express bus.
So, the i82574L/LA has a 5 entry IVAR. Typical drivers will use only the first three IVARs and ignore activities on entries 4 and 5. (Technically, 3 and 4, as it starts at 0. So I'm going to be starting from 0 here.) ESX/ESXi uses or touches all of the IVAR table, 0-4. The i82574 can operate in a number of modes, which are identified by the Function registry entry. In normal operation on most systems, Function will be 0, which indicates the following list of items:
- Operating as LAN0
- Operating as LAN1
- Operating as LAN0 shared with IPMI/BMC
- Operating as LAN1 shared with IPMI/BMC
Yes. A single Function mode indicator, indicates FOUR functions. So how do we control whether we're doing operations strictly for the host, or we're doing operations for the IPMI? By asserting via MSIX and the IVAR table.
Here's where ESX/ESXi's e1000 driver breaks in a predictable and reproducible fashion. I can't explain why it's breaking, only exactly HOW it's breaking.
When operating normally, the e1000 driver will LOSE the MSIX vector completely. This results in the Interrupt Status Register being lost, causing the driver to lose awareness of the controller state as well as halting all network traffic. This is not enough to crash ESX/ESXi, and the driver continues operation without asserting an error, even though interrupts are doing nothing. This also means that the driver is unaware of link state changes, so any HA/FT features will be rendered useless as far as that host is concerned. (Remote hosts will have to assert failure condition on network going unreachable and yank control.) If you attempt to use ethtool to diagnose the i82574L/LA at any point during operation in either online or offline mode, you will get this failure:
PCPU 0 locked up. Failed to ack TLB invalidate (0 others locked
up).
cr2=0x0 cr3=0x400ed000 cr4=0x16c
0:8353/ethtool 1:6231/sfcbd *2:4109/helper1-0 3:5287/sfcbd
4:5101/openwsman 5:5001/hostd 6:5100/openwsman 7:8450/prop_of_i
--
Saved backtrace from: pcpu 0 TLB NMI
Sanitizing and rewriting to make it make sense, here's the actual code path of the failure.
FindIRQInfo+0x69
RemoveIRQInfo+0x41
vmklnx_request_irq+0x32f
e1000_diag_test+0xc5f
ethtool_self_test+0xfc
__ethtool_ioctl+0xe62
vmklnx_ethtool_ioctl+0x7a
netdev_ioctl+0x101
NicCharOpsIoctl+0x65
VmkApiCharDevIoctl+0xe6
DevFSIoctl+0x3e5
FSS_Ioctl+0x17d
UserFile_PassthroughIoctl+0x44
LinuxFileDesc_Ioct+0x7e
User_LinuxSyscallHandler+0xa3
So what's the exact problem? When it allocates the interrupt vector, it promptly loses it. It appears to be IVAR entry 5, as ESXi reports looking for 0x5a5a5a5a and instead getting 0xffffffff.
Exposure of the problem appears to be tied to a change in board settings which most users will make, meaning it's very easy to trigger. (The initial build was slightly different.) Until I get this issue escalated within VMware to the point where they're actually guaranteeing further investigation as well as a fix for this problem, I can't tell you what parts to buy because obviously, they no longer work right now, even though they are on the VMware HCL.
Sorry folks. I'm working my tail off here to get VMware to look at this. I have the crash dumps available on request. I just don't have a support contract.
Subscribe to:
Posts (Atom)
