Monthly Archives: August 2011

SSD Storage: not ready for the enterprise yet?

This week, our whole SAP production environment was wiped out for a day. I wasn’t directly involved in the sleepless process that was the restore and subsequent tests, but it was very interesting to watch from my ivory tower.

Background

First, it’s worth noting that Bluefin runs its whole SAP environment on VMWare ESX. We have some 40-something SAP systems in total (mostly demo systems) and we run something like 80% of our business on SAP technologies as well. It looks like we should get most of the other 20% onto SAP this year – completing our entire Demand to Cash process on the latest SAP technologies including ERP, CRM, BW, Portal, BPC and BI4. We eat our own dog food.

We invested in VMWare very early on and it has paid massive cost reduction dividends. We also invested in storage infrastructure in 2007, and it started to creak early this year, and a proposal to replace it is underway. In the interim – some 8 months ago, we invested in some Solid State Storage (SSD) for our productive environments.

On the face of it, this seemed like a really smart idea. Huge performance increases – allowing a BW upgrade in 2 hours for example – and a stop gap to keep our production systems fast whilst we decided what to do with our main storage fabric.

Why is SSD so fast?

Well SSD in the Enterprise is much like it is for my MacBook Air. You don’t have any moving parts and so it doesn’t have to search to find stuff. Instead, access to information is much faster – around 50x – compared to the spinning plates of metal you have in regular hard disks.

This means that for loads that require a lot of data to be accessed, SSD massively outperforms regular magnetic storage. It sped our productive environment right up and everyone in the company got to see a benefit of some sort.

It’s worth noting at this stage that we put the SSD disks in a RAID5 configuration – meaning that there was some redundancy. There are 6 disks in this case – and it possible to lose one disk, with the others picking up the slack whilst the failed disk is replaced. It turns out that this was our downfall, as you will see later.

It’s also worth noting that the SSD storage that goes into Enterprise equipment isn’t the same stuff you get in the MacBook Air and other consumer devices. It’s faster and it has a much lower fault rate. SSD storage, by the way, has a fault rate which is the amount of information that’s written to it. Past a certain point, your chances of failure go up exponentially.

What happened next?

So around a month ago, we lost all our production systems for a short while. Dell replaced the backplane (the part that connects the disks to the system) on the storage system and it started working again; they tested it in their labs and didn’t find a problem but we had assumed at that time that it was a blip. In retrospect, it was probably a warning shot.

And on Monday night, the first SSD drive failed. This was promptly replaced and during the ensuing rebuild process, all of the other 5 drives failed. We lost all of our systems and had to restore to our other VMWare farm – a time consuming process that involved turning other systems off.

At first glance, this looks utterly bizarre, but it turns out that this is a feature of SSD storage – a dirty little secret, if you like. Remember these disks were only 8 months old.

Why do SSD drives fail en masse?

The answer turns out to be really simple and it is very well technically explained in some dude called Ray’s blog.

But basically SSD disks last for a certain number of write operations. And with RAID5, which we use, that number is halved for technical reasons. Using a database like Microsoft SQL Server and putting logs and data on the same disk group (which we did, and is quite normal in VMWare), you massively decrease that again. It turns out that in the environment we have, SSD disks just won’t last.

So one of the SSD disks will fail sooner or later, which is fine (you replace it). But what happens next isn’t fine. During the rebuild process – which takes an hour or so, there is a substantial additional load placed on the other SSD disks. Because they all have the same failure write rate, and RAID5 balances the load very well across the 6 disks, there is a huge risk that they are all ready to fail at the same time.

Add this to the increased load and you have a very likely possibility that the failure will just cascade and ruin the rest of the disks in the group.

What can we do about this?

Well, short of not using SSD storage? There seem to be a few options:

1) Don’t use RAID1 or RAID5 (or RAID50, RAID10 etc.). Instead use an unbalanced RAID like RAID4 that puts all the stress on a single disk. This one will fail much more often, which is fine.

2) Don’t put database logs on SSD. Put them on regular magnetic storage – they just massively decrease the time time to failure.

3) Advances in SSD storage mean that they will be able to predict failure soon and tell you when to replace them before they go bang.

4) Mix age and/or vendors of SSD drives within a storage group. This will mean that not all the drives fail at once.

5) SSD technology is advancing fast and they are becoming more reliable with each release. This is partially down to smart electronics that balance writes and increase overall reliability.

Conclusions

It seems that despite massive advances in SSD storage, it’s not ready for the prime time yet. If you are considering investing in SSD then research whether the storage system you are buying takes the 4 aspects above into account – and challenge the vendor to how they will work around the problems described.

It’s also fair to say that our IT organisation wasn’t to blame for this. It’s not a well documented phenomenon and we wanted to know how SSD storage worked in the real world of Enterprise IT. They worked long and hard to fix the problems and Chris turned up in the office last night at 7pm looking like death warmed up.

That said, it is clear that SSD is the future. Spinning plates of metal so that we can store information is plain daft.

HP buys software company Autonomy: bank or bust?

I’m sitting here pondering the HP acquisition of Autonomy. I can’t really make sense of it. Can you?

On the one hand it is a tacit admission that the PC hardware business is commoditised and therefore not valuable any more. IBM figure that out years ago and ditched its PC division to Lenovo. Big Blue is smart and it built an excellent services division around it – which has grown, whilst the hardware division shrinks year on year. People just don’t buy those expensive UNIX servers any more.

HP are pretty woeful right now. They’re dead in the UNIX server market – Intel is sunsetting the Itanium CPU it is built on and Oracle have removed support for it. This makes me laugh, because HP are suing Oracle over a platform that they also know is dead. What for? Presumably to show some face to their beleaguered customers.

They’ve also made some bizarre decisions in mobile – the botched Palm acquisition, the TouchPad which has a proprietary OS and isn’t selling well: Apple have them stitched in the market and Best Buy have 245,000 out of 270,000 sitting on a shelf somewhere. Or probably on a delivery lorry back to HP.

To add to this, they bought EDS (which UK magazine Private Eye called “probably the worst consultancy in the world” – like the Carlsberg ad) in 2006 – pulling it into HP Consultancy Services – but not getting anything like the traction that IBM have.

And in an interesting move this year, they employed ex-SAP CEO Leo Apotheker – who left SAP after a hellish 18 months at the helm. Leo never worked well in SAP and in the market we’ve been wondering if he would fare better in HP’s culture. I don’t think this has played out yet and we’ll see what he has to offer in coming months.

But as to the Autonomy acquisition, it just looks bizarre at first glance. It’s a standalone software company that does niche database search products and sells to some big customers. Great, it will diversify HP’s software business but I don’t see the attraction of such a niche?

Either I’m missing something and it is an Apotheker stroke of genius, or HP made a bad call employing him. Let’s sit on the sideline and see how it plays out in 6 months from now. Because HP has already lost 25% of its stock value in 2011 and if this is another bad business decision, HP will pay dearly.

Could Google’s acquisition of Motorola Mobility be a good thing for innovation?

It was a bit of a shock when Google’s acquisition of Motorola Mobility hit the news today. Moto was once a cellphone superpower, and it created the first commercial cellular phone, the DynaTAC 8000X. It had a demise in the early 2000s after failing to innovate around software for its phones.

More recently it has attempted a comeback in the smartphone market by implementing on Google’s Android platform and now the majority of their product lines are built on this basis.

But whilst Android sales have been sold, nothing can match the build quality and usability of Apple’s iPhone. The reason why Apple outclasses the other smartphone vendors in this respect is because it builds the software for the device – thereby limiting the number of varieties of device out there.

This has pros and cons. On the one side there isn’t diversity and choice, and on the other, this makes building software that works well much easier. And this is where Android falls over – there are so many variants that software doesn’t really work well. Updates work even less well and vary from manufacturer to manufacturer and from device to device.

So Google’s acquisition of Motorola Mobility is a great thing in one respect. It allows google to control one of the devices in the Android market and to set a gold standard for which other vendors like HTC and Samsung to adhere to. It can control updates and make really great devices that can really go up against Apple. This can only be a good thing for a market which is starting to become dominated by one major player.

The downside seems to be that the big players have started to play hardball in the courts around patents. Apple are suing Samsung around its Galaxy tablet PC and if history is our guide, the fights will get bloodier and messier. Google’s acquisition of Motorola Mobility means that they acquire a large and varied patent pool.

The question is: will Google flex its muscles in the court as Apple is starting to do, or will it instead use the acquired company to innovate and drive competition in the marketplace for smartphones. We will see.

Why Apple has redefined the laptop with the new MacBook Air

So I’m a few weeks into owning the new MacBook Air. I had this feeling from the start – but now I’m certain of it. The new Air defines a changing point for Apple and for the laptop business as a whole. Here’s why:

Technology Innovation

There’s nothing new in the Air – I mean that. There’s no technology innovation here – all the components are commodity and all were available before. What Apple has done is to put them together at the right time in the right place. And crucially, Intel has made its new i5 and i7 processors available in low-power forms, which allow performance which is sufficient for most people, whilst giving excellent battery life.

The old Air had good battery life (5h) but poor performance, and had a habit of overheating. Apple have added SSD storage (low power, great performance), a screen big enough for most uses, backlit keyboard etc. etc.

Price Point

The next point is price. To get good performance on the old Air, you had to pile on the extras, and when you did this, it went near the $2500/£2000 mark. With the new model, the base configuration is sufficient for most people – and that is just $1299/£1099. It’s true that you can get a netbook for less than half that, but I think they’re rubbish.

More to the point – let’s compare with its serious competitors. An equivalent Dell E4310 is £1650 – compared to the base Apple i5 Air. The equivalent Sony Vaio SA is £1379. The Lenovo Thinkpad X1 is £1699. Apple wipes the competition clean.

Build Quality

The old Air felt a bit fragile, it has to be said. Mine went into repair for the hinge after a while. The new one feels just as well built as the MacBook Pro and feels like it should last its 3 year service life with me. And if it can do that, it will last most people just fine – I travel a lot and use my laptop more than most.

I’ve not used equivalents from Sony or HP but I have used the Dell E4310 and it feels flimsy by comparison. The battery hangs out the back and feels like you could snap it off with 2 fingers and a thumb. I’ve seen the new Lenovo X1 and that is well built – but very expensive.

Battery Life

And here’s the killer blow – battery life. On my old Air I got 4 hours when it was new, and more like 2.5-3h after 2 years. What’s more, it took hours and hours to charge, which meant it was always hard to keep topped up. I’ve just been using this laptop for a good few hours, and i have 65% battery left. In real terms this means I don’t need to carry a charger with me for an average day.

Compare this to the Dell E4310, where you have to have a battery with a bit that hangs out the back of the laptop to get a decent battery life of 4 hours. Or worse the Sony Vaio SA, which claims 7 hours, but owners complain they are lucky to get 2-3h.

And then there’s charging the new Air – an hour gets you 80% charge, which means you are never without. Plus the battery saving features of Lion mean instant-on and deep sleep, so you can leave it in sleep for up to 30 days. nice.

Gestures

The new Lion gestures on the touchpad are worth a short paragraph because it makes using the Air like using an iPad. You can browse and use the computer generally, faster than any device before it. It’s all slightly counter-intuitive to start with (scrolling is back to front for example) but once you get going, you won’t stop.

Criticisms

There will always be critics, and no doubt there are those who say that the lack of expandability – no upgradeable RAM, hard disk or battery, are a problem. For me I’d rather have the better build quality and lower weight that comes from having a sealed unit. I’d also like the option of GPS and 3G wireless integration – Apple seem to expect you to tether it to an iPhone, which I find inconvenient.

Other than that there’s the matter of my iPad2. It’s been lying unused at the bottom of my bag for 2 weeks, which I find troubling. More on that some other time.

Conclusion

Apple have taken technology elements together and blown the market wide open. The MacBook Air will be the killer selling laptop this year. It’s all the computer that someone like me that travels a lot and needs a powerful computer can want, and my Dell laptop is confined to the back of the cupboard.

There is the wider question of the future of the laptop market, as tablets and laptops converge and authoring and consuming content becomes blurred – but I don’t think that will hamper sales.

Attempting to balance consulting with health – Part 1: Self-Awareness

The bane of most consultants is trying to stay fit and healthy. The lifestyle is almost completely incompatible with this and it’s so, late in 2010, that I found myself at some 220lb (100kg or 16 stone). It’s not that I was terribly unfit – I could still run 7-8 miles – but the weight had slowly put itself on over the previous 3 years.

It’s something that we see in a lot of people in consulting. The lifestyle that lures us into the career in the first place – travel, exciting places and interesting roles – is exactly the lifestyle that is our downfall. In my case it’s the long days and late evenings – usually fuelled by 3 meals a day out of the house. For others it’s long stints away from home, digesting fatty hotel food.

To compound this, we find ourselves mentally exhausted from work and unwilling or unable to exercise effectively. How can you exercise when you leave home at 7am and get back at 9pm? And for many, this is further exacerbated by the long days leading to the desire to have a drink – or two… – to relax more quickly.

What’s interesting is that this becomes a vicious circle – because the late nights, fatty food and booze make people sleep worse. We are a little less focussed at work and so work longer hours. We’re a little grouchier too. See where it’s going? Do you see a bit of yourself in this?

So some time in early 2011 I decided to try to kick it. In my case, it’s still a work in progress but I thought it was time to share my experience – in the hope that it might strike a chord with someone else.

The first step – in my opinion at least – is nothing more than realization and self-awareness. We’re supposed to be self-aware in everything we do, as consultants and leaders. Are we really self-aware in how we look and how we treat our bodies? I’m not so sure.

So look in the mirror and think back to college days. I don’t know about you but I was 82kg and had a 32″ waist. When I seriously looked in the mirror at the end of 2010 I was at least 100kg (I started to lose weight before I weighed myself, so I can’t be 100% sure) and my 34″ jeans were – honestly – at least a size too small. My T-Shirts and shirts were full, but not with muscle.

If you look in the mirror and see this person, the first thing you need to do is to relax. Most people out there feel this way at some stage or another. What you do next is up to you but I’d caution against trying to do something extreme. In most cases, extreme reactions (severe diets and regimes) don’t last. They do for some people, but not for most.

The other thing I would caution against is making some purchase to spur on a fitness reaction. Don’t buy the bike, or the rollerblades or treadmill – or even the gym membership at this stage. Rewarding yourself prior to results does not enforce behavioural change. “I just need a XXX and then I will be able to do YYY” just isn’t compatible with human psychology.

For most people, self-awareness combined with small change is all that’s required. The question is – what change to make? That’s for the next part, but for now just look in the mirror and try to be self-aware.

Does Apple have a serious competitor in Microsoft and Nokia?

I’ve been a loyal iPhone customer since its launch in 2007, with just one brief foray into Microsoft’s new Windows Phone 7 platform when it first came out. They redefined the cellphone industry and their focus on profits rather than handset sales has paid off.

RIM is all but dead and for my money at least, there is not – yet – an Android device that competes in quality and battery life for what Apple are offering. Whether you like Android or not is a separate point, and it does have a lot of things going for it and a loyal following, particularly amongst the technical community. Plus, of course the iPhone is expensive.

The thing is though, the cellphone market has a habit of reinventing itself every few years. Nokia, RIM and now Apple have at different times dominated the industry – for 5-10 years at a time. Apple has reached a dominant position and this means that everyone is gunning for their slot.

And enter Microsoft with a completely rewritten Windows Phone 7 platform. As I’ve written before, Microsoft have created an Phone which works like we think. It’s truly social and integrated with Facebook, Twitter and you move effortlessly from pane to pane – unlike the silos within which Apple’s iOS operates. What’s more its integration for email far surpasses what Apple, RIM and Google are doing.

There are essentially two problems with Windows Phone 7. The first is that it’s immature. Microsoft haven’t been agile in releasing new features and the over-the-air updates have been fraught with problems – which is all very reminiscent of early iPhone releases.

The second problem has been more endemic, which is the lack of a decent handset with decent battery life. Sorry HTC and Samsung, but your handsets suck. They’re too big and clunky and they don’t feel like a device that I’d like to hold. And at the price of some of the high-end handsets, you may as well have an iPhone.

Enter Nokia. Purveyor of quality handsets with great build quality and a reputation for excellent battery life. What then if Microsoft and Nokia co-innovate. What if Windows Phone 7 – with bugs fixed and better power consumption was available on a phone as nice as the new Nokia N8? I’ve held a N8 in my hand and it is a quality device – one I’d be happy to own. Shame the platform it is built on is obsolete at launch.

This combined with a decent number of apps (and the developers will come) and Microsoft’s reputation for integration and security may well woo the Enterprise IT community, if not the consumer alike. And from a technical perspective all is well.

The problem unfortunately is that Microsoft and Nokia have become political and social internal disasters – with many levels of management that throttle innovation and agility.

To Microsoft & Nokia: Create a skunk labs for this. Cut the crap out of your organisations and focus only on delivery. Prevent the management levels from stifling innovation or you won’t deliver. If you don’t deliver, remember what will happen. Nokia will become a dinosaur producing cheap mass-produced handsets. And Microsoft, your Windows Phone 7 platform will become another expensive product failure.

To Apple: Be worried. The iOS platform isn’t social and it’s already technologically behind what Microsoft is doing. Despite the fact that you have the advantage of creating beautiful products and a mature platform that works, you will lose this advantage when someone else innovates. And you’re not innovating fast enough with iOS. Use some of the $75bn and rewrite it from the ground up if you need to – it’s OK if it takes 5 years in the making. iOS will last that long, but it is on a meteoric rise, which will inevitably have a fall.