Over in the PACSGroup UK forum there is a discussion thread that has been quite busy the last week or so.  The overall topic is the specification of the notion of ‘Vendor Neutral Archive’ – in particular, at least as I read it, what problems VNA are supposed to solve.  Lest I be misinterpreted at any point – I am a big supporter of any disruption in the Radiology/PACS market that empowers the end user to a greater extent than they have been in the past.  That disruption has come in the shape of VNAs over the last couple of years and I tip my cap.

However, there is one suggestion regarding the value of VNAs that makes me pause for thought.  That is that VNA can be used as a Business Continuity facility to mitigate against the risk of PACS becoming unavailable.  That just in its own right is a limited view of how PACS BC can be approached.  A VNA has the capability of maintaining an independent copy of images – which can be used as the distribution hub for clinical locations outside Radiology.  Lets assume for the moment that the viewing capabilities outside of Radiology are themselves independent of PACS (with the state of affairs as well as many procurements under way even as I speak), the fact is (elephant in the room?) if PACS is down – RADIOLOGY is down. That is NOT providing any kind of continuity of service at all.

And, in my mind – it is a profoundly oversimplified perspective of what BC is and the challenges that face it, as a process.

The foundation of business continuity are the standards, program development, and supporting policies; guidelines, and procedures needed to ensure a firm to continue without stoppage, irrespective of the adverse circumstances or events. All system design, implementation, support, and maintenance must be based on this foundation in order to have any hope of achieving business continuity

Wikipedia: Business Continuity
Emphases mine.

It is important to realise that Business Continuity measures cannot be easily plugged-into existing systems and in a moment I’ll give a couple of examples where BC processes can be hampered by decisions made in the delivery of basic infrastructure – quick possibly many years prior to the procurement of a given system.

Business continuity is sometimes confused with disaster recovery, but they are separate entities. Disaster recovery is a small subset of business continuity.

DR is where the old-fashioned off-site backups come in.  Lets say you suffer the nightmare scenario – you have a bunch of servers running in a server room which is consumed in fire.  DR is then about recovering to a ‘normal’ state of affairs.  You may need to provision a new server room.  You’ll probably need to order up a bunch of hardware in a hurry (some thoughts on Vendor Relationship Management to follow!) and mobilise your software vendors to help restore from whatever backup strategy you have in place – which – yes – can include a VNA (as long as it was on a seperate site).  Depending on the number of systems involved and the amount of preparation, that process is likely to be measured on a timescale of days.  Clearly, DR in this extreme sense, is NOT about continuity of business process.

Where BC comes in in this case – is in protecting the viability and integrity of core business processes while the DR process is ongoing.  And BC must consider a whole gamut of potential risks other than the ‘nightmare scenario’.  Some of these risks can be identified, analysed, and mitigated.  Some that are identified may simply be carried, and there will always be the scenario that comes out of left field.  One the the core messages in the superb book “The Back Swan” by Nassim Nicholas Taleb is that the risks we identify should never do us significant damage – precisely because we saw it coming and can prepare in advance.  It is the disasters and events we don’t see coming (like the existence of black swans) that will have the biggest effect – because we didn’t.  It is the job of the Business Continuity Process to provide the flexibility and structures to deal with the black – as well as the white – swans.

A couple of examples that are reasonable extrapolations of real events that have occurred in any number of sites around the world:

Example 1:

Lets say one has a server room with a number of servers – most of which have disk storage served from a SAN.  The hardware for the servers, the SAN and the network has been carefully specified and implemented to eliminate any Single Point of Failures (SPoF) with multiple layers of redundancy, integrity checking and clustering.  Fire suppression is state-of-the-art.  What can go wrong?  The air conditioning fails, and will take 3 hours to fix.

With the heat generated by the various kit, the room temperature will go out-of-spec in 1 hour.  All servers must then be shut down until the AC is fixed and the environment restored.  It is possible to identify a subset of servers that simply MUST be kept online.  Can the remainder be shut down for the duration while the handful stay up – yes they can – but it doesn’t make any difference.  By far the biggest generator of heat is the SAN.  The SAN is either up – or it isn’t.  If it isn’t – all servers are down anyway.

Lesson: Its really easy to miss a risk at an early point in an overall infrastructure deployment that turns out to have a much bigger impact when facilities have grown.

How does BC handle this?  Probably by fallback to a manual process.  Does that mean special stationary needs to be held in stock? Does everybody involved know how to fallback to manual?  Will you need a team of facilitators mobilised to ease that process?  It WILL have an impact on the performance of involved departments – does there need to be an escalation plan to limit service demand by (for example) diverting emergency patients to alternative facilities? This is Business Continuity Planning, and requires a holistic approach.

Example 2:

Lets say you have all of that in place, but that the ‘event’ was a fire outside the server room.  Lets face it, fire suppression works, but fires still happen. The suppression kicked in and as soon as the Fire Safety Officer has given the all-clear, servers are restarted within 30 minutes and the server room back to normal in good time.  In the meantime the arrangements are that test orders are written on card (a supply of pink and blue cards is maintained in central stores) and urgent orders telephoned through to the relevant department. Additional staff are drafted in to deal with the extra legwork and data entry.  All fine.  Except the fibre optic cables from the server room were housed in metal conduits.  The conduit got hot, and melted the cables so although the servers are up – they are still offline.  So manual fallback is still running.  Except for Orthopaedic Theatre – whose telphones run over VoIP – controlled – or not – by a server in the (currently) offline server room.  Oops.

Even the best laid plans sometimes need to pivot and adapt to retain BC.  And therein lies the key to true Business Continuity Planning – people.

If people on the ground (in clinical, admin and operational areas as well as technical) have the right skills and knowledge – technical, communications, and management, then whatever threat comes out of left field is vulnerable.  The more static and rigid the available skills – the more likely the Black Swan will bite.

The big change is a decisive movement away from the GUI console as the primary means of local administration.  That means no point-and-click with right-click context menus, no visual clues, and no ‘wizards’ – at least in the form that most folk are accustomed to.

The changes (for Windows Server aren’t an immediate turnaround.  WS8 will have an optional minimal-GUI for the time being.  But even the official line from Microsoft is:

In Windows Server 8, the recommended application model is to run on Server Core using PowerShell for local management tasks and then deliver a rich GUI administration tool capable of running remotely on a Windows client.

.. which of course is the model that MS themselves have been using in tools like AD admin for many years.  But as RedmondMag opines:

Anyone who thinks “minimal GUI” mode is anything more than a holding measure is crazy. To me, this clearly says Microsoft is trying to get us off the console for good.

In most cases, in my opinion, the inconvenience will be quickly forgotten:

• The vast majority of app-level administration for any software that truly belongs in an ‘enterprise’ infrastructure follows the recommended model already, and at worst, needs only a modest change in practice.
• I expect Microsoft to expand their own range of remote-GUI tools to extend deeper into the OS/Hardware stack so that the range of admin functions that can only be performed directly on the server using PowerShell will be quite limited.  System Recovery and AV cleanup may be challenges but not by any means insurmountable.

However, healthcare offers a different category of challenge.  Not uniquely, but probably to a greater extent than many other industries.  That is one of the desktop app hosted on a server.

It happens less nowadays than in the past but is still (in my experience) a practice that derives from the way Health IT has evolved over the last 15 years or so in an environment where individual departments/specialties have more autonomy than equivalents in other industries.  I’m not debating the rights and the wrongs – just the reality.

I have seen applications coming through the door which have (to cut a long story short) questionable ‘enterprise’ credentials.  yes, there are MSDE databases (theoretically easily migratable to SQLServer but – in theory….), and yes Access databases, and HL7 interface engines written in VB or Delphi that run only on a desktop.  The requirements of clinical departments are many and varied and sometimes it is a necessary evil to accept such software.  Sometimes, the best that can be made of the situation is that they are hosted in a server environment so that some of the benefits can accrue – like redundant power supplies, controlled environment, or OS that doesn’t need to be rebooted once a week.

For those applications, either a new strategy is required or they need to be replaced.  There isn’t – necessarily – a panic.  Extended support for Server 2003 continues to 2015 and for Server 2008 until 2018, so there is time to consider carefully.  And another consequence of the HIT evolutionary process is that everyone accepts we’ll be living with multiple versions of Windows Server – as well as the LINUX, VMS, AIX, HP-UX and even (gasp) OSX Server installations routinely in use.

To keep Firefox, then, has meant hosting applications on a platform that no longer has ongoing support or security fixes.

Which meant that any hospital looking to move towards the undeniable future of zero-footprint web image viewers, would end up relying on Internet Explorer.

I did say that one way out of the quandary was for Mozilla to volte-face. I offered an opinion that that was unlikely. Thankfully, I was wrong.

As covered in various places, Mozilla has changed its mind and will not only be providing a Long-Term-Support version, but has resurrected the Enterprise User Working Group for collaborative input to make Firefox even more Enterprise-friendly.

Game back on.

I agree, but I do disagree with their definition of what ‘cloud’ actually means.

Indeed what some vendors are referring to as cloud PACS is simply a virtualization of locally hosted systems, further hindering market adoption of “true” cloud PACS. True cloud PACS is defined as having both application and storage provisioning and hosting by a third-party server offsite from the end user (i.e., public cloud).

Well – having app and storage hosted by a third party off-site is today – and always has been – call co-location (or co-lo for geeks).  That isn’t the same as ‘cloud’.  Wikipedia defines cloud as computing-as-a-service, which is, I think, closer, but I would go even further.  I would tend more towards computing-as-an-amorphous-and-opaque-service.

Lets take GMail as an example. I use GMail as my primary service provider and its a classic example of what most people would think of as ‘cloud’.  From which of Google’s many datacentres is GMail served?

Credit: Techcrunch

I’ll be honest.  I haven’t a clue.  It could be any.  In fact as a service, different elements could be served from different locations – mail from Europe, contacts from US and search from South America.  I don’t know and importantly I don’t care.

Another example is the Amazon Elastic Computing Cloud (EC2).  Within EC2, it is possible to rent servers in units of compute cycles, memory and storage, and point roughly to where they should be located – as in ‘somewhere in Europe’ or ‘somewhere in the US’ – but the Amazon infrastructure could move those around for a number of reasons.  You may not be aware they are being moved, but moved they are.

In Amazon’s case it is able to do this by virtual of virtualisation – the technology that the article in AuntMinnie specifically says is not ‘cloud’.  Virtualisation also offers a concept that is quite opposite to AM’s definition – that of the ‘private cloud’.

A private cloud is one which is dedicated to the use of a particular organisation. It can be externally hosted, an internal implementation or indeed a hybrid.  But the properties of amorphous and opaque still apply.  A private cloud architecture often implementation might be 2 or 3 datacentres in different locations – say 2 within the grounds of a hospital, and one off-site.  This can be organised in a way to maximise disaster tolerance, for example.  Individual servers/services can be moved between datacentres without the user being aware.   The user just knows the service/data is coming from ‘somewhere’ *.

So the distinction is not between ‘cloud’ and virtualisation – or even between public and private.  It is between local and remote.  Services hosted remotely suffer from two handicaps:

1. Trust.  Both from a security and privacy point of view, remote cloud services are still looked at with a slightly leery eye.
2. Infrastructure.  Remote cloud services will almost inevitably end up going through public infrastructure.  Even in the ‘developed’ world, the high specifications required for bandwidth, latency and reliability are not available everywhere.

Both of those are improving – but I think for data-intensive applications like PACS, ‘the cloud’ is still to come over the horizon.

Paring the idea of private cloud to the bone, we could end up with another of AM’s predictions – that PACS as a ‘Managed Service’ will grow in popularity over the next couple of years.  I personally think such a solution is a seriously retrograde step in PACS as an industry – although it may have benefits in a minority of (small) organisations.  Why?  Another of AM’s articles that landed in the mail this morning  points to ESR guidelines for the communication of urgent and unexpected findings.  It is interested reading in itself, but the bottom line is that while technology can provide part of the solution, it can’t do everything.  That is because the delivery of healthcare in any organisation of meaningful size is a complex thing that will always be at least as much to do with people as technology.

That is the reason integration between the many silos of information that have evolved in a typical healthcare organisation has progressed so slowly.  Any to keep the wheels moving smoothly, I would argue that what is needed is more knowledge within the organisation that know the systems, the data, and how they interact, and not less.  Managed Services in any but the simplest of organisations is the wrong direction to go.

But that’s my opinion.

Martin Peacock

* It should be noted that virtualisation isn’t the only way to achieve this – but virtualisation makes the job of ‘clouding’ an existing service much easier.  Services can be engineered such that they can be implemented as a cloud service – software engineering techniques such as MVC help. And infrastructure measures such as clustering and load balancing go a long way to achieving similar objectives.  Whatever mechanism and architecture is used – this aspiration should be high on the list for any any mission-critical system.

It has been patched for dcm4chee-2.17.1 but the fix is easy enough to apply to previous versions.  In the file server/default/deploy/jmx-console.war/WEB-INF/web.xml find the following block of code (probably towards the bottom of the file):

<security-constraint>
<web-resource-collection>
<web-resource-name>HtmlAdaptor</web-resource-name>
<description>An example security config that only allows users with the
role JBossAdmin to access the HTML JMX console web application
</description>
<url-pattern>/*</url-pattern>
<http-method>GET</http-method>
<http-method>POST</http-method>
</web-resource-collection>
<auth-constraint>
<role-name>JBossAdmin</role-name>
</auth-constraint>
</security-constraint>

.. and remove the lines:

<http-method>GET</http-method>
<http-method>POST</http-method>

This ensures that all verbs are routed through the security checks by default.

The release of Firefox 5 has, in the eyes of Mozilla, signalled the end of life (EOL) for Firefox version 4 – which means no more security updates.  It is expecting users that continue to use Firefox to upgrade to version 5 if they require security updates.  The new-style aggressive release cycle that Mozilla have put in place (not by any means a bad thing in itself) means security conscious users must go through a major-release upgrade every 3 months.

But- for enterprise users – that is nigh on impossible.  Before any major release upgrade all (or at the very least all critical) browser-based applications must be tested on the new platform.  It is inevitable that some will suffer some level of failure which must be relayed to vendors to implement corrective action.  That process requires time, resources, and on occasion, additional funding for upgrade licenses.  That this process (particularly the vendor-contributed part) is difficult is precisely why IE 6 still lurks balefully.  Even if this occurs – in the meantime there are still no security patches for the version actually installed.

To insist that this process be followed every three months is insane.

For vendors in healthcare and other regulated industries the vista is even worse.  I am currently building a product which – while currently can just about squeeze past certification as a medical device – will almost certainly later this year be classified as a medical device (at least in EU).  I do not have a problem with that – I firmly believe all software in any clinical context should be regulated (although I don’t think the medical device classification scheme does software QA processes little justice).  But I cannot rationally promote the product as suitable for use on a platform which either changes every 3 months or languishes in a security limbo.

Does it matter?  Do the browsers not all support HTML 5 with the Canvas tag?  Yes, they do. But Safari and Opera are still niche browsers so expecting those to fill in the gap is effectively ditching the ‘zero-footprint’ aspiration, and AFAIK Safari doesn’t support WegGL on Windows.  Microsoft has quite roundly criticised WebGL as a browser-based 3D rendering technology.  The Microsoft arguments have been roundly – and rightly – rebutted by Mozilla but it seems likely that IE will be, at best, a late adopter of WebGL.  Since 3D presentation is squarely in my product roadmap, I must reconsider building on WebGL and instead look at Silverlight.

So – reminiscent of the bad days 8-10 years ago, but realistically – I’m building a product for IE.  Unless something changes.

There are a number of things that could make this situation better:

• Mozilla could volte-face on this issue and continue to provide security updates.  From the language used in various communications that seems unlikely.
• Given the language, it is perhaps more possible that Mozilla mirror the Ubuntu-style release pattern of Long Term Support (LTS) releases every 4 or 8 non-LTS releases. Those speaking publicly for Mozilla so far have distanced themselves from this however.
• Of course, Firefox is Open Source Software, which, in principle, anyone can support.  It is quite conceivable that a commercial entity appears to back-port security fixes to the equivalent of an LTS release.  That entity could easily be a commercial wing of Mozilla itself.
• Microsoft could change their stance on WebGL.  That wouldn’t fix the Mozilla release cycle issue but would be a small help.

For the present, however, it would appear ‘zero-footprint’ ≡ IE

Increasingly, Hospitals are looking to augment their existing PACS installation with an archive sourced independently of their current vendor.  There are a number of reasons why this may be:

• For some installations, much historic data is held offsite with a per-study charge for retrieval.  A seperate, cost-effective archive on commodity hardware can allow for historical data to be made available without incurring such charges.
• The capital cost of expanding on-site storage can with many vendors be at an unreasonable premium.
• Migrating to a new vendor can be a fraught process unless all the image data is held on an open, standards-based archive.

One of – if not the – most popular option for acheiving just this is the open source archive software DCM4CHEE.  It is unquestionably one of the most robust and feature-full archives (either proprietary or open source) in th emarket.

A barrier, however, to implementation has been a lack of formal support and services options.  This is why we are offering the Fidelity range of support packages for DCM4CHEE – for your peace of mind and to facilitate your efforts to take back control over your data.

Click here for more information.

WebGL is a mechanism to extend JavaScript to allow HTML5-compliant browsers to render 3D graphics with hardware acceleration.

While other parts of the HTML5 specification is important for the future of browser-based imaging applications, 3D features such a virtualoscopies have become increasingly important with the explosion of data volumes generated by recent generation modalities such as CT.

WebGL is an important stepping stone therefore to a fully featured brpwser based Radiography viewing application.

However, as is noted here, there is a small paragraph slipped into the end of the press release that offers even greater hope for the future:

WebCL creates the potential to harness GPU and multi-core CPU parallel processing from a Web browser, enabling significant acceleration of applications such as image and video processing….

Now that is only in an exploratory phase at the moment but if that can be driven to widespread browser adoption we can truly say bye bye to ‘fat’ clients.

DICOM Troubleshooting is a 1 day on-site course to understand how to diagnose DICOM communications issues – particularly between vendors.  Click through to read more.