Yesterday the Khronos Group – a consortium of media-centric companies including Dell, Google, HP, IBM, AMD, Apple, Intel, Nokia, nVidia, and many more, released the 1.0 version of the WebGL standard.
While other parts of the HTML5 specification is important for the future of browser-based imaging applications, 3D features such a virtualoscopies have become increasingly important with the explosion of data volumes generated by recent generation modalities such as CT.
WebGL is an important stepping stone therefore to a fully featured brpwser based Radiography viewing application.
However, as is noted here, there is a small paragraph slipped into the end of the press release that offers even greater hope for the future:
WebCL creates the potential to harness GPU and multi-core CPU parallel processing from a Web browser, enabling significant acceleration of applications such as image and video processing….
Now that is only in an exploratory phase at the moment but if that can be driven to widespread browser adoption we can truly say bye bye to ‘fat’ clients.
We’ve noted before there are continued advances in 3D visualisation. While my own prediction for 2011 (as early as it is yet) is that we will see browser-based 3D modelling based on HTML5 and WebGL, there are other threads of development also. The video below (thanks to New Scientist) seems to be the same as this Sony device we noted in October last year , but does put more of a medical context to the technology.
I have a small role in the setting up of the Irish national RIS/PACS system. I hope I provide more solutions than problems but I’m sure that could be debated. One of the questions that has appeared is of a category that appears in the implementation of pretty much any clinical system: Can access to (some entity) be restricted by (some data element/property). An example of this might be:
Can access be limited to the originating institution for 24 hours after data generation?
The stock developer’s answer, is of course:
Yes. But. You have to define that element/property in advance and we can schedule it into a future release, based on available development time.
.. which isn’t terribly helpful.
Implementing a user-accessible scripting language into the system is one way of providing such a feature dynamically. But its a tough job to retro-fit a scripting language into a system if the system was not originally designed with that in mind. There may be another way – using data transformation.
XSLT is an XML-based language which provides transformation services for XML-based datasets. So a dataset, when expressed in XML, can be transformed into some other representation – say HTML. Importantly, the transformed dataset may contain some subset of the original data, but need not. It could equally be a dataset that is driven by the contents of the original dataset, but actually contain none of it.
The DCM4CHEE archive uses this idea, for example in defining forwarding rules based on the contents of DICOM fields in incoming images. It applies an XSLT template such that, based on some set of criteria within the DICOM headers, the output is (in XML form), a parameterised list of destinations to which that image should be forwarded.
So how can this be used as a User Access Protocol? Easy. For each data item which is to be subject to User Access Control, package up its own element values, and element values from the wider context which are relevant, as an XML stream. Run that through an XSLT parser with an output that defines what access the current user has.
So, for example, the dataset for an ‘order’ object may include, from its own data fields as well as wider context, the following data:
<?xml version=”1.0″ encoding=”UTF-8″?>
<?xml-stylesheet type=”text/xsl” href=”Order-Access.xsl”?>
<attr tag=”procedure”>CT Head</attr>
<attr tag=”orderplacer”>Jane Doe</attr>
<attr tag=”orderinstitution”>St Elsewhere</attr>
<attr tag=”userinstitution”>not St Elsewhere</attr>
.. that can be processed via XSLT:
<?xml version=”1.0″ encoding=”UTF-8″?>
<xsl:stylesheet xmlns:xsl=”http://www.w3.org/1999/XSL/Transform” version=”1.0″>
<xsl:output method=”xml” indent=”no”/>
<xsl:variable name=”orderinstitution” select=”attr[@tag='orderinstitution']“/>
<xsl:variable name=”userinstitution” select=”attr[@tag='userinstitution']“/>
<xsl:variable name=”hours-since-order” select=”attr[@tag='hours-since-order']“/>
<xsl:when test=”$hours-since-order > 24″>
…to form output XML, based on :
<?xml version=”1.0″ encoding=”UTF-8″?>
So in this case, we can dynamically define quite complex access rules to restrict access to an order to the originating institution only, for the first 24 hours of its lifespan. Cool.
Note there is a bit of a kludge in the XML-packaged dataset. I’ve put in a derived field (hours-since-order) which in itself requires some pre-casting of requirements. The main reason is that, since I’ve used DCM4CHEE as an example, DCM4CHEE straight out of the box, is limited (by virtue of JBOSS, itself by virtue of Apache XALAN) to XSLT version 1. The really useful date-processing functions are, alas, defined in XSLT version 2. So the next logical stage I guess is to work out how to upgrade!
A couple of items have appeared in the last couple of days which may be of interest to those developing for browser platforms:
- Browser cache behaviour is one the the things that annoyed me most about my now-RIP iPhone. Ryan Grove has quantified the behaviours of the major mobile platforms that really demonstrates why iBrowsing sucks and I’m so much happier with ‘droid. Seriously – any web app or even mobile-oriented site should take cognizance of these numbers to optimize UX.
- We’ve seen IE9 is due to support hardware accelerated CANVAS and thats a great step forward. The next step is WebGL. What is WebGL? Heres a new FAQ. With WebGL, support for 3D visualisation is native to the browser. More later today on 3D.
As reported by El Reg and Ajaxian, despite some confusion in the past whether IE 9 will support the CANVAS tag, it seems it not only will, but will be hardware accelerated where appropriate. This is big news for MI applications.
The prospect of fully features browser-based PACS clients without resorting to plug-ins is fast approaching.
Ajaxian starts what is undoubtedly only the beginning of optimisations around HTML5 – notably the CANVAS element which is of particular interest to anyone in imaging circles. There is a sweet window/levelling proof-of-concept within the Oviyam (DCM4CHEE) project which not only does window/levelling but handles FULL 16 BIT IMAGES. Sorry for shouting.
Despite committing to HTML in principle, however, the Microsoft position on CANVAS is still unclear. Google fills in the gaps with ExCanvas (which it appears has performance issues for intensive use). Mozilla has an ActiveX plugin solution to retro-fit CANVAS, but I’m not sure its releas-quality at this point and it would clearly be better if IE supported it natively.
On my ever-increasing “gotta get finished” list, is a benchmark for performing window/levelling of DICOM images in a browser using the HTML 5 CANVAS element. I might drop it off altogether now there is a proof-of-concept up on youtube. One of the reasons I’ve dropped it down the priority list is that while it does do the job, it requires first a drop down to 8 bit jpeg from the original (generally 12 or even 16-bit). So really, its only half a job. The bottom line is, to perform this job properly, one of three things is required:
- Browser standards start supporting multi-byte grayscale jpegs. Not very likely. HTML5 has struggled to gain universal acceptance (and even now, not really in its entirety). Another step back to the table won’t be for a long time.
- Plugins. Over the last few years we’ve seen a significant increase in the variety of browsers out there and that is only likely to continue (even if Gecko/Webkit are at the core of most). Maintaining plug-ins for all would be a nightmare and certainly, a retrograde step.
- Trips back to the server. Personally this is, in my opinion, the most likely. It is necessary to minimise the size of the image being processed and transmitted – perhaps using heavy compression/resizing during the window/levelling operation.
For the short- to medium-term, I am leaning towards the concept of trips back to the server as the best option:
- Once a user has indicated (mousedown) that W/L is required, a very low resolution image is generated (server-side) from the full image. For a 2kx2k CR, for example, a image 256×256 is generated and displayed in the browser – either as a thumbnail/preview or overlaid on the original, full-res image. It would clearly be advantageous to have the low-res image cached in memory in some form.
- Each W/L action (mousemove) is performed on the low-res image and displayed in real-time.
- Once the user is happy with the settings (mouseup), the W/L is performed on the full-resolution image, and piped down to the browser.
An example of this (albeit not involving server-trips) is the volume rendering engine included in Osirix (ITK). When the view of the volume is rotated – it drops down to a low resolution under the view perspective is final, when it returns to full resolution. Clearly not as good as fully hardware-assisted rendering, but quite usable all the same.
So, while the decision to write up in PHP was made some time ago (and hasn’t changed), the actual development of a browser-based PACS viewer has (hurray!) made some progress. Focus for now is on the simple stuff – CR first, then a means for viewing CT/MR. But on the roadmap (that’s in the future, BTW) is MPR and that poses a challenge.
MPR (AFAIK) necessarily requires two stages:
- Build the 3D model from the original data.
- For each coordinate change, generate a projection against the 3D model.
Now while PHP can happily manage most of the requirements, in this case it suffers a limitation that doesn’t apply to platforms like J2EE. Each interaction is a discrete, non-persistent entity. At the simplest level, that means that every time a change to the XYZ coordinate is made, the 3D model has to be rebuilt. Well that’s not going to work.
So, (and this is largely for my own notes), there are ways to address the issue:
- When a client initiates MPR, exec off a seperate PHP process to build the 3D model, and remain persistent (in a big loopy thing). Subsequent requests for changes in XYZ then communicate with this detached process (say, ActiveMQ) to retrieve projections. That would work, but without the infrastructure provided by something like J2EE, it’s little better than a hack.
- Have the initiation build the 3D model, but dump a representation of that onto a RAM-based storage area for subsequent XYZ requests.
- Taking that even further, maintaining the model in a database table that itself is maintained in RAM. This raises the potential for some smart SQL/DB optimisations, which in itself, makes it my (currently) preferred option. Hell that may change.
Addendum: One of the principles I’m working on is database neutrality so the ‘smart DB optimisations’ may not work out although I’ll cross that bridge when I come to it.
Latest from the Blog:
- High Availability
- Lessons learned
- Open Source
- PACS General
- Project matters
- Martin Peacock on Virtual Servers and PACS
- Edward Mangiola on Virtual Servers and PACS
- Globalstorage on Business Continuity Planning in Health IT
- Martin P on Window/Levelling in a browser – CANVAS or server-trips?
- Martin P on 3D can be a dangerous game