{"id":5263,"date":"2015-05-27T08:16:41","date_gmt":"2015-05-27T00:16:41","guid":{"rendered":"https:\/\/www.curtin.edu.au\/news\/big-data-big-ideas-at-the-helm\/"},"modified":"2022-12-07T13:07:17","modified_gmt":"2022-12-07T05:07:17","slug":"big-data-big-ideas-at-the-helm","status":"publish","type":"post","link":"https:\/\/www.curtin.edu.au\/news\/big-data-big-ideas-at-the-helm\/","title":{"rendered":"Big data: big ideas at the helm"},"content":{"rendered":"

Big data: it\u2019s the glue of the digital universe. If you\u2019re reading this, chances are that you are one of the two billion people in the world who live and work online and, according to the International Data Corporation, generate an annual \u201cdigital shadow\u201d of around 1.8 million gigabytes each \u2013 whether it\u2019s the data from streaming digital TV or just your smartphone pinging your location to Google.<\/p>\n

Related: I Like Big Data and I Cannot Lie<\/a><\/p>\n

For Curtin researchers Dr Andrew Hutchison, Dr Andrew Woods, Dr Petra Helmholz and Joshua Hollick (who is also undertaking a PhD), their digital shadows are going to be larger than most this year. Unlike most of us, their big data isn\u2019t just restricted to the mass data generated on the Internet and from smart devices. In cooperation with the Western Australian Museum, they are looking to create their own sets of big data and \u2013 with the aid of the Curtin HIVE\u2019s 3D technology \u2013 generate virtual reconstructions of the shipwrecks HMAS Sydney (II)<\/em> and HSK Kormoran<\/em>.<\/p>\n

Related: Researchers abuzz over HIVE and immersive visualisation<\/a>.<\/p>\n

Sixty-seven years lost at sea<\/h2>\n

In 1941 Australian HMAS Sydney (II) <\/em>sunk with all hands in an encounter with German HSK Kormoran<\/em> approximately 200 km off the Western Australian coast. Both ships sank in the encounter and until 2008 their final resting places were unknown.<\/p>\n

\"\"<\/a>
The HMAS Sydney (II)<\/em> in 1940.<\/figcaption><\/figure>\n

\u201cWe\u2019ve just revisited the wreck sites of the HMAS Sydney (II)<\/em> and HSK Kormoran<\/em> and taken many many<\/em> images to create impressive 3D models for both ships,\u201d Hollick says.<\/p>\n

\u201cWe\u2019re planning to use the 3D models to create a museum exhibit so people can experience what the wreck sites are like, because being 2.5km underwater they are something that almost no one can get to.\u201d<\/p>\n

Related: Two Lost Ships<\/a><\/p>\n

\"\"<\/a>
A DOF Subsea ROV inspecting the engine room of HSK Kormoran<\/em> – Sydney Kormoran Expedition 2015. Image courtesy of WA Museum and Curtin University. Copyright WA Museum.<\/figcaption><\/figure>\n

Beating the big data problem<\/h2>\n

In their four-day expedition to the shipwrecks, the team has accumulated approximately 50 terabytes of data from two underwater remotely operated vehicles (ROVs), each equipped with seven digital still cameras and two 3DHD video cameras (in addition to the other cameras already installed on the robots). Fresh off the back of the expedition Hollick estimates he has almost 700,000 images to process and analyse \u2013 along with 3D multibeam sonar surveys of each wreck site.<\/p>\n

\u201cAnd that\u2019s where we start falling into the realm of big data,\u201d says Hollick.<\/p>\n

\"HMAS<\/a>
A multibeam sonar survey of the main hull of HMAS Sydney (II). Image courtesy of WA Museum and Curtin University. Copyright WA Museum.<\/figcaption><\/figure>\n

Watch: ROV footage of the HMAS Sydney (II), 1 May 2015<\/a><\/p>\n

As powerful as the HIVE is, with almost 700,000 images to process to generate his 3D models, if they only used their existing processing techniques Hollick expects the rendering time would be about 1.8 millennia<\/em>.<\/p>\n

It\u2019s a common problem for big data. The sheer volume of it is too much to process. Fortunately Hollick has a few ideas of how to get around this.<\/p>\n

First is to use a supercomputer, specifically the Pawsey Supercomputer, located here in Perth. However, cutting the rendering time of the images depends on how many of the supercomputer\u2019s \u201cnodes\u201d can be allocated to process the images \u2013 and even with enough nodes it\u2019s only slightly quicker.<\/p>\n

Enter metadata: librarian of the digital universe.<\/p>\n

\u201cEssentially, we are trying to be smarter with what data we collect and process,\u201d Hollick says. The usual process of sorting the images for a 3D model is a bit like putting a jigsaw puzzle together, each image needs to be compared and matched against every other image until it finds where it fits \u2013 which is fine until your puzzle has 700,000 pieces in it.<\/p>\n

Using the sonar system of the underwater ROVs, Hollick is able to geo-tag where the images were taken with an accuracy of around ten metres. When it\u2019s time to link all the images together for the 3D reconstruction, the computer will compare images taken at similar locations, rather than images from opposite ends of the wreck. So rather than sorting through 700,000 puzzle pieces at random, the computer now knows that say these<\/em> 10,000 images are all pieces from the same corner.<\/p>\n

But even knowing where those 10,000 images were taken, it is still a puzzle \u2013 just slightly smaller. Why not take it one step further?<\/p>\n

In any sort of bulk photography or filming, images close together in time will also be close together in position. And by collecting timestamp data from the images of the HMAS Sydney (II)<\/em> and HSK Kormoran<\/em>, Hollick can teach the computer to recognise this.<\/p>\n

\u201cThe idea is that when you are comparing images, each image will match with the next image in the sequence or series,\u201d Hollick says.<\/p>\n

It\u2019s not unlike an old film reel. Each frame is an image and when strung together they form a motion picture. Instead of the computer being dominated by having to compare this<\/em> image here to that<\/em> image all the way over there \u2013 at the other end of the reel \u2013 the timestamp instructs the computer to only compare images that are close to one another in the sequence.<\/p>\n

Last of all, a new algorithm is needed so the computer can combine all these different forms of data (and metadata) in an intelligent way. This is the focus of Hollick\u2019s PhD. Essentially Hollick\u2019s algorithm links the images and data (multibeam sonar, geo-tags, and timestamps) together to guide the 3D\u00a0reconstruction processes.<\/p>\n

But the algorithm does more than simply speed up the 3D reconstruction side of things. It also creates a more accurate 3D model, and that\u2019s because errors can affect the data. Moreover, each type of data can present errors unique to its data-type. But, with the algorithm and its ability to pull information from multiple datasets and compare it, Hollick can reduce, or even remove, these errors entirely.<\/p>\n

\u201cThe algorithm will help us find any inconsistencies between the metadata, and the images\u00a0and trajectory that each ROV recorded when they navigated through the shipwrecks,\u201d Hollick says.<\/p>\n

And why is that so important? Well, without the algorithm to iron out the kinks, you might wind up with a wonky ship.<\/p>\n

\u201cWith it [the algorithm] we should also be able to correct distortions in the model by combining the different datasets and possibly fill in missing geometry using the multibeam data,\u201d Hollick explains.<\/p>\n

In a nutshell, where one piece of data fails, others can take its place.<\/p>\n

So what will you get when you combine big data, metadata to catalogue it, an algorithm to collate it, and the raw processing power of a supercomputer?<\/p>\n

Put on a pair of 3D glasses in the HIVE and, by Jove, we\u2019ve got ourselves a ship.<\/p>\n