Orbica: Using machine vision in GIS

Last week I had the opportunity to sit down with Orbica CEO’s Kurt Janssen and data scientist  Sagar Soni.

Kurt has worked in the Geographic Information Systems (GIS) industry for more than 14-years. Last year he started his own company, Orbica, which does GIS consulting for organisations in the public and private sector. Orbica invests some of its consulting revenue into developing its own product. A major – and rewarding – investment has been hiring data scientist Sagar.

Sagar was taught machine learning during his master’s degree and had the opportunity to put it into practice developing an earth rock image classification system at Dharmsinh Desai University and using deep learning algorithms like Recurrent Neural Networks to solve medical entity detection problems at US health care solutions provider ezDI. Last year he immigrated to NZ and had just the skills and experience Orbica was looking for.

Orbica’s first product automatically identifyies buildings and waterways from aerial photos. This manually intensive job is traditionally done by geographers and cartographers who draw polygons on maps identifying these features using digitising techniques. The first product identifies buildings in urban areas. The 15 million-pixel (4800×3200 ) photos have each pixel covering a 7.5×7.5cm square . Sagar has built a convolution neural network that takes these photos and outputs the vectors representing the polygons where it believes the buildings are.

They have a good amount of training, test and validation data from Land Information New Zealand that consists of the images and polygons that have been hand drawn. Because of the size of the image, Sagar has tiled them into 512×512 images . He built the model over a couple of months with a little trial and error testing the various hyper parameters. The existing model has nine layers, with the standard 3×3 convolutions. He’s currently getting 90 per cent accuracy on the validation set.

Building outlines

RiverDetection_AIThe water classification is very similar, working with 96 million pixel(12000×8000)  images, but with smaller resolution 30x30cm  pixels. The output is the set of polygons representing the water in the aerial images, but the model also classifies the type of water body, e.g. a lake, lagoon, river, canal, etc.

The commercial benefits of these models are self-evident: Orbica can significantly improve the efficiency of producing this data, whether it does this for a client, or it is sold as a service to city and regional councils. These are done regularly – to identify buildings that have been added or removed, or to track how waterways have changed.

WaterBodiesClassification'

Another opportunity has come from the Beyond Conventions pitch competition in Essen, Germany, where Orbica won the Thyssenkrupp Drone Analytics Challenge and the People’s Choice Award. Orbica’s pitch was to use machine vision to analyse drone footage of construction sites to automatically generate a progress update on the construction project. This is a more complex problem given its 3-dimensional nature. Thyssenkrupp has now resourced Orbica to put together a proof of concept, which Sagar is busy working on. Should this go well, Orbica will probably hire at least one other data scientist. DroneImage_Output

Because the technology is developing quickly, Sagar keeps up to date with the latest developments in deep learning through Coursera and Udacity courses. He’s a fan of anything Andrew Ng produces.

To me, Orbica’s use of machine vision technology is an excellent case study for how New Zealand companies can use the latest advances in artificial intelligence. They have a deep knowledge in their own vertical; in this case GIS. They develop an awareness of what AI technologies are capable of in general and have a vision for how those technologies could be used in their own industry.  Finally, they make an investment to develop that vision. In Orbica’s case, the investment was reasonably modest: hiring Sagar. A recurring theme I’m seeing here is hiring skilled immigrants. New Zealand’s image as a desirable place to live – coupled with interesting work – will hopefully make this a win-win for all involved.

For those that would like to hear more. Kurt is speaking at AI Day in Auckland next week.

 

 

 

 

Jade: developing AI capability for chatbots and predictive modelling

Jade logo

A couple of weeks ago I sat down with Eduard Liebenberger who is the head of digital at Jade to find out a little about their AI capabilities and plans. Eduard is passionate about AI and the possibilities it brings to transform the way we communicate with businesses.

In Eduard’s words, Jade’s core focus is around freeing people from mundane/repetitive tasks and instead allow them to apply their creativity/expertise to more challenging tasks – and the JADE development, database and integration technologies. Eduard and the team at Jade have been watching recent developments in AI and identifying which of these they can use to help their customers. Their first foray has been into conversation interfaces (chatbots). They’ve developed a number of showcases, including an insurance chatbot called TOBi which shows how the technology can be used to make a claim, change contact details etc. From their they have started rolling out this technology into existing customers.

The chatbot uses natural language processing and sentiment analysis. It aims to make businesses interactions with their customers more efficient by allowing them to communicate via conversations that don’t have to be in real time, like a phone call and are more intuitive than a web form. Jade’s main advantage with their existing customers is that they have already done the tricky integration work with the back-end systems and so can fairly quickly add a chatbot as an alternative to an existing interface. Jade’s focus on the digital experience means they invest heavily into making this a natural and human-like interaction. For non-Jade customers their attraction is their ability to deliver a whole solution and not just the chatbot.

A77B63FA-FD17-4EF9-9643-0F01F13BF576

Another advantage Jade has is that through their existing customers they have access to a lot of data that can be used to power machine learning applications. One example Eduard talked about was a summer intern project with a NZ university to try and identify students at risk of dropping out.  This was done using the data in student record database which is powered by Jade and contains several years’ of records. In just a few weeks the interns built a predictive model that was able to predict which students were likely to drop out with 90%+ accuracy. Ed is a big fan of rapid development for these types of proof of concept projects and doesn’t believe it should cost a fortune to get value from AI.

Overall, I think it’s fair to say that Jade’s AI capability is nascent. However, it’s positive to see that they are looking to build capability, understandably with a focus on the business benefits to their customers. I’m keen to see how it develops.

For those that want to find out more, Eduard is delivering the keynote at Digital Disruption X 2018 in Sydney, and presenting at DX 2018 and the AI Day in Auckland, all later this month. He’s a busy man.

Imagr: NZ’s version of Amazon Go

Last Friday during a visit to Auckland I took the opportunity to catch up with Will Chomley, CEO of Imagr. Will is an ambitious entrepreneur, with a finance background. He has surrounded himself with a group of engineers skilled in deep learning. This start-up is focused on using machine vision to identify products as they are added to a shopping cart: NZ’s version of Amazon Go. But this is aimed at cart rather than the whole store. Their product is called SMARTCART and integrates with your phone. 

When I met Will they had just announced their first trial with Foodstuffs New Zealand in their Four Square Ellerslie store . They were busy building their training dataset which at the time of writing contains over 2 million product photos for their machine vision models.

The 12-person company has a handful of machine vision engineers. A portion of these are immigrants because the skills are hard to find in New Zealand. Will is very enthusiastic about running the company from New Zealand because it is such a desirable place to live and it’s easy and quick to move here for people with the right qualifications.

The capabilities of their machine vision look impressive. They’re able to identify very similar looking packages with only subtle differences. I saw an example of two ELF mascara products that on first inspection looked identical, but on closer inspection one had a small brush.  They’re able to identify, with high accuracy, occluded products – partially covered by a hand, or high speed objects being thrown into a basket that I couldn’t recognize, but the blurred images are able to be recognised by their 200+ layer convolutional neural network.

They have designed the cart so the inference engine, which is making the decision about what is going into the basket, can either run on a computer on the basket, or the information can be sent to a server. To get the speed from this model they have developed their own software using C, rather than relying on the easier to use but slower  frameworks such as TensorFlow. This gives the capability to identify and engineer around bottlenecks.

In parallel they’re working on the hardware, having recently decided to use small, cheaper, lower resolution cameras with some added lighting. These can provide the same high accuracy rate as higher resolution, expensive cameras. They have their challenges. Designing and building hardware that can be manufactured at scale and operate reliably and easily is no mean feat. However, they have some engineering chops: their CTO Jimmy Young was Director of Engineering for PowerByProxi, the wireless charging company bought by Apple last year.

They have some backers with deep pockets to help resource these challenges.  Last year they received an undisclosed investment from Sage Technologies Ltd, the technology venture of private investment company QuantRes founder, billionaire Harald McPike.

There’s a large opportunity in front of them and they’re moving quickly. Adding smart carts has to be a lot cheaper than fitting out a store Amazon Go style. They may be able to get a piece of the global grocery market, grabbing a cut of the value of the cart, saving costs for the grocer, improving the experience for the shopper and opening up a world of possibility for the company once they are collecting the shopping data.

One of their challenges is to stay focused. There are so many applications of machine vision technology, even if they stick to retail. They’ve experimented with smart fridges that can identify the gender and age of people taking products, as well as knowing which products they take. They’re talking to others in the industry about applications for their technology.

If they can keep their focus and execute they have a bright future ahead of them.  Their product has global appeal. Retailers can see the threat of technology giants such as Amazon and Alibaba who are moving quickly into bricks and mortar retail. This accelerated last year with Amazon’s purchase of Whole Foods. The convenience of online shopping is coming to a store near you and Imagr may be the one’s powering it.

Cacophony: Using deep learning to identify pests

This is the first of a series of posts I intend to write on organisations that are using artificial intelligence in New Zealand. I am closer to this organisation than most because it was started by my brother, Grant.

Cacophony is a non-profit organisation started by Grant when he observed that the volume of bird song increased when he did some trapping around his section in Akaroa. His original idea was simply to build a device to measure the volume of bird song in order to measure the impact of trapping. Upon examining trapping technology, he came to the conclusion there was an opportunity to significantly improve the effectiveness of trapping by applying modern technology. So he set up Cacophony to develop this technology and make it available via open source. This happened a little before the establishment of New Zealand government’s goal to be predator free by 2050. He managed to get some funding and has a small team of engineers working to create what I refer to as an autonomous killing machine. What could possibly go wrong?

Because most of the predators are nocturnal the team have chosen to use thermal cameras. At the time of writing they have about 12 cameras set up in various locations that record when motion is detected. Grant has been reviewing the video and tagging the predators he can identify. This has created the data set that has been used to by the model to automatically detect predators.

They hired an intern, Matthew Aitchison, to build a classifier over the summer and he’s made great progress. I’ve spent a bit of time with Matthew, discussing what he is doing. Matthew completed Standford’s CS231n computer vision course that I’m also working my way through.

He does a reasonable amount of pre-processing: removing the background, splitting the video into 3 second segments and detecting the movement of the pixels, so the model can use this information. One of his initial models was a 3 layer convolution neural network with long short-term memory.  This is still a work in progress and I expect Matthew will shortly be writing a description of his final model, along with releasing his code and data.

However, after just a few weeks he had made great progress. You can see an example of the model correctly classifying predators below, with the thermal image on the left and on the right the image with the background removed, a bounding box around the animal and instantaneous classification at the bottom with the cumulative classification at the top.

 

A version of this model is now being used to help Grant with his tagging function, making his job easier and providing more data, faster.

The next thing is to work out how to kill these predators. They’re developing a tracking system that you can see a prototype working below.

From my perspective it feels like they are making fantastic progress and it won’t be too long before they can have a prototype that can start killing predators. If you ask Grant he thinks we can be predator free well before the government’s goal of 2050.

One final point on this from a New Zealand AI point of view, is how accessible these technologies are that are driving the Artificial Intelligence renaissance. Technologies such as deep learning can be learnt from free and low-cost courses such as the CS231n. Those doing so, not only have a lot of fun, but open up a world of opportunity.