AI day 2018: My take

I noticed the AI day videos were released a few days ago and I’d like to share my thoughts on the day. First I”d like to congratulate the organisers Ben Reid and Justin Flitter for putting this event together. Michelle Dickinson did a great job making the day flow as master of ceremonies. This type of event is just what NZ needs to help people understand how different organisations are using AI so they can make more informed decisions on how they could use the ever evolving set of technologies.

AI day 2018 videos
The AI day 2018 videos

I’d characterise the event as having presentations from small and large organisations, a couple of panels, a politician and a good dose of networking. The highlight for me was from the small companies because they were the ones who had taken various AI technologies and applied them in a way to give them an advantage. In my mind these are the stories that are most likely to inspire other NZ companies. This included:

  • R& D coordinator for Ohmio, Mahmood Hikmet describing the self-driving shuttle that they are building and how their AI team is building a sensor fusion model. This combines data from GPS, lidar and odometry sensors to estimate the position of the shuttle, that is then used for navigating.
  • Kurt Janssen, the founder of Orbica described how they’re using machine vision with aerial and drone footage to automate various GIS tasks.
  • Grant Ryan (or Bro as I call him) describing how Cacophony are using machine vision with thermal cameras to automatically identify pests, and how they might then kill them.
  • Sean Lyons had the most entertaining presentation where he described how Netsafe are using bots to waste scammers time in a project they call Rescam. They’re using IBM Watson for sentiment analysis. It’s been hugely successful, wasting over 5 years of scammers time with 1 million emails.
    netsafe bot
  • Mark Sagar and team are doing some of the most interesting AI work globally at Soul Machines. Unfortunately, his presentation had a few technical glitches, but it was nice to see the latest version of BabyX, complete with arms. Mark talked a little bit about how they are using neural networks for perception and control. I’d love to find out more details.
    Babyx

The other small company that presented was Centrality.ai. Founder Aaron McDonald spent most of the presentation explaining blockchain and how it can be used for contracts. I didn’t come away with any understanding that the company is using AI, or with any comprehension of what the company actually does.

The panels had a selection of interesting entrepreneurs and academics. However, I personally find the panel format a little too unstructured to get much useful information from. I may be an outlier here, Justin told me they got very good feedback about the panels from their post conference surveys.

The other highlight of the conference for me was the networking during the breaks. Everyone you spoke to had some involvement in AI: Entrepreneurs, practitioners, academics and investors. This was an added benefit to an already very stimulating day. I wasn’t able to attend the 2nd day of workshops.

To Justin and Ben: Well done! I look forward to attending next year and hearing how a host of other NZ companies are using AI in interesting ways. For those that didn’t make it, check out the videos.

 

Cognitive modelling, self aware robots Tensorflow, & adversarial attacks

 

This week I’ve been learning about cognitive modelling, self aware robots,  adversarial attacks in reinforcement learning and starting to play with Tensorflow.

Cognitive Modelling

The latest MIT AGI video was released a few days ago. In this Nate Derbinsky gives an overview of different types of cognitive architectures, including SPAUN, ACT-R, Sigma and Soar (his baby). This reminds me of old school AI: symbolic processing. My supervisor’s PURR-PUSS would belong in this category. These are a lot less sexy than deep learning, but in many way’s they are complementary with applications in robotics, game playing & natural language processing.

 

TWiML podcasts

SUM cognitive architectureThis week I listened to an interesting podcast with Raja Chatila on robot perception and discovery. In this Raja talked about the necessity of robot self awareness for true intelligence and the ethics of intelligent autonomous systems. It’s interesting to see that the sort of architectures used for exploring artificial consciousness in robotics have a lot of overlap with the cognitive models described by Nate Derbinsky in the MIT AGI series.

I also had the chance to listen to Google Brainers Ian Goodfellow & Sandy Huang discussing adversarial attacks used against reinforcement learning  Adversarial attacks highlight some of the weakness of deep learning. When used for image classification the image is just a series of numbers that has a set of mathematical operations performed on it to produce a classification. By subtly changing some of the numbers you can fool the classifier, even though to a human the image looks exactly the same. The example of a Panda below was taken from a 2015 Google paper.

panda

In the podcast Ian and Sandy discuss how this can be used against a reinforcement learning agent that has been trained to play computer games. Even changing one pixel can significantly degrade the performance.

Tensorflow

I’m up to the part in my CS231n course where you start to train CNNs using Tensorflow or Pytorch. Despite reading a compelling argument for using Pytorch over Tensorflow on Quora, the people I’ve spoken to locally are using Tensorflow – so I’m going to go with that. I found this introduction useful.

I managed to get the software installed and run Hello World. Apparently there is more you can do with it…

Tensorflow hello world

 

Orbica: Using machine vision in GIS

Last week I had the opportunity to sit down with Orbica CEO’s Kurt Janssen and data scientist  Sagar Soni.

Kurt has worked in the Geographic Information Systems (GIS) industry for more than 14-years. Last year he started his own company, Orbica, which does GIS consulting for organisations in the public and private sector. Orbica invests some of its consulting revenue into developing its own product. A major – and rewarding – investment has been hiring data scientist Sagar.

Sagar was taught machine learning during his master’s degree and had the opportunity to put it into practice developing an earth rock image classification system at Dharmsinh Desai University and using deep learning algorithms like Recurrent Neural Networks to solve medical entity detection problems at US health care solutions provider ezDI. Last year he immigrated to NZ and had just the skills and experience Orbica was looking for.

Orbica’s first product automatically identifyies buildings and waterways from aerial photos. This manually intensive job is traditionally done by geographers and cartographers who draw polygons on maps identifying these features using digitising techniques. The first product identifies buildings in urban areas. The 15 million-pixel (4800×3200 ) photos have each pixel covering a 7.5×7.5cm square . Sagar has built a convolution neural network that takes these photos and outputs the vectors representing the polygons where it believes the buildings are.

They have a good amount of training, test and validation data from Land Information New Zealand that consists of the images and polygons that have been hand drawn. Because of the size of the image, Sagar has tiled them into 512×512 images . He built the model over a couple of months with a little trial and error testing the various hyper parameters. The existing model has nine layers, with the standard 3×3 convolutions. He’s currently getting 90 per cent accuracy on the validation set.

Building outlines

RiverDetection_AIThe water classification is very similar, working with 96 million pixel(12000×8000)  images, but with smaller resolution 30x30cm  pixels. The output is the set of polygons representing the water in the aerial images, but the model also classifies the type of water body, e.g. a lake, lagoon, river, canal, etc.

The commercial benefits of these models are self-evident: Orbica can significantly improve the efficiency of producing this data, whether it does this for a client, or it is sold as a service to city and regional councils. These are done regularly – to identify buildings that have been added or removed, or to track how waterways have changed.

WaterBodiesClassification'

Another opportunity has come from the Beyond Conventions pitch competition in Essen, Germany, where Orbica won the Thyssenkrupp Drone Analytics Challenge and the People’s Choice Award. Orbica’s pitch was to use machine vision to analyse drone footage of construction sites to automatically generate a progress update on the construction project. This is a more complex problem given its 3-dimensional nature. Thyssenkrupp has now resourced Orbica to put together a proof of concept, which Sagar is busy working on. Should this go well, Orbica will probably hire at least one other data scientist. DroneImage_Output

Because the technology is developing quickly, Sagar keeps up to date with the latest developments in deep learning through Coursera and Udacity courses. He’s a fan of anything Andrew Ng produces.

To me, Orbica’s use of machine vision technology is an excellent case study for how New Zealand companies can use the latest advances in artificial intelligence. They have a deep knowledge in their own vertical; in this case GIS. They develop an awareness of what AI technologies are capable of in general and have a vision for how those technologies could be used in their own industry.  Finally, they make an investment to develop that vision. In Orbica’s case, the investment was reasonably modest: hiring Sagar. A recurring theme I’m seeing here is hiring skilled immigrants. New Zealand’s image as a desirable place to live – coupled with interesting work – will hopefully make this a win-win for all involved.

For those that would like to hear more. Kurt is speaking at AI Day in Auckland next week.

 

 

 

 

Robots opening doors, driverless cars and finding exoplanets

Here’s some things I’ve been watching/listening to lately…

The latest video from Boston Dynamics is cool. They seem to be having a lot of fun.

I’m continuing to watch the MIT series on Artificial General Intelligence. They’re currently releasing one video a week. The latest is from Emilio Frazzoli on self driving cars. I’ve been enjoying this series.

I’m also listening to the TWiML interview with Chris Shallue about using Deep Learning to hunt for exoplanets. Also pretty cool. I thought Chris’s accent might have been kiwi – but nah he’s an Aussie.

exoplanets

 

 

Jade: developing AI capability for chatbots and predictive modelling

Jade logo

A couple of weeks ago I sat down with Eduard Liebenberger who is the head of digital at Jade to find out a little about their AI capabilities and plans. Eduard is passionate about AI and the possibilities it brings to transform the way we communicate with businesses.

In Eduard’s words, Jade’s core focus is around freeing people from mundane/repetitive tasks and instead allow them to apply their creativity/expertise to more challenging tasks – and the JADE development, database and integration technologies. Eduard and the team at Jade have been watching recent developments in AI and identifying which of these they can use to help their customers. Their first foray has been into conversation interfaces (chatbots). They’ve developed a number of showcases, including an insurance chatbot called TOBi which shows how the technology can be used to make a claim, change contact details etc. From their they have started rolling out this technology into existing customers.

The chatbot uses natural language processing and sentiment analysis. It aims to make businesses interactions with their customers more efficient by allowing them to communicate via conversations that don’t have to be in real time, like a phone call and are more intuitive than a web form. Jade’s main advantage with their existing customers is that they have already done the tricky integration work with the back-end systems and so can fairly quickly add a chatbot as an alternative to an existing interface. Jade’s focus on the digital experience means they invest heavily into making this a natural and human-like interaction. For non-Jade customers their attraction is their ability to deliver a whole solution and not just the chatbot.

A77B63FA-FD17-4EF9-9643-0F01F13BF576

Another advantage Jade has is that through their existing customers they have access to a lot of data that can be used to power machine learning applications. One example Eduard talked about was a summer intern project with a NZ university to try and identify students at risk of dropping out.  This was done using the data in student record database which is powered by Jade and contains several years’ of records. In just a few weeks the interns built a predictive model that was able to predict which students were likely to drop out with 90%+ accuracy. Ed is a big fan of rapid development for these types of proof of concept projects and doesn’t believe it should cost a fortune to get value from AI.

Overall, I think it’s fair to say that Jade’s AI capability is nascent. However, it’s positive to see that they are looking to build capability, understandably with a focus on the business benefits to their customers. I’m keen to see how it develops.

For those that want to find out more, Eduard is delivering the keynote at Digital Disruption X 2018 in Sydney, and presenting at DX 2018 and the AI Day in Auckland, all later this month. He’s a busy man.

Imagr: NZ’s version of Amazon Go

Last Friday during a visit to Auckland I took the opportunity to catch up with Will Chomley, CEO of Imagr. Will is an ambitious entrepreneur, with a finance background. He has surrounded himself with a group of engineers skilled in deep learning. This start-up is focused on using machine vision to identify products as they are added to a shopping cart: NZ’s version of Amazon Go. But this is aimed at cart rather than the whole store. Their product is called SMARTCART and integrates with your phone. 

When I met Will they had just announced their first trial with Foodstuffs New Zealand in their Four Square Ellerslie store . They were busy building their training dataset which at the time of writing contains over 2 million product photos for their machine vision models.

The 12-person company has a handful of machine vision engineers. A portion of these are immigrants because the skills are hard to find in New Zealand. Will is very enthusiastic about running the company from New Zealand because it is such a desirable place to live and it’s easy and quick to move here for people with the right qualifications.

The capabilities of their machine vision look impressive. They’re able to identify very similar looking packages with only subtle differences. I saw an example of two ELF mascara products that on first inspection looked identical, but on closer inspection one had a small brush.  They’re able to identify, with high accuracy, occluded products – partially covered by a hand, or high speed objects being thrown into a basket that I couldn’t recognize, but the blurred images are able to be recognised by their 200+ layer convolutional neural network.

They have designed the cart so the inference engine, which is making the decision about what is going into the basket, can either run on a computer on the basket, or the information can be sent to a server. To get the speed from this model they have developed their own software using C, rather than relying on the easier to use but slower  frameworks such as TensorFlow. This gives the capability to identify and engineer around bottlenecks.

In parallel they’re working on the hardware, having recently decided to use small, cheaper, lower resolution cameras with some added lighting. These can provide the same high accuracy rate as higher resolution, expensive cameras. They have their challenges. Designing and building hardware that can be manufactured at scale and operate reliably and easily is no mean feat. However, they have some engineering chops: their CTO Jimmy Young was Director of Engineering for PowerByProxi, the wireless charging company bought by Apple last year.

They have some backers with deep pockets to help resource these challenges.  Last year they received an undisclosed investment from Sage Technologies Ltd, the technology venture of private investment company QuantRes founder, billionaire Harald McPike.

There’s a large opportunity in front of them and they’re moving quickly. Adding smart carts has to be a lot cheaper than fitting out a store Amazon Go style. They may be able to get a piece of the global grocery market, grabbing a cut of the value of the cart, saving costs for the grocer, improving the experience for the shopper and opening up a world of possibility for the company once they are collecting the shopping data.

One of their challenges is to stay focused. There are so many applications of machine vision technology, even if they stick to retail. They’ve experimented with smart fridges that can identify the gender and age of people taking products, as well as knowing which products they take. They’re talking to others in the industry about applications for their technology.

If they can keep their focus and execute they have a bright future ahead of them.  Their product has global appeal. Retailers can see the threat of technology giants such as Amazon and Alibaba who are moving quickly into bricks and mortar retail. This accelerated last year with Amazon’s purchase of Whole Foods. The convenience of online shopping is coming to a store near you and Imagr may be the one’s powering it.

The limits of deep learning

There are a couple of articles I’ve read recently that have gelled with my own about the limits of deep learning. Deep learning simply refers to multi layered neural networks, that typically learn using back-propagation to train. These networks are very good at pattern recognition, and are behind most recent advances in artificial intelligence.  However, despite the amazing things they are capable of, I think it’s important to realize that these networks don’t have any understanding of what they’re looking at or listening to.

Gödel, Escher, BachThe first article was by Douglas Hofstadter, a professor of cognitive science and author of Gödel, Escher, Bach. I read that book many years ago and remember getting a little lost. However, his recent article titled The Shallowness of Google Translate clearly demonstrates how  the deep learning powered Google Translate successfully translates the words, but often fails to translate the meaning. Douglas believes that one day machines may be able to do this but they’ll need to be filled with ideas, images, memories and experiences, rather than sophisticated word clustering algorithms.

The second article by Jason Pontin at Wired discusses the Downsides of Deep Learning:

  • They require lots of data to learn
  • They work poorly when confronted with examples outside of their training set
  • It’s difficult to explain why they do what they do
  • They don’t gain any innate knowledge or common sense.

Jason argues that for artificial intelligence to progress we need something beyond deep learning. There are many others saying the same types of things. I’ve recommend watching MIT’s recent lectures on Artificial General Intelligence that covers this as well.

 

Thought Experiment logo

image-1A quick word on the thought experiment logo. I had this logo designed using Freelancer – a great site for getting all sorts of things done. I’ve seen these guys present at investor meetings in Sydney and Auckland and I previously used the service to get a design for my gates at home: for just $20 I got 18 designs, selected one and my father-in-law built it with me as manual labour. I’m still pretty proud of those gates.

PURR-PUSS

Again for $20, I ran a competition, this time for a logo. My Ph.D supervisor, John Andreae used a stylised cat, to illustrate his PURR-PUSS AI system. I suggested in the competition notes that they use this combined with the thought experiment Schrödinger’s cat to come up with a logo. I received about 20 designs and ended up choosing the one on the top right. It’s a lot better than what I could have come up with myself.img_1246-1.jpg

Cacophony: Using deep learning to identify pests

This is the first of a series of posts I intend to write on organisations that are using artificial intelligence in New Zealand. I am closer to this organisation than most because it was started by my brother, Grant.

Cacophony is a non-profit organisation started by Grant when he observed that the volume of bird song increased when he did some trapping around his section in Akaroa. His original idea was simply to build a device to measure the volume of bird song in order to measure the impact of trapping. Upon examining trapping technology, he came to the conclusion there was an opportunity to significantly improve the effectiveness of trapping by applying modern technology. So he set up Cacophony to develop this technology and make it available via open source. This happened a little before the establishment of New Zealand government’s goal to be predator free by 2050. He managed to get some funding and has a small team of engineers working to create what I refer to as an autonomous killing machine. What could possibly go wrong?

Because most of the predators are nocturnal the team have chosen to use thermal cameras. At the time of writing they have about 12 cameras set up in various locations that record when motion is detected. Grant has been reviewing the video and tagging the predators he can identify. This has created the data set that has been used to by the model to automatically detect predators.

They hired an intern, Matthew Aitchison, to build a classifier over the summer and he’s made great progress. I’ve spent a bit of time with Matthew, discussing what he is doing. Matthew completed Standford’s CS231n computer vision course that I’m also working my way through.

He does a reasonable amount of pre-processing: removing the background, splitting the video into 3 second segments and detecting the movement of the pixels, so the model can use this information. One of his initial models was a 3 layer convolution neural network with long short-term memory.  This is still a work in progress and I expect Matthew will shortly be writing a description of his final model, along with releasing his code and data.

However, after just a few weeks he had made great progress. You can see an example of the model correctly classifying predators below, with the thermal image on the left and on the right the image with the background removed, a bounding box around the animal and instantaneous classification at the bottom with the cumulative classification at the top.

 

A version of this model is now being used to help Grant with his tagging function, making his job easier and providing more data, faster.

The next thing is to work out how to kill these predators. They’re developing a tracking system that you can see a prototype working below.

From my perspective it feels like they are making fantastic progress and it won’t be too long before they can have a prototype that can start killing predators. If you ask Grant he thinks we can be predator free well before the government’s goal of 2050.

One final point on this from a New Zealand AI point of view, is how accessible these technologies are that are driving the Artificial Intelligence renaissance. Technologies such as deep learning can be learnt from free and low-cost courses such as the CS231n. Those doing so, not only have a lot of fun, but open up a world of opportunity.

Rodney Brooks’ AI predictions

IMG_3465Their is a lot of hype around artificial intelligence, what the technology will bring and its impact on humanity. I thought I’d start my blogging by highlighting  some more grounded predictions from someone who has a lot of experience at the practicalities of AI implementation: Rodney Brooks. Rodney is a robotics pioneer who co-founded iRobot which bought us the Roomba robot vacuum cleaner. I had the pleasure of meeting Rodney when I did global entrepreneurship program at MIT Sloan School of Management. I was a little star struck…

Earlier this month Rodney made some dated predictions about self driving car cars (10 years before driverless taxis are in most big US cities), AI (30 years before we reach dog level), and space travel (humans on Mars in 2036). Rodney calls himself a techno-realist. His experience has shown that turning ideas into reality at scale takes a lot longer than people think. Undoubtedly his predictions will be wrong because that is the nature of predicting the future. However this is a useful perspective given the pace at which the field is advancing. The recent posts from the Google Brain team reviewing 2017 (part 1, and part 2) give a great view of how much progress was made in just the last year. Rodney’s assertion is that turning this progress into products is hard and will take longer than most people think.