Auckland University Robotics Lab

I recently had the chance to catch up with Professor Bruce MacDonald who chairs the Auckland University Robotics Research Group. Although we had never met before, Bruce and I have a connection, having the same PhD supervisor, John Andreae.

Bruce took me through some of the robotics projects that him and his team have been working on. The most high profile project is a kiwifruit picking robot that has been a joint venture with Robotics Plus, Plant and Food Researchand Waikato University. This multi armed robot sits atop an autonomous vehicle that can navigate beneath the vines. Machine vision systems identify the fruit and obstacles and calculate where they are relative to the robot arm which is then guided to the fruit. A hand gently grasps the fruit and removes it from the vine using a downward twisting movement. The fruit then rolls down a tube.

Kiwifruit picking robot

The work has been split between the groups with the Auckland University team focused on the machine vision and vehicle navigation, Waikato on the control electronics and software, and Robotics Plus on the hardware. The team estimates that the fruit picking robot will be ready to be used in production in a couple of years. The current plan is to use it to provide a fruit picking service for growers. This way their customers don’t need to worry about robot repairs and maintenance and the venture can build a recurring revenue base. They are already talking to growers in New Zealand and the USA.

Along with Plant and Food Research, the group is also researching whether the same platform can be used to pollinate the kiwifruit flowers. Declining bee populations are expensive to maintain, and this may provide a cost effective alternative.

The group has just received funding of $17m to improve worker performance in orchards and vineyards. The idea is to use machine vision to understand what expert pruners do and translate that into a training tool for people learning to prune and for an automated robot.

Bruce’s earlier work included the use of robotics in healthcare. This included investigating if robots could help people take their medication correctly and the possibility of robots providing companionship to those with dementia who are unable to keep a pet.

Therapeutic robot

I asked Bruce whether Auckland University taught deep learning at an undergraduate level. He said that they don’t, but it is widely used by post grad students. They just pick it up.

Bruce is excited by the potential of reinforcement learning. We discussed whether there is the possibility of using our supervisor’s goal seeking PURR-PUSS system with modern reinforcement learning. I think there is a lot of opportunity to leverage some of this type of early AI work.

At the end of the meeting Bruce showed me around the robotics lab at the new engineering school. It was an engineer’s dream – with various robot arms, heads, bodies, hands and rigs all over the place. I think Bruce enjoys what he does.

Robotics lab

NEC NZ: AI for smart cities

While I was in Wellington as part of the excellent AI panel put together by WREDA, I caught up with Tim Rastall who recently left NEC to start his own consulting firm. Tim gave me some insight into the interesting AI work NEC have been doing in New Zealand. NEC is the Japanese tech giant best known for their IT services and products. One of the core focuses for NEC’s New Zealand arm is on smart cities technologies.

Car counting was the first project Tim discussed. This was a very early project at a time when there were limited tools available. The project used machine vision to count vehicles in a stream of traffic. One of the problems they had with these early models is that they were trained in cities with more smog that Wellington.  The crisp NZ light would create shadows that would confuse the model. This combined with different viewing angles than the model was trained with meant, out of the box, the accuracy was not as high as hoped for. There are now a few third part solutions that do a reasonable job of car counting. What they really learned from the project was that the market wanted vehicle categorisation solutions but using surveillance cameras/analytics presented a raft of technical and practical issues. This led them developing a road based sensor that provides much higher accuracy and reliability and is about to be deployed for some field trials in Wellington.

One of NEC’s higher profile projects was the Safe City Living Lab. This is a research project that includes microphones and cameras placed around the city that can detect events such breaking glass, screaming, beggars, or fighting. The idea being that if an event is detected then an agency could be alerted. There were concerns around privacy but these were addressed to the satisfaction of the Office of the Privacy Commissioner. Tim also shared that they worked hard to remove bias from the models.

Safe City Beggar

NEC also worked with Victoria university to build a model to identify birds from audio recordings. Researcher Victor Anton collected tens of thousands of hours of recordings. The idea was to use deep learning to automatically identify which birds have been recorded. Like a lot of deep learning tasks, the biggest problem was to get a good data set to train from. This involved having people listen to audio recordings and identify which if any birds were in the clip. This is difficult because a lot of the recordings contain other sounds: trees rustling in the wind, other animals or other urban sounds like door bells. So, the first tool they built was a model to identify whether a particular clip contained a bird song or not, that would then make the tagging task easier. From the sounds of it (no pun intended), this is not solved yet for sensors that had novel environmental noise. For those sensors they created good training data they could categorise discreet species very well. Listen to a Radio NZ interview with Victor and Tim about the birdsong AI project for more information.

This research is related the Cacophony project that was inspired by the idea that measuring the volume of bird song would be a way of measuring the impact of predator control. Cacophony pivoted and are now creating an AI powered predator killing machine.

full_The_acoustic_sensor_technology_(Yadana_Saw)

NEC’s current focus is aggregating city data from various sources and making it easily accessible. This includes powerful visualisation tools that allow the data be overlaid on a map and viewed in 3D, either on a screen or in VR. This makes it easier to understand and helps people make decisions with the data more efficiently. While this isn’t directly related to AI, having the data accessible makes it much easier for people to use in AI applications.

3D_Capture_of_Wellington_fewer_pixels

In summary, NEC New Zealand has worked on many interesting smart city projects using deep learning and data visualisation. They have a sizeable team that is growing in experience. I’m interested to see what they come up with next.

Meanwhile Tim has left NEC to work on a gaming start up and do some consulting around artificial intelligence, augmented and virtual reality, and Internet of Things. I’m sure he’ll do well.

 

Autonomous weapons, running robots, Open AI and more

Here’s some highlights of AI reading, watching, listening I’ve been doing over the past few weeks.

A couple of videos from the MIT AGI series. First, Richard Moyes, co-Founder and Managing Director, Article36 on autonomous weapons and the efforts he and others are doing to reduce their impact.

The second is Ilya Sutskeve, co-founder of open AI, on neural networks, deep reinforcement learning, meta learning and self play. He seems pretty convinced that that it is a matter of when, not if, we will get machines with human level intelligence.

Judea Pearl, who proposed  that Baysian networks could be used to reason probabilistically, laments AI can’t compute cause and effect and summarises Deep Learning as simply curve fitting.

After reading a post about mask R-CNN being used on the footage of Ronaldo’s recent bicycle kick goal I took a look at the description of the code here.

Mask RCNN street Mask RCNN Football

Jeff Dean TWIMLAI_Background_800x800_JD_124An interview with Jeff Dean, Google Senior Fellow and head of the company’s deep learning research team Google Brain, on core machine learning innovations from Google and future directions. This guy is a legend.

 

 

 

Jose TWIMLAI_Background_800x800_JH_137

An interviewwith Jose Hernandez-Orallo on the kinds of intelligence.

 

 

 

 

And of course the video of the running, jumping robot from Boston Dynamics. If you want to find out a little bit more about the company I recommend the lecture from their CEO Marc Raibert below.

 

Ohmio automation: self driving buses

Last month I had lunch with Yaniv Gal, the artificial intelligence manager at Ohmio. Yaniv is an interesting character who grew up in Israel and has focused his career on computer vision and machine learning, both in academia and industry. Although a lot of his experience was in medical imaging, in New Zealand he had been working in the fresh produce industry as research program manager at Compac Sorting Equipment which uses machine vision to automatically sort fruit. At Ohmio he’s built one of the largest AI teams in New Zealand.

Yaniv explained that Ohmio emerged from electronic sign provider HMI technologies. HMI has been around since 2002 and has thousands of signs operating throughout NZ. To me it seemed unusual that an electronic sign company would spawn a self-driving vehicle company. There were a couple of core reasons:

  1. They had some experience using machine vision with traffic: cameras attached to their signs could be used to count traffic in a much more cost effective and reliable way than digging up the road to install induction powered sensors.
  2. They had experience at installing infrastructure alongside roads. This type of infrastructure could be used to aid a self-driving vehicle along a fixed path

This is a crucial differentiator for Ohmio. They are not trying to compete with the myriad of large companies that are trying to develop level 5 autonomous vehicles: those that can drive without human input in all conditions. This is a difficult problem. Sacha Arnold, the director of engineering at Waymo (owned by Alphabet, Google’s parent) recently said they are about 90% of the way there, but they still have 90% to go. Instead Ohmio are going for the more tractable problem of building a vehicle platform that can navigate along a fixed path. They call this level 4+ autonomy. While this doesn’t have the same broad opportunity as level 5, they believe it is something they can build and that there is still a large market opportunity.

Ohmio LIFT

Their first customer is Christchurch airport. This will allow them to prove the concept and refine the technologies. The economics are obvious, with no driver this just ends up cheaper. It’s not just about the money though, Yaniv is confident it will be safer and with electric vehicles, greener. Since our meeting Ohmio have announced a sale of 150 shuttles to  Korean Company Southwest Coast Enterprise City Development Co Ltd.

For fixed path navigation the path can be learnt and if necessary additional infrastructure can be added along the path to aid the vehicle in localising and navigating. Most of this is done on the vehicle using a variety of sensors. To establish exactly where it is odometery, GPSs and LIDAR are combined to get a more accurate location, with more redundancy than possible with a single sensor. Combining data from multiple sensors like this is called sensor fusion. Company R&D coordinator Mahmood Hikmet described this in his recent AI day presentation.

Machine vision is used primarily for vehicle navigation and collision avoidance. Here collision avoidance is detecting whether there is an object in the vehicle’s path, or if an object may come into its path. Ohmio use a variety of machine vision techniques, including deep learning. Yaniv’s experience predates the recent rise in popularity of neural networks. He is aware of the few disadvantages deep learning suffers compared to more “traditional” machine learning techniques and so he doesn’t feel like every machine vision problem needs the hammer that is a deep neural network.

Yaniv confessed that it will be a nervous moment the first time the system is driving passengers with no safety driver. However, he is confident it is going to be safer than a vehicle with a driver. We talked about how both of us would be even more nervous using a self driving, flying taxi that Kitty Hawk is testing in Christchurch, even though it should be safer than a ground based vehicle because there are less objects to crash into.  We compared this to the fear people felt when elevators were first self-operating, without an elevator operator. It seems like a silly fear now. Maybe the next generation will laugh at our anxiety about self-driving vehicles.

After lunch I told my Uber driver about our conversation. He expressed concern about whether the tech should be developed and the loss of jobs that will come with it. This is an understandable concern given his career (although his masters in aeronautical engineering should see him right). There are too many people working on this type of technology now. If the genie is not out of the bottle, he has his head and shoulders out. The economics are too strong and the world should be a better place with this technology. It’s nice to see a NZ company contributing.

Lincoln Agritech: using machine vision for estimating crop yield

I recently had the opportunity to visit Lincoln Agritech where I met with CEO, Peter Barrowclough, Chief Scientist Ian Woodhead and their machine vision team of Jaco Fourie, Chris Bateman and Jeffrey Hsiao. Lincoln Agritech is an independent R&D provider to the private sector and government employing 50 scientists, engineers and software developers. It is 100% owned by Lincoln University, but distinct from their research and commercialisation office.

Lincoln Agritech have taken a different approach to developing AI capability. Rather than hiring deep learning experts, they have invested in upskilling existing staff by allowing Jaco, Chris and Jeffrey to take a Udacity course in machine vision using deep learning. The investment is in the time of their staff. Having three of them take the course together means that they can learn off each other.

The core projects they are working on involve estimating the yield of grape and apple crops based on photos and microwave images. The business proposition for this is to provide information for better planning for the owners of the crops, both in-field and in-market. Operators of these vineyards and orchards can get a pretty good overall crop estimate based on historical performance and weather information. However, they can’t easily get plant by by plant estimates. To do this they need an experienced person to walk the fields and make a judgement call. A machine vision based approach can be more efficient with better accuracy.

The team elected to tackle the problem initially using photos. They had to take the images carefully at a prescribed distance from the crop, using HDR (this is where you combine light, medium and dark images to bring out the detail in the shadowy canopy). Like most machine learning tasks the biggest problem was getting a tagged data set. The tagging involved drawing polygons around the fruit in the images, including fruit partially occluded by leaves. There was a lot of work trying to train people to do this properly. Inevitably at some stage there were guys with PhDs drawing shapes, such is the glamour of data science. This problem is similar to that faced by Orbica who built a model to draw polygons around buildings and rivers from aerial photography.

In this image labels of a fixed size are added to an image to tell the model where the grape bunches are.
This image shows the result of a trained network automatically finding the areas in the image where the grape bunches are.

They used a convolution neural network to tackle this problem. Rather than train a network from scratch and come up with their own architecture, they used the image net winning inception architecture and adapted that. This network was already trained to extract the features from an image that are required to classify 1000 different classes of images. This technique is called transfer learning. This model works well, with 90% accuracy (on their validation data set).

However, part of the challenge here is that the images do not show all of the fruit that is on the plant. The only way to get the “ground truth” is to have someone go under the canopy and count all the fruit by hand. This is where the microwave technology comes into play.

The company is recognised internationally for their microwave technology in other projects. The way it works is a microwave transmitter emits microwaves and then detects the reflections. The microwaves will travel through leaves, but will be reflected by the water content in reasonably mature fruit.

The machine vision team is working to create a model that can use the microwave image and photo to get superior performance. This is a harder problem because this type of sensor fusing is less common than regular image processing.

The team is using the Tensorflow and Keras platforms on high end machines with Nvidia Titan GPUs. There were a few raised eyebrows from the company when the team were asking for what essentially looked like high end gaming machines.

I applaud Lincoln Agritech for investing in their deep learning capability. The experience they will have gained from their first projects will make each subsequent project easier to apply the technology. The fact they have three people working on this, provides redundancy and the ability to learn off each other. This is a model that other New Zealand organisations should consider, particularly if they’ve having problems finding data scientists. Applying the latest AI technologies to agriculture seems like a real opportunity for New Zealand.

AI day 2018: My take

I noticed the AI day videos were released a few days ago and I’d like to share my thoughts on the day. First I”d like to congratulate the organisers Ben Reid and Justin Flitter for putting this event together. Michelle Dickinson did a great job making the day flow as master of ceremonies. This type of event is just what NZ needs to help people understand how different organisations are using AI so they can make more informed decisions on how they could use the ever evolving set of technologies.

AI day 2018 videos
The AI day 2018 videos

I’d characterise the event as having presentations from small and large organisations, a couple of panels, a politician and a good dose of networking. The highlight for me was from the small companies because they were the ones who had taken various AI technologies and applied them in a way to give them an advantage. In my mind these are the stories that are most likely to inspire other NZ companies. This included:

  • R& D coordinator for Ohmio, Mahmood Hikmet describing the self-driving shuttle that they are building and how their AI team is building a sensor fusion model. This combines data from GPS, lidar and odometry sensors to estimate the position of the shuttle, that is then used for navigating.
  • Kurt Janssen, the founder of Orbica described how they’re using machine vision with aerial and drone footage to automate various GIS tasks.
  • Grant Ryan (or Bro as I call him) describing how Cacophony are using machine vision with thermal cameras to automatically identify pests, and how they might then kill them.
  • Sean Lyons had the most entertaining presentation where he described how Netsafe are using bots to waste scammers time in a project they call Rescam. They’re using IBM Watson for sentiment analysis. It’s been hugely successful, wasting over 5 years of scammers time with 1 million emails.
    netsafe bot
  • Mark Sagar and team are doing some of the most interesting AI work globally at Soul Machines. Unfortunately, his presentation had a few technical glitches, but it was nice to see the latest version of BabyX, complete with arms. Mark talked a little bit about how they are using neural networks for perception and control. I’d love to find out more details.
    Babyx

The other small company that presented was Centrality.ai. Founder Aaron McDonald spent most of the presentation explaining blockchain and how it can be used for contracts. I didn’t come away with any understanding that the company is using AI, or with any comprehension of what the company actually does.

The panels had a selection of interesting entrepreneurs and academics. However, I personally find the panel format a little too unstructured to get much useful information from. I may be an outlier here, Justin told me they got very good feedback about the panels from their post conference surveys.

The other highlight of the conference for me was the networking during the breaks. Everyone you spoke to had some involvement in AI: Entrepreneurs, practitioners, academics and investors. This was an added benefit to an already very stimulating day. I wasn’t able to attend the 2nd day of workshops.

To Justin and Ben: Well done! I look forward to attending next year and hearing how a host of other NZ companies are using AI in interesting ways. For those that didn’t make it, check out the videos.

 

Cognitive modelling, self aware robots Tensorflow, & adversarial attacks

 

This week I’ve been learning about cognitive modelling, self aware robots,  adversarial attacks in reinforcement learning and starting to play with Tensorflow.

Cognitive Modelling

The latest MIT AGI video was released a few days ago. In this Nate Derbinsky gives an overview of different types of cognitive architectures, including SPAUN, ACT-R, Sigma and Soar (his baby). This reminds me of old school AI: symbolic processing. My supervisor’s PURR-PUSS would belong in this category. These are a lot less sexy than deep learning, but in many way’s they are complementary with applications in robotics, game playing & natural language processing.

 

TWiML podcasts

SUM cognitive architectureThis week I listened to an interesting podcast with Raja Chatila on robot perception and discovery. In this Raja talked about the necessity of robot self awareness for true intelligence and the ethics of intelligent autonomous systems. It’s interesting to see that the sort of architectures used for exploring artificial consciousness in robotics have a lot of overlap with the cognitive models described by Nate Derbinsky in the MIT AGI series.

I also had the chance to listen to Google Brainers Ian Goodfellow & Sandy Huang discussing adversarial attacks used against reinforcement learning  Adversarial attacks highlight some of the weakness of deep learning. When used for image classification the image is just a series of numbers that has a set of mathematical operations performed on it to produce a classification. By subtly changing some of the numbers you can fool the classifier, even though to a human the image looks exactly the same. The example of a Panda below was taken from a 2015 Google paper.

panda

In the podcast Ian and Sandy discuss how this can be used against a reinforcement learning agent that has been trained to play computer games. Even changing one pixel can significantly degrade the performance.

Tensorflow

I’m up to the part in my CS231n course where you start to train CNNs using Tensorflow or Pytorch. Despite reading a compelling argument for using Pytorch over Tensorflow on Quora, the people I’ve spoken to locally are using Tensorflow – so I’m going to go with that. I found this introduction useful.

I managed to get the software installed and run Hello World. Apparently there is more you can do with it…

Tensorflow hello world

 

Orbica: Using machine vision in GIS

Last week I had the opportunity to sit down with Orbica CEO’s Kurt Janssen and data scientist  Sagar Soni.

Kurt has worked in the Geographic Information Systems (GIS) industry for more than 14-years. Last year he started his own company, Orbica, which does GIS consulting for organisations in the public and private sector. Orbica invests some of its consulting revenue into developing its own product. A major – and rewarding – investment has been hiring data scientist Sagar.

Sagar was taught machine learning during his master’s degree and had the opportunity to put it into practice developing an earth rock image classification system at Dharmsinh Desai University and using deep learning algorithms like Recurrent Neural Networks to solve medical entity detection problems at US health care solutions provider ezDI. Last year he immigrated to NZ and had just the skills and experience Orbica was looking for.

Orbica’s first product automatically identifyies buildings and waterways from aerial photos. This manually intensive job is traditionally done by geographers and cartographers who draw polygons on maps identifying these features using digitising techniques. The first product identifies buildings in urban areas. The 15 million-pixel (4800×3200 ) photos have each pixel covering a 7.5×7.5cm square . Sagar has built a convolution neural network that takes these photos and outputs the vectors representing the polygons where it believes the buildings are.

They have a good amount of training, test and validation data from Land Information New Zealand that consists of the images and polygons that have been hand drawn. Because of the size of the image, Sagar has tiled them into 512×512 images . He built the model over a couple of months with a little trial and error testing the various hyper parameters. The existing model has nine layers, with the standard 3×3 convolutions. He’s currently getting 90 per cent accuracy on the validation set.

Building outlines

RiverDetection_AIThe water classification is very similar, working with 96 million pixel(12000×8000)  images, but with smaller resolution 30x30cm  pixels. The output is the set of polygons representing the water in the aerial images, but the model also classifies the type of water body, e.g. a lake, lagoon, river, canal, etc.

The commercial benefits of these models are self-evident: Orbica can significantly improve the efficiency of producing this data, whether it does this for a client, or it is sold as a service to city and regional councils. These are done regularly – to identify buildings that have been added or removed, or to track how waterways have changed.

WaterBodiesClassification'

Another opportunity has come from the Beyond Conventions pitch competition in Essen, Germany, where Orbica won the Thyssenkrupp Drone Analytics Challenge and the People’s Choice Award. Orbica’s pitch was to use machine vision to analyse drone footage of construction sites to automatically generate a progress update on the construction project. This is a more complex problem given its 3-dimensional nature. Thyssenkrupp has now resourced Orbica to put together a proof of concept, which Sagar is busy working on. Should this go well, Orbica will probably hire at least one other data scientist. DroneImage_Output

Because the technology is developing quickly, Sagar keeps up to date with the latest developments in deep learning through Coursera and Udacity courses. He’s a fan of anything Andrew Ng produces.

To me, Orbica’s use of machine vision technology is an excellent case study for how New Zealand companies can use the latest advances in artificial intelligence. They have a deep knowledge in their own vertical; in this case GIS. They develop an awareness of what AI technologies are capable of in general and have a vision for how those technologies could be used in their own industry.  Finally, they make an investment to develop that vision. In Orbica’s case, the investment was reasonably modest: hiring Sagar. A recurring theme I’m seeing here is hiring skilled immigrants. New Zealand’s image as a desirable place to live – coupled with interesting work – will hopefully make this a win-win for all involved.

For those that would like to hear more. Kurt is speaking at AI Day in Auckland next week.