At the end of the camp I took part in a hackathon and worked in a team of three to achieve second place. We developed a chatbot for tourists in Belfast that allowed the user to take a photo of a local landmark and send it to the bot which would then identify the landmark and send information about it and things nearby. The frontend was developed using Microsoft Bot Framework (NodeJS version). The backend used Python and the Keras package to create a convolutional neural network model which classified landmark images. The backend also had a module for supplying relevant landmark information that used datasets from OpenDataNI, Google Maps API and Wikipedia’s API.
In the future I plan to improve this application. I’d like to make full use of Bot Builder cards to present location info better. The CNN model is currently only trained to identify either Belfast City Hall or QUB. I’d like to play about with adding more landmarks into the mix or using the Azure computer vision API. I want to also use geolocation data stored in image metadata to improve the accuracy of decision.
This was an incredibly fun opportunity and a brilliant introduction to the world of machine learning. A massive thanks goes out to the folks at Kainos who did a superb job at teaching us.
View the code for the project here.