Google Summer of Code (GSoC) 2022
Project Details
- Contributor: Neelay Shah
- Mentors: Dr. James Turner and Prof. Thomas Nowotny
- Project Title: Creating Benchmark Datasets for Object Recognition with Event-based Cameras
- Project-related links:
Project Abstract
Event-based vision is a subfield of computer vision that deals with data from event-based cameras. Event cameras, also known as neuromorphic cameras, are bio-inspired imaging sensors that work differently to traditional cameras in that they measure pixel-wise brightness changes asynchronously instead of capturing images at a fixed rate. Since the way event cameras capture data is fundamentally different to traditional cameras, novel methods are required to process the output of these sensors. In addition to dealing with ways for capturing data with event cameras, event-based vision encompasses techniques to process the captured data - events - as well, including learning-based techniques and models, spiking neural networks (SNNs) being an example. This project aims to create benchmark datasets for object recognition tasks with event-based cameras. Using machine learning solutions for such tasks requires a sufficiently large and varied collection of data. The primary goal of this project is to develop Python utilities for augmenting event camera recordings of objects captured in an academic setting in various ways to create benchmark datasets.
Progress Log
Pre-coding Period
- Met with mentors (virtually), got to know about each other, and discussed project goals
- Set up GitHub code repository for the project
- Received data relevant to the project from mentors along with some starter code
Week 0 ( 6th June - 12th June )
- Added code for generating 2D Perlin noise
- Added code for spike encoding of image / video data using rate coding
- Set up continuous integration (CI) for the package which includes unit tests and linting and formatting checks
Week 1 ( 13th June - 19th June )
- Added code coverage check to CI workflow
- Created a GitHub pages site for the project
- Added code for spike encoding of video data using a method based on thresholding differences in intensities of pixels in neighbouring frames
- Added code to generate streams of progressive Perlin noise
- Worked on creating an automatic changelog generation workflow using GitHub Actions
Week 2 ( 20th June - 26th June )
- Added code for end-to-end spike encoding of videos using the threshold-based method
- Tested the above method on real-world videos; happy with the results!
Week 3 ( 27th June - 3rd July )
- Added code for injecting events in spike-encoded custom videos into existing event recordings to augment them
- Worked on analyzing spike-encoded 3D Perlin noise
Week 4 ( 4th July - 10th July )
- Added code for downloading clips of desired duration from YouTube videos
- Experimented with different kinds of resampling methods with 3D Perlin noise to get smoother spike encodings
Week 5 ( 11th July - 17th July )
- Experimented with different settings and parameters for generation of 3D Perlin noise
Week 6 ( 18th July - 24th July )
- Worked on writing code for converting encoded videos / 3D noise to event sequence representation
Week 7 ( 25th July - 31st July )
- Completed the functionality for transforming spike encodings to the event sequence format
Week 8 ( 1st August - 7th August )
- Augmented multiple pre-existing object recordings using Perlin noise and videos from YouTube
Week 9 ( 8th August - 14th August )
- Created an IPython notebook to demonstrate the entire workflow for augmenting event-camera datasets using movies / videos
Week 10 ( 15th August - 21th August )
- Created another similar IPython notebook for augmentation using Perlin / Fractal noise
Week 11 ( 22nd August - 28th August )
- Created documentation website
- Set up continuous deployment (CD) workflow