Building Lizzie – I


Recently I have been busy building a personal assistant that I would be fitting in my car. Currently I am in experimentation mode and I am experimenting with speech capabilities. I would start with a description of my journey so far.

First let me show off a little bit with these videos that I created while experimenting.

Here is another one with grammar based speech recognition.

Lizzie (read Lizzie Hearts) would be the name of the personal assistant and I have planned to use a Raspberry Pi to accommodate her. I am using a Raspberry Pi 3 which is quite powerful considering the form factor and size. I started with installing Windows 10 IoT core because I have been primarily working on Microsoft Windows technologies for a while and the platform is quite comfortable both in terms of fundamentals and programming technologies. After experimenting for a while I moved to Linux (I tried several variants of Linux which include Raspbian, Cent OS, Ubuntu Core and Linux Mint), played with PocketSphinx and Sphinx 4 for offline speech recognition on Linux. While speech recognition using these was easy but the accuracy and most importantly dictation mode was lacking (not unless you are willing to build a large language model with very limited set of tools). I came back to Microsoft Windows 10 IoT core and started my experiments with a clear set of goals and functionality. The features that I am working on right now are:

  1. Emails
    1. Announce count of unread emails.
    2. Search emails using keywords or senders.
    3. Read emails aloud.
    4. Send/Reply emails.
    5. Accept/Reject meeting invites
  2. Calendar (from Google Calendar)
    1. Read schedule for today or any other day (past or future).
    2. Create an appointment.
    3. Create and send meeting invites.
    4. Cancel/Reschedule any appointment.
    5. Cancel/Reschedule any meeting and send updates.
  3. Navigation(using Google Maps and GPS from connected phone – by running a service on phone as a web server)
    1. Distance/Time calculation (using Distance Matrix API from Google Maps API)
    2. Show places on maps (using Google Maps Geocoding API and Google Maps Embed API/Google Static Maps API)
    3. Directions API (need to decide the approach but most probably would be using the GPS data from phone on a predefined interval and call the URL from java script and then update the markers and directions on the Map application).
    4. Log the path in file and then plot that on the Map using Roads API.
  4. Show vehicle statistics using ELM327 (OBD-II Bluetooth adapter)
    1. Speed:
    2. RPM
    3. Fuel pressure
    4. Engine Load
    5. Engine temperature
    6. Intake air temperature
    7. Throttle position
    8. Engine Oil temperature
    9. Engine Run-time
  5. Music player
    1. Play music
    2. Playlist support
    3. Music search
    4. Play streaming radio

Well this is a long list and I am really excited to explore the unknown. I have already experimented with speech recognition and synthesis part now the next step is to experiment on vehicle statistics, I have got a very simple PoC (proof of concept) with simulator already working and the next step is to formalize and organize the bits. I have already worked on other items like Mail, Calendar, Music and Maps so they would be relatively easy to integrate with the speech controlled framework.

One more last note, for doing NLP (natural language processing) and getting intents and parameters out of natural language commands I am using Api.Ai cloud platform (I thought of writing my own but then decided to make use of Api.Ai because the speech recognition happens locally on the device and only the text is sent to the server for processing and the results so far are amazing – see first video in case you have any doubts; so I thought why re-invent the wheel).

Keep watching this space for updates, hoping to be back soon.

Until next time…

, , , , ,


2 responses to “Building Lizzie – I”

  1. mortoray Avatar

    One would think points 3,4,5 would be higher on the list than 1,2. 🙂

    1. thegeeksdiary786 Avatar

      Agreed, given the fact that Lizzie would live in my car those features would be on top of the list. Emails and calendar are good to have features and I put them on top because I wouldn’t need to experiment about them. Thanks for your feedback.😊☺

Leave a Reply

Other posts

  • Object Extraction using Image Segmentation: A Comprehensive Tutorial with Detectron2 and Mask2Former

    Object Extraction using Image Segmentation: A Comprehensive Tutorial with Detectron2 and Mask2Former

    Discover how to perform object extraction using image segmentation with Detectron2 and Mask2Former in our step-by-step tutorial. Learn to set up the environment, configure the model, and visualize segmentation results, extracting objects from images with ease. Boost your computer vision skills and optimize your image processing projects with this comprehensive guide.

  • Building Your First Neural Network with TensorFlow – Deep Learning 2

    Building Your First Neural Network with TensorFlow – Deep Learning 2

    Neural networks are a fundamental concept in deep learning and are used for a wide range of applications such as image and speech recognition, natural language processing, and much more. In this article, we will walk you through the process of building your first neural network using TensorFlow, a popular open-source machine learning library. We'll…

  • Introduction to Deep Learning with TensorFlow – Deep Learning 1

    Introduction to Deep Learning with TensorFlow – Deep Learning 1

    In this article, we provide an introduction to deep learning with TensorFlow. We cover what deep learning is, what it can do, why TensorFlow is a great choice for deep learning, and an overview of TensorFlow itself. We also explore the different types of neural networks used in deep learning, and demonstrate how to build…

  • How To: Set Up PyTorch with GPU Support on Windows 11 – A Comprehensive Guide

    How To: Set Up PyTorch with GPU Support on Windows 11 – A Comprehensive Guide

    Introduction Hello tech enthusiasts! Pradeep here, your trusted source for all things related to machine learning, deep learning, and Python. As you know, I’ve previously covered setting up TensorFlow on Windows. Today, I’m excited to bring you a detailed guide on setting up another popular deep learning framework, PyTorch, with GPU support on Windows 11.…

  • Solving a Complex Logistics Optimization Problem using the Pulp Library in Python – Part 4

    Solving a Complex Logistics Optimization Problem using the Pulp Library in Python – Part 4

    In this article, we demonstrate how to solve a logistics optimization problem using the Pulp library in Python. By defining the variables, objective function, and constraints, and using the solve method to find the optimal solution, we are able to minimize the total cost of transportation while satisfying the constraints. This article concludes the multi-part…

  • Linear Programming in Python using PuLP – Part 3: Optimizing Investment Portfolios with Multi-Objective Optimization

    Linear Programming in Python using PuLP – Part 3: Optimizing Investment Portfolios with Multi-Objective Optimization

    In this article, we used the Pulp library in Python to solve a linear programming problem to find the optimal investment portfolio. We defined variables, added constraints, defined objectives, and solved the problem to find the optimal solution that balances the trade-off between maximizing returns and minimizing risk. The code was concise and easy to…

  • Linear Programming in Python using Pulp – Part 2

    Linear Programming in Python using Pulp – Part 2

    In this article, we delve deeper into linear programming and explore how to solve a multi-objective optimization problem using the Pulp library in Python. We present a problem in which a nutritionist must find the optimal meal plan for a patient suffering from anemia, balancing the intake of Vitamin B12 and fat. We demonstrate how…

  • Linear Programming in Python using PuLP – Part 1

    Linear Programming in Python using PuLP – Part 1

    Linear programming is an optimization technique used to find the best outcomes for a given problem. This technique relies on a set of constructs which are all expressed using a system of linear equations. It is important to understand that you should be able to express your objective as a linear equation dependent on an…

  • How To: Setup Tensorflow With GPU Support using Docker

    How To: Setup Tensorflow With GPU Support using Docker

    Previously I published a guide for setting up tensorflow in an anconda environment with GPU support. A lot of people liked it and I have been working with this environment myself for more than a year now. I am happy with the results however the process is a bit involved and requires quite a bit…

  • How To: Setup Tensorflow With GPU Support in Windows 11

    How To: Setup Tensorflow With GPU Support in Windows 11

    It's been just 2 days since Windows 11 came out and I am already setting up my system for the ultimate machine learning environment. Today we are going to setup a new anaconda environment with tensorflow 2.5 with GPU support using NVIDIA CUDA 11.4 and CUDNN 8.2.4 along with Python 3.8. This is going to…

  • Tools of The Trade – II

    Tools of The Trade – II

    In continuation of my previous post today I will talk about the website tanooja.com. I did this project on request of my wife because she wanted to pursue blogging and didn't want to go through the ordeal needed to write, publish and manage SEO using most of the prominent blogging platforms like WordPress, Joomla, Drupal…

  • Tools of The Trade – I

    Tools of The Trade – I

    In this post I will share a few tools and technologies that I am using to run a couple of blazing fast websites using latest modern tools and technologies. The caveat here is that I don't pay any infrastructure/hosting costs for any of these websites and they can scale infinitely in terms of supported users…

  • Building Lizzie – IV

    Building Lizzie – IV

    Another post about Lizzie. I started off with a Raspberry Pi 3 to build a personal assistant for my car and I have come a long way both in terms of the concept and the functionality. Most importantly I have formalized the application flow and also extended the scope from one device to almost all…

  • OBD-II with Raspberry Pi3

    OBD-II with Raspberry Pi3

    I am writing this article in response to a question posted on my YouTube channel. Here I would be talking about communicating to an OBD-II device (ELM327 chip with Bluetooth) hooked into your car’s OBD-II port. The OS I am using is Windows 10 IoT core. This information is important because it makes a difference…

  • Building Lizzie – III

    Building Lizzie – III

    As mentioned in previous article today I would be talking about OBD-II integration in Lizzie using a Bluetooth serial communication with an ELM327 adapter that fits on a OBD-II port in your car. OBD stands for On Board Diagnostics which is connected to the ECU (Engine Control Unit) and provides a ton of information (both…

  • Building Lizzie – II

    Building Lizzie – II

    In the previous post I described my experiments around building an intelligent artificial personal assistant – Lizzie. The pseudo intelligent agents available today around us (Siri, Cortana or Google Next) are all great feats of engineering given the fact that they reside on small devices like mobile phones and are able to do powerful things…

  • Building Lizzie – I

    Building Lizzie – I

    Recently I have been busy building a personal assistant that I would be fitting in my car. Currently I am in experimentation mode and I am experimenting with speech capabilities. I would start with a description of my journey so far. First let me show off a little bit with these videos that I created…

  • How To: Super Fast Tensorflow 2 Setup with GPU Support in VS Code

    I am publishing this article in response to a comment I received on How To: Setup Tensorflow With GPU Support using Docker article. Docker is all good and nice but all of us agree that we use IDEs (integrated dev environment – think VSCode) on our machines. So let’s say you want to get the…

%d bloggers like this: