Steve Breese

A Chicago-based Full-stack JavaScript Developer

Make an app that uses machine learning to analyze your images.

Introduction

This tutorial provides step-by-step instructions on building a web application that consumes Google's awesome free service, the Cloud Vision API. Google Cloud's Vision API is a REST service that allows you to assign labels to images and quickly classify them into millions of predefined categories. Add best of all... 1000 image reads are free every month!

Step 1 - Set Up Google Cloud Vision API

  1. Create a new Google Cloud project
  2. From your Google Cloud Platform console, click "Go to APIs overview" (at the bottom of APIs card)
  3. From the APIs & Services page, click "ENABLE APIS AND SERVICES" button as shown in this image: Enable APIs button
  4. On the API Library page, find the "Search for APIs & Services" search box as shown in this image: Search for APIs & Services box and populate it the with "Cloud Vision API".
  5. From the results list, click "Cloud Vision API" to be taken to the Cloud Vision API overview page.
  6. Click the Enable button Cloud Vision APIs Enable button
  7. Return to the APIs & Services page for your project
  8. On the left-hand side of the page, click "Credentials" APIs & Services Credentials button
  9. On the APIs & Services Credentials page, click Create credentials dropdown and select "Service account key" APIs & Services Service Account Key button
  10. On the Create Service Account page, click the Service Account dropdown and select "New Service Account" New Service Account
  11. In the "Service account name" box, type "what-am-i"
  12. From the Role dropdown list, select Project > Owner as show in this image: Service Accout Name and Role
  13. Make sure the Key Type is set to JSON, then click Create
  14. Select a location on your laptop folder to save the file and make note of its path. Copy and paste the path name here so that the command in Step 4-1 will work properly:

Step 2 - Setup Project Locally

  1. Launch Finder
  2. Navigate to your development folder
  3. Right click on your development folder and click "New Terminal at Folder" to launch Terminal
  4. Create what-am-i folder and cd into it
    mkdir what-am-i && cd $_
  5. Initialize project
    $ git init
    $ npm init -y
  6. Setup a basic React project
    npx create-react-app client

Step 3 - Setup empty application in Heroku

  1. If necessary, create a free account on Heroku.
  2. Navigate to the Heroku Dashboard.
  3. In the top-right corner, click New > Create new app.
  4. Give your app a suitable name, then from the Deploy tab, select Heroku Git as the Deployment Method.
  5. If necessary, install the Heroku CLI

    (Download and install the Heroku CLI.

  6. Log in to your Heroku account and follow the prompts to create a new SSH public key.
    $ heroku login
  7. Since we already created our Heroku app, we can easily add a remote to our local repository with the heroku git:remote command. All you need is your Heroku app’s name:
    $ heroku git:remote -a what-am-i
    set git remote heroku to https://git.heroku.com/what-am-i.git
    
  8. Deploy your changes

    Make some changes to the code you just cloned and deploy them to Heroku using Git.

    $ git add .
    $ git commit -am "make it better"
    $ git push heroku master

Step 4 - Setup environment variables

The end result of Step 1 was exporting a JSON file to our computer. Since we actually don't want to store this file in our repository, we will convert them into a Node.js environment variables (for our local development enviornment), as well as set them as Herou Config Variables for our Production environment. This allows us to keep the credentials out of the Git repository, and it also provides a way to easily change the credentials without having to re-deploy the entire app.

  1. Run this comman in Terminal to open up your Google Cloud Service Account Key in TextEdit
    open -a TextEdit {file path}
  2. Launch Code Beautify's JSON Minifier app and copy & paste the complete JSON text from TextEdit to the box on the left side of the Minifier screen.
  3. Click "Minify/Compress" button to minify the JSON.
  4. Copy the minified JSON to your laptop's clipboard
  5. Create a server directory:
    mkdir server
  6. Create a .env file to store the local environment varibles:
    touch server/.env
  7. Append a SERVICE_ACCOUNT_JSON variable to the environment file:
    echo 'SERVICE_ACCOUNT_JSON={paste your minified JSON here}' >> server/.env
  8. Append a GOOGLE_APPLICATION_CREDENTIALS variable to the file as well. The variables's value (gcloud-credentials.json) will refer to a file that our code will create automatically.
    echo 'GOOGLE_APPLICATION_CREDENTIALS=gcloud-credentials.json' >> server/.env
  9. Go to your Heroku dashboard and click on "Settings" tab
  10. In the Config Vars section, click Reveal Config Vars. This is where you can add environment variables for your Heroku app.
  11. Add the same keys/values as you just added to the .env file.
  12. The Config Vars section should resemble the following:

OK, so we got done with some of the rote boring stuff that every application requires. Now let's move on to some real coding and build the core of our application. I will breakdown each line for you so you know exactly what it does.

You will notice that I used shell commands to write our code files. Using the command line interface like this allows me to honor the step-by-step theme of the tutorial. If you are more comfortable using an integrated development environment for code entry, please go ahead and do this.

Step 5 - Create a Backend Server with Node.js

  1. Navigate into the server directory
    cd server
  2. At the top of the file, we need to initialize dotenv to read our .env file, which we created in Step 4 above
    cat <<EOT >> index.js
    // load environment variables from .env file, if it exists
    require('dotenv').config();
    EOT
    
  3. Import all the dependencies that we’ll need.
    cat <<EOT >> index.js
    // import dependencies
    const path = require('path');
    const util = require('util');
    const fs = require('fs');
    const express = require('express');
    const multer = require('multer');
    const { ImageAnnotatorClient } = require('@google-cloud/vision');
    EOT
    
  4. Let's promisify some file-system functions to make our code a bit cleaner later on.
    cat <<EOT >> index.js
    // promisify the filesystem functions we need
    const readdirAsync = util.promisify(fs.readdir);
    const statAsync = util.promisify(fs.stat);
    const unlinkAsync = util.promisify(fs.unlink);
    EOT
    
  5. Next, we’ll need to write out the Google Cloud Service Account Key from the environment variable to a file on the server. This is done because Google Cloud SDK needs a JSON file for authentication.
    cat <<EOT >> index.js
    // we are using the synchronous version writeFileSync, because this needs to be finished before booting up the server
    fs.writeFileSync(path.join(__dirname, 'gcloud-credentials.json'), process.env.SERVICE_ACCOUNT_JSON);
    
    // create Cloud Vision client
    const visionClient = new ImageAnnotatorClient();
    EOT
    
  6. Let’s initialize an Express app and create an uploads folder.
    cat <<EOT >> index.js
    // create express app
    const app = express();
    
    // define path for file uploads
    const uploadPath = path.join(__dirname, 'uploads');
    
    // create the upload folder if it doesn't exist
    if (!fs.existsSync(uploadPath)) {
        fs.mkdirSync(uploadPath);
    }
    
    // configure multer to use the uploads folder (utilized in our post request handler)
    const upload = multer({ dest: 'uploads/' });
    EOT
    

Step 6 - Handler for Image Uploads

  1. Setup a skeleton post request handler
    cat <<EOT >> index.js
    // handle post requests with images to the /upload path
    app.post('/api/upload', upload.single('image'), async (req, res) => {
      try {
        if (!req.file) {
            res.sendStatus(500);
            return;
        }
    
        // handler code
      } catch (err) {
        console.error(err);
        res.sendStatus(500);
      }
    });
    EOT
    
  2. Extract the uploaded file path using the multer middleware
    cat <<EOT >> index.js
        // get the file path uploaded via multer
        const filePath = req.file.path;
    EOT
    
  3. Here is the meat of the application... sending the uploaded images to the Google Cloud Vision client!
    cat <<EOT >> index.js
        // send the image to gcloud for label detection
        const results = await visionClient.labelDetection(filePath);
    EOT
    
  4. Google Cloud sends back a lot of information about the image. We are only interested in its labels. Here's how we get them:
    cat <<EOT >> index.js
        // pull label data out of the response from google
        const labels = results[0].labelAnnotations.map(x => x.description.toLowerCase());
    EOT
    
  5. Let's send back the labels to the frontend for display to the user:
    cat <<EOT | perl -pe 'chomp if eof' >> index.js
        // check if labels have been assigned
        if (labels.length) {
          res.status(201).json({ message: 'Labels were successfully detected!', labels });
        }
    EOT
    
    Note the use of the syntax | perl -pe 'chomp if eof'. This removes the newline at the end of the heredoc. This is so we can easily append an else block to our if block in the next step.
  6. And in the small chance Google Cloud Vision does not detect anything, let's delete the image and let the user know:
    cat <<EOT >> index.js
     else {
          // remove this bizarre, undetectable image from our server
          await unlinkAsync(filePath);
          res.status(400).json({ message: 'Nothing was detected in this image.  Deleted!' });
        }
    EOT
    
  7. Finally, let's close up our post request handler:
    cat <<EOT >> index.js
      } catch (err) {
        console.error(err);
        res.sendStatus(500);
      }
    });
    EOT
    

Step 7 - Handler for Image Retreival

Now we’ll be able to upload images to the server. Now we also want to display our images! First we need a get request handler for retrieving individual images. This is a simple function which simply sends back a file from the server based on the image ID.

  1. Create a route handler to retreive images from the images directory:
    cat <<EOT >> index.js
    // handle requests to individual cats
    app.get('/api/images/:id', (req, res) => {
    EOT
    
  2. Grab the id parameter from and use it to complete the image path.
    cat <<EOT >> index.js
      const { id } = req.params;
      const imgPath = path.join(uploadPath, id);
    EOT
    
  3. Send the image back to the client and close the handler.
    cat <<EOT >> index.js
      res.sendFile(imgPath);
    });
    EOT
    

Step 8 - Handler to Return All Images

Next, we need a way to get the feed of latest uploaded images. We’ll send back the most recent 10 images uploaded to the server. To do this, write another get request handler.

  1. Create a route handler to retreive the last 10 uploaded images
    cat <<EOT >> index.js
    // handle get requests to retrieve the last 20 uploaded images
    app.get('/api/images', async (req, res) => {
      try {
    EOT
    
  2. Obtain a list of all images in the upload directory
    cat <<EOT >> index.js
        // read our uploads directory for files
        const files = await readdirAsync(uploadPath);
    EOT
    
  3. Prepare to call our promisfied stat function for every image in the files array. Since we need to wait for multiple promises, we can wrap them in Promise.all and then await:
    cat <<EOT >> index.js
        // read file stats asyncronously
        const stats = await Promise.all(
          files.map(filename =>
    EOT
    
  4. Call stat to obtain a set of details for each file and return the filename string and stat object.
    cat <<EOT >> index.js
            statAsync(path.join(uploadPath, filename))
              .then(stat => ({ filename, stat }))
    EOT
    
  5. Close the map and Promise.all methods:
    cat <<EOT >> index.js
          )
        );
    EOT
    
  6. Sort the files chronologically
    cat <<EOT >> index.js
        // sort files chronologically
        const images = stats
          .sort((a, b) => a.stat.mtime.getTime() - b.stat.mtime.getTime())
          .map(stat => stat.filename)
    EOT
    
  7. Sent the images back to the client
    cat <<EOT >> index.js
        res.status(200).json({ images, message: 'Images successfully retreived!' });
    EOT
    
  8. Catch any file system errors and close the route handler
    cat <<EOT >> index.js
      } catch (err) {
        console.error(err);
        // if there's an error, just send an empty array
        res.status(500).json({ images: [], message: 'Internal server error' });
      }
    });
    EOT
    

Step 9 - Complete Node.js Web Server

Finally, finish off the server with the code we previously had for serving the built react site and listening on the proper port.

  1. Serve static frontend from all other routes
    cat <<EOT >> index.js
    // serve static frontend from all other routes
    app.use(express.static(path.join(__dirname, '../client/build')));
    EOT
    
  2. Start the server
    cat <<EOT >> index.js
    // start the server
    const port = process.env.PORT || 8080;
    app.listen(port, () => console.log(`Server listening on port ${port}`));
    EOT
    

That’s it! Our server is ready to go. Continue to the next step where we build our frontend. If you want to test the server so far, you can use Postman to test out these endpoints.

Step 10 - Build the Frontend in React - Presentation Source Code

We have a backend going, so now it’s time to build a nice frontend for ImageLabeler. Change the directory back into the client folder and install two dependencies: http-proxy-middleware, which we’ll use for development, and reactstrap, which will make it easy to build a nice interface.

  1. Navigate into the client directory:
    cd ../client
  2. Install http-proxy-middleware, reactstrap, and bootstrap:
    npm i http-proxy-middleware reactstrap bootstrap
  3. Remove all the demo files in the src folder, since we’ll be creating our own from scratch.
    rm -rf src/*
  4. Move inside the src directory and create an index.jsx file to act as the homepage.
    cd src && touch index.jsx
    cat <<EOT >> index.jsx
    import React from 'react';
    import ReactDOM from 'react-dom';
    import App from './app';
    import 'bootstrap/dist/css/bootstrap.min.css';
    import './index.css';
        
    ReactDOM.render(<App />, document.getElementById('root'));
    EOT
    
  5. Also, create a proxy file, which will tell our react-scripts to proxy any requests to the /api route back to the server running on localhost:8080. (This only impacts development, but it’s important to have in order for our API calls to work locally.)
    touch setupProxy.js
    cat <<EOT >> setupProxy.js
    const proxy = require('http-proxy-middleware');
    
    module.exports = function(app) {
      app.use(proxy('/api/**', { target: 'http://localhost:8080' }));
    }
    EOT
    
  6. Next, create a CSS file to hold some basic CSS. (Yes, I realize there are better ways to use styles in React (such as Styled Components or Radium), but those are outside the scope of this tutorial.
    touch index.css
    cat <<EOT >> index.css
    html, body {
      margin: 0;
    }
    
    .crossed {
      display: inline-block;
      width: auto;
      position: relative;
      overflow: hidden;
    }
    
    .crossed:before, .crossed:after {
      position: absolute;
      content: '';
      background: red;
      display: block;
      width: 100%;
      height: 30px;
      -webkit-transform: rotate(-45deg);
      transform: rotate(-45deg);
      left: 0;
      right: 0;
      top: 0;
      bottom: 0;
      margin: auto;
    }
    
    .crossed:after {
      -webkit-transform: rotate(45deg);  
      transform: rotate(45deg);
    }
    EOT
    
  7. Lastly, we’ll need a button to click which will upload our images. We’ll create an UploadButton component, which accepts a loading prop to make it disabled while loading. This will provide some nice feedback to the user while uploading images.
    touch upload-button.jsx
    cat <<EOT >> upload-button.jsx
    import React, { PureComponent } from 'react';
    import { Button } from 'reactstrap';
    
    class UploadButton extends PureComponent {
      fileInputRef = React.createRef();
    
      handleClick = () => {
        this.fileInputRef.current.click();
      }
    
      render() {
        const { children, loading } = this.props;
    
        return (
          <div>
            <input
              ref={this.fileInputRef}
              accept="image/*"
              style={{display: 'none '}}
              type="file"
              onChange={this.props.onUpload}
            />
            <Button
              color="primary"
              disabled={loading}
              onClick={this.handleClick}
            >
              {children}
              </Button>
          </div>
        )
      }
    }
     
    export default UploadButton;
    EOT
    

Step 11 - Build the Frontend in React - Controller Source Code

Now, it’s time to create the meat of our frontend application. The App component will handle all the API calls and display the main content of Image Analyzer.

  1. First, start by importing what we need, including our newly created UploadButton component, and set up the App component with some state.
    touch app.jsx
    cat <<EOT >> app.jsx
    import React, { Component } from 'react';
    import { Nav, NavItem, NavLink, Navbar, NavbarBrand } from 'reactstrap';
    import UploadButton from './upload-button';
    
    class App extends Component {
      state = {
        loading: false,
        success: null,
        message: '',
        preview: null,
        cats: []
      };
    EOT
    
  2. We'll give this component a function to fetch images from the server. This will fetch a list of the latest 20 images from the /api/images endpoint, then individually fetch each image and shift it into the component state. We’ll also run this function when the component mounts.
    cat <<EOT >> app.jsx
      componentWillMount() {
        this.fetchImages();
      }
     
      fetchImage = (id) => {
        return new Promise(async (resolve) => {
          // fetch the specified image from our server
          const res = await fetch(`/api/images/${id}`);
          const imageBlob = await res.blob();
          // create an object URL to display in an <img> element
          const url = URL.createObjectURL(imageBlob);
          // shift the image into state
          this.setState(prevState => ({
            images: [{ id, url }, ...prevState.images]
          }), resolve);
        })
      };
     
      fetchImages = () => {
        this.setState({ images: [] }, async () => {
          const res = await fetch('/api/images');
          const { images } = await res.json();
          for (const image of images) {
            await this.fetchImage(image);
          }
        })
      };
    EOT
    
  3. Now we can receive images, but we need a function to upload them. This handler will be used in our UploadButton’s onUpload event prop, which triggers when a file is selected. Here, we create some FormData from the selected file, update the state to loading (for our loading spinner on the UploadButton), and send the file to the server. If the image file cannot be labeled by Google, we’ll display it as a crossed-out preview to show the user that he or she cannot upload images that are unidentifiable.
    cat <<EOT >> app.jsx
      handleUpload = async (event) => {
        const file = event.currentTarget.files[0];
        const formData = new FormData();
      
        // show loading spinner
        this.setState({ loading: true, preview: null, message: '' });
      
        // add the file to the form data
        formData.append('image', file);
    
        try {
          // send the form data to our server
          const res = await fetch('/api/upload', {
            method: 'POST',
            body: formData
          });
     
          // parse the server response as json
          const { message } = await res.json();
          // we should receive a 201 response if successful
          const success = res.status === 201;
          this.setState({ success, message });
     
          // read the uploaded file
          const reader = new FileReader();
          reader.onload = (e) => {
            console.log(e.target.result);
            if (success) {
              // shift the uploaded cat onto the state
              this.setState(prevState => ({
                cats: [{ id: prevState.cats.length, url: e.target.result }, ...prevState.cats]
              }));
            } else {
              this.setState({ preview: e.target.result });
            }
          }
          reader.readAsDataURL(file);
     
        } catch (err) {
          console.error(err);
        }
     
        // hide loading spinner
        this.setState({ loading: false });
      };
    EOT
    
  4. Finally, we want to rendering our Element into the browser's DOM. The render function in our App component combines everything together and binds all the proper state values to the page.
    cat <<EOT >> app.jsx
        return (
          <>
            <Navbar color="light" light>
              <NavbarBrand href="/">Image Analyzer</NavbarBrand>
              <Nav>
                <NavItem>
                  <NavLink href="https://github.com/sbreese" target="_blank">GitHub</NavLink>
                </NavItem>
              </Nav>
            </Navbar>
            <div style={{ padding: 32 }}>
              {message && <h6>{message}</h6>}
              {preview && (
                <div className="crossed">
                  <img src={preview} alt="upload preview" style={{ maxHeight: 300 }} />
                </div>
              )}
              <UploadButton
                onUpload={this.handleUpload}
                loading={loading}
                success={success}
                >
                Upload Image
              </UploadButton>
              <br />
              <br />
              <hr />
              <br />
              <h6>Recent images:</h6>
              <br />
              {images.map(image => (
                <div key={image.id}>
                  <img src={image.url} alt="image" style={{ maxHeight: 300 }} />
                </div>
              ))}
            </div>
          </>
        );
      }
    }
     
    export default App; 
    EOT
    

Step 12 - Deploy your App to Heroku

  1. Commit and push your changes to Heroku master and watch as it automatically deploy. Very slick!
      $ git commit -am "making it better" && git push heroku master
    

Congrats!

Great job! We’ve now built a fully featured Image Analyzer, both front and back. You saw how easy it is to use the Vision API, and the kind of utilities that it provides that to are at a remarkably less cost. Be sure you check out Cloud ML, which is the whole suite of Machine Learning APIs that Google provides.

Heroku makes it simple to automatically deploy fast and scalable applications directly connected to your local Git repository.