Uploading measured data using Socket.IO

Task

The current task is it to upload a .csv file to the web server using Socket.IO. Socket.IO is a realtime framework for web applications, but can be used for communication between two NodeJS instances as well. It provides improved communication mechanisms over websockets and some fallback options as well.

What we want to implement will consist of a client, which will read data from a .csv file and transmit this data row by to the web server, which will receive the data, parse and transform it into JSON and then save it in the database.

The Client

The client consists only of a few lines of code. A connection to the server will be established. After that, the file will be read and the content will be split up into rows (by splitting between each new line: ‘\n’ ). Then each row will be sent via the websocket to the server. After sending each row, a ‘done’ sequence will be sent as a simple workaround for the server to see whether the whole file was sent.

var io = require('socket.io-client'),
 fs = require('fs');
 _ = require('lodash');

var socket = io.connect('http://localhost:3002');

fs.readFile('test_data/newFormat.csv', function(err, data) {
   data = data.toString('utf-8');
   data = data.split('\n');
   console.log('Scanned file with '+data.length+' rows');
   _.each(data, function(row) {
     socket.emit('upload', row);
   });
   socket.emit('upload', "####done####");
});

The Server

The server consists of a simple http server, on which the Socket.IO framework will listen for incoming connections. For each connection, a new socket will be opened and each it will receive the data row by row. The data is then parsed into JSON and stored in an array in order to buffer. Only if 1000 elements are in our buffer, the content will be saved in the database. This is because MongoDBs maximum bulk insert size is 1000 objects, and these settings resulted in the fastest execution of both receiving and storing the objects.

'use strict';
var app = require('express')();
var server = require('http').Server(app);
var io = require('socket.io')(server);
var _ = require('lodash');
var Promise = require('bluebird');
var mongoose = Promise.promisifyAll(require('mongoose')),
Schema = mongoose.Schema,
Measurement = Promise.promisifyAll(require('./model/Measurement'));

mongoose.connectAsync('mongodb://localhost/roadstar_csv')
.then(server.listen(3002));

io.on('connection', function (socket) {
    socket.on('upload', receiveData);
});

var buffer = [];
var ops = [];
var firstTime = true;

function receiveData(chunk) {
     var op;

    if(firstTime) {
       console.time('receiving rows');
       console.time('writing to db');
       firstTime = false;
    } else if(chunk === '####done####') {
       op = Measurement.collection.insertAsync(buffer);
       ops.push(op);
       buffer = [];
 
       console.timeEnd('receiving rows');
       Promise.all(ops)
       .then(function() {
         console.timeEnd('writing to db');
         firstTime = true;
     });
   } else {
         try {
             chunk = dataToJSON(chunk);
             if(buffer.length < 1000) {
                 buffer.push(chunk);
             } else {
                 op = Measurement.collection.insertAsync(buffer);
                 ops.push(op);
                 buffer = [];
                 buffer.push(chunk);
            }
         } catch (err) {
             console.log(err);
         }
     }
}

Recording a test ride

To test the GPS receiver and to get a first feel for handling the messages sent by it, we did a short test ride on bus line 63 in Dresden, Germany. While recording a message log using the official u-center Windows software, we were also able to observe the current speed, location and course.

What we have

After the ride, we extracted the recorded GPS data by simply reading out the file created by u-center when recording was activated. It turned out, that this file was basically a log containing the messages from the receiver: Read More

Testing MPU-6050 with Node.js and Websockets

After setting up and testing the hardware we need to get going on the software side of the measurement setup.

We decided that each of our four MPU-6050 sensors gets its own RPi (Model B) for sensing the accelerometer and gyroscope data. These samples get directly transferred to a Server-RPi (Model 2-B) where they receive a timestamp and are stored for later use. As a protocol we keep it simple and use websockets for the beginning. We are going to switch to a UDP protocol if the network delays get problematic.

Node.js library for the MPU-6050 sensor

Fortunately there are npm-packages for our sensor out there, let’s try to install them:

  • npm mpu6050 (didn’t compile)
  • npm i2c-mpu6050 (worked!)

Using the example code in the i2c-mpu6050 package provided sensor data with ~35 Hz. After hacking around a bit it went up to ~46Hz. As we don’t need the temperature provided in the package’s regular index.js we uncommented those code lines and got a sensing frequency of ~55Hz (RPI Model B) which should work for us (if the car drives max. 160km/h we still have a sample at least every 80cm). With a RPI 2 we would get ~76Hz.

First sample data

To get a first idea of the sensed data, the delays and if a quake of the underground can be seen in the data we did put 4 RPis with sensors on a common ground and shaked it (by knocking) 5 times while sensing with 55 Hz and sending the data to our server. Here are first results:

received_samples_booklet5_4PI-PI2B-55Hz_1PIsample

received_samples_booklet5_4pi-pi2b-55hz_allsamples1

With a sensing frequency of 55Hz a difference between the timestamps up to 18ms is regular.

Size of stored data

The data volume we produce and need to store for 4 sensors with 55Hz is ~100MB per hour, thus our class 6 SD card should be able to manage that.