Lessons, Surprises, Mistakes II.

Next set of lessons, surprises and mistakes; this time various issues in no particular order.

Hardware issues

We use some cheap Android Sencor Element tablets. From 10 tablets, 2 were faulty and we had to replace them. You can recognize faulty tablet by its freezing. Yet there were also more interesting manifestations (appeared only on one tablet):

• Time speeding up by minutes per hour. That produced transactions and shifts with wrong time. Automatic time synchronization does not help as I assume it is synchronized only once per day.
• Touch events were not registered in some areas of the screen or were shifted (touching on one place triggers action from different place).

User feedback

It's always good to observe and talk to your very end users. Based on seeing the POS in action and talking to one cashier, I realized the buttons were too small. The cashier had hard time to aim at the right button. So I've made all buttons and texts as big as possible:

Wrong development browser size

I use Chrome Resolution Test plugin to easily resize browser window to tablet's resolution (1024 x 786). But I made a stupid mistake. The POS web page on tablet spreads across the full tablet screen (minus Android menu bar) while the plugin resizes the whole browser window to the specified size. So page areas were different. To have the same area in browser and on tablet (1024 x 720), the correct resolution for the plugin is 1034 x 826.

Browser zooming

Cashiers had also problem with browser zooming triggered by double tap. For a long time, I did not know how to disable it because I thought it is as a browser functionality and looked for an option in browser. Eventually, I've found out it is a HTML5 meta viewport property and you can turn it off by:

<meta name="viewport" content="user-scalable=0" />

JavaScript caching

I was surprised that browser loads JavaScript files from cache even when you ask to bypass it (e.g. by Ctrl+F5). I have not found a way how to force reloading JavaScript files. As a fix, I append a timestamp to main.js in the release package (and index.css because it can be cached too).

Automatic updates

Uptake of a new version was slow. It was normal that an old version was used several days after the new version was released because page refresh was needed on POS and only company employees could do it. Luckily, there was no critical update needed.

I've implemented automatic updates like this: successful ping request returns the actual server version. When it comes back to a tablet, it means there is internet connection (necessary because of mobile internet - see the previous post). When the server version does not match with the client version and there is no cashier signed in, the web page is refreshed which performs the update. The server version is set manually as part of release deployment.

Lessons, Surprises, Mistakes I. - Internet Connection

This is the first of several posts about my mistakes, surprises and lessons learned. I'm always eager to see other's experiences so I can learn from them. Here are mines so you can learn from me.

Going Live

1st April morning. It kind of works. Most kiosks are online and I see signed in cashiers and some transactions. Some kiosks are offline. I guessed internet connection problems.

It was very similar during the next couple of days. There were rare moments when all kiosks were online. Most of the time at least one kiosk was offline. We discussed the cause of the issues and it was clear that the culprit was the internet connection. The mobile internet is ***. It disconnects every now and then, it drops packets, it is sloooooow. Especially in the kiosk which is a big metal box. I had to mitigate the problems.

Status emails

First step was to have better visibility of the problems. Having status page with kiosks overview was nice but it was not enough - nobody periodically checked it. That's why I've implemented status emails. AMS offers scheduled jobs and I use them to periodically check kiosk statuses and send warning email with problematic kiosks. Initially, the check was every 30 minutes but there were too many emails and nobody cared about them. Currently, the status is checked once per hour during the opening hours and a warning is sent only when kiosk does not ping for more than one hour.

Automatic router restarts

We've discovered the biggest issue in routers. They disconnect from the internet quite often. Luckily for us, routers have monitoring functionality. They can ping specified servers and if no reply comes back, they restart themselves. Once turned on (with 30 minutes threshold), the internet reliability increased dramatically.

Inability to sign in/out

Pages in Durandal are stored in separate files. That means there has to be internet connection to successfully navigate to another page (e.g. starting/ending shift). I thought (but not tested) that browser would cache all pages. But I was wrong and with such bad internet connection, some cashiers could not sign in or out - the screen just turned white because the target page could not be loaded.

The solution is to build the whole app into three files (index.html, index.css, and main.js) which are loaded at once during POS initialization. To do that, I use grunt-durandal and grunt-uncss tasks when creating release package. The only communication then is sending data to server which is backed up in local storage.

Double transactions

Another sign of bad internet connection is double transactions. POS sends data, data is written into database, but the confirmation is lost on a way back and POS sends data again. I have actually seen several triple transactions in the database. To solve this issue, I have added a check before inserting a new data into database.

It did not fully mitigated the issue though. I'm still seeing some duplicate transactions. AMS provides createdAt column and according to this column these duplicate transactions are usually milliseconds apart or even at the same time. I have no idea how these duplicate transactions are created. It can be strange network behavior, bug in AMS…

Success transaction visibility

This is a small tweak. POS displays whether the last server operation was successful or not. It's not for cashiers because they don't care. It's mainly for company employees when they check the kiosk on site to know that everything is OK.

Summary
After implementing above fixes, the reliability is quite good. It is rare to see offline kiosk. There is still some work I'd like to do though. For example, I do not properly handle authentication token expiration so somebody has to manually re-login kiosks every ~30 days otherwise they would be offline because of missing authentication.

POS Architecture

The initial design proved itself to be working nicely so I've stuck to it:

• Android tablet in each kiosk connected to the Internet via 3G Wi-Fi router. Tablets with 3G modem are more expensive and every kiosk had already been equipped with 3G Wi-Fi router because of the previous system.
• HTML SPA (Single Page Application) as client POS application running in the browser on the tablet
• JavaScript Azure Mobile Services (AMS) with SQL database as server

The software architecture is guided by using Durandal (MVVM design pattern) on the client and AMS on the server. I decided to use Durandal because I like the testability of MVVM applications and AMS because of its easy use.

POS as web page
The whole POS software runs as a HTML5 application in full screen browser. Cashiers cannot use other apps on the tablet because the special tablet holder hides main Android menu, all hardware buttons and prevents edge swiping which is the only way how to control the browser. Here is the picture of the real device:

Authentication

All AMS endpoints require authenticated user. AMS supports several authentication providers. I've chosen Google provider as all company employees have Google account. Special Google account has been created for every kiosk. The kiosk account is signed in in the POS browser all the time so no cashier needs to know its password.

Authorization

There are only two groups of users:

1. Cashiers
   They can only insert data (transactions, shifts). Every cashier has assinged a PIN which is used to start her shift.
2. Company employees
   Additionally to cashiers, they can also edit some data (cashier management) and read the data (reports). As there are only 3 employees at this moment, their Google ids are hardcoded in scripts.

Offline mode

Because kiosks are connected to the Internet via mobile network, I expected they can be offline from time to time. That’s why every operation is first saved to browser local storage and only when server confirms it is saved in the database, it is deleted from the local storage. When confirmation comes, then next unsaved operation is sent to the server until the queue is empty. Operations are sent one by one to not trigger multiple requests at the same time (imagine sending all transactions from one day at once).

Operations

As the launch day approached, I started to wonder/worry how to monitor the health of the kiosks. There is no way to connect to them from the Internet and check the POS status. So I did it other way round. Every POS pings the server in specified interval (10 minutes) and reports its status (signed in cashier, locally stored operations, POS version). Company employees and I have access to the web page which displays statuses of all kiosks. The most important information is whether kiosk is alive (pinging) or dead. Repair is simple - restart tablet and/or router.

Release package

Grunt task for creating release package contains steps to uglify JavaScript and uncss/mincss (uglify for CSS rules). Initially, the Durandal pages were separate files even in the release package but I had to change it (more on that later). Now, the whole POS consists of three files: index.html, index.css, and main.js.