Tuesday, October 7, 2014

Maker Faire Rome 2014

Maker Faires have been around for a while now in the US but they're new to Europe, last weekend I decided to pop down to the second year of Maker Faire Rome. It was super - the scale was amazing (I wonder what the NY one must be like) and the level of enthusiasm from all the Makers was great, and from the big corporates too.

Everyone and his dog seemed to have a new 3D printer or something in some way connected to 3D printing - Wasp Italy (who I'd never heard of) had the most impressive setup, in this area, with a whole tent full of printers working away on everything from the standard small plastic items (the little red monster being created in the picture above) to large concrete constructions.

But BLE is my area of obsession at the moment. There are lots of short-run one off and no-name BLE boards (often with limited UART like features) - Redbear are about the only people who make boards that are really readily available from multiple suppliers and which support pretty much everything you'd want to do. I already have one of their BLE Mini boards but at the Faire I also picked up these two:
Just look at the size of them compared with a standard USB connector! The new BLE Nano, on the left, which is a Nordic nRF51822 SoC and the Blend Micro, on the right, which is an Arduino plus BLE. The Nano is similar to the older BLE Mini in that it has central support but unlike the Mini it looks like you really can program the SoC yourself without (with the Mini you have to talk to the SoC with something else unless you're willing to shell out for TI's industry priced development tools).

At the IoT Zurich stand I met Michael Kroll and Matthias Ringwald. Michael is one of those people whose name just keeps on popping up if you're looking into things BLE related (him and Jeff Rowberg) and he has joined up with Matthias to produce some new BLE shields where the Bluetooth stack runs on the host system, i.e. your Arduino or whatever, rather than being hidden away in proprietary firmware behind some UART or SPI based interface. The stack takes up a noticeable amount of memory (so you probably want something a bit bigger than an UNO) but you do get full control of the stack and can hack it as you want if required rather than having to treat it as a black box as you do with all other such products. It was good to talk briefly with Michael and Matthias and I hope I get to see them again at a future IoT Zurich meetup :)

Before I flew down to Rome I swore I wouldn't buy an Intel Edison. I think it's a cool piece of hardware but I already have way too many devices that I need to devote proper time to before I can justify buying yet another platform. But on seeing it in reality (and considering it's impossible to get online) I cracked and got one with a mini breakout:
I'm surprised at some of the flack the Edison has received since launch. It's an amazing little device. With BLE, wifi, 1GB of RAM and 4GB of eMMC flash, two Atom cores and a Quark core in one tiny package it does seem wildly overspec'd compared to most other IoT offerings. I certainly don't care that it's not quite SD card sized (Electric Imp went with this form factor initially but on the whole once people have got started with the Imp I think most people end up wanting more GPIO and go with something like the P3V3). At USD$50 you're probably not going to use the Edison in your slightly-smart lightbulb Kickstarter but it'll be amazing for prototyping, for demonstrating initial ideas in action, for one off items and for IoT devices that do require real power. Most hobbyist devices used to be 5V devices but 3.3V has become quite common and now the Edison is 1.8V so it's time to start buying yet more level shifters! As I understand it the Quark core isn't currently accessible to developers which is a bizarre (I presume one was meant to run Linux of the Atom cores and timing critical classical embedded stuff on the Quark core) but I guess this will change with a future update.

Intel found the right guy to be enthusiastic about Edison. Irishman David Hunt describes himself as maker, photographer, climber and coder on his card. He works for emutex in Limerick where they produce embedded systems drivers and software for various companies including Intel. David demonstrated his own projects, including the next generation of the CamIO project that he'd shown off last year in Rome, this time using the much smaller Edison in place of the Galileo board used previously.
Half the people at the Faire seemed to have completed a Kickstarter - it really made you realize how amazing Kickstarter has been in enabling a lot of the innovation seen today.

There were many crowd pulling items at the Faire, like life size androids, that have been covered by the press elsewhere (e.g. Forbes) and many cool, if sometimes slightly disturbing!, things like a 3D printed hexapod robot. The following are a few of the generally smaller, less immediately dramatic things that were interesting in one way or another for me.

Various people were printing circuits. The Fingies people at Bauhaus-Universität Weimar managed to bring together printed circuits and the IoT with sensor cards (that could be joined together electrically through magnets) hooked up to a non-printed wifi central component. Fujifilm were there with the Fingies people demonstrating their impressive Digimatix materials printer that was used to print the circuits.

I'd love to have my own pick-and-place machine. The guys at BotFactory not only have a desktop pick-and-place, the machine also prints the PCB onto which the parts will be placed! BotFactory have recently got their Squink project Kickstarter funded. At USD$3,000 per device it's at the very high end in terms of price for a Kickstarter technology project, but when you see what it's doing the price is fairly understandable. Line up your individual SMD components as best you can on the feeder tray and their camera technology will work things out, picking up the items and rotating them properly to compensate for any clumsiness on your part (these items are tiny). It can only do so much at the moment - the PCBs are printed paper so no multilayer designs, the smallest resistors it can handle are 0603 (pretty small by most people's standards) and while it can handle SOIC it can't handle say QFN (see the Adafruit PCB ruler for a good guide to these sizes).
Lots of people seem to be coming to the conclusion that BLE and wifi make a good combination when it comes to IoT, e.g. the already mentioned Edison or the recent McThings Kickstarter. Berlin based Wunderbar does this as well and brings together a main module with wifi and BLE and six mini sensors (most with multiple functions, e.g. humidity plus temperature) in a really cool chocolate bar style packaging. And unlike McThings (and too many other people) they don't seem to gone with yet another makey-up scripting language but have proper normal APIs. Like most of these things it comes with its own cloud framework. I must admit initially I preferred the idea of devices that didn't require any additional infrastructure (what happens to your device if the manufacturer goes bust and the cloud service goes offline?) but I've flipped and now on the whole think the cloud can solve more problems than it creates for IoT devices. Note: Wunderbar uses the GainSpan GS1500 module - I'd be interested in how this compares to something like the TI CC3200 seen in a lot of other projects coming out now.

Among the many people there who'd completed a Kickstarter were the Munich based Mr. Beam team who were showing off their hobbyist laser cutter - they told me it was coming along great and would be waaay better than the apparently similar LazerBlade Kickstarter that got funded at almost exactly the same time. I'm sure there's just friendly competition and both teams do wish each other the best :)
Both TI and ST had enthusiastic and interesting representatives showing off their products suitable for hobbyists. ST had their super cheap mbed compatible Nucleo boards (USD$10 for 96KB RAM, 512KB flash ARM M4 SoC board!) along with various add-on boards (e.g. a BLE one). While mbed compatible their representative did feel you'd get more out of them using their own STM32Cube software (that includes FreeRTOS). TI were showing off their LaunchPad evaluation kits - including the CC3200 one. They were pretty up front that the CC3000 had in many ways been a rather sucky product and swore the CC3200 was a massive improvement. They also pointed me at Energia - a platform that brings the Arduino framework to LaunchPad and provides a more hobbyist friendly environment than the traditional TI tools (or the environments like IAR that they generally recommend).

Atmel unsurprisingly were a big sponsor. Thank you for the free ATmega328P Xplained Mini - they were giving this away to anyone who showed enough interest their stuff. The Xplained Mini is pretty cool - it's very like an Arduino but with proper debugger support without requiring a separate expensive JTAG debugger - just hook it up via USB to Atmel Studio. I haven't tried it out but they said you could build and upload Arduino sketches from Atmel Studio and set breakpoints etc. in the code that's running on the MCU on the Mini board. I really liked this feature in LPCXPresso - shame that, unlike LPCXPresso, Atmel Studio doesn't run on Mac or Linux, it's just available for Windows.

The ridges that you see on all hobbyist 3D prints detract a lot from the finished product. The 3DFinisher crowd think they've come up with a solution - you place your completed printed item in a sealed glass container where it is exposed to a hot vapor that removes the ridges by both slightly melting the item's surface and by applying a thin coating. The results do look nice - but I suspect it may result in over smoothing for certain applications.

Despite my interest in wifi for embedded system I've never really looked into Carambola, so it was good to get a chance to talk with the guys from 8 devices and see their new NUY shield which allows you to add Yún like capabilities to your existing Arduino board.
Who doesn't need their own CNC router? After recently doing a one-day course on using a full size CNC machine I was interested to see a number of table top hobbyist CNC routers - in particular one from Makesmith, a group of nice guys from SF. I'd love to have see their 3D (yes they said 3D, when I said 2.5D) model of SF but unfortunately someone had walked off with it an hour earlier :(

Autodesk seem to be trying all sorts of things at the moment - so you can't quite be sure what's just a fun idea and what will really still be around, properly supported and developed in future years. Having said that 123D Circuits looked interesting - construct your Arduino controlled circuit, in a very Fritzing like web app, but then go on to write the Ardunio code for it there too, simulate it and then lay it out and order it as a PCB - all in one app.

Need an 8x8 grid of buttons - each of which can be lit with whatever color you want and the whole thing looks beautiful? Try bhoreal - funded via the Spanish crowdfunding site Goteo. Kind of like the Adafruit UNTZtrument but with full color!

Like Wunderbar the Apio team bring together a central controller that talks to the world via wifi and with its slave devices via BLE. Unlike Wunderbar its not about sensor but about controlling things. Developers or even end users to hook things together with triggers and such like. The central node's relationship with other nodes is somewhat like the IoT equivalent of the Logitech Harmony remote controls that can interact with a whole load of AV devices etc. to complete a given task, e.g. "boot" your home cinema setup. The Apio team skipped the hassle of separate apps for iOS, Android etc. and went with a HTML5 web app (I know that didn't work out for Facebook a few years back but presumably things have gotten better since then).

A drone made out of cardboard? There was a guy there called Angelo Aussiana who was making pretty much anything out of cardboard including drones. He's trying to get a mass production run funded via Indiegogo but he seems to have all the work done and had kits at the show (you can see all the bits being popped out of a couple of pieces of card in the video on the Indiegogo project page).

There were various manufacturers there showing off their wares and their ability to turn maker ideas into mass produced products, but I'll choose just one to mention - Seeed. They were showing off tons of little items that they've helped bring to the Maker community. They're the people who've done the actual manufacturing for all kinds of third parties like Spark Core and Redbear Lab. If you're looking for something they probably make it. And if you want something made they'll make it for you - I'd love to try out their maker targeted PCB assembly service one day, not only will they print your PCB (up to 4 layers) in low volume (minimum quantity 2) they'll also place any of the standard components from their open parts library - so even if there are some specialist parts you're going to solder yourself you can get all the standard resistors etc. done for you (and using SMD components that you'd be hard put to solder yourself).
For cool artistic output I really liked the DoubleCLICK project from Officine Sistemiche - I wish I'd taken a photo. It reminded me of the technology behind lightpainting. It consisted of a a small cart that could be pushed around by hand over a piece of paper. The cart contained a pen and how hard the pen was pushed against the paper was motor controlled. This cart, with pen and motor, was connected by a long flexible cable to a main machine. Working from a gray scale photo an application there, knowing the location of the cart at all times, would adjust the pressure applied to the pen according to the gray scale value appropriate for the current location relative to the original image. The pen was never lifted entirely from the paper so you got a continuous line of varying thickness as you moved the cart. The original image was clearly visible if you pushed the cart around enough, and you could produce very nice effects by pushing the cart around in a structured manner, e.g. back and forward but not too close between lines. The DoubleCLICK project page describes a somewhat different system where the pen basically has just two modes - either on or off the paper and it's not just the the cart that moves the pen in the x-y plane but also the servos connected to the pen so the whole effect is achieved in quite a different way to just varying the line thickness drawn. I hope I wasn't hallucinating about the behavior I saw at the Faire :)

Andrea Bruno has a project to add NFC to phone's that only have Bluetooth (i.e. every iOS device until the 6s and even there the NFC APIs aren't exposed to developers for anything other than Apple Pay). Though actually Andrea's project currently just works with Android.
Roland has a lot of what they called 3D modelling machines. Unlike most of the 3D printers on display these were non-hobbyist machines that could produce really professional looking consumer ready end products out of all kinds of materials. I particularly enjoyed the chocolates that were being churned out.

Lastly Microsoft must win a prize for bizarre product naming with what they were calling Windows on Devices - but there are no windows as it's Microsoft's IoT offering and intended for headless devices!

I really enjoyed Maker Faire Rome 2014 - the amount of inventiveness and creativity was amazing. If this is anything to go by the future is going to be great! Hopefully I'll get to see the World Maker Faire in NY sometime soon and get to compare.

Monday, July 28, 2014

Imp powered In-System Programmer


The Atmel AVR MCUs are very popular in the hobbyist electronics market, in particular the ATmega328 found in the Arduino UNO. The MCU on an UNO and similar boards is generally programmed via a bootloader, a piece of code that needs to be preinstalled and makes the chip more convenient to program.

I wanted to use an Atmel AVR MCU in combination with an Electric Imp in a circuit and thought wouldn't it be nice if the Imp could completely setup the AVR MCU in place without any preprogramming or pre-existing bootloader being required. This can be done via a technique called in-system programming (see also the Atmel In-System Programming Guide).

Once I started down this route I thought why not try creating a completely general purpose Electric Imp based AVR MCU programmer. This blog entry describes the result.

Simple standalone AVR programmers typically only have the details for a few AVR MCUs hardcoded into their logic. This programmer can query any AVR MCU for its type (signature) and then lookup all the details for programming from an another online service I created (that uses avrdude.conf as a basis and serves up MCU details as JSON).

Many Imp applications come with a fairly basic web interface - I tried to create something a little fancier but still very lightweight using the now ubiquitous Bootstrap. You can also interact with the programmer in a REST like manner.

Note: some AVR MCUs, e.g. the ATtiny85, do not provide hardware bootloader support and can only be programmed using in-system programming (having said that one can emulate bootloader like behavior as Adafruit do with their Gemma and Trinket products).


You can see the programmer's web interface on GitHub. Pressing any of the buttons will just result in an error as it's not being served by an Electric Imp agent, but you can see the various features - which are:
  • Querying and displaying the MCU's signature (along with the name associated with the signature, e.g. "ATtiny85").
  • Uploading new programs (sketches in Arduino terminology) to the MCU.
  • Querying and setting the MCU's fuses and lock bits.


Setup is very easy. You need to wire up your Imp to your AVR MCU. At the top of this page you can see an Imp wired up to an ATtiny85 and down below you can see an Imp wired up to an ATmega328 (in one case the ATmega328 is taken straight from an Arduino and so requires an external clock, a ceramic resonator, and in the other case it's an ATmega328 with factory defaults, i.e. running using its own internal 1MHz clock).

I've used an April with an imp001 with the Imp pins connected as follows:
  • Pin 1 to SCK.
  • Pin 2 to reset.
  • Pin 8 to MOSI.
  • Pin 9 to MISO.
  • 3V3 and ground as required.
You can lookup the MOSI, MISO, SCK and reset pins, along with Vcc and ground pins, in the data sheet for your AVR MCU.

Once the hardware is setup the software side is easy. The Electric Imp agent and device code is available on GitHub. You just need to:
  • Open up the Electric Imp IDE.
  • Create a new model called e.g. "InSystemProgrammer".
  • Associate a real device with the model.
  • Copy the agent code into the agent pane and the device code into the device pane and then save.
If your Imp is powered up you should see output from both the agent and device as shown below. The important thing is the programmer URL (highlighted here with a red border):

Open this URL in your normal web browser. You should see the programmer's web interface and, if you've correctly wired up your Imp to your AVR MCU, you should now be able to query the MCU for its type, reprogram it etc.

Note: the wiring I've shown above for the ATtiny85 could be simpler. I did it this way to match the style seen for the more complex ATmega328 circuits shown below. For the ATtiny85 you could wire 3V3 from the April directly to the MCU's Vcc (rather than using three red wires as shown) and also move the LED's cathode pin so you just need one ground wire (instead of the two black wires shown).


Note: the programmer depends on a service running on Heroku to lookup the name and details of an AVR MCU. This service uses Heroku's free hobbyist tier which means that if the service is unused for more than an hour it is suspended. When this happens it can take ten seconds or more to respond to the next incoming request as it takes a while to come out of suspended state. To avoid a pause due to this when using the programmer you can first click a link, like this one for the details of the ATmega328, to query this underlying service directly for something - just to make sure that it is currently warmed up.

Usage of the programmer's web interface should hopefully be fairly self evident - here's a quick run through.


If you press Query you should see the MCU's signature as three hex bytes along with the MCU name associated with that signature.


At the risk of alienating low level AVR enthusiasts I've called this the Sketch section as this is what programs are known as by Arduino users. Here you can upload the HEX files generated by the Arduino IDE (and other AVR toolchains) to the MCU.
Once you've selected a HEX file (more on these in the next section) and pressed Upload things should whir away for a while (depending on the size of the file) and if all goes well you should see a nice "Programming succeeded" response in green.


This is the dangerous bit - if you don't know what a fuse is then leave this section well alone - random experimenting may well result in getting the MCU into a state in which it is essentially bricked.
You can query the current fuse settings and set new values. By default there are some safety checks built in that only allow you to change bits that should not impact further interaction with the MCU. If you know exactly what you're doing you can tick the "Allow unsafe changes" checkbox and enter any value you want.

The safety checks only work if using an ATtiny or ATmega MCU, other AVR MCUs have different fuse assignments and the safety logic does not attempt to cover all AVR families. The safety checks prevent changes to:
  • the clock source and startup bits of the low fuse - changes here won't really brick the MCU but may force you to change the circuit, e.g. to add in an external clock if the external clock bit is programmed.
  • the reset, debug-wire and SPI enable bits of the high fuse - changes here really will brick the MCU - requiring a high voltage programmer (like the Rescue Shield from MightyOhm) to recover the situation.
Using this feature is entirely at your own risk, I've used it successfully but I'm not guaranteeing that it can't contain bugs that may brick your device even if the values you enter make sense!

Lock bits

Like the fuses you can query and set the lock bits. Lock bits aren't terribly interesting outside commercial settings where you may e.g. want to restrict the ability to retrieve the code running on the MCU.
Note: once a particular lock bit is programmed it cannot be unprogrammed without erasing the chip. The easiest way to do this is to upload a new program as the chip is erased as part of this process.

HEX files

When you compile a sketch in the Arduino IDE it generates files in a format called Intel HEX which can then be uploaded to the board being programmed. Generally this all happens under the covers and you never see these files. All other AVR toolchains can also generate these files (though some may use an alternative format by default).

The GitHub repository for the programmer also contains a few sample HEX files that can be used to demonstrate programming the ATtiny85 and ATmega328 MCUs in the breadboard layouts shown on this page so that they flash the attached LED on and off.

To upload your own programs that you've created using the Arduino IDE you need to be able to find the HEX files it creates. First open the Arduino IDE's Preferences dialog - it will tell you where you can find your preferences.txt file:
It's important to exit the IDE before editing the preferences.txt file otherwise any changes you make will be overwritten when the IDE exits.

First create a new folder where your HEX files will end up, e.g. add a new folder called "build" in your usual documents folder. Then find your preferences.txt file. Unfortunately the standard location for preferences.txt may be hidden by default on your system. If you don't know how to show hidden files just google for something like "how to show hidden files on windows" or "mac" or "linux" as appropriate. Once you've found the preferences.txt file open it in a normal text editor and add the line:
You'll need to replace /full-path/build with the full path to the build folder you just created. If you created it in your documents folder then this will typically be something like:
  • C:\Users\<username>\Documents\build on Windows.
  • /Users/<username>/Documents/build on Mac OS X.
  • /home/<username>/build on Linux.
It doesn't matter where in the file you add this line (in fact if you later reopen the preferences.txt file after running the IDE you'll find the IDE has probably reordered the lines and this new line will end up somewhere random in the file).

Save your changes and restart the Arduino IDE.

Now whenever you press Verify in the IDE it will save the HEX file for your sketch in the build folder you created (along with many other intermediary files that aren't of interest in this situation).
If your sketch was called "Blink" then you'll find a HEX file in this folder called "Blink.cpp.hex"

Important: before you press Verify make sure you've got the right board selected under the Arduino IDE Tools → Board menu. E.g. maybe you've only used the Arduino IDE with your Arduino UNO but now want to generate a HEX file for an ATtiny85 connected to your Electric Imp - so make sure to select the ATtiny85 as the target board. You may need to install hardware descriptions for the MCU if it is not supported by default by the Arduino IDE, e.g. as described for the ATtiny family on High-Low Tech. HEX files generated for one MCU type will not be compatible with another MCU type, e.g. you cannot use files generated for an ATtiny85 with an ATmega328 and vice-versa.

Once you've found the HEX file generated by the Arduino IDE for your sketch you can upload it to your MCU as described in the section up above that describes using the programmer.

Sample HEX files

I've provided a few small sample HEX files with the programmer code that can be used to demonstrate the programmer with the breadboard layouts shown on this page:
  • attiny85_blink_pin_4.hex - this blinks the LED in the ATtiny85 layout above.
  • attiny85_nop.hex - this programs the ATtiny85 to do absolutely nothing and can be used to demonstrate the MCU behavior changing between when programmed to blink the LED and when programmed to do nothing.
  • atmega328_blink_pin2.hex - this blinks the LED in either of the ATmega328 layouts shown below.
  • atmega328_nop.hex - this programs the ATmega328 to do absolutely nothing.
  • atmega328_blink_pin13.hex - this blinks the onboard LED on an Arduino UNO and can be used to demonstrate the programming of an ATmega328 that will be returned to an UNO board.
The blink files are simply compiled from the Arduino IDE File → Examples → 01.Basics → Blink sketch with the pin value in the code changed as appropriate and the correct board selected under Tools → Board. Similarly the nop files are compiled from the 01.Basics → BareMinimum sketch with nothing added.

Programming bits

If you're not used to AVR terminology where they talk of programming and unprogramming bits, rather than of setting or unsetting them, then it may come as a surprise to find that:
  • Programming a bit means unsetting it, i.e. setting it to 0.
  • Unprogramming a bit means setting it, i.e. setting it to 1.
So if e.g. all bits of say the extended fuse are unprogrammed this means they are all set to 1 and the fuse has value 0xff, i.e. 1111-1111 in binary. Similarly if the SPI-enable bit of the high fuse is to be enabled it must be programmed, i.e. set to 0.

Useful libraries

The programmer code base contains a few libraries that may be of general use elsewhere.

Multipart Parser

The standard agent API provides support for dealing with form data that came in via an HTTP GET request but doesn't provide any standard support for dealing with the multipart data of a POST request. POST requests have to be used when uploading files. I came across special purpose POST parsers in a few Imp projects that I looked at - but they were hardcoded to just deal with the data for that project, e.g. just a single file upload. I wrote a small library that can deal with multiple file uploads and any other standard form inputs. Here's a quick example showing how to use it:
const PARAM_LOCATION = "location";
const PARAM_DATA_FILE = "dataFile";

function requestHandler(request, response) {
    // Check the request is multipart.
    if (multipartParser.isMultipart(request)) {
        // Parse the request.
        local params = multipartParser.getParams(request);

        // Check for the presence of a parameter.
        if (!params.hasParam(PARAM_LOCATION)) {
            throw "...";

        // Retrieve a parameter.
        server.log("location: " + params.getParam(PARAM_LOCATION).content);

        // Retrieve a parameter that may have multiple values.
        local files = params.getListParam(PARAM_DATA_FILE);

        // Iterate over those values.
        for (file in files) {
            server.log("file content - " + file.content);

As shown every retrieved parameter item has a field called content, for a file upload this is the actual content of the file, for a simple form element like a text field it is just the value of the field. For uploaded files the item may also have the fields filename and type which respectively provide the original name of the file and the content-type, e.g. "text/plain".


An agent can check if a device is connected using device.isconnected(), however as discussed in this thread it may take a long time to notice that the device is no longer there - in fact on unplugging an Imp, i.e. an unannounced disconnect, in the evening I've seen the agent still reporting it as connected late the next day. I started using the code Hugo provided in the aforementioned thread and made it smarter and smarter until it grew to a surprising 60 lines (surprising for such a simple task).

Then I realized all I wanted was something that would confirm that a send would probably succeed and otherwise generate an error. So I wrote a function that takes the same arguments as device.send(name, value) plus an additional error function argument. The underlying agent logic pings the device, if the device responds within a second or two a normal send is done with the original send arguments provided otherwise the error function is called. You can use it like this:
sender.send("setspeed", 20, function () {
    server.log("could not contact device");
Note: if the device is not connected the error function will be invoked - it will also be invoked if the device was connected but busy doing something that prevented it responding promptly to the ping sent to it to determine whether it was available.

REST usage

All the discussion above covers using the programmer via its web interface. But if you're a command line fan like me, or want to control the programmer from another application, you can interact with it in a REST like style. The following command line session shows all the programmer functionality being exercised with curl (a tool installed as standard on Mac OS X and most Linux installations and available for Windows):
$ AGENT_ID=x5evDQQr51tF
# You should replace this ID with the one in the URL shown for your agent in the Electric Imp IDE.

# Get signature.
$ curl https://agent.electricimp.com/$AGENT_ID/programmer/action/getSignature
{ "signature": [ 30, 147, 11 ], "description": "ATtiny85" }

# Get fuse values.
$ curl https://agent.electricimp.com/$AGENT_ID/programmer/action/getFuses
{ "lfuse": 98, "hfuse": 223, "efuse": 255 }

# Set fuse values, note the double quotes due to the ampersands.
$ curl "https://agent.electricimp.com/$AGENT_ID/programmer/action/setFuses?lfuse=0x62&hfuse=0xdf&efuse=0xff"
{ "lfuse": 98, "hfuse": 223, "efuse": 255, "unsafe": false }

# Set fuse values with all safety checks disabled.
$ curl "https://agent.electricimp.com/$AGENT_ID/programmer/action/setFuses?lfuse=0x62&hfuse=0xdf&efuse=0xff&unsafe=on"
{ "lfuse": 98, "hfuse": 223, "efuse": 255, "unsafe": true }

# Get the lock bits.
$ curl https://agent.electricimp.com/$AGENT_ID/programmer/action/getLockBits
{ "lockBits": 255 }

# Set the lock bits.
$ curl https://agent.electricimp.com/$AGENT_ID/programmer/action/setLockBits?lockBits=0xfe
{ "lockBits": 254 }

# Upload a HEX file, the '@' tells curl to interpret what follows as a file from which content should be read.
$ curl --form hexFile=@/path/to/my-sketch.hex https://agent.electricimp.com/$AGENT_ID/programmer/action/uploadHex
{ "action": "uploadHex", "timestamp": 1405802184 }
Every request returns a JSON structure as shown. The returned JSON also includes a few other values that have been excluded here for clarity and which may be useful for debugging. The JSON fields are not returned in any particular order.

All the requests that require byte values expect them to be formatted as hex, e.g. 0xff, however all values returned are decimal as JSON does not support hex literals.

If an action succeeds it returns HTTP status code 200. If it fails it returns code 500 and includes an error message in the JSON response as shown here.
# Use --include with curl to show the HTTP response headers including the status, in this case 500.
$ curl --include https://agent.electricimp.com/$AGENT_ID/programmer/action/setLockBits?lockBits=0xff
HTTP/1.1 500 
Content-Type: application/json
{ "lockBits": 255, "message": "programmed lock bits can only be reset with chip erase" }


The breadboard layout at the start of this page shows an April board wired up to an ATtiny85. The breadboard layouts here show the slightly more complicated wiring required for the ATmega328, i.e. the MCU found on the Arduino UNO board.

If you've levered your ATmega328 straight off an Arduino UNO board then it will have had its fuses set to expect an external clock. So you also need to provide an external clock in the breadboard layout. I've used a ceramic resonator here as, unlike a crystal, it doesn't require external capacitors.
I used a 16MHz ceramic resonator borrowed from a Boarduino kit I had knocking around.

If you've got a factory fresh ATmega328 (that has not been preconfigured in any way by the reseller) then it will come with its fuses set to use its own internal 1MHz clock and you can omit the external clock source altogether:

Note: you can obviously also program an ATmega328 in place in an Arduino. However if you're using the Imp as a programmer for an Arduino, i.e. a setup where the MCU definitely has a bootloader, there's little or no advantage in using an in-system programmer over one that talks to the bootloader.

Aside: I always thought the crystal oscillator you can clearly see on the UNO board provided the clock signal for everything that needed one on the board, but it turns out it's just for the SMD ATmega8u2 used to interface with USB. The main MCU has a separate less accurate ceramic resonator (for photos, and an explanation of why this is, see this FAQ entry and the following ones from Adafruit).


All the code related to this project can be found on GitHub in the in-system-programmer subdirectory of my Electric Imp repository. GitHub Pages allows you to also host web pages in your repository by creating a separate gh-pages branch - I've used this feature for the programmer's web interface.

The agent part of the programmer retrieves the main index.html file for its interface from GitHub and feeds it out to the user. Nothing else is served out by the agent - all other content, e.g. included graphics or Javascript files, is either handled via HTTP redirection or standard CDNs (for things like Bootstrap and jQuery).

The README.md for the web interface covers all the third party tools used to create it.

Before implementing this programmer I looked for any existing projects that did the same thing. For the Electric Imp I only came across programmers that depended on a bootloader. These included Sparkfun's Wireless Arduino Programming with Electric Imp (as with all Sparkfun tutorials this project comes up with a great write-up that's worth reading for all kinds of background information). And on the Electric Imp forums I found:
  • Aron Steg's Impeeduino which is included in the Electric Imp reference repository.
  • Gene Jones's AVR Rascal.
Once I found there didn't seem to be an existing in-system programmer for the Electric Imp I started looking at the code behind any such programmers based around the Arduino or similar hardware.

An Arduino board can be setup as an in-system programmers as described on the Arduino site using the ArduinoISP sketch that can be found in the Arduino IDE under File → Example → ArduinoISP.

The ArduinoISP code isn't terribly easy to follow but I used it, in combination with avrdude, as a definitive source for answers to questions for which I couldn't find clear or reliable answers elsewhere.

However ArduinoISP depends on being connected to a computer where under the covers avrdude does most of the hard work. So I looked for standalone programmers - the two most useful projects I found were:
Much of the programmer logic in my project was developed by working through the code of these two projects.

Towards the end of this project's development I stumbled across the AVR target programming in Nut/OS and picked up their ideas of encoding in-system programming commands as four byte integers and of providing some safety checks when programming fuses (though I take a slightly different approach to theirs).

I switched back and forward between looking at Nick Gammon's and Adafruit's implementations for the various parts of the programmer logic. Each implementation has strong and weak points and each had one or two areas where they were clearer than the other for a given issue. In particular Nick Gammon's code can handle more MCU types than Adafruit's code (which has more hardcoded logic, e.g. the page mask they use only works for the 128 byte page size of the ATmega16 and ATmega32 families).

Both Nick Gammon's and Adafruit's programmers are standalone with limited memory resources and so can only contain the hardcoded details needed for programming a limited number of MCU types. The Electric Imp has the advantage of being able to access the web and query an external service for any details it needs.

Avrdude is one of the most popular tools used to upload programs to AVR MCUs (and is used under the covers by the Arduino IDE) and it has one of the most comprehensive collections of the details needed to interact with different AVR MCU types. You can see all these details in the avrdude.conf.in file in their svn repository and you can see that it uses its own ad-hoc format. I decided to create a queryable web service (described in another blog post) that parses these details and serves them up as JSON (a format supported by the Electric Imp APIs and pretty much everything else out there). You can see this in action by clicking e.g. this link that takes the three byte signature of an ATmega328 and returns its details:


This returns a nice human readable response suitable for browsers. For a pure JSON response try the following on the command line:
$ curl --header "Accept: application/json" http://avrdude-conf.herokuapp.com/conf/parts/signatures/0x1e9514


The following section covers various largely unrelated points that didn't fit in anywhere else.

The programming speed of the programmer is dictated by the SPI rate which is currently set to 234KHz, this allows it to program even the slowest 1MHz MCUs (the SPI rate must be a quarter or less of the target AVR MCU's clock frequency, as mentioned in Pololu's Atmel Studio 6 guide and other sources). The programmer could probably query the MCU and determine if it could use a higher rate and so increase programming speed, but it does not do so at the moment.

If experimenting with fuses be very careful and read up about them first. In working on this project I've found the Engbedded Atmel AVR Fuse Calculator extremely useful. If you're using a recent version of the Arduino IDE you can also include fuse settings in your code (these are one-off settings that are used when your program is uploaded - fuses cannot be updated dynamically from your MCU code). This is described in the fuse API section of the AVR libc user manual (you'll need to search down for the section covering C++ usage if working with the Arduino IDE).

I think including fuse settings with your firmware code makes sense as you can more clearly state what you're trying to achieve (using the various #defined names etc.) than you you can using raw hex values programmed as part of a different step to uploading your code. However at the moment my programmer simply ignores any fuse settings in a HEX file it is given to upload - you must use the separate fuse setting functionality covered up above.

In the ATmega328 section above I used a ceramic resonator as a clock source. If you'd rather use a crystal oscillator, with external capacitors, take a look at this tutorial on creating an Arduino UNO setup on a breadboard. It covers wiring up a ceramic resonator first and then wiring up a crystal oscillator to do the same job.

At the moment the programmer can only program up to 64KB (actually lower due to the device's limited memory space and due to limits on the blob sizes that can be passed between an Electric Imp agent and device). This is good enough for any MCU used in e.g. the Arduino AVR range which all have at most 32KB of flash memory - with the exception of the Arduino Mega 2560 which has an ATmega2560 with 256KB of flash memory. Nick Gammon's programmer, mentioned above, can deal with programs over 64KB (this requires handling extra Intel HEX record types) and similar logic (and some kind of chunking approach to get around blob size issues) could be added to my programmer.

The programmer can request the details for any AVR MCU known to avrdude. This doesn't mean it can necessarily program every AVR MCU - it only takes into account some of the information included in the details for a given MCU. It should be flexible enough to work with many or all ATtiny and ATmega MCUs - which covers two of the most popular AVR lines. An example of an incorrect assumption it makes is that it assumes that all MCUs support polling for RDY. Some AVR MCUs do not support this and one has to wait a fixed amount of time rather than polling.

When the programmer starts up you'll see that in addition to the main web interface URL it also outputs a test URL. Something like:


Requesting this URL in a browser toggles a test mode on and off (simply press refresh to switch back and forward between the two modes, you'll see whether test is enabled in the simple JSON response received). When test mode is on then the device behaves as if an exception occurred in any action it is asked to perform by the agent - this allowed me to check the agent side error handling and reporting to the end user.

Final notes

The breadboard layout images were all produced using Fritzing.

Hopefully I haven't let slip my real agent identifier in any of the screenshots or text above. The dummy agent and device identifiers seen here were generated using the string and byte generators of random.org.

Sunday, June 1, 2014

Smart Config for consumer products? Alternatives?

Smart Config looks like magic the first time you see it - what it's doing seems impossible if you know a bit about encrypted wifi networks.

Back in October 2013, out of a sense of intellectual curiosity, I spent an unreasonable amount of time looking at decompiled Java and wireshark dumps working out what was going on.

In the end it turns out to be fairly simple - a clever trick, the core of which has been around for a while - Paul Edwin Charles Martin covered it in his 2007 thesis "Covert Channels in Secure Wireless Networks."

The best ideas are often simple - so would I use this one in a consumer device?


It's insecure - you can make it secure but in doing so one loses most of the convenience the whole thing seemed to offer (a long unique AES key needs to be programmed into each device, included in the packaging shipped with the device and entered by the end user during setup).

And it's based on a "trick" rather than an established protocol, when it works then great but when it doesn't there's no easy way to diagnose what went wrong (and no one to complain to).

It's a broadcast only mechanism - the only point at which the device can respond to you is at the end if the whole setup process goes successfully.

The setup process can fail for various odd reasons (MIMO, MTU size etc.) where all the other networking hardware is operating correctly but just not in a way that Smart Config can handle.

So what would I use to communicate wifi credentials to a headless device?

Either a physical connection over USB or a wireless technology like BLE that allows for two way communication.

Smart Config sounds attractive in that the setup can be done with no extra hardware beyond the wifi module that the device is going to use anyway in its day-to-day activity.

Using USB or BLE requires additional hardware that's probably only ever going to be used once for the initial setup operation.


First USB - it's a technology that even the most technophobic end user is familiar with and I suspect that, in the case of people who own their own wifi access point (AP), a larger percentage of people have a desktop or laptop with USB than necessarily have an Android or iOS smart phone (unless tablets really have replaced laptops to a greater extent than I imagine).

USB is an established bullet proof technology and the physical connection eliminates pretty much any security issues.

The downsides are that your devise design has to include a USB port and you have to package a USB cable with every device (even though 99% of consumers probably have spare cables knocking around).


For a wireless solution I think BLE probably provides the best option.

It's a cheap and relatively well established device-to-device communications technology that properly supports two way communication.

It's been included in Apple laptops since around mid-2011 and in many Windows based machines, the iPhone has had it since the 4S. Google originally bet on NFC and BLE has only been supported as standard in Android since 4.3, i.e. mid 2013 - however certain manufacturers have had their own support for longer, e.g. the Samsung Galaxy S range has had BLE support since the S III (released in May 2012).

So you can use many modern laptops or phones with BLE.

Note: laptops are in general out-of-bounds for Smart Config as it doesn't play well with the MIMO capabilities of modern laptop wifi chipsets.

So why would I use BLE that's only been around for a few years and not available in every device over Smart Config that works using wifi that every smart phone has supported since forever?

Because I'd rather know that my device will work with those devices that have BLE rather than offer a consumer device that can be setup with a wider range of phones but will mysteriously fail for a subset of those users for essentially undiagnosable reasons.

Let's look at setup - we've got three main devices:
  • a phone running the setup app.
  • the IoT device being setup.
  • the wifi AP that we want the IoT device to connect to.
What are the most likely issues we'll see during the setup process? Probably:
  • The phone can't see the IoT device.
  • The IoT device can't see the wifi AP.
  • The user provided wifi credentials are incorrect.
  • The IoT device has connected but for some reason cannot see the public internet.
If using BLE all of these things can be sensibly reported (the first one by the phone, the others by the IoT device via the phone) and intelligently responded to, e.g. by moving the IoT device nearer to the wifi AP. Similarly if there is an issue that results in an unresolvable problem for many end users the IoT device can report all kinds of detailed diagnostics to the setup app that can be forwarded back to the manufacturer in order to track down the problem.

With Smart Config none of these issues can be distinguished. The whole Smart Config process could fail at the last step, where the CC3000 device tries to announce its successful connection by advertising its presence via (bastardized) DNS-SD, and the end user wouldn't see any difference between some kind of multicast failure at this point and a failure at the first-step (with the setup app being unable to even talk to the IoT device).

Only complete trial and error can resolve the source of any Smart Config failure - hardly ideal for a consumer device.

Smart Config tries, in the setup app, to pre-diagnose issues using the phone's own wifi hardware to check for things the CC3000 cannot handle, e.g. if the network is 802.11n only, operating in the 5GHz band, has too small an MTU size etc. This mitigates some of the issues, but ideally you still want the device to be able to report the issues it experiences.

And unlike Smart Config (without AES) the setup via BLE will be secure. To be honest I haven't looked into how security is achieved but as BLE is used in payment terminals (and end users are clearly not required to enter long AES like keys as part of the process) I presume they've got this covered in a consumer friendly manner.

Other CC3000 concerns

The CC3000 is one of the first really popular hobbyist wifi modules for use with Arduino etc. Adafruit and Sparkfun have produced shields and breakout boards for it.

The most active CC3000 community is based around the Spark Core from Spark Labs. The Spark.io forums show that a noticeable number of users, of presumably above average technical ability, have issues with the initial setup of their devices (quite apart from subsequent programming/development issues).

However many hobbyists, using boards like the Adafruit CC3000 breakout, simply hardcode their wifi credentials into e.g. their Arduino sketches which then configure the CC3000 directly rather than configuring it OTA and so use the CC3000 happily without ever using or running into issues with Smart Config.

If one uses the CC3000 without Smart Config are there still issues that concern me regarding its usage in consumer devices?

The main one would be confidence in the firmware.

The wifi chipset world is one in which companies like Broadcom, Atheros (now part of Qualcomm) and Intel dominate. TI is a much smaller player in comparison.

The CC3000 has been around since late 2011 - a while now - however rather than having settled into a phase of small fixes and minor improvements the release notes that accompany the CC3000 firmware updates still seem to regularly cover surprisingly major issues.

And some problems have required immense community effort to get resolved - e.g. the "cyan flash of death" seen in the Spark Core (the root cause of which was a TI issue). But hopefully projects like the Spark Core are finally driving TI to iron out the remaining major issues with the CC3000 and related chipsets.

On the TI forums it's nice that the actual engineers involved in the CC3000 project answer questions, however the quality of some of the answers are more disturbing than reassuring. In the case of the CC3000 firmware it seems clear that it's being developed by people primarily from a hardware background rather than some kind of expert hardware/software hybrid types and perhaps this explains somethings.

Hobbyist BLE development

So if you want to try out BLE where do you start?

First some terminology - Bluetooth Low Energy is often abbreviated as Bluetooth LE or BLE. In consumer marketing it's generally referred to as Bluetooth Smart and some people call it Bluetooth 4.0 even though it's only one part of the Bluetooth 4.0 specification.

Most hobbyist BLE breakout boards are based around chips from Bluegiga, e.g. the BLE112, or Nordic, e.g. the nRF8001.

Individual SMD BLE chips from Digikey etc. are extremely cheap, however BLE breakout boards from established names like Adafruit are relatively expensive, e.g. the nRF8001 breakout from Adafruit for $19.95.

There have been a number of one-off Kickstarter BLE projects and eBay is full of cheap Arduino friendly modules from random Chinese manufacturers.

There are numerous people offering BLE solutions for hobbyists, some of the things to look at include:
  • RFduino - one of the few BLE/Arduino hybrids that's really available to purchase from sources like Mouser.
  • BLEduino - one of the most talked about BLE/Arduino hybrids, much delayed but apparently now back on track.
  • Many people have produced short run BLE boards etc., but some people's names come up again and again - e.g. Michael Kroll and Jeff Rowberg.
  • The Arduino-BLE kit that has come out of the widely publicized Coin project.
But perhaps the most interesting crowd at the moment are RedBearLab - they clearly work very closely with people like Nordic and Arduino and produce a number or products including the new Blend Micro (a BLE/Arduino hybrid) and the BLE shield.

But perhaps most interesting is their BLE Mini as it, unlike most other BLE hobbyist products, has central role support. If you just want a simple device that can be controlled e.g. by your phone this isn't so important but if you want the device itself to play a more sophisticated role in a BLE setup then you probably need central role support. For an introduction to what central and peripheral roles and other terminology mean see one of these quick intros from Apple, Bluegiga (authored by Jeff Rowberg) or Android.

Note: there's quite a lot to BLE but many of the hobbyist solutions don't provide much more than a serial port style device-to-device wireless connection. If you look into BLE and decide that features like GATT sound interesting, e.g. you'd like to be able to record the heart rate information published by something like a Mio Alpha watch (that uses a standard GATT profile), then make sure to buy a device that actually supports that feature.

BLE/Wifi combos

Various people are pairing up BLE and wifi. E.g. the two are included in Intel's upcoming and much hyped (but no longer quite SD card sized) Edison.

Interestingly TI are also now producing a BLE/wifi combo module - the WL1835MOD (part of their WL18xx range).

The CC3000 can be used with very low end MCUs like the ATmega328 found in Arduinos, however TI clearly intend that the WL1835MOD be used with higher end processors like the Sitara AM3358 (based on the ARM Cortex-A8 core).

There's not much hobbyist support for the WL1835MOD - there is a cape, that was designed for the original BeagleBone but can now also be used with the BeagleBone Black, though the instructions aren't for the faint hearted (and, while the instructions mention Bluetooth once, they only actually go into testing out wifi).

Wednesday, May 7, 2014

avrdude.conf-as-JSON REST service using Heroku

Subtitle: Why do in a day what you can spend two weeks automating?

Credit to Terrence Parr from whose motto this is derived.

This post discusses creating a REST service to provide all or specific parts of the Avrdude configuration file as JSON.

Atmel produce a wide range of MCUs and these MCUs are a common component of many circuit designs. But before they can become useful in a given circuit they need to be programmed. For hobbyist projects this is commonly done using a bootloader. However for this the MCU needs to be preprogrammed with the bootloader itself.

An alternative that requires no preprogrammed bootloader is in-system programming (ISP). Typically the MCU is mounted along with all the other components on a given PCB and the PCB includes an ISP header which allows the MCU to be programmed directly on the board using a programmer.

The ISP header is generally a 6-pin header that just exposes a few pins of the MCU (reset and 3 SPI pins) along with ground and the positive power supply.

You can use an Arduino as a programmer or use a purpose built programmer like the USBtinyISP.

The programmer is mainly just hardware, you then need a utility that uses the programmer to e.g. actually upload a sketch to the flash ROM of the chip being programmed.

One of the best known tools for doing this is Avrdude (the Arduino IDE uses Avrdude under the covers).

To be able to do its work Avrdude needs to know various details about the programmer being used and the chip being programmed (the kind of details that can be found in the datasheet for a given MCU).

Avrdude currently knows the details for about 90 programmers and 150 MCUs (parts in Avrdude terminology).

Note: many of the programmers defined for Avrdude really just correspond to different configurations of particular given programmers.

The definitions for these programmers and parts are found in avrdude.conf, the source for which can be viewed in the Avrdude SVN repository as avrdude.conf.in.

For standalone projects where people want to be able to program just one MCU type or a limited number of MCU types, without requiring a computer capable of running Avrdude, the easiest thing is to just scrape the details for the relevant MCU or MCUs from avrdude.conf and hard code them into the logic that will be handling the programming.

E.g. see the code for the Adafruit standalone AVR chip programmer or Nick Gammon's ATmega board programmer (both of which are based on or inspired by Bill Westfield's OptiLoader).

The Electric Imp

I've been using the Electric Imp in a number of projects. Now I'm using it in a circuit that also includes an ATtiny85. I thought it would be nice if the Electric Imp could also handle the programming of the ATtiny85.

The simple solution would have been to program up something like the logic found in the Adafruit standalone AVR chip programmer and hardcode in the details for the ATtiny85.

But as the big feature of the Electric Imp is its web connectivity I thought it would be nice if the programmer logic could avoid any hand coding and derive the relevant Atmel chip details directly from the information available via the web interface to the contents of the Avrdude SVN repository.

Unfortunately the avrdude.conf file format is fairly ad hoc and not easily consumable by anything other than the parser built into the Avrdude application.

So I decided to parse avrdude.conf and make the content available in a format consumable by pretty much anything - JSON.

And make it available in bite size chunks, e.g. one could request the entire contents of avrdude.conf as JSON (around 545KB) or one could request just some specific subset, e.g. a list of all programmers or all parts, or the details for a specific part (around 4.2KB for something like the ATmega328).

Parts should be queryable by the arbitrary names assigned to them by Avrdude, e.g. "m328" for the ATmega328, or by the 3 byte signature that can be retrieved from any Atmel MCU and is unique to each MCU type.

This could have been done in the cloud based agent logic of an Electric Imp.

However this would have been quite complex in Squirrel (a Lua inspired language with C like syntax), the language used for Imp and agent logic.

So I decided to separate this logic off into an independent REST based service, i.e. make it available in a fashion that's easy for applications to interact with but which human beings can also interact with via a web browser.

As a Java programmer this proved quite simple once I'd worked through:

Jackson is a JSON library for Java. Heroku is a PaaS that makes it extremely easy to make your web service publicly available (the free hobbyist tier provides enough container hours to run a single process essentially non-stop). And Jersey is a RESTful web service framework for Java - their getting started guide very conveniently covers how to get a simple example running on Heroku.

REST allows one to request resources via URLs and to tailor the response according to the format or formats the requester expresses a preference for. E.g. a web browser will implicitly express a preference for some form of HTML while an application can explicitly express a preference for JSON.

So one could request JSON like so:
$ curl -H "Accept: application/json" http://avrdude-conf.herokuapp.com/conf/parts/ids/m328
And receive:
  "id" : "m328",
  "desc" : "ATmega328",
  "has_debugwire" : true,
  "flash_instr" : [ 182, 1, 17 ],
  "eeprom_instr" : [ 189, 242, 189, 225, 187, 207, 180, 0, 190, 1, 182, 1, 188, 0, 187, 191, 153, 249, 187, 175 ],
  "stk500_devcode" : 134,
  "signature" : [ 30, 149, 20 ]
And then one could request exactly the same URL using a web browser and get:

The web page includes a brief standard introduction, followed by the JSON shown previously. However this JSON is formatted to be human readable, in particular it may include comments, e.g. for the list of parts the part descriptions are included as comments and for the list of signatures the part names are included as comments, and some of the integer values are displayed as hex, e.g. 0xFF rather than 255. Neither comments nor hex are allowed in valid JSON (so there are no comments and all integers are formatted as decimals if one explicitly requests JSON). At the end of the web page is a summary of the URLs for all available resources.

Try it now - click the following URL to request the details for the ATmega328:


Note: Heroku may be slow to respond initially if no one else has used the service recently as Heroku will put a service to sleep after an hour of no requests if the service has only one dyno (the Heroku unit of scalability) - one dyno is the default and typical for hobbyist services. Once out of sleep the service will respond swiftly to subsequent requests.

The source code for this service is available from GitHub here and can be cloned like so:
$ git clone https://github.com/george-hawkins/avrdude-conf.git
It can be built like so:
$ mvn package
And then run locally like so:
$ java -cp target/classes:target/dependency/* net.betaengine.avrdude.heroku.Main
Then specify http://localhost:8080/conf as the location in your web browser to access it.

Assuming you've worked through the Getting started with Java on Heroku guide mentioned above and created an Heroku account and installed the Heroku toolbelt etc. then deploying the above service so that it's available from anywhere on the web is trivial.

This was done, after going to the base directory of the cloned git repository, as follows:
$ heroku create avrdude-conf
$ git push heroku master
$ heroku info
That's it!

So after doing this the service could be accessed using "avrdude-conf.herokuapp.com" rather than "localhost:8080".

The command heroku create takes a name as an argument (in this case I chose "avrdude-conf"), this name must be unique across all Heroku users (so as you can imagine all the more generic names are already gone). The name will ultimately become a subdomain of herokuapp.com.

You can omit the name argument and Heroku will create a unique name itself, something fairly unusual like "immense-inlet-4196". However if you own your own domain you can map a subdomain (named however you choose) of your domain to this unusual name, as described here, for something more memorable.

Note: the Savannah SVN web interface is extremely slow so there is logic to cache rather than rerequest the underlying avrdude.conf.in file each time it's needed. Heroku automatically provides a small PostgreSQL database with each service created and this is used as a persistent cache.


  • Do I approve of REST? REST works very nicely for this service. It's simple to get going and being able to interact with it via a web browser is very convenient. However on the whole I'm unconvinced by REST. Every few years something comes along in this domain because people believe what exists already has become a monster. I'm sure REST people will argue it's addressing a different problem to say SOAP or CORBA but in a few years I suspect the average REST library will have a similar feature set (because all those features are actually useful in some given scenario). At the moment as people discover they want a particular feature that might have been available in one of the supposedly bloated "old" solutions they have to code up their own ad hoc solutions. Soon standardized solutions to those problems will be included into the REST libraries and things will be little different to earlier "monsters".
  • Is Heroku the best PaaS option going? Heroku allows for laziness at the development phase - one can pretty much push something that runs on your local machine straight into a service running in the cloud. That's pretty cool - but for a non-hobby project I think I might prefer to live with the increased restrictions of Google App Engine. I started writing out some reasoning for this but it quickly grew and grew - that's a whole separate topic.

Sunday, October 13, 2013

Ideas for improving Smart Config

This blog posting is me thinking out loud about how to improve the flawed Smart Config process. However by the end I essentially come to the conclusion that there's not much that can be done. So unless you're interested in seeing what ideas I had and why they don't work in practice I'd skip this post. I do look at workable alternatives in my later posting "Smart Config for consumer products? Alternatives?"

As Smart Config exists at the moment I think it has two main problems:

  • The user has to enter a longish AES key in order for their network password to be securely transmitted to a CC3000 device.
  • The CC3000 may not pick up the transmitted data if the Smart Config client is running on a machine talking to its wifi access point (AP) using 802.11n.
It is possible to use the CC3000 without requiring an AES key but in a previous post I shown how easy it is for an outside party to recover passwords transmitted in this way. And in another post I explain why I think this isn't just a hypothetical issue for end users and that as such TI should not provide the option to use the CC3000 without AES.

In normal operation the CC3000 only supports 802.11b and 802.11g. During the setup phase for a CC3000 enabled device the 802.11 protocol used by the device running the Smart Config application should ideally not be relevant to the process. Unfortunately it is - as discussed in this post the CC3000 can receive Smart Config data from some devices that talk to the AP using 802.11n but not from others. And it's clear the situation will not improve as newer protocols such as 802.11ac become available.

Below I look at using an asymmetric key algorithm, rather than AES (which uses symmetric keys), in order to create a better end user experience in relation to key handling.

And I look at using wifi probes to communicate information in a manner independent of the 802.11 protocol version so avoiding the 802.11n and MIMO issues seen with the current approach.

However in the end the best features of both ideas depend on functionality that while available on Android, Mac OS X, Windows and Linux is crucially not available via any public interface for iOS (see later for the exact details).

So on the whole they're an interesting thought experiment, however a few smaller improvements are also covered that would be possible to make.


If we accept that encryption keys are required then is there a way to make the handling of these keys easier or even invisible to the end user?

Entering random character sequences is difficult for human beings and as the user will get no feedback if the sequence is entered incorrectly, other than the failure of the given CC3000 enabled device to connect to their network, one should ensure one catches entry errors up front.

This can be done by including a checksum as part of the sequence that needs to be entered, this allows the Smart Config application to verify the sequence and point out that it's incorrect. The current TI Smart Config applications do not do this.

Unfortunately adding a checksum makes the sequence even longer. The CC3000 supports 128 bit AES, to enter a complete 128 bit key would require a sequence of 32 hex digits, not even accounting for the checksum. If instead of hex digits we use all printable ASCII characters we can encode a little more than 6 bits per character so reducing the number of characters required to around 20 but introducing a different problem - characters that are hard to distinguish for the end user who has to enter them, e.g. zero and capital O.

A 128 bits is the maximum supported key length. Are shorter keys secure enough? I can't find a definitive reference - I see that in the 2002 RSA challenge they found it would take the computing power of 45,000 2GHz AMD processors running for 790 days to work through the complete 64 bit keyspace. RSA is obviously quite a different algorithm to AES and costs drop off very rapidly as the key size decreases.

This recent article by one of the people behind 1Password claims that for AES it would require all the computer power on earth a year to check all the keys possible with 75 bits. A totally untuned unscientific experiment in Java on my 1.8GHz Intel Core i5 Macbook Air using 100% of all cores could only check all keys possible with 38 bits in a day, presumably tuned assembler GPU logic could do orders of magnitude better (but remember e.g. that 3 orders of magnitude would still only be equivalent to 10 additional bits). Note: if instead of truly random keys the attacker knows you're creating keys from human friendly text like "LetMeIn" then the search space is reduced massively in size. So while "LetMeIn" may be encoded as 7 bytes, i.e. 56 bits (or 42 bits if you take into account that each character only really encodes 6 bits if we consider just the printable ASCII characters) don't equate this number of bits with a random key of anything like equivalent length.

Side note: TI use the simplest AES encryption mode, called electronic cookbook (ECB) mode. It's interesting to note that on the relevant Wikipedia page it's claimed ECB "is not recommended for use in cryptographic protocols at all." However to be fair the issues outlined don't really come into play in the way AES is used with the CC3000.

Asymmetric keys

There might be a different way though to reduce down the size of the key that needs to be entered to something the length of a bank PIN.

On one of my previous posts someone asked if using RSA rather than AES would make a difference. AES is a symmetric key algorithm - the same key is used for both encryption and decryption so one has to be very careful with the distribution of the key. RSA is an asymmetric key algorithm, basically one key is used to encrypt the data but a different key is needed to decrypt it - so you can give out one key (called the public key) without taking any care to keep it secret - if someone wants to send you data they encrypt it with this key but no one can use this key to decrypt the resulting data - this can only be done with the other key (called the private key) that you have never given out to anyone.

Note: RSA and AES aren't really alternatives to each other - RSA is only used to encode small amounts of data while AES is used for arbitrarily large amounts of data. In fact the two are often used together, e.g. in TLS, the protocol used by https, the data is exchanged using AES but at the start of the connection process the AES key is first randomly generated and then exchanged using RSA.

So would using an asymmetric algorithm make any difference? My initial thought was that it wouldn't.

With AES there is just one key which is known to the CC3000 enabled device and is also included on a sticker, or such like, with the device so the end user can enter it in the Smart Config application.

With RSA the key included on the sticker would be different to the one known to the CC3000 enabled device.

So even if one knew the key on the sticker one wouldn't be able to use it to decode a password encrypted with it and transmitted by Smart Config.

For a closed source device it would make a difference - one could could safely use a single well known public key for all your devices if the key known to the devices was kept secret. However hardcoding keys into devices doesn't have a very successful history - the keys are either eventually recovered by hacking the physical devices or leaked - see e.g. the history of DVD encryption (DVD keys are now brute forced in a matter of seconds but DeCSS started with a key recovered by disassembling a software DVD player).

If instead of one public/private key pair, common to all instances of a given device, one had a different key pair per device would this provide any advantage versus one AES key per device (as would be required for any sensible use of AES with the CC3000) ?

It would mean that recovering the private key of one device would have no implications for any other device and it would mean that absolutely no special care would need to be taken with packaging the public key with the device, whereas with an AES key one should take some care to ensure it cannot easily be recorded by another party before one takes delivery of it. However even I admit in a previous post that if someone who wants to decrypt your network traffic has had physical access to a device you intend to connect to that network then you probably have bigger problems, and that in the end an AES key printed on a sticker within the physical shrink wrapped packaging is probably completely adequate.

Side note: I've seen it suggested that keys cannot be kept private in an open source based project. This isn't the case - keys do not have to be hard coded into the microcontroller code and thus visible to all. For a situation where one intends to mass produce a given product the MCU code can e.g. reference a particular EEPROM memory location at which it expects to find the key and this location can be written with a unique key value generated per device as part of the process that loads the compiled code into each individual MCU. Or one could have the keys pairs preprogrammed onto tiny and cheap serial EEPROM chips that are then included as a part of each device (keeping track of which public key went with each device would still be an issue). This process would be no different for a closed or open source project. Simpler schemes could be constructed for projects intended for individual makers to force them to create a unique key for their instance of the project rather than providing a common hard coded one.

So if there isn't much advantage in using asymmetric keys in the same manner as we currently use AES keys is there some other way they could be used that wouldn't be possible with AES keys?

I thought there might be but it depended on being able to exchange keys. On all platforms except iOS this could be done using wifi probes, however on iOS there is no public interface for sending or receiving them. All low level network activity is handled by iOS itself, applications can basically only determine if they are connected or not via wifi or mobile/cellular (using Reachability in the System Configuration Framework) and then use the higher level services of the CFNetwork Framework.

The following section depends on being able to exchange keys so unless a solution for iOS can be found it's currently unimplementable.

Side note: RSA is considerably more expensive space wise to implement and space and time wise to run that AES - this is a serious issue for an embedded system implementation, e.g. see this implementation for the PIC18 which requires 5KB of flash, 700 bytes of RAM and 120 seconds (yes, really) for a single block decode operation. The decode cost effectively excludes RSA from use irrespective of other issues.

Public Key Exchange

Instead of the end user entering the public key of a public/private key pair into the Smart Config application the CC3000 could broadcast its public key and the Smart Config application could pick this up without the user needing to do anything, any password then encrypted with this key could only be decoded by the device that had broadcast it and also had the corresponding private key - the public key being of no use for decryption to anyone who captured it.

The CC3000 wouldn't even have to be preconfigured with a unique public/private key pair - it could generate its own just as e.g. ssh-keygen does (or it could use something similar to the Atmel ATSHA204 crypto chip).

However there's an obvious problem, the solution I described for recovering non-AES protected passwords just involved passive monitoring but with an active component one could overcome this asymmetric approach too.

The Smart Config application would have to do something to tell the CC3000 to announce its public key, e.g. the Smart Config application could send a wifi probe request and the CC3000 could include the key in a wifi probe response (encoded it as a dummy SSID). Or the CC3000 enabled device could be put into a setup mode, e.g. by pressing a button on its case, that caused the CC3000 to repeatedly announce its key in some way.

If I was an outside party and saw a Smart Config application requesting a public key I'd simply respond with my own key and hope the device saw my response before the real one. Similarly if I saw a CC3000 repeatedly announcing its public key I would do the same and hope the Smart Config application would see my key before the real one.

The end user would of course see that their CC3000 enabled device didn't then connect to the network but this would just look like a random failure. The system generating the fake public keys would stop interfering once it had received the network password sent by Smart Config and the user would probably just restart the Smart Config application and try again with everything apparently going without problem the second time.

The Smart Config application could look out for these issues - but it couldn't do anything much about them other than report them to the end user and refuse to allow them to setup the device they'd just bought.

In many security protocols each party has their own public/private key pair and before communication starts between them they exchange their public key with the other.

So another approach would be for both the CC3000 enabled device and the Smart Config application to generate their own public/private key pairs and for the CC3000 to in addition have a unique short PIN code, e.g. 4 digits long. First the Smart Config application would announce its public key such that the CC3000 could pick it up, the CC3000 would use this key to encrypt its PIN and include this along with its public key when it announces it such that the Smart Config application can pick it up in return.

The PIN would also be included on a sticker with the device - and the end user would have to enter it in the place where the AES key goes currently - the Smart Config application could compare this with the PIN it decrypts using its private key from the response assumed to be from the CC3000 device, if it matches the response is accepted otherwise it is simply ignored.

Note: if one was using a request and response to retrieve the public key from the CC3000 enabled device it might seem acceptable to just send the PIN unencoded in the response (and so not bother with a key pair for the Smart Config application), but as wifi packets are often discarded for one reason or another despite having been visible on the network this is probably not a great idea. If everything doesn't go perfectly for the first packet seen that contains the PIN then it will become known to the outside party and can be used to create a fake response before any followup real response.

The user still has to enter something (or one could instead display the PIN received from the device and ask them to confirm it matches the one on the device. However user interface testing shows users generally don't read dialog messages and just press whatever button they think will get the process finished).

A 4 digit PIN, using just the digits 0 to 9, is far easier for an end user to handle than a long sequence of random characters.

Why is a 4 digit PIN acceptable here, while an AES key of random characters twice this length would be completely unacceptable? Because with the current AES approach one can record the traffic and work through all the possible permutations at leisure whereas with this approach it's the Smart Config application that effectively dictates the number of chances, i.e. one, to generate the correct value.

This new approach would be far more complex to implement but it does make the security aspect, a 4 digit PIN vs a long random character sequence, easier for the end user. The implementation would only need to be done once and would prevent many end user problems and support issues.

However as noted it's currently not implementable at all, at least with wifi probes, on iOS.

802.11 Protocol

The original IEEE 802.11 standard was established in 1997 (unfortunately the IEEE standards are not available free of charge - they cost $250 each to download as PDFs).

The subsequent protocols, e.g. 802.11b, 802.11g or 802.11n, are amendments to this original standard.

Ideally Smart Config would use an element common to all the protocol versions in order to communicate the SSID etc.

This would eliminate the problem where the Smart Config application was running on a machine that used a protocol that the CC3000 does not know or uses an aspect of a protocol, e.g. MIMO, that affects the ability of the CC3000 to monitor the relevant traffic.

Wifi probes are just such an element.

Wifi probes contain 32 bytes that are usually used to encode an SSID but which could be used to encode arbitrary information (in fact TI used to use probes in this way in their old, and very unsatisfactory, First Time Configuration process).

However again iOS is the problem platform - as noted above it provides no low level network access so one cannot send or receive wifi probes with an iOS app. One must use a mechanism that works with high level network constructs, e.g. UDP as with the current Smart Config approach.


Copyright @ 2013 Depletion Region.

Designed by Templateify & Sponsored By Twigplay