Imp powered In-System Programmer

Introduction

The Atmel AVR MCUs are very popular in the hobbyist electronics market, in particular the ATmega328 found in the Arduino UNO. The MCU on an UNO and similar boards is generally programmed via a bootloader, a piece of code that needs to be preinstalled and makes the chip more convenient to program.

I wanted to use an Atmel AVR MCU in combination with an Electric Imp in a circuit and thought wouldn't it be nice if the Imp could completely setup the AVR MCU in place without any preprogramming or pre-existing bootloader being required. This can be done via a technique called in-system programming (see also the Atmel In-System Programming Guide).

Once I started down this route I thought why not try creating a completely general purpose Electric Imp based AVR MCU programmer. This blog entry describes the result.

Simple standalone AVR programmers typically only have the details for a few AVR MCUs hardcoded into their logic. This programmer can query any AVR MCU for its type (signature) and then lookup all the details for programming from an another online service I created (that uses avrdude.conf as a basis and serves up MCU details as JSON).

Many Imp applications come with a fairly basic web interface - I tried to create something a little fancier but still very lightweight using the now ubiquitous Bootstrap. You can also interact with the programmer in a REST like manner.

Note: some AVR MCUs, e.g. the ATtiny85, do not provide hardware bootloader support and can only be programmed using in-system programming (having said that one can emulate bootloader like behavior as Adafruit do with their Gemma and Trinket products).

Features

You can see the programmer's web interface on GitHub. Pressing any of the buttons will just result in an error as it's not being served by an Electric Imp agent, but you can see the various features - which are:

• Querying and displaying the MCU's signature (along with the name associated with the signature, e.g. "ATtiny85").
• Uploading new programs (sketches in Arduino terminology) to the MCU.
• Querying and setting the MCU's fuses and lock bits.

Setup

Setup is very easy. You need to wire up your Imp to your AVR MCU. At the top of this page you can see an Imp wired up to an ATtiny85 and down below you can see an Imp wired up to an ATmega328 (in one case the ATmega328 is taken straight from an Arduino and so requires an external clock, a ceramic resonator, and in the other case it's an ATmega328 with factory defaults, i.e. running using its own internal 1MHz clock).

I've used an April with an imp001 with the Imp pins connected as follows:

• Pin 1 to SCK.
• Pin 2 to reset.
• Pin 8 to MOSI.
• Pin 9 to MISO.
• 3V3 and ground as required.

You can lookup the MOSI, MISO, SCK and reset pins, along with Vcc and ground pins, in the data sheet for your AVR MCU.

Once the hardware is setup the software side is easy. The Electric Imp agent and device code is available on GitHub. You just need to:

• Open up the Electric Imp IDE.
• Create a new model called e.g. "InSystemProgrammer".
• Associate a real device with the model.
• Copy the agent code into the agent pane and the device code into the device pane and then save.

If your Imp is powered up you should see output from both the agent and device as shown below. The important thing is the programmer URL (highlighted here with a red border):

Open this URL in your normal web browser. You should see the programmer's web interface and, if you've correctly wired up your Imp to your AVR MCU, you should now be able to query the MCU for its type, reprogram it etc.

Note: the wiring I've shown above for the ATtiny85 could be simpler. I did it this way to match the style seen for the more complex ATmega328 circuits shown below. For the ATtiny85 you could wire 3V3 from the April directly to the MCU's Vcc (rather than using three red wires as shown) and also move the LED's cathode pin so you just need one ground wire (instead of the two black wires shown).

Usage

Note: the programmer depends on a service running on Heroku to lookup the name and details of an AVR MCU. This service uses Heroku's free hobbyist tier which means that if the service is unused for more than an hour it is suspended. When this happens it can take ten seconds or more to respond to the next incoming request as it takes a while to come out of suspended state. To avoid a pause due to this when using the programmer you can first click a link, like this one for the details of the ATmega328, to query this underlying service directly for something - just to make sure that it is currently warmed up.

Usage of the programmer's web interface should hopefully be fairly self evident - here's a quick run through.

Signature

If you press Query you should see the MCU's signature as three hex bytes along with the MCU name associated with that signature.

Programming

At the risk of alienating low level AVR enthusiasts I've called this the Sketch section as this is what programs are known as by Arduino users. Here you can upload the HEX files generated by the Arduino IDE (and other AVR toolchains) to the MCU.

Once you've selected a HEX file (more on these in the next section) and pressed Upload things should whir away for a while (depending on the size of the file) and if all goes well you should see a nice "Programming succeeded" response in green.

Fuses

This is the dangerous bit - if you don't know what a fuse is then leave this section well alone - random experimenting may well result in getting the MCU into a state in which it is essentially bricked.

You can query the current fuse settings and set new values. By default there are some safety checks built in that only allow you to change bits that should not impact further interaction with the MCU. If you know exactly what you're doing you can tick the "Allow unsafe changes" checkbox and enter any value you want.

The safety checks only work if using an ATtiny or ATmega MCU, other AVR MCUs have different fuse assignments and the safety logic does not attempt to cover all AVR families. The safety checks prevent changes to:

• the clock source and startup bits of the low fuse - changes here won't really brick the MCU but may force you to change the circuit, e.g. to add in an external clock if the external clock bit is programmed.
• the reset, debug-wire and SPI enable bits of the high fuse - changes here really will brick the MCU - requiring a high voltage programmer (like the Rescue Shield from MightyOhm) to recover the situation.

Using this feature is entirely at your own risk, I've used it successfully but I'm not guaranteeing that it can't contain bugs that may brick your device even if the values you enter make sense!

Lock bits

Like the fuses you can query and set the lock bits. Lock bits aren't terribly interesting outside commercial settings where you may e.g. want to restrict the ability to retrieve the code running on the MCU.

Note: once a particular lock bit is programmed it cannot be unprogrammed without erasing the chip. The easiest way to do this is to upload a new program as the chip is erased as part of this process.

HEX files

When you compile a sketch in the Arduino IDE it generates files in a format called Intel HEX which can then be uploaded to the board being programmed. Generally this all happens under the covers and you never see these files. All other AVR toolchains can also generate these files (though some may use an alternative format by default).

The GitHub repository for the programmer also contains a few sample HEX files that can be used to demonstrate programming the ATtiny85 and ATmega328 MCUs in the breadboard layouts shown on this page so that they flash the attached LED on and off.

To upload your own programs that you've created using the Arduino IDE you need to be able to find the HEX files it creates. First open the Arduino IDE's Preferences dialog - it will tell you where you can find your preferences.txt file:

It's important to exit the IDE before editing the preferences.txt file otherwise any changes you make will be overwritten when the IDE exits.

First create a new folder where your HEX files will end up, e.g. add a new folder called "build" in your usual documents folder. Then find your preferences.txt file. Unfortunately the standard location for preferences.txt may be hidden by default on your system. If you don't know how to show hidden files just google for something like "how to show hidden files on windows" or "mac" or "linux" as appropriate. Once you've found the preferences.txt file open it in a normal text editor and add the line:

build.path=/full-path/build

You'll need to replace /full-path/build with the full path to the build folder you just created. If you created it in your documents folder then this will typically be something like:

• C:\Users\<username>\Documents\build on Windows.
• /Users/<username>/Documents/build on Mac OS X.
• /home/<username>/build on Linux.

It doesn't matter where in the file you add this line (in fact if you later reopen the preferences.txt file after running the IDE you'll find the IDE has probably reordered the lines and this new line will end up somewhere random in the file).

Save your changes and restart the Arduino IDE.

Now whenever you press Verify in the IDE it will save the HEX file for your sketch in the build folder you created (along with many other intermediary files that aren't of interest in this situation).

If your sketch was called "Blink" then you'll find a HEX file in this folder called "Blink.cpp.hex"

Important: before you press Verify make sure you've got the right board selected under the Arduino IDE Tools → Board menu. E.g. maybe you've only used the Arduino IDE with your Arduino UNO but now want to generate a HEX file for an ATtiny85 connected to your Electric Imp - so make sure to select the ATtiny85 as the target board. You may need to install hardware descriptions for the MCU if it is not supported by default by the Arduino IDE, e.g. as described for the ATtiny family on High-Low Tech. HEX files generated for one MCU type will not be compatible with another MCU type, e.g. you cannot use files generated for an ATtiny85 with an ATmega328 and vice-versa.

Once you've found the HEX file generated by the Arduino IDE for your sketch you can upload it to your MCU as described in the section up above that describes using the programmer.

Sample HEX files

I've provided a few small sample HEX files with the programmer code that can be used to demonstrate the programmer with the breadboard layouts shown on this page:

• attiny85_blink_pin_4.hex - this blinks the LED in the ATtiny85 layout above.
• attiny85_nop.hex - this programs the ATtiny85 to do absolutely nothing and can be used to demonstrate the MCU behavior changing between when programmed to blink the LED and when programmed to do nothing.
• atmega328_blink_pin2.hex - this blinks the LED in either of the ATmega328 layouts shown below.
• atmega328_nop.hex - this programs the ATmega328 to do absolutely nothing.
• atmega328_blink_pin13.hex - this blinks the onboard LED on an Arduino UNO and can be used to demonstrate the programming of an ATmega328 that will be returned to an UNO board.

The blink files are simply compiled from the Arduino IDE File → Examples → 01.Basics → Blink sketch with the pin value in the code changed as appropriate and the correct board selected under Tools → Board. Similarly the nop files are compiled from the 01.Basics → BareMinimum sketch with nothing added.

Programming bits

If you're not used to AVR terminology where they talk of programming and unprogramming bits, rather than of setting or unsetting them, then it may come as a surprise to find that:

• Programming a bit means unsetting it, i.e. setting it to 0.
• Unprogramming a bit means setting it, i.e. setting it to 1.

So if e.g. all bits of say the extended fuse are unprogrammed this means they are all set to 1 and the fuse has value 0xff, i.e. 1111-1111 in binary. Similarly if the SPI-enable bit of the high fuse is to be enabled it must be programmed, i.e. set to 0.

Useful libraries

The programmer code base contains a few libraries that may be of general use elsewhere.

Multipart Parser

The standard agent API provides support for dealing with form data that came in via an HTTP GET request but doesn't provide any standard support for dealing with the multipart data of a POST request. POST requests have to be used when uploading files. I came across special purpose POST parsers in a few Imp projects that I looked at - but they were hardcoded to just deal with the data for that project, e.g. just a single file upload. I wrote a small library that can deal with multiple file uploads and any other standard form inputs. Here's a quick example showing how to use it:

const PARAM_LOCATION = "location";
const PARAM_DATA_FILE = "dataFile";

function requestHandler(request, response) {
    // Check the request is multipart.
    if (multipartParser.isMultipart(request)) {
        // Parse the request.
        local params = multipartParser.getParams(request);

        // Check for the presence of a parameter.
        if (!params.hasParam(PARAM_LOCATION)) {
            throw "...";
        }

        // Retrieve a parameter.
        server.log("location: " + params.getParam(PARAM_LOCATION).content);

        // Retrieve a parameter that may have multiple values.
        local files = params.getListParam(PARAM_DATA_FILE);

        // Iterate over those values.
        for (file in files) {
            server.log("file content - " + file.content);
        }
    }
}

http.onrequest(requestHandler);

As shown every retrieved parameter item has a field called content, for a file upload this is the actual content of the file, for a simple form element like a text field it is just the value of the field. For uploaded files the item may also have the fields filename and type which respectively provide the original name of the file and the content-type, e.g. "text/plain".

Sender

An agent can check if a device is connected using device.isconnected(), however as discussed in this thread it may take a long time to notice that the device is no longer there - in fact on unplugging an Imp, i.e. an unannounced disconnect, in the evening I've seen the agent still reporting it as connected late the next day. I started using the code Hugo provided in the aforementioned thread and made it smarter and smarter until it grew to a surprising 60 lines (surprising for such a simple task).

Then I realized all I wanted was something that would confirm that a send would probably succeed and otherwise generate an error. So I wrote a function that takes the same arguments as device.send(name, value) plus an additional error function argument. The underlying agent logic pings the device, if the device responds within a second or two a normal send is done with the original send arguments provided otherwise the error function is called. You can use it like this:

sender.send("setspeed", 20, function () {
    server.log("could not contact device");
});

Note: if the device is not connected the error function will be invoked - it will also be invoked if the device was connected but busy doing something that prevented it responding promptly to the ping sent to it to determine whether it was available.

REST usage

All the discussion above covers using the programmer via its web interface. But if you're a command line fan like me, or want to control the programmer from another application, you can interact with it in a REST like style. The following command line session shows all the programmer functionality being exercised with curl (a tool installed as standard on Mac OS X and most Linux installations and available for Windows):

$ AGENT_ID=x5evDQQr51tF
# You should replace this ID with the one in the URL shown for your agent in the Electric Imp IDE.

# Get signature.
$ curl https://agent.electricimp.com/$AGENT_ID/programmer/action/getSignature
{ "signature": [ 30, 147, 11 ], "description": "ATtiny85" }

# Get fuse values.
$ curl https://agent.electricimp.com/$AGENT_ID/programmer/action/getFuses
{ "lfuse": 98, "hfuse": 223, "efuse": 255 }

# Set fuse values, note the double quotes due to the ampersands.
$ curl "https://agent.electricimp.com/$AGENT_ID/programmer/action/setFuses?lfuse=0x62&hfuse=0xdf&efuse=0xff"
{ "lfuse": 98, "hfuse": 223, "efuse": 255, "unsafe": false }

# Set fuse values with all safety checks disabled.
$ curl "https://agent.electricimp.com/$AGENT_ID/programmer/action/setFuses?lfuse=0x62&hfuse=0xdf&efuse=0xff&unsafe=on"
{ "lfuse": 98, "hfuse": 223, "efuse": 255, "unsafe": true }

# Get the lock bits.
$ curl https://agent.electricimp.com/$AGENT_ID/programmer/action/getLockBits
{ "lockBits": 255 }

# Set the lock bits.
$ curl https://agent.electricimp.com/$AGENT_ID/programmer/action/setLockBits?lockBits=0xfe
{ "lockBits": 254 }

# Upload a HEX file, the '@' tells curl to interpret what follows as a file from which content should be read.
$ curl --form hexFile=@/path/to/my-sketch.hex https://agent.electricimp.com/$AGENT_ID/programmer/action/uploadHex
{ "action": "uploadHex", "timestamp": 1405802184 }

Every request returns a JSON structure as shown. The returned JSON also includes a few other values that have been excluded here for clarity and which may be useful for debugging. The JSON fields are not returned in any particular order.

All the requests that require byte values expect them to be formatted as hex, e.g. 0xff, however all values returned are decimal as JSON does not support hex literals.

If an action succeeds it returns HTTP status code 200. If it fails it returns code 500 and includes an error message in the JSON response as shown here.

# Use --include with curl to show the HTTP response headers including the status, in this case 500.
$ curl --include https://agent.electricimp.com/$AGENT_ID/programmer/action/setLockBits?lockBits=0xff
HTTP/1.1 500 
Content-Type: application/json
{ "lockBits": 255, "message": "programmed lock bits can only be reset with chip erase" }

ATmega328

The breadboard layout at the start of this page shows an April board wired up to an ATtiny85. The breadboard layouts here show the slightly more complicated wiring required for the ATmega328, i.e. the MCU found on the Arduino UNO board.

If you've levered your ATmega328 straight off an Arduino UNO board then it will have had its fuses set to expect an external clock. So you also need to provide an external clock in the breadboard layout. I've used a ceramic resonator here as, unlike a crystal, it doesn't require external capacitors.

I used a 16MHz ceramic resonator borrowed from a Boarduino kit I had knocking around.

If you've got a factory fresh ATmega328 (that has not been preconfigured in any way by the reseller) then it will come with its fuses set to use its own internal 1MHz clock and you can omit the external clock source altogether:

Note: you can obviously also program an ATmega328 in place in an Arduino. However if you're using the Imp as a programmer for an Arduino, i.e. a setup where the MCU definitely has a bootloader, there's little or no advantage in using an in-system programmer over one that talks to the bootloader.

Aside: I always thought the crystal oscillator you can clearly see on the UNO board provided the clock signal for everything that needed one on the board, but it turns out it's just for the SMD ATmega8u2 used to interface with USB. The main MCU has a separate less accurate ceramic resonator (for photos, and an explanation of why this is, see this FAQ entry and the following ones from Adafruit).

Details

All the code related to this project can be found on GitHub in the in-system-programmer subdirectory of my Electric Imp repository. GitHub Pages allows you to also host web pages in your repository by creating a separate gh-pages branch - I've used this feature for the programmer's web interface.

The agent part of the programmer retrieves the main index.html file for its interface from GitHub and feeds it out to the user. Nothing else is served out by the agent - all other content, e.g. included graphics or Javascript files, is either handled via HTTP redirection or standard CDNs (for things like Bootstrap and jQuery).

The README.md for the web interface covers all the third party tools used to create it.

Before implementing this programmer I looked for any existing projects that did the same thing. For the Electric Imp I only came across programmers that depended on a bootloader. These included Sparkfun's Wireless Arduino Programming with Electric Imp (as with all Sparkfun tutorials this project comes up with a great write-up that's worth reading for all kinds of background information). And on the Electric Imp forums I found:

• Aron Steg's Impeeduino which is included in the Electric Imp reference repository.
• Gene Jones's AVR Rascal.

Once I found there didn't seem to be an existing in-system programmer for the Electric Imp I started looking at the code behind any such programmers based around the Arduino or similar hardware.

An Arduino board can be setup as an in-system programmers as described on the Arduino site using the ArduinoISP sketch that can be found in the Arduino IDE under File → Example → ArduinoISP.

The ArduinoISP code isn't terribly easy to follow but I used it, in combination with avrdude, as a definitive source for answers to questions for which I couldn't find clear or reliable answers elsewhere.

However ArduinoISP depends on being connected to a computer where under the covers avrdude does most of the hard work. So I looked for standalone programmers - the two most useful projects I found were:

• Nick Gammon's programmer.
• Adafruit's standalone AVR chip programmer.

Much of the programmer logic in my project was developed by working through the code of these two projects.

Towards the end of this project's development I stumbled across the AVR target programming in Nut/OS and picked up their ideas of encoding in-system programming commands as four byte integers and of providing some safety checks when programming fuses (though I take a slightly different approach to theirs).

I switched back and forward between looking at Nick Gammon's and Adafruit's implementations for the various parts of the programmer logic. Each implementation has strong and weak points and each had one or two areas where they were clearer than the other for a given issue. In particular Nick Gammon's code can handle more MCU types than Adafruit's code (which has more hardcoded logic, e.g. the page mask they use only works for the 128 byte page size of the ATmega16 and ATmega32 families).

Both Nick Gammon's and Adafruit's programmers are standalone with limited memory resources and so can only contain the hardcoded details needed for programming a limited number of MCU types. The Electric Imp has the advantage of being able to access the web and query an external service for any details it needs.

Avrdude is one of the most popular tools used to upload programs to AVR MCUs (and is used under the covers by the Arduino IDE) and it has one of the most comprehensive collections of the details needed to interact with different AVR MCU types. You can see all these details in the avrdude.conf.in file in their svn repository and you can see that it uses its own ad-hoc format. I decided to create a queryable web service (described in another blog post) that parses these details and serves them up as JSON (a format supported by the Electric Imp APIs and pretty much everything else out there). You can see this in action by clicking e.g. this link that takes the three byte signature of an ATmega328 and returns its details:

http://avrdude-conf.herokuapp.com/conf/parts/signatures/0x1e9514

This returns a nice human readable response suitable for browsers. For a pure JSON response try the following on the command line:

$ curl --header "Accept: application/json" http://avrdude-conf.herokuapp.com/conf/parts/signatures/0x1e9514

Miscellaneous

The following section covers various largely unrelated points that didn't fit in anywhere else.

The programming speed of the programmer is dictated by the SPI rate which is currently set to 234KHz, this allows it to program even the slowest 1MHz MCUs (the SPI rate must be a quarter or less of the target AVR MCU's clock frequency, as mentioned in Pololu's Atmel Studio 6 guide and other sources). The programmer could probably query the MCU and determine if it could use a higher rate and so increase programming speed, but it does not do so at the moment.

If experimenting with fuses be very careful and read up about them first. In working on this project I've found the Engbedded Atmel AVR Fuse Calculator extremely useful. If you're using a recent version of the Arduino IDE you can also include fuse settings in your code (these are one-off settings that are used when your program is uploaded - fuses cannot be updated dynamically from your MCU code). This is described in the fuse API section of the AVR libc user manual (you'll need to search down for the section covering C++ usage if working with the Arduino IDE).

I think including fuse settings with your firmware code makes sense as you can more clearly state what you're trying to achieve (using the various #defined names etc.) than you you can using raw hex values programmed as part of a different step to uploading your code. However at the moment my programmer simply ignores any fuse settings in a HEX file it is given to upload - you must use the separate fuse setting functionality covered up above.

In the ATmega328 section above I used a ceramic resonator as a clock source. If you'd rather use a crystal oscillator, with external capacitors, take a look at this tutorial on creating an Arduino UNO setup on a breadboard. It covers wiring up a ceramic resonator first and then wiring up a crystal oscillator to do the same job.

At the moment the programmer can only program up to 64KB (actually lower due to the device's limited memory space and due to limits on the blob sizes that can be passed between an Electric Imp agent and device). This is good enough for any MCU used in e.g. the Arduino AVR range which all have at most 32KB of flash memory - with the exception of the Arduino Mega 2560 which has an ATmega2560 with 256KB of flash memory. Nick Gammon's programmer, mentioned above, can deal with programs over 64KB (this requires handling extra Intel HEX record types) and similar logic (and some kind of chunking approach to get around blob size issues) could be added to my programmer.

The programmer can request the details for any AVR MCU known to avrdude. This doesn't mean it can necessarily program every AVR MCU - it only takes into account some of the information included in the details for a given MCU. It should be flexible enough to work with many or all ATtiny and ATmega MCUs - which covers two of the most popular AVR lines. An example of an incorrect assumption it makes is that it assumes that all MCUs support polling for RDY. Some AVR MCUs do not support this and one has to wait a fixed amount of time rather than polling.

When the programmer starts up you'll see that in addition to the main web interface URL it also outputs a test URL. Something like:

https://agent.electricimp.com/<imp-id>/programmer/action/test

Requesting this URL in a browser toggles a test mode on and off (simply press refresh to switch back and forward between the two modes, you'll see whether test is enabled in the simple JSON response received). When test mode is on then the device behaves as if an exception occurred in any action it is asked to perform by the agent - this allowed me to check the agent side error handling and reporting to the end user.

Final notes

The breadboard layout images were all produced using Fritzing.

Hopefully I haven't let slip my real agent identifier in any of the screenshots or text above. The dummy agent and device identifiers seen here were generated using the string and byte generators of random.org.

Read More