It’s taken a while to get back to this point, but there we are. Verison 1.4 of the board finally works well enough it will be accepted by a real ntpd.

This year was also the first time I’d presented at the conference, although just in one of the miniconfs. The Arduino miniconf is usually quite popular as the first half of the day is spent assembling a project – this year a full Arduino clone with a bunch of extra peripherals - and then more conventional presentations on things related.

I’ve embedded my talk below, you can probably guess the subject given how often I’ve posted about it. (The clip is a bit of a wonky encode, the AV team didn’t quite trim it properly so it doesn’t start until 57m into the video. It’s not really an hour an a half long!)

In other news, I’ve started to embark on designing an Arduino clone based off the XMEGA MCU line, probably a D3 to begin with as it’s what I’m most familar with. The rough bought layout has been done, although it’s missing switching power sources between the DC jack and USB. It’s also deliberately not using the USB capabilities of some of the XMEGAs since there’s a lack of good code around to support it and it complicates the board layout a fair bit.

The rest of the projects are slowly making their way towards further progress!

Now that I’ve started to run the prototypes in the target case I have some idea of the temp of the board, components, and how the case feels. And it’s warmer than I want.

This is somewhat of an issue since the stability of the clock is related to the temperature. If a crystal has a stability of, say, 20ppm, it is always stated with a temperature range at which that holds true. For (most) cheap crystals that’s 25 degrees. As it gets colder or warmer it gets worse.

During the design some of this was taken into consideration. Firstly, since we are aligning ourselves to GPS sources, the stability is less critical: we have a stable external reference less influenced by our local temperature. It does mean we’re limiting how long we can run without GPS sources (it’s currently in the “minutes” range, which is not far off what a $4k device does and this costs nothing like that). This is currently implemented in a coarse-grain way, we just adjust our internal clock being fed by the DS3234 output.

Secondly, the decision to use the DS3234 RTC instead of a watch crystal was driven by the fact it’s a Temperature Compensated Crystal Oscillator, that is it has a temp sensor and can tune the crystal inside it to compensate for the local temperature. Over the range 0 to 40 degrees it claims 2ppm out of the box, and 3.5ppm between -40 and +85. So even with a lot of heat it’s still significantly more stable than a cheap watch crystal.

There are some things I haven’t implemented yet which I’ll now need to consider.

The DS3234 exposes it’s tuning interface, and the datasheet mentions that with an external trusted reference (oh hai GPS) you can feed it extra offset information to further compensate for both aging and to obtain finer precision (you get about +/- 0.1ppm of control). I’ll need to add an extra step after GPS lock to tune the DS3234 against the GPS pulses, and retune it from time to time.

Once the case is closed up properly (it’s not fully closed right now), it may need some holes cut in specific places to improve natural convection. Likely places will be right under the (not very efficient, therefore a source of a lot of heat) LDO regulator. It also will likely need holes in both endplates to ensure any air that is around can get into the case.

We’re also running the main MCU at full clock rate without any regard for clocking down based on demand. While it’s a lot of work to implement, I could look at power management for the MCU by implementing dynamic clocking.

Those things can be done without changing any aspect of the board design.  But there’s board design changes I’m looking at as well.

The 1.5 board now has space for a 16x16x4mm fan in one endplate, with PWM control so we can run it as slow as possible. Like many things, it’s there if I finally decide it needs a fan, but it’s something I’m going to try to avoid. The fan is 3.3V so doesn’t need it’s own LDO (since a lot of fans are 12V, fewer 5V), but should probably have a transistor between the MCU and it to source as little current from the MCU as possible.

The LDO has also been re-arranged to make it easier to attach a small heatsink to. Ideally this will move heat away from going into the board so much, as I suspect that’s what is really causing the reported temps to be as high as they are. It might only work best with real airflow over it, but we’ll see how that goes.

There’s also some evidence the Ethernet section is drawing more power than it should do, as a result of not quite getting the front-end right. There’s fixes for that in 1.5 as well.

All in all, it feels like I’m getting into the real tail end of the project now to be considering these sorts of issues. There’s less making-it-work-at-all, and more making-it-awesome.

If you want to see any of the pre-XMEGA code, there’s a release tagged “board-1.3″ which was the last working version for the MEGA MCU on a rev 1.3 board.

Should it come to pass we need a rev 1.5, I will tag the last working 1.4 release in the same way.

Unexpectedly this week a bunch of the PSUs sold, and more needed to be built in a (relatively) short time. This is quite different as an experience to the prototyping phase, where every build is unique and new, and the timeline is as long or short as you feel like. I’m glad that I chose something simple to start with, because I’ve very quickly found the things I need to think about for larger and more complex designs going into production.

Inventory management is important

During the prototyping phase on all of the projects I have underway at the moment, I’ve built up quite a stack of parts. Some of these parts overlap with other designs, but most are design-specific. For example, every design will probably involve a ton of 100nF power supply decoupling caps scattered all around various ICs. But this is the exception, not the rule.

For a while, “inventory management” was a pile of parts in bags. It started out in small bins, when it was mostly PTH parts which really were easier to store in little bins, but as the number of parts escalated it fell off a little. I only knew if I could build one by actually finding all the parts at the time.

Inventory management is really critical if you need to know quickly how many you can build beyond just one. One is easy. Five or ten imposes more need to know how many you have of all your components. At least twice now I’ve been caught out by problems in inventory – one part important part wasn’t tracked at all, and a second wasn’t updated properly to reflect builds completed.

Some of this comes from the fact upstream part suppliers probably won’t sell you one of a part, but often if you want good prices you’ll need to buy 10, or 50, or 150 at a time anyway. And they’ll get used at different rates, so you need to track the build usage carefully as well.

Plastic stencils last; stencilling jig needs work

I’ve only ever used plastic stencils for assembling my mostly-SMD boards, and had some concerns about how long they would last. It turns out more than long enough, and now I have a cheap source to make them (Ponoko) I’m less worried about exactly how long they’ll last. Still, I’ve gotten sixteen builds out of the first stencil and they’ve effectively cost a buck for each stencil, so it’s a good run.

The more significant problem with the stenciling approach is the jig I use is .. primitive. I’ve just used spare boards I had from prior prototypes to form something to wedge the target board in place, and it does the job, but only just. It would be easier with something more purpose cut to surround the board as much as possible. One option is to get something laser cut by Ponoko but I’ll need to figure out how close their material thicknesses will get to a PCB thickness.

Frypan works, reflow oven nicer

Despite how primitive the frypan cooking method for board soldering is, it actually does work, and I can manage to get a few boards at a time being soldered up. The reliability has been reasonable so far, although we’ll see how these turn out in the field. The main problem is actually the need to watch it. A controlled oven I could put a set of boards on a tray, stick them in, and just run the cook profile.

This will mean I might have to step up efforts to work on building a reflow oven myself. A full commercial one is both far too large, too expensive, and probably not a lot better than a hacked up toaster oven, but something with control is better than none.

SMD parts are great for production

It feels very strange to be so in favour of surface mount parts, but they are so much easier to solder together a bunch of components than PTH by hand with a soldering iron. I had already come to the conclusion SMD parts made things easier in general, but it wasn’t until I realised I was spending 33% of the build time on just a single PTH component that it really hit home how much easier.

There are still some challenges, getting tiny parts orientated out of the tape they often come in is a pain, and tweezers are a bit prone to too much force causing the part to fly off and disappear to never be seen again .. but those are mostly going to come down to practice. I’ve also ordered a cheap vacuum pencil to see how that works for part pick-up and placement, as a sort of poor human pick-and-place machine.

Production is confidence-building

Lastly, production builds confidence in two ways. Being able to reproduce a build and have them pass basic QA tests first time, every time, gives you a lot more certainty about the way something has been designed. It works, so your understanding of how it works looks a bit better each time.

Having other people using your builds is also helps out a bit with motivation to build other things. It’s fun, no doubt about it I have enjoyed all the things I’ve built whether they worked or not, but it’s a very nice feeling having new devices you made, put for the first time into an anti-static bag, all clean and ready to be used by someone else.

The new revision of the NTP Server and a test build of the LUFA AVR Programmer boards have arrived. The service from Seeedstudio was pretty quick and excellent for pricing, although one of the boards had some drills not correctly done. Still, it’s far cheaper than getting larger runs made so I am quite happy.

Sadly, I appear to have made some mistakes with the stencils that were cut by Ponoko – the detail has been lost a lot more than previous cuts. I suspect I didn’t take into account laser kerf enough, but I also think they were cut without protective paper on both sides as previous ones were done. That seems to make a lot of difference. I shall get them re-cut in the next few days.

Now off to order more parts from Element 14 to prepare for the first builds of these new designs!

The LUFA board is an quick prototype to see how badly I’ve misread the USB AVR datasheets. It is based off the hints provided in a readme associated with some LUFA code, and vague attempts to read the USB sections of the Mega 16U2 datasheets. It includes a level shifter to support 1.8 – 5V targets which covers all the useful ranges of both Mega and XMega MCUs.

I’m not sure how the little LUFA board is going to turn out, but it was fairly cheap to get made and I’m bound to learn something out of it as a result.

These are the first boards being made by Seeedstudio’s Fusion PCB service, which for very short runs looks cheaper than previous options. It’s only 10 boards (compared to the 25 or so I was getting from other runs) so it’s more expensive per board but much cheaper to do a run because it’s a smaller number of  boards. We’ll see how they turn out.

The most recent scare with the design was what looked like a short in the antenna cable causing various things to reset and may now be damaged. It’s unclear to me exactly what happened, but we’re no longer getting a stable consistent GPS lock. The antenna is active – it has 3.3V fed up into it to power an LNA in the weather-sealed puck antenna – but there’s no current limitation on what’s fed up this path.

The datasheet for the GPS has a large example circuit to provide hints to the GPS chip that either the antenna is disconnected (only works for active antennas), or has a short and automatically disables the power into the antenna. But the design is far more than I really need, and takes up enough board space I’d have to do some serious rearrangement. It’d be nice to use the GPS chip’s LNA control line and reporting antenna modes, but the main thing is just limit current and suppress shorts.

I’ve generally ignored putting any significant additional current limitation into the design because none of it is exposed to any external connections. All the failure modes of any of the parts would just result in the board being tossed rather than fixed, and none of them actually are subject to external connectors – except now realising the antenna is directly. (The ethernet jack does have power going into the internal side of the transformer, but since it’s internal side a broken cable plugged into the jack doesn’t cause more power to be drawn.)

Instead, I’ve decided to just add a simple PTC fuse to the power input side of the antenna, limiting it to 125mA which is comfortably more than any active antenna should need, but should blow fast enough to prevent significant damage to the board. Whether I add some remote control of that power using a MOSFET or some sort of reporting of the failure to the GPS or MCU I haven’t decided. It depends on how much board space I need. A PTC fuse is small enough to just throw one on there without much concern.

(PTC fuses are tiny resistors which have a Positive Temperature Coefficient, that is as they try to carry more current they heat up, and a high enough current/temp will result in a very high resistance, enough to stop the current flowing. They reset themselves by cooling down. Their self-resetting nature and simplicity makes them useful for this sort of case. PTC fuses are commonly used in external power ports such as USB.)

Hopefully, that’ll reduce the risk of a short! Soon it will be ready!

On a Mega, the choice of system clock is set by fuses. It generally isn’t intended that you change fuses from your main code, instead they should be changed mostly by external programming or when asked by a bootloader. That is, you set fuses when doing a brand new build on an out of the box chip and never at any other time.

Mega fuses are a pain to work with. Apart from being inverted in sense (ie, 0 and 1 are called programmed and unprogrammed, and programmed means “active”) a single incorrect write to them can brick the chip and disable important programming interfaces. If you wanted to run a Mega at full clock speed, you had to carefully ensure an external crystal was present and then write the appropriate fuses. And hope.

(Pedant alert: technically there was one clocking option you could change from within your main code, that was system clock prescaler. It’s initalised from a fuse (CLKDIV8) to divide by 8 for a safe default from a variety of possible sources, but you could switch that to undivided in your main code quite easily and non-destructively. That still limited you do the internal 8MHz RC oscillator, however.)

The XMega does away with this disaster, and instead allows you to change the system clock source from within your main code at any time. It allows you to run the chips at their full speed without external components. You can use external components of a variety of speeds and multiply the clock in addition to dividing it. It’s a vastly better design.

By default, an XMega will always boot off the internal 2MHz RC oscillator, with the PLL multipler disabled and no prescaler divisions, giving us a nice simple reliable default clock to work with. Should any clock option you set later fail (and, yes, it has clock failure detection) the chip will always force itself back to the 2MHz RC oscillator. (It also generates a non-maskable interrupt for this situation, so you main code can do something about it.)

This means, in general, whatever you do to the chip is a simple flash of some new code to apply a different set of clocking options, even if you have pieces missing off the board you’ve built.

All XMegas are capable of running at 32MHz @ 3.3V and there’s a variety of ways to get this. The easiest to start with is using the internal 32MHz RC oscillator and run everything at the same clock rate. The only trick to enabling this is to ensure the oscillator is running before attempting to use it, and then changing the clock via a protection mechanism:

OSC.CTRL = OSC_RC32MEN_bm; /* start 32MHz RC oscillator */
while (!(OSC.STATUS & OSC_RC32MRDY_bm)); /* wait for ready */
CCP = CCP_IOREG_gc; /* allow changing CLK.CTRL */
CLK.CTRL = CLK_SCKLSEL_RC32M_gc; /* system clock is internal 32MHz RC */

This does mean all peripheral modules are running at  32MHz as well, which all modules will accept by default. Some modules (EBI, HiRes Timer Extensions) run off faster clocks, which can be twice (PER2) or four times (PER4) the maximum system clock speed. Setting up the clock system for those I might cover at a later date.

Otherwise, that’s all which is required to get an XMega with no external components running at full clock speed. A lot nicer than a Mega!

An external crystal is not much harder, again you need to start the appropriate oscillator, wait for it to be ready, and change the clock via a protection mechanism. But in this case, we’re going to start with an 8MHz crystal and multiply it up to 32MHz using the PLL:

OSC.XOSCCTRL = OSC_FREQRANGE_2TO9_gc | OSC_XOSCSEL_XTAL_256CLK_gc; /* configure the XTAL input */
OSC.CTRL |= OSC_XOSCEN_bm; /* start XTAL */
while (!(OSC.STATUS & OSC_XOSCRDY_bm)); /* wait until ready */
OSC.PLLCTRL = OSC_PLLSRC_XOSC_gc | 0x4; /* XTAL->PLL, 4x multiplier */
OSC.CTRL |= OSC_PLLEN_bm; /* start PLL */
while (!(OSC.STATUS & OSC_PLLRDY_bm)); /* wait until ready */
CCP = CCP_IOREG_gc; /* allow changing CLK.CTRL */
CLK.CTRL = CLK_SCLKSEL_PLL_gc; /* use PLL output as system clock */

While this is quite a lot more code than using a crystal clock on a Mega, it is still better since we get flexibility about what kind of crystal we’re using and the desired real clock rate we want from it. We could have done this with just a 32MHz crystal, and avoided the PLL. But only needing a cheaper 8MHz one and being able to multiply it up to the desired clock rate is a nice feature.

Part of the problem is I’m not settled on what “done” looks like. I was originally planning on not migrating from a simple watch crystal RTC to the DS3234 RTC until after 1.4 was completed. But working with the XMega RTC has given me a lot of niggly issues around getting it to behave well with aligning the clock.

The problems with the XMega RTC are a mix of just the result of using a 20ppm crystal (ie, you get 20ppm accuracy, and that’s not so cool) and some interesting choices about how it’s built.

Some AVR Megas have an async clock timer, usually it’s the single 8-bit timer. It’s pretty basic, but 8-bit is actually a good choice since you can cover off the rest of the needed bits for a 32.768kHz crystal off interrupts, once every 256 ticks is not a heavy interrupt workload. The Xmega doesn’t have an async clock timer, instead there’s an explicit 16-bit RTC with a crystal input.

At first (mostly when reading the datasheet) this looks pretty good. But, there’s a few annoyances. A number of Megas (and, more specifically, the ones I’ve been using like the 1284P) allow not only a 32.768kHz crystal but an external clock input. This allows you to use a variety of other clock options. The Xmega RTC has no such option, not in the A and D series anyway, and that’s almost all of the ones actually shipping. (You can brute force an external clock into a crystal input but I’d prefer to stay within what the datasheet says the chip is expecting.)

The other issue with the RTC is the way you access it. Because it’s running in it’s own clock domain – specifically off whatever 32-ish-kHz source you chose – you have to jump through a lot of busywaiting to get configuration pushed into it. Busywait. Set something. Busywait. Set something else. And so on. The same problem plagues accessing the current count, if you want to change it yet more busywaiting.

You also don’t get any capture options with the RTC, it has just compare and period registers, that’s it.

This makes it quite difficult to align the RTC to a fine degree. Sure, it’s great if you just need a simple timer to fire, say, a 1Hz interrupt to do something else in your code. It’s actually very good at that. But if you actually care what “now” really is, then it doesn’t cut it.

Falling out of that has been a large redesign, and involving the DS3234 RTC to replace the simple crystal. Thankfully while they did take away async clocking for a timer, they replaced it with a much more awesome system and the result is any of the normal 16-bit timers can provide a better replacement with finer control and more reliable capture. I’ll probably post about that some other time.

I’ll still be using the Xmega RTC, but it’ll just be there as a system clock for trivial event timing that we don’t really care about alignment for. Hell, 1% RC oscillators used as an RTC will be fine for many other timing needs in the code.