Jekyll2020-02-26T10:19:35+13:00http://hairy.geek.nz//feed.xmlhairy.geek.nzhairy.geek.nz is a rarely updated blog, about things geeky
OpenStack Heat: Split pets by AZ Part 22020-02-25T00:00:00+13:002020-02-25T00:00:00+13:00http://hairy.geek.nz//2020/02/25/heat-split-by-az-pt2<p>So in <a href="/2020/02/21/heat-split-by-az.html">the first part of this story</a>
I mentioned
nested stacks would allow us to pass <code class="highlighter-rouge">%index%</code> from the
<code class="highlighter-rouge">OS::Heat::ResourceGroup</code> resource type in Heat, which combined
with making each server a child stack gives us some way to
look up a list with that index.</p>
<p>But typically we don’t have one AZ for each server, we would spread
the servers out over two to three AZs. How can we make Heat do that?
The answer is YAQL.</p>
<!--more-->
<p>YAQL is a “query language” for YAML, with a library of functions suitable
for manipulating data presented as YAML objects. There’s a ton of stuff
you could do with YAQL, but I’m mostly interested in the math functions.</p>
<p>What we need is a way of taking the index and finding the modulo of
the length of the list of AZs passed to us. Just that simple!</p>
<p>Heat supports YAQL in a few different ways. You can use it in place of
literal values (like <code class="highlighter-rouge">str_replace</code> does), or more commonly you can
simplify your structure using <code class="highlighter-rouge">OS::Heat::Value</code> resources.</p>
<p>This is a resource that just holds a value like this:</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">resources</span><span class="pi">:</span>
<span class="na">MyValue</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">OS::Heat::Value</span>
<span class="na">properties</span><span class="pi">:</span>
<span class="na">value</span><span class="pi">:</span> <span class="s">myhostname</span></code></pre></figure>
<p>We can now retieve the value with resource attribute lookups in another
part of the stack:</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">resources</span><span class="pi">:</span>
<span class="na">MyInterestingResource</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">OS::Nova::Server</span>
<span class="na">properties</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="pi">{</span> <span class="nv">get_attr</span><span class="pi">:</span> <span class="pi">[</span> <span class="nv">MyValue</span><span class="pi">,</span> <span class="nv">value</span> <span class="pi">]</span> <span class="pi">}</span></code></pre></figure>
<p>Now, there’s an obvious problem with using an <code class="highlighter-rouge">OS::Heat::Value</code> in a
template: it only gets created once. You can’t think of it as a function
call, it has to be resolved to a single value when created like any
other resource. So we can’t use <code class="highlighter-rouge">OS::Heat::Value</code> in our main stack
if we’re passing it <code class="highlighter-rouge">%index%</code> anyway, even if the substitution of
<code class="highlighter-rouge">%index%</code> worked. Which it doesn’t.</p>
<p>We can however use it if we create a child stack for each and every
instance we were launching, so we can put code to modulo <code class="highlighter-rouge">%index%</code> in
a value. Great. How do we do this?</p>
<p>app_server.yaml:</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">paramaters</span><span class="pi">:</span>
<span class="na">index</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">number</span>
<span class="na">az_list</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">comma_delimited_list</span>
<span class="na">az_count</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">number</span>
<span class="na">resources</span><span class="pi">:</span>
<span class="na">az_index</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">OS::Heat::Value</span>
<span class="na">properties</span><span class="pi">:</span>
<span class="na">value</span><span class="pi">:</span>
<span class="na">yaql</span><span class="pi">:</span>
<span class="na">expression</span><span class="pi">:</span> <span class="s">$.data.index mod $.data.count</span>
<span class="na">data</span><span class="pi">:</span>
<span class="na">index</span><span class="pi">:</span> <span class="pi">{</span> <span class="nv">get_param</span><span class="pi">:</span> <span class="nv">index</span> <span class="pi">}</span>
<span class="na">count</span><span class="pi">:</span> <span class="pi">{</span> <span class="nv">get_param</span><span class="pi">:</span> <span class="nv">az_count</span> <span class="pi">}</span>
<span class="na">AppServer</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">OS::Nova::Server</span>
<span class="na">properties</span><span class="pi">:</span>
<span class="na">availability_zone</span><span class="pi">:</span> <span class="pi">{</span> <span class="nv">get_param</span><span class="pi">:</span> <span class="pi">[</span> <span class="nv">az_list</span><span class="pi">,</span> <span class="pi">{</span> <span class="nv">get_attr</span><span class="pi">:</span> <span class="pi">[</span> <span class="nv">az_index</span><span class="pi">,</span> <span class="nv">value</span> <span class="pi">]</span> <span class="pi">}</span> <span class="pi">]</span> <span class="pi">}</span>
<span class="pi">[</span><span class="nv">...</span><span class="pi">]</span></code></pre></figure>
<p>It’s as easy as that. Kind of. We need to now pass this stack a count of the
AZ, the AZ list, and the index. So we can do this in a resource group
like this:</p>
<p>app_main.yaml:</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">paramaters</span><span class="pi">:</span>
<span class="na">count</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">number</span>
<span class="na">default</span><span class="pi">:</span> <span class="m">3</span>
<span class="na">az_list</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">comma_delimited_list</span>
<span class="na">default</span><span class="pi">:</span> <span class="s2">"</span><span class="s">az_a,az_b"</span>
<span class="na">resources</span><span class="pi">:</span>
<span class="na">AppServers</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">OS::Heat::ResourceGroup</span>
<span class="na">properties</span><span class="pi">:</span>
<span class="na">count</span><span class="pi">:</span> <span class="pi">{</span> <span class="nv">get_param</span><span class="pi">:</span> <span class="nv">count</span> <span class="pi">}</span>
<span class="na">resource_def</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">app_server.yaml</span>
<span class="na">properties</span><span class="pi">:</span>
<span class="na">index</span><span class="pi">:</span> <span class="s1">'</span><span class="s">%index%'</span>
<span class="na">az_list</span><span class="pi">:</span> <span class="pi">{</span> <span class="nv">get_param</span><span class="pi">:</span> <span class="nv">az_list</span> <span class="pi">}</span>
<span class="na">az_count</span><span class="pi">:</span>
<span class="na">yaql</span><span class="pi">:</span>
<span class="na">expression</span><span class="pi">:</span> <span class="s">$.data.list_param.count()</span>
<span class="na">data</span><span class="pi">:</span>
<span class="na">list_param</span><span class="pi">:</span> <span class="pi">{</span> <span class="nv">get_param</span><span class="pi">:</span> <span class="nv">az_list</span> <span class="pi">}</span></code></pre></figure>
<p>Now we just need to pass arguments to our main stack of the list of AZs,
and how many servers we want, and the stack will ensure they are split over
the available AZs automatically.</p>
<p>There’s a bunch of other things you can do with nested stacks, yaql, and
resource groups. This is just an example of the power that Heat has in
making it easier to write less template code and have fixed resources
built out easily.</p>So in the first part of this story I mentioned nested stacks would allow us to pass %index% from the OS::Heat::ResourceGroup resource type in Heat, which combined with making each server a child stack gives us some way to look up a list with that index. But typically we don’t have one AZ for each server, we would spread the servers out over two to three AZs. How can we make Heat do that? The answer is YAQL.OpenStack Heat: Split pets by AZ2020-02-21T00:00:00+13:002020-02-21T00:00:00+13:00http://hairy.geek.nz//2020/02/21/heat-split-by-az<p>It’s been ages since I posted anything, so here’s something that is
probably not very useful. OpenStack Heat has reosurce groups which
allow you to stamp out the same resource multiple times. Better yet,
it provides a magic token <code class="highlighter-rouge">%index%</code> by default for which copy of a
resource your template is run for.</p>
<p>But <code class="highlighter-rouge">%index%</code> doesn’t get substituted by intrinsic functions. This
unfortunately means you can’t use it in really obvious places like
<code class="highlighter-rouge">str_replace</code> or use it to lookup a list by index with <code class="highlighter-rouge">get_param</code>.
This would be useful for figuring out which AZ we want to launch
an instance in by being passed a list of AZs and mangling the index
from the resource group.</p>
<p>However, all is not lost. There’s a way to do this. You just need
to use nested stacks.</p>
<!--more-->
<p>Nested stacks in Heat allow you to define a psuedo-resource which is
itself another Heat stack, complete with parameters. Effectively, the
properties of the resource become paramaters in the nested stack.</p>
<p>For example, we can write:</p>
<p>main.yaml:</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">resources</span><span class="pi">:</span>
<span class="na">NestedStack</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">innerstack.yaml</span>
<span class="na">properties</span><span class="pi">:</span>
<span class="na">some_param</span><span class="pi">:</span> <span class="s">value1</span>
<span class="na">another_param</span><span class="pi">:</span> <span class="s">value2</span></code></pre></figure>
<p>innerstack.yaml:</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml">
<span class="na">parameters</span><span class="pi">:</span>
<span class="na">some_param</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">string</span>
<span class="na">another_param</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">string</span>
<span class="na">resources</span><span class="pi">:</span>
<span class="na">SomeResource</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">OS::Nova::Server</span>
<span class="na">properties</span><span class="pi">:</span>
<span class="na">flavor</span><span class="pi">:</span> <span class="pi">{</span> <span class="nv">get_param</span><span class="pi">:</span> <span class="nv">some_param</span> <span class="pi">}</span>
<span class="na">image</span><span class="pi">:</span> <span class="pi">{</span> <span class="nv">get_param</span><span class="pi">:</span> <span class="nv">another_param</span> <span class="pi">}</span>
<span class="pi">[</span><span class="nv">...</span><span class="pi">]</span></code></pre></figure>
<p>In this case, when we call the child stack we are not just doing a
simple #include-like practice, but creating an new child stack with
parameters and reosurces like any other stack. The parameters are
passed to it from the parent stack.</p>
<p>Obvious use cases for this include having a canoncial version of what
say an app server looks like, and passing in just the variables you
need for a specific implementation. A canonical version might have
metadata that must be applied, and you can turn those into parameters
passed into the child stack.</p>
<p>Why is this useful beyond just a standard version of a resource? Well
Heat includes an interator to stamp out multiple copies of a resource
from a single resource definition in the template.</p>
<p>The resource type is <code class="highlighter-rouge">OS::Heat::ResourceGroup</code>, and it accepts a resource
to stamp out, in the same format as a ‘top level’ resource (or whatever
you want to call them). It has <code class="highlighter-rouge">%index%</code> by default that will be substituded
for the index into the count you provide. So:</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">parameters</span><span class="pi">:</span>
<span class="na">Hostnames</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">comma_delimited_list</span>
<span class="na">default</span><span class="pi">:</span> <span class="s2">"</span><span class="s">first,second,third,forth,fifth"</span>
<span class="na">resources</span><span class="pi">:</span>
<span class="na">MyResourceGroup</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">OS::Heat::ResourceGroup</span>
<span class="na">properties</span><span class="pi">:</span>
<span class="na">count</span><span class="pi">:</span> <span class="m">5</span>
<span class="na">template</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">OS::Nova::Server</span>
<span class="na">properties</span><span class="pi">:</span>
<span class="na">flavour</span><span class="pi">:</span> <span class="s">abcd1</span>
<span class="na">namea</span><span class="pi">:</span> <span class="pi">{</span> <span class="nv">get_param</span> <span class="pi">[</span> <span class="nv">Hostnames</span><span class="pi">,</span> <span class="s1">'</span><span class="s">%index%'</span> <span class="pi">]</span> <span class="pi">}</span></code></pre></figure>
<p>Oh, but THIS DOES NOT WORK. You can’t use <code class="highlighter-rouge">%index%</code> in the function
like that. You can use it as literal string-sub values, but not inside
a function. There’s a <a href="https://bugs.launchpad.net/heat/+bug/1470614">bug</a>
which discusses this problem.</p>
<p>Here come nested stacks. Instead of stamping the actual resource, you
stamp out a nested stack, and can pass the index (and say, in the example
above, the list which is being indexed) into the child stack:</p>
<p>main.yaml:</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">parameters</span><span class="pi">:</span>
<span class="na">count</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">number</span>
<span class="na">names</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">comma_delimited_list</span>
<span class="na">resources</span><span class="pi">:</span>
<span class="na">ServerCluster</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">OS::Heat::ResourceGroup</span>
<span class="na">paramters</span><span class="pi">:</span>
<span class="na">count</span><span class="pi">:</span> <span class="pi">{</span> <span class="nv">get_param</span><span class="pi">:</span> <span class="nv">count</span> <span class="pi">}</span>
<span class="na">template</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">appserver.yaml</span>
<span class="na">properties</span><span class="pi">:</span>
<span class="na">names</span><span class="pi">:</span> <span class="pi">{</span> <span class="nv">get_param</span><span class="pi">:</span> <span class="nv">names</span> <span class="pi">}</span>
<span class="na">index</span><span class="pi">:</span> <span class="s1">'</span><span class="s">%index%'</span></code></pre></figure>
<p>appserver.yaml:</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">parameters</span><span class="pi">:</span>
<span class="na">names</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">comma_delimited_list</span>
<span class="na">index</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">number</span>
<span class="na">resources</span><span class="pi">:</span>
<span class="na">AnAppServer</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">OS::Nova::Server</span>
<span class="na">properties</span><span class="pi">:</span>
<span class="na">flavor</span><span class="pi">:</span> <span class="s">abcd1</span>
<span class="na">name</span><span class="pi">:</span> <span class="pi">{</span> <span class="nv">get_param</span> <span class="pi">[</span> <span class="nv">names</span><span class="pi">,</span> <span class="nv">index</span> <span class="pi">]</span> <span class="pi">}</span>
<span class="pi">[</span><span class="nv">...</span><span class="pi">]</span></code></pre></figure>
<p>This can be used to index any list as a paramter to the parent stack, and
it’s a useful trick for reducing the number of resources that need to be
defined, and provides some flexibility.</p>
<p>In another post, I’ll discuss how to abuse this to assign servers to
availability zones in a balanced distribution. Hint: YAQL.</p>It’s been ages since I posted anything, so here’s something that is probably not very useful. OpenStack Heat has reosurce groups which allow you to stamp out the same resource multiple times. Better yet, it provides a magic token %index% by default for which copy of a resource your template is run for. But %index% doesn’t get substituted by intrinsic functions. This unfortunately means you can’t use it in really obvious places like str_replace or use it to lookup a list by index with get_param. This would be useful for figuring out which AZ we want to launch an instance in by being passed a list of AZs and mangling the index from the resource group. However, all is not lost. There’s a way to do this. You just need to use nested stacks.Enterprise Architecture Modelling is Dead. All Hail Enterprise Architecture Modelling2019-07-23T00:00:00+12:002019-07-23T00:00:00+12:00http://hairy.geek.nz//architecture/2019/07/23/eam-is-dead-all-hail-eam<p>In this post, I’m going to argue that Enterprise Architecture Modelling
is dead, it was right to die, and it should be replaced with Enterprise
Architecture Modelling. Confused? Let me explain.</p>
<!--more-->
<p>When I first started as an Architect (with the actual title, rather than
just “I figured out how to make this and no-one argued with me”) the place
I was working at had a large, comprehensive, and detailed model of the
entire enterprise. It covered many different layers and showed the deeply
interconnected nature of the complex environment the company operated in
and had to operate in IT.</p>
<p>It was truly beautiful, and in the years which followed I spent a great
deal of time helping to maintain this model. It assisted in identifying
the root cause of faults, and helped work out who was responsible for
what, and helped with ensuring a consistent standard of design and
documentation was followed.</p>
<p>It made sure we had a clear understanding of what we had, where it was,
and how it interacted. It is incredibly valuable to have that understanding
because most organisations have no idea what they have. And therefore,
they can’t gauge their security posture, or understand their costs, or do
many other things that business stakeholders will ask from time time.</p>
<p>It was also a massive pain in the bum.</p>
<p>The hours which had to be sunk into it to keep it up to date was huge,
more than we could ever put it. Even if it could be automated with discovery
of “assets” you needed a person to filter all the noise and keep it
organised. Projects would duplicate resources all over the place, and
would not file what they had back into the main model structure.</p>
<p>It was used to document past intentions and future state, but since it
was a view on the current state only lots of diagrams would rot and become
invalid - maintaining those as well as the main model was too much.</p>
<p>And it could only really live in a world that everything was heavily
governed, that was strongly waterfall based with projects and explicit
design before work began. If, as we did, your organisation starts to
pivot towards “Digital” and towards speed as a priority - your model
is not going to survive that.</p>
<p>It died. I miss it, but I’m glad it died, because it wasn’t maintainable
before and it certainly isn’t maintainable in an “agile” (whatever spin
you put on that term) environment.</p>
<p>So why then am I arguing that an Enterprise Architecture Model is needed?</p>
<p>The consequence of not having a model of some kind is you end up with
no documentation, and scattered documentation in a dozen different formats
even when you do get it. You get solutions that claim to automatically
discover you application landscape that really are guesses and not curated
enough to be dependable.</p>
<p>And you still need to document <em>something</em> of your environment and your
landscape. That problem is not going to go away no matter how much the
belief in speed at any cost is pushed. (Why I despise developer-driven
agile is another rant in and of itself. Suffice to say, just pushing
everything into prod and undocumented might be fine for a startup, you
are playing with fire in any complex environment without a very
consistent culture of co-operation and agreement!)</p>
<p>The problem with Enterprise Architecture Models of the past is they were
just too verbose and decomposed to be maintained. It wasn’t that they
didn’t provide value, they did, they just made it too hard to get value
out of them.</p>
<p>If you look at the industry favoured metamodels for Enterprise Architecture
you will find far too much decomposition of systems, and a tendancy to
expect enumerating the very bottom of the layers (down in the weeds of
metal and containers and networks) when that’s highly unstable and
the most expensive to maintain.</p>
<p>What doesn’t tend to change are high-level needs - capabilities, applications
and what other applications they talk to, who owns these things. Those
high-level concepts are much more stable and durable over time.</p>
<p>Stop documenting what datacentre contains what rack contains what physical
server contains what OS contains what JVM contains what EAR contains what
application contains what applicaiton function and realizes what business
function and realizes what business process and and and and..</p>
<p>We need an Enterprise Architecture Model that is light, and has very few
different objects in the system. And given one starting from those high
level concepts, it’ll be useful to help understand what we have, and
be maintainable, and ensure even the most agile environments are still
something that can be documented.</p>
<p>I don’t have a metamodel to sell here - I wish I did but I don’t - so
this is more a call to put down the metamodels will hundreds of different
reosurce types and get back to documenting those simple high-level layers.</p>
<p>Everything else, well you can fill out if you really need it, but don’t add
anything assuming it has no cost to maintain.</p>In this post, I’m going to argue that Enterprise Architecture Modelling is dead, it was right to die, and it should be replaced with Enterprise Architecture Modelling. Confused? Let me explain.PriorityQueue in C#2019-06-27T00:00:00+12:002019-06-27T00:00:00+12:00http://hairy.geek.nz//2019/06/27/csharp-prio-queue<p>For a game I am working on, I need to perform some pathfinding on top of a
graph. The graph contains edges with a weight to allow us to pick the “best”
path that reflects a cost to use each path, rather than merely the smallest
number of nodes.</p>
<p>I’m using <a href="https://en.wikipedia.org/wiki/Dijkstra%27s_algorithm">Dijkstra’s Algorithm</a> which requires among other things a queue that is sorted by priority.</p>
<p>The game is being built in Unity, which uses C# as it’s preferred
programming language. C# has a <code class="highlighter-rouge">Queue<T></code>, and it has <code class="highlighter-rouge">SortedList<T></code> and
other structures by default that look useful, but have various
limitations, and this has annoyed me a great deal.</p>
<p><code class="highlighter-rouge">Queue<T></code> is strictly FIFO, you can <code class="highlighter-rouge">Queue<T>.Enqueue()</code> something to the
end, and <code class="highlighter-rouge">Queue<T>.Dequeue()</code> the head of the queue. And you can peek into
the queue etc. The only way to insert mid-queue is convert to a <code class="highlighter-rouge">List<T></code>
or an array and manipulate it, then re-create the queue from that
list/array. That sucks.</p>
<p><code class="highlighter-rouge">SortedList<T></code> is great, you can insert something into the list and it’ll
be automatically sorted, presuming you have an <code class="highlighter-rouge">IComparable</code> interface
on whatever type you’re using. But <code class="highlighter-rouge">SortedList<T></code> doesn’t accept duplicate
keys, so if you have two edges in your graph with equal weight it dies.</p>
<p>Okay, so we could just use a plain <code class="highlighter-rouge">List<T></code> and do the sorting ourselves.
And that’s fine, but if we’re doing that we take a penalty to insert into
the middle of <code class="highlighter-rouge">List<T></code> because under the hood it really resizes the array.
Which is more of a hit than we need to take - the queue will be adjusted
constantly while walking the graph.</p>
<p>C# doesn’t have <code class="highlighter-rouge">PriorityQueue<T></code>, even though it’s clearly a useful
structure to have (based on the number of implementations around!). It
seems to be stuck in limbo in tickets on GitHub about what corner cases
it would trip up on based on how someone expects to use it. Okay,
fine, we pick one.</p>
<p>Like I said, there’s a bunch of implementations. Some impose rules about
what type is used to sort the queue, most are probably more than I need.
But I also wanted to learn a bit more about C# so why not write something
myself. So, here it is .. a <code class="highlighter-rouge">PriorityQueue</code> that might work
okay and definitely isn’t optimised.</p>
<p>This uses <code class="highlighter-rouge">LinkedList<T></code> internally. Enqueue is O(n) because we have
to walk the linked list to find a place to insert. However, the insert
into the list itself is O(1) due to it being a linked list. It is a stable
queue, in that the order of enqueue is respected (effectively, equal priority
items behave like a FIFO). Dequeue is O(1) since it’s just the head.</p>
<p>I have no view on whether the “right” enqueue/dequeue costs should be O(n)
and O(1) respectively, but there’s probably not much overall gain in
shifting it around to try to make enqueue cheaper at the expense of dequeue.</p>
<p>You can use any type with <code class="highlighter-rouge">IComparable</code> hooks as the sort key. I’m using
it with <code class="highlighter-rouge">int</code> but if you had some reason to do it with <code class="highlighter-rouge">string</code> or
<code class="highlighter-rouge">MyCrazyType</code> then it should work fine.</p>
<p>Implemnentation follows. It probably needs <code class="highlighter-rouge">IEnumerable</code> support at some
point to allow LINQ support, but I don’t currently need it.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>using System.Collections.Generic;
using System;
namespace DataStructures.Queue {
public class PriorityQueueEntry<TPrio,TItem> {
public TPrio p { get; }
public TItem data { get; }
public PriorityQueueEntry(TPrio p, TItem data) {
this.p = p;
this.data = data;
}
}
public class PriorityQueue<TPrio,TItem> where IPrio : IComparable {
private LinkedList<PriorityQueueEntry<TPrio,TItem>> q;
public PriorityQueue() {
q = new LinkedList<PriorityQueueEntry<TPrio,TItem>>();
}
public int Count { get { return q.Count(); } }
public void Enqueue(TPrio p, TItem data) {
if (q.Count == 0) {
q.AddFirst(new PriorityQueueEntry<TPrio,TItem>(p,data));
return;
}
// This is a bit classical C but whatever
LinkedListNode<PriorityQueueEntry<TPrio,TItem>> current = q.First;
while (current != null) {
if (current.Value.p.CompareTo(p) > 0) {
q.AddBefore(current,new PriorityQueueEntry<TPrio,TItem>(p,data));
return;
}
current = current.Next;
}
q.AddLast(new PriorityQueueEntry<TPrio,TItem>(p,data));
}
public TItem Dequeue() {
// LinkedList -> LinkedListNode -> PriorityQueueEntry -> data
var ret = q.First.Value.data;
q.RemoveFirst();
return ret;
}
}
}
</code></pre></div></div>For a game I am working on, I need to perform some pathfinding on top of a graph. The graph contains edges with a weight to allow us to pick the “best” path that reflects a cost to use each path, rather than merely the smallest number of nodes. I’m using Dijkstra’s Algorithm which requires among other things a queue that is sorted by priority. The game is being built in Unity, which uses C# as it’s preferred programming language. C# has a Queue<T>, and it has SortedList<T> and other structures by default that look useful, but have various limitations, and this has annoyed me a great deal. Queue<T> is strictly FIFO, you can Queue<T>.Enqueue() something to the end, and Queue<T>.Dequeue() the head of the queue. And you can peek into the queue etc. The only way to insert mid-queue is convert to a List<T> or an array and manipulate it, then re-create the queue from that list/array. That sucks. SortedList<T> is great, you can insert something into the list and it’ll be automatically sorted, presuming you have an IComparable interface on whatever type you’re using. But SortedList<T> doesn’t accept duplicate keys, so if you have two edges in your graph with equal weight it dies. Okay, so we could just use a plain List<T> and do the sorting ourselves. And that’s fine, but if we’re doing that we take a penalty to insert into the middle of List<T> because under the hood it really resizes the array. Which is more of a hit than we need to take - the queue will be adjusted constantly while walking the graph. C# doesn’t have PriorityQueue<T>, even though it’s clearly a useful structure to have (based on the number of implementations around!). It seems to be stuck in limbo in tickets on GitHub about what corner cases it would trip up on based on how someone expects to use it. Okay, fine, we pick one. Like I said, there’s a bunch of implementations. Some impose rules about what type is used to sort the queue, most are probably more than I need. But I also wanted to learn a bit more about C# so why not write something myself. So, here it is .. a PriorityQueue that might work okay and definitely isn’t optimised. This uses LinkedList<T> internally. Enqueue is O(n) because we have to walk the linked list to find a place to insert. However, the insert into the list itself is O(1) due to it being a linked list. It is a stable queue, in that the order of enqueue is respected (effectively, equal priority items behave like a FIFO). Dequeue is O(1) since it’s just the head. I have no view on whether the “right” enqueue/dequeue costs should be O(n) and O(1) respectively, but there’s probably not much overall gain in shifting it around to try to make enqueue cheaper at the expense of dequeue. You can use any type with IComparable hooks as the sort key. I’m using it with int but if you had some reason to do it with string or MyCrazyType then it should work fine. Implemnentation follows. It probably needs IEnumerable support at some point to allow LINQ support, but I don’t currently need it. using System.Collections.Generic; using System; namespace DataStructures.Queue { public class PriorityQueueEntry<TPrio,TItem> { public TPrio p { get; } public TItem data { get; } public PriorityQueueEntry(TPrio p, TItem data) { this.p = p; this.data = data; } } public class PriorityQueue<TPrio,TItem> where IPrio : IComparable { private LinkedList<PriorityQueueEntry<TPrio,TItem>> q; public PriorityQueue() { q = new LinkedList<PriorityQueueEntry<TPrio,TItem>>(); } public int Count { get { return q.Count(); } } public void Enqueue(TPrio p, TItem data) { if (q.Count == 0) { q.AddFirst(new PriorityQueueEntry<TPrio,TItem>(p,data)); return; } // This is a bit classical C but whatever LinkedListNode<PriorityQueueEntry<TPrio,TItem>> current = q.First; while (current != null) { if (current.Value.p.CompareTo(p) > 0) { q.AddBefore(current,new PriorityQueueEntry<TPrio,TItem>(p,data)); return; } current = current.Next; } q.AddLast(new PriorityQueueEntry<TPrio,TItem>(p,data)); } public TItem Dequeue() { // LinkedList -> LinkedListNode -> PriorityQueueEntry -> data var ret = q.First.Value.data; q.RemoveFirst(); return ret; } } }Baby Status Liveblog 20192019-02-13T00:00:00+13:002019-02-13T00:00:00+13:00http://hairy.geek.nz//2019/02/13/baby-status<p>This will be a frequently updated page. No, it won’t be, but it’ll at
least have a broad status. Maybe.</p>
<h2 id="2019-03-04">2019-03-04</h2>
<p>Yes! All are doing well, and will be home soon :D :D</p>
<h2 id="2019-03-01">2019-03-01</h2>
<h3 id="late-update">Late Update</h3>
<p>Radio silence.</p>
<h3 id="early-update">Early update</h3>
<p>Ah, no.</p>
<h2 id="2019-02-28">2019-02-28</h2>
<p>No?</p>
<h2 id="2019-02-27">2019-02-27</h2>
<h3 id="afternoon-update">Afternoon Update</h3>
<p>Nope.</p>
<h3 id="early-morning-update">Early Morning Update</h3>
<p>Officially no, but hmmm..</p>
<h2 id="2019-02-26">2019-02-26</h2>
<p>Launch hold remains.</p>
<h2 id="2019-02-24">2019-02-24</h2>
<p>Revised plan. 0% chance at work is now 100% for a few more days.</p>
<h2 id="2019-02-22">2019-02-22</h2>
<p>More monitoring, and no bad news. Graphing data from various scans
suggests one place that started freaking out probably just messed up
and is far outside a reasoanble error margin.</p>
<h2 id="2019-02-20">2019-02-20</h2>
<p>Scan was fine. Little ones are decidedly not interested in going anywhere.</p>
<h2 id="2019-02-19">2019-02-19</h2>
<p>Scan day! Because this is far away, it’ll end up taking the whole day.
But for work people reading: I might not be around but no update!</p>
<h2 id="2019-02-18">2019-02-18</h2>
<p>An uneventful weekend, we are still in a launch count hold.</p>
<h2 id="2019-02-15">2019-02-15</h2>
<p>Set sail for .. oh, we’re still docked. Okay, well, maybe another day?</p>
<h2 id="2019-02-14">2019-02-14</h2>
<p>Valentines day! No news! Plan has been adjusted and approved.</p>
<h2 id="2019-02-13">2019-02-13</h2>
<p>No news is good news! I am still around.</p>This will be a frequently updated page. No, it won’t be, but it’ll at least have a broad status. Maybe. 2019-03-04 Yes! All are doing well, and will be home soon :D :D 2019-03-01 Late Update Radio silence. Early update Ah, no. 2019-02-28 No? 2019-02-27 Afternoon Update Nope. Early Morning Update Officially no, but hmmm.. 2019-02-26 Launch hold remains. 2019-02-24 Revised plan. 0% chance at work is now 100% for a few more days. 2019-02-22 More monitoring, and no bad news. Graphing data from various scans suggests one place that started freaking out probably just messed up and is far outside a reasoanble error margin. 2019-02-20 Scan was fine. Little ones are decidedly not interested in going anywhere. 2019-02-19 Scan day! Because this is far away, it’ll end up taking the whole day. But for work people reading: I might not be around but no update! 2019-02-18 An uneventful weekend, we are still in a launch count hold. 2019-02-15 Set sail for .. oh, we’re still docked. Okay, well, maybe another day? 2019-02-14 Valentines day! No news! Plan has been adjusted and approved. 2019-02-13 No news is good news! I am still around.Analysis of FENZ OIA response to data licensing2018-05-24T00:00:00+12:002018-05-24T00:00:00+12:00http://hairy.geek.nz//2018/05/24/analyis-fenz-oia-data<h1 id="introduction">Introduction</h1>
<p>Before going too much further, I wish to state that I am not a lawyer, and
this should not be read as legal interpreation. I’m a geek. I am writing
from the perspective of a geek. Take that into consideration.</p>
<p>Secondly, it helps if you read some of the background material:</p>
<ul>
<li><a href="https://medium.com/@phirate/the-boundaries-of-new-zealand-suburbs-must-be-publicly-released-9f0190dbb0b5">Phirate’s blog post on CC licensing of geometry data</a></li>
<li><a href="https://fyi.org.nz/request/7533/">The OIA request Phirate sent</a> which includes the response by FENZ</li>
</ul>
<h1 id="the-oia-response-as-a-whole">The OIA response as a whole</h1>
<p>The response from FENZ to the request for communications about licensing the
data is oddly filled with justifications for the decision about the license.</p>
<p>FENZ appears to be more concerned that releasing their discussions about
licensing will undermine their policy, than acting in a transparent manner
about those discussions. This is a concerning interpretation of the OIA,
the purpose of which is transparency and not a vehicle for an organisation
to spin their line on the information.</p>
<p>It suggests that FENZ internally, in at least some parts of the organisation,
recognise the policy is largely nonsense. That their response tries to bury
or cover-up this is not in the public interet, nor respecting transparency
expected of public service organisations.</p>
<h1 id="fenz-justification-for-restrictive-licenses">FENZ justification for restrictive licenses</h1>
<p>The core justification provided by FENZ to the restrictive license is
roughly summarised as:</p>
<ul>
<li>Previously, inconsistent data between agencies resulted in confusion
and potential public impact (ie, emergency services don’t go where
they were expected to)</li>
<li>Data used by others must be kept current, to ensure any use by the
public in an emergency is only using the current data</li>
</ul>
<p>The first of these is entirely an internal matter between the organisations
and has no bearing on the use of the data by anyone else. It does not
stand up as a good reason to prevent the distribution of the data under
an open license.</p>
<p>Those organisations need explicit policies and bilaterial agreements over
location information, simply because they are explicitly providing a
co-ordinated service. Those obligations are arguably steeper than the
criteria FENZ uses for the ‘public’ release of the data.</p>
<p>In other words, the presumption seems to be that the public shouldn’t have
a copy of the data because organisations who need to care about the data
aren’t doing a good job of that. That’s unfortunate, but that’s a bad
argument to withhold the data from an open license.</p>
<p>The second of these seems willfully ignorant of the likely public use in
an emergency. The assumption is that location information provided by
the public is based on the FENZ data. But the emergency service system
must still operate when a member of the public is not staring at a GPS
or positioning system.</p>
<p>Therefore, the system will always need to tolerate the degree of actual
accuracy a person can provide in an emergency, which is often not going
to be first by looking up the reference data set provide by FENZ.</p>
<p>Secondly, the more alarming assumption is that where the data is used by
device makers etc, the positioning accuracy is itself reliable. GPS and
other positioning systems are not perfect, so the locality information will
still have significant errors in it.</p>
<p>For example, if location app uses a blend of point data from wifi signals,
and uses the FENZ data to present the “correct” location, we have no idea
if the wifi signal data is correct anyway, and so the FENZ “correct”
location is arguably worse than a guess. That is, you can’t get a more
correct position if you start with bad data in the first place.</p>
<p>It is unclear then what the apparent public risk really is. The public
are not likely to consult the FENZ data for the “correct” location - just
witness the size of “Epsom” in any property listings for an example
of that syndrome - and where devices use the FENZ data we are presuming
their positioning is more accurate than we should trust.</p>
<p>The core justification just doesn’t stack up.</p>Introduction Before going too much further, I wish to state that I am not a lawyer, and this should not be read as legal interpreation. I’m a geek. I am writing from the perspective of a geek. Take that into consideration. Secondly, it helps if you read some of the background material: Phirate’s blog post on CC licensing of geometry data The OIA request Phirate sent which includes the response by FENZ The OIA response as a whole The response from FENZ to the request for communications about licensing the data is oddly filled with justifications for the decision about the license. FENZ appears to be more concerned that releasing their discussions about licensing will undermine their policy, than acting in a transparent manner about those discussions. This is a concerning interpretation of the OIA, the purpose of which is transparency and not a vehicle for an organisation to spin their line on the information. It suggests that FENZ internally, in at least some parts of the organisation, recognise the policy is largely nonsense. That their response tries to bury or cover-up this is not in the public interet, nor respecting transparency expected of public service organisations. FENZ justification for restrictive licenses The core justification provided by FENZ to the restrictive license is roughly summarised as: Previously, inconsistent data between agencies resulted in confusion and potential public impact (ie, emergency services don’t go where they were expected to) Data used by others must be kept current, to ensure any use by the public in an emergency is only using the current data The first of these is entirely an internal matter between the organisations and has no bearing on the use of the data by anyone else. It does not stand up as a good reason to prevent the distribution of the data under an open license. Those organisations need explicit policies and bilaterial agreements over location information, simply because they are explicitly providing a co-ordinated service. Those obligations are arguably steeper than the criteria FENZ uses for the ‘public’ release of the data. In other words, the presumption seems to be that the public shouldn’t have a copy of the data because organisations who need to care about the data aren’t doing a good job of that. That’s unfortunate, but that’s a bad argument to withhold the data from an open license. The second of these seems willfully ignorant of the likely public use in an emergency. The assumption is that location information provided by the public is based on the FENZ data. But the emergency service system must still operate when a member of the public is not staring at a GPS or positioning system. Therefore, the system will always need to tolerate the degree of actual accuracy a person can provide in an emergency, which is often not going to be first by looking up the reference data set provide by FENZ. Secondly, the more alarming assumption is that where the data is used by device makers etc, the positioning accuracy is itself reliable. GPS and other positioning systems are not perfect, so the locality information will still have significant errors in it. For example, if location app uses a blend of point data from wifi signals, and uses the FENZ data to present the “correct” location, we have no idea if the wifi signal data is correct anyway, and so the FENZ “correct” location is arguably worse than a guess. That is, you can’t get a more correct position if you start with bad data in the first place. It is unclear then what the apparent public risk really is. The public are not likely to consult the FENZ data for the “correct” location - just witness the size of “Epsom” in any property listings for an example of that syndrome - and where devices use the FENZ data we are presuming their positioning is more accurate than we should trust. The core justification just doesn’t stack up.What happened?2016-01-13T00:00:00+13:002016-01-13T00:00:00+13:00http://hairy.geek.nz//2016/01/13/what-happened<p>So obviously the design of the blog has changed rather radically.</p>
<p>What happened was that the WordPress instance I was hosting myself was
let’s face it, poorly maintained, and thus had a whole lot of crap
injected into the pages and content. That crap was then hijacking people
browsering into a bunch of scam sites. Ugh.</p>
<p>Thankfully I have backups, and have largely restored the content back
to the way it was. But fed up with having to patch WordPress, I’ve
decided to switch to a static blog system called Jekyll.</p>
<p>It’s a little rough to use, but I appreciate just being able to write
markdown in an SSH session, and the hosting can just be an S3 bucket.
Which let’s face it, is a whole lot more awesome than my own Apache
instance running PHP.</p>
<p>So this will be a bit of a work in progress while I resurrect the
content and restyle it, but the meat is all still here. It just doesn’t
look so great right now.</p>So obviously the design of the blog has changed rather radically. What happened was that the WordPress instance I was hosting myself was let’s face it, poorly maintained, and thus had a whole lot of crap injected into the pages and content. That crap was then hijacking people browsering into a bunch of scam sites. Ugh. Thankfully I have backups, and have largely restored the content back to the way it was. But fed up with having to patch WordPress, I’ve decided to switch to a static blog system called Jekyll. It’s a little rough to use, but I appreciate just being able to write markdown in an SSH session, and the hosting can just be an S3 bucket. Which let’s face it, is a whole lot more awesome than my own Apache instance running PHP. So this will be a bit of a work in progress while I resurrect the content and restyle it, but the meat is all still here. It just doesn’t look so great right now.BBPSU 3.0 on the way!2015-01-23T08:53:32+13:002015-01-23T08:53:32+13:00http://hairy.geek.nz//projects/production/2015/01/23/bbpsu-3-0-on-the-way<p>It's been a few years since I did a major revision of the Breadboard PSU, but that time has come. Below is a render of the most recent revision about to enter manufacture:</p>
<p><a href="http://hairy.geek.nz/wp-content/uploads/2015/01/bbpsu30.png"><img class="aligncenter size-medium wp-image-683" src="http://hairy.geek.nz/wp-content/uploads/2015/01/bbpsu30-300x276.png" alt="bbpsu30" width="300" height="276" /></a>This version includes better thermal performance for the 5V regulator, by switching to a D-PAK (TO-252) package. This has more surface area and a larger heat sink area on the board itself, and provides a more stable surface to install a small heatsink.</p>
<p>The other major change is that rather than driving the 3.3V regulator directly off the DC input, it is chained off the 5V regulator. This actually allows more current to be drawn from the 3.3V regulator, since it has a smaller voltage drop to regulate down to. This does mean the total output is limited by the 5V regulator, but it is a considerable improvement in what you can draw from the 3.3V rail.</p>
<p>The remaining changes are mostly cosmetic. This happens to be the first version I've used Kicad for, which is unexpected to me given I've used Kicad on a number of projects now!</p>
<p>These should be available soon from nicegear.co.nz!</p>
<p> </p>dave2It's been a few years since I did a major revision of the Breadboard PSU, but that time has come. Below is a render of the most recent revision about to enter manufacture: This version includes better thermal performance for the 5V regulator, by switching to a D-PAK (TO-252) package. This has more surface area and a larger heat sink area on the board itself, and provides a more stable surface to install a small heatsink. The other major change is that rather than driving the 3.3V regulator directly off the DC input, it is chained off the 5V regulator. This actually allows more current to be drawn from the 3.3V regulator, since it has a smaller voltage drop to regulate down to. This does mean the total output is limited by the 5V regulator, but it is a considerable improvement in what you can draw from the 3.3V rail. The remaining changes are mostly cosmetic. This happens to be the first version I've used Kicad for, which is unexpected to me given I've used Kicad on a number of projects now! These should be available soon from nicegear.co.nz! Kakapo net+sd shield2014-10-07T08:06:28+13:002014-10-07T08:06:28+13:00http://hairy.geek.nz//projects/production/2014/10/07/kakapo-netsd-shield<p>Those of you who have been following my personal twitter account will have seen endless pictures of the shiny new net+sd shield for the Kakapo.</p>
<p><a href="http://hairy.geek.nz/wp-content/uploads/2014/10/20140927_160940.jpg"><img class="aligncenter size-medium wp-image-669" src="http://hairy.geek.nz/wp-content/uploads/2014/10/20140927_160940-300x168.jpg" alt="20140927_160940" width="300" height="168" /></a></p>
<p>There's plenty of shields with ethernet and SD card support, but there's a few different things about this one compared to others:</p>
<ul>
<li>The latest WizNet W5500 hardware TCP/IP chip
<ul>
<li>32kB SRAM, 8 sockets</li>
<li>Automatic circular buffer management</li>
<li>Compact 48-pin LQFP chip</li>
<li>SPI support up to 80MHz</li>
<li>Variable length transfers to reduce overhead</li>
<li>10/100 Full and Half Duplex support</li><br />
</ul><br />
</li></p>
<li>Built-in I2C MAC address EEPROM</li>
<li>MicroSD card socket</li>
<li>W5500 chip is automatically reset by Kakapo reset sources</li>
<li>W5500 and SD card MISO shield pin is buffered; does not drive pin at all when deselected</li>
<li>Four status LEDs: Link, Activity, 100M and Full Duplex</li>
<li>Jumpers for easy disable/enable features; no soldering to free shield pins from features not required!
<ul>
<li>Jumpers for W5500 !CS line and !INT lines</li>
<li>Jumper for SD card !CS line</li>
<li>Jumper for I2C 4.7k pull-up resistors</li>
<li>Opening !CS jumpers enforces part remains deselected</li><br />
</ul><br />
</li></p>
<li>MAC address sticker for easy identification</li>
<li>All pins used explained on the bottom side of the shield</li>
<li>Same pins used as the common Arduino Ethernet shield</li><br />
</ul><br />
The shield is going through some final testing, and will be available soon!</p>dave2Those of you who have been following my personal twitter account will have seen endless pictures of the shiny new net+sd shield for the Kakapo. There's plenty of shields with ethernet and SD card support, but there's a few different things about this one compared to others: The latest WizNet W5500 hardware TCP/IP chip 32kB SRAM, 8 sockets Automatic circular buffer management Compact 48-pin LQFP chip SPI support up to 80MHz Variable length transfers to reduce overhead 10/100 Full and Half Duplex support Built-in I2C MAC address EEPROM MicroSD card socket W5500 chip is automatically reset by Kakapo reset sources W5500 and SD card MISO shield pin is buffered; does not drive pin at all when deselected Four status LEDs: Link, Activity, 100M and Full Duplex Jumpers for easy disable/enable features; no soldering to free shield pins from features not required! Jumpers for W5500 !CS line and !INT lines Jumper for SD card !CS line Jumper for I2C 4.7k pull-up resistors Opening !CS jumpers enforces part remains deselected MAC address sticker for easy identification All pins used explained on the bottom side of the shield Same pins used as the common Arduino Ethernet shield The shield is going through some final testing, and will be available soon!API changes and new stuff coming2014-09-03T00:48:24+12:002014-09-03T00:48:24+12:00http://hairy.geek.nz//projects/2014/09/03/api-changes-and-new-stuff-coming<p>Kakapo API updates first. The SPI API was changed yet again to include providing the byte to send when stuffing the TX side with irrelevant data. The "txdummy" param to spi_conf() configures what the byte should be when stuffing.</p>
<p>This is mostly useful for dealing with SD cards over SPI, as to push them into SPI mode you need to issue >=74 SCK cycles with its data-in pin high. Stuffing the output with 0xff does the trick for this.</p>
<p>Since I've mentioned SD cards, the Net+SD shield is nearly complete and should see the first prototype ordered next week. This includes a WizNet W5500 TCP/IP chip and a microSD card socket.</p>
<p>The shield is roughly similar to the official Arduino Ethernet shields (using the same pins as that shield), but includes a MAC EEPROM on I2C and all used pins can be disabled with simple jumpers. This means you can use just the Ethernet or SD functions of the board. It also includes pull-up resistors on the I2C port, with a jumper to disable them if you already have pull-ups somewhere else.</p>
<p>To ensure the W5500 reset is consistent with the main Kakapo board, there is a buffered reset handler. The output pins of both the SD card and the W5500 are also buffered and controlled by the CS pin for each function to ensure no disturbance of header pins they are connected to.</p>
<p>The W5500 is an upgraded version of the W5100 featured on Arduino official shields. It includes 32k SRAM (up from 16), 8 sockets (up from 4), and allows variable length transfers (instead of fixed 1, 2 or 4 byte transfers). The latter means we can push larger blocks of data into or pull out of sockets with less overhead.</p>
<p>Since I know people will want to hit the ground running, a basic driver for the W5500 for TCP client and UDP in both directions has been pushed into libkakapo. There is a stub which still needs to be completed for TCP server support. The TCP code supports using standard IO functions with TCP connections, like the USART driver.</p>
<p>At this point, I'm still working on the SD card support, and it's likely the board will be released before this support is completed in libkakapo. The W5500 driver also still uses fixed IP configuration, a DHCP client and other high-level protocols (like DNS) are still to come.</p>dave2Kakapo API updates first. The SPI API was changed yet again to include providing the byte to send when stuffing the TX side with irrelevant data. The "txdummy" param to spi_conf() configures what the byte should be when stuffing. This is mostly useful for dealing with SD cards over SPI, as to push them into SPI mode you need to issue >=74 SCK cycles with its data-in pin high. Stuffing the output with 0xff does the trick for this. Since I've mentioned SD cards, the Net+SD shield is nearly complete and should see the first prototype ordered next week. This includes a WizNet W5500 TCP/IP chip and a microSD card socket. The shield is roughly similar to the official Arduino Ethernet shields (using the same pins as that shield), but includes a MAC EEPROM on I2C and all used pins can be disabled with simple jumpers. This means you can use just the Ethernet or SD functions of the board. It also includes pull-up resistors on the I2C port, with a jumper to disable them if you already have pull-ups somewhere else. To ensure the W5500 reset is consistent with the main Kakapo board, there is a buffered reset handler. The output pins of both the SD card and the W5500 are also buffered and controlled by the CS pin for each function to ensure no disturbance of header pins they are connected to. The W5500 is an upgraded version of the W5100 featured on Arduino official shields. It includes 32k SRAM (up from 16), 8 sockets (up from 4), and allows variable length transfers (instead of fixed 1, 2 or 4 byte transfers). The latter means we can push larger blocks of data into or pull out of sockets with less overhead. Since I know people will want to hit the ground running, a basic driver for the W5500 for TCP client and UDP in both directions has been pushed into libkakapo. There is a stub which still needs to be completed for TCP server support. The TCP code supports using standard IO functions with TCP connections, like the USART driver. At this point, I'm still working on the SD card support, and it's likely the board will be released before this support is completed in libkakapo. The W5500 driver also still uses fixed IP configuration, a DHCP client and other high-level protocols (like DNS) are still to come.