Caffeinated fumblings

Using visualizations to help understanding

Wed, 17 Aug 2022 00:00:00 +0000

I’ve recently started (re)reading “Principles of Product Development Flow” by Donald G. Reinertsen. I’ve read it before but it did not stick. I forgot most of what it had to say and I’m not sure I actually understood much of it the last time around.

A tactic I’m trying out to better retain what I read is to look out for opportunities to create visualizations.

For me visualizations are powerful tools to help guide understanding and I’ve been trying to use utilize that power as I read.

One section of “Principles of Product Development Flow” talks about random processes and uses the following as an example: Image you flip a coin while keeping a running tally. If you get heads you add one from your tally and if get tails you subtract one.

Reinertsen then continues:

In general, even numerate people (such as engineers and scientists) have limited intuition for the behavior of random processes.

Sounds like a good opportunity for a visualization!

Lets draw the running tally of the coin flips as we run trough the random process. The length of the process shown below 685. Once a process has run trough its 685 steps we start a new process and begin drawing it on top of the old result.

Sure enough! That graph did not match my intuition. My expectation was that the graph would continuously hover around the zero line. But it doesn’t!

Boo For my intuition! Hoorah for visualizations!

The fun with random process does not stop there though, because Reinertsen continues:

Although the most likely value of the cumulative sum is zero, this value becomes less likely with time.

Sounds like it’s time for histograms! My idea was to run a lot of the coin flip random processes and plot the frequency of results.

For each histogram below a random process (of the length indicated in the captions) is run for every frame, the result saved and the histogram updated.

You should see the distribution slowly settle down and it becomes visually obvious that the longer the sequence the lower the likelihood of hitting zero - even though zero is still the most likely result. The distribution flattens out.

Why I Think This Helps

Creating a visualization offers me an opportunity to work trough the text and understand it well enough to turn it into a visualization. I can’t code visualizations for what I don’t understand.

And, I’m hoping, that will help me better retain what I read.

Are we pair programming too much?

Sun, 08 Aug 2021 21:00:00 +0000

The team I’m part of had been working towards making pair programming the default mode of work, when one of the team members asked: “Are we pair programming too much?”.

“It’s at least twice as fast as when everyone works on their own” is something I’ve heard, it’s something I’ve said and it’s something I’ve always just sort of taken for solid fact. We couldn’t be doing too much pair programming if it’s the most efficient way of working.

But it is though? The answer may be that it’s nowhere as simple as that. In a review of 18 studies¹ the conclusion was:

Our meta-analysis suggest that pair programming is not uniformly beneficial or effective

The review of pair programing looked at the effectiveness of pair programming on:

Duration: The calendar time it takes to complete a task
Effort: The joint effort used to complete a task
Quality: The quality of the work, by looking at passed tests, grades and other metrics

The results of the review can be summarized as: Pair programming gives you a a small improvement on quality and a medium improvement in duration, but you pay for those with a medium negative effect of effort.

The negative effect on effort means that we less back from pair programming than if we just let everyone work on their own.

Seems like if we take the research at its word then perhaps, we are indeed pair programming too much? But what’s going on and is there something we can do to improve the situation? Or at least supply some guidance on when pair programming is likely to be the most effective?

Part of the reason for the results and a potential source of guidance may be found in the phenomenon of social loafing.

Loaf: to avoid activity, especially work

Social loafing is the phenomenon that people tend to exert less effort when working collaboratively than they would have if they had worked on the same task alone.

The classical example is that of pulling on a rope to lift a weight - you might expect two people to be able to pull twice as hard and three people to lift three times as hard.

That turns out not to be the case: You get less joint effort than you’d expect from simple addition of what the individuals are capable on their own.

It’s worth noting that this is separate from any coordination loss you might expect.

Size of the Effect

How big of an effect is social loafing? “Social Loafing: A Meta-Analytic Review and Theoretical Integration”² on which I’m leaning heavily on terms it as:

within the middle range of effects in the domain of social behavior

More formally the review found an effect size of 0.44 expressed in terms of Cohen’s D.

I’m going to tread very carefully here because I’m by no means a statistician, but Cohen’s D puts a value on how big an effect is - more specifically how big of a difference there is between the mean values of two groups. In this case the two groups are the group working alone and the group working together. A Cohen’s D value of 1 means that the groups differ by 1 standard deviation.

This does not at once make 0.44 more intuitively understandable but an effect size 0.44 falls between a “small” and a “medium” size as per the rule of thumb interpretation of effect sizes suggested by Cohen. Medium being “probably big enough to be noticed by the naked eye”.

I’ll go out on a limb and summarize that to mean: The effect is there, and it is large enough for us to care about.

But I don’t loaf!

About now you might have the same objection that I had: “I definitely don’t loaf!”.

Well, I’m sorry to say but you probably do - it’s just that you may not be aware of it. In experiments where participants were asked to self-report on their effort, they tended to report a similar level of effort as when working alone when they in fact had not.

So, we’re either unaware or unwilling to admit that we loaf.

But Programming is Different!

The effect has also been proven to be robust across a range of activities from the simple physical task of pulling on a robe, to shouting, brainstorming, to maze solving, to evaluation of information, to typing and pushing buttons.

I think we would be misleading ourselves if we think programming to be except from the effects of social loafing.

So social loafing is a thing and most if not all of us do it. Now what?

Social loafing does not occur at an equal strength for all tasks nor does it under all circumstances but instead seems to be affected by several factors.

Can knowledge of these factors help us in deciding when to pair program. I think they can or at least some of them can be helpful in guiding us when we build teams and try to steer our engineering culture.

I’ll skip the discussion of factors related to gender and ethnicity (women tend loaf less and so does people from eastern cultures). I’ll also skip as factors that I think are detrimental to building great teams and cultures (for example we loaf less when we think our coworkers will do a bad job). Instead, I’ll focus on the factors I believe gel well with build great teams.

Go check out the article if your curious to know all the factors.

Task Valance

We loaf less when we feel we’re working on tasks that are intrinsically important. This shouldn’t be much of a surprise.

What may come as a surprise is that it is likely that we are sensitive to how others feel and talk about a task.

Therefore: Don’t talk down the importance of tasks. Not only are you likely to bring some you own performance, but you’ll likely bring down the performance of those working with you.

While the reverse effect also exists (that is you can talk a task up and get increased performance) I’d be hesitant to use it. Instead, I’d save pair programming to talks that are actually interesting and important.

Task complexity

The complexity of the task at hand plays a role on social loafing. We loaf more on simple tasks than on complex tasks.

On complex tasks the effect of social loafing may even be reversed, and individuals put more effort into collaborative working than they would working on their own.

Uniqueness of Individual Input

We loaf less when we feel we bring something unique to the table.

While unsurprising it is still worth keeping in mind, especially when doing novice-expert pairs where the novice might feel that their input is redundant.

Stressing that learning and gaining familiarity with the codebase (and not just task solving) is the focus in novice-expert pairs might help keep the motivation up in that specific context.

Team Cohesiveness and Identity

Team cohesiveness and and feeling of identity plays a strong role in social loafing.

This is probably the best news of this article: Strong teams may be able to more or less completely eliminate the effect of social loafing.

Group Size

Social loafing increases with group size. The bigger the group the larger the effect.

Take away: Mob sessions are at a higher risk of social loafing, so reserve them for situations where the other factors play in your favor.

Closing thoughts

The next time someone asks me about the reasonability of doing pair programming I think I’ll change may answer from “It’s at least twice as fast” to:

“I believe that pair programming is a worthwhile investment in knowledge sharing, team building and learning - but for some tasks it will be an investment”.

I still strongly believe that pair programming is worth the investment, but I also believe that we should be honest about it being an investment.

We invest effort and get back knowledge sharing that help reduce the risk of knowledge silos.

We invest effort and (when it works well) we help build a more cohesive team.

We invest effort and increase learning.

But a silver bullet it is not, and it comes with its own set of constraints. Knowing the factors that affect social loafing can help you make a conscious decision on when to do pair programming and get more out of the pair programming you do.

Key Takeaways

Social loafing scales with group size: Be especially mindful of the effect when doing mob programming
Strong team cohesiveness is the strongest factor affecting social loafing
Focus on the complex tasks: Quality improves with pair programming on complex tasks and complexity affects social loafing

Materiale lavet til Coding Pirates Horsens [In Danish]

Tue, 16 May 2017 15:00:00 +0000

Undervisning er svært, virkeligt svært. Det har været min primære erfaring efter knap to sæsoner hos Coding Pirates Horsens.

Minecraft

Første forsøg var at skrive et hæfte om at lave mods til Minecraft via JavaScript. Tanken var at ungerne kunne læse lidt, løse opgaver, få hjælp når det var svært.

Forløbet skulle vare fire uger, og så skulle de videre til næste forløb. Det fungerede tåleligt, specielt delen hvor “programmeringen” foregik direkte i Minecraft.

Så kom sæson to. Nu skulle der kød på så fortsætterne følte de kom fremad. De skulle lave et Battle Royale mod i Minecraft. Dele af forløbet gik godt; Det var sjovt at lave tryllestave (eller fisk) der skød med ildkugler og fik modstanderen til at flyve op i luften. Men forløbet mindede om klassisk klasserumsundervisning, og det har ungerne allerede haft rigeligt af når de møder op klokken fem på en varm eftermiddag.

Forløbet var for svært og de begyndte at kede sig. Det skriftlige element kom i vejen. Det er svært at få skrevet et for-loop rigtigt, der går kludder i parenteserne og fejlbeskederne er uforståelige. Der må prøves noget nyt.

Hæftet kan i øvrigt ses her.

Flipped Classroom og GameMaker

Det nye forsøg er inspireret af Flipped Classroom. Der er skiftet til GameMaker. Bogen er smidt ud til fordel for små YouTube videoer. Det er nemmere at gå til, man kommer hurtigere frem og man kan fokusere på andre ting end programmering. Man kan lege med graffik, lyd, level-design og masser af andre ting. Og med GameMakers Drag-and-drop programmering fjernes en stor del af det skriftlige element der kom i vejen for Minecraft forløbet.

Adding a Elm compilation to you Mix project

Wed, 22 Mar 2017 20:00:00 +0000

We’ve been using cowboy for some time at work for writing small REST-endpoints and lately Elm as been added to the mix to provide front-ends for those endpoints.

At the beginning the work flow was to compile the Elm assets, copy the result to the priv/ in our cowboy projects and let cowboy serve them from there. That work flow feelt a bit clunky. If you made changes to both the cowboy handlers and the Elm code you would have to run both elm make and mix compile.

To make that work flow a bit more smooth we have integrated the Elm compile cycle into mix compile. You can see the end result here: elm-compile.

If you’re just looking an easy way to integrate Elm into your mix projects just stop reading now and follow the link above. If on the other hand you’re interested in how to add external compilers to a mix project read on.

Two approaches

There are two approaches to adding Elm compilation to your Mix project: Override mix compile with an alias or add Elm compilation to the list of compilers used by mix compile.

An aside on writing your own Mix tasks

Both approaches will require us to write our own Mix tasks. Luckily thats easy.

A Mix task can be defined by simply using Mix.Task in a module starting with Mix.Tasks. and defining the run/1 function: – Mix.Task behaviour docs

It basically boils down to createing a module in the Mix.Task namespace, adding use Mix.Task and implementing run(args) as required by the Mix.Task behaviour. You can find an example here .

Using aliases

If you’ve already encountered Mix aliases if you’ve ever created a Phoenix. If you look at the mix.exs file in a Phoenix project you’ll see the following:

  def project do
    [app: .....
     version: .....
     elixir: .....
     elixirc_paths: .....
     aliases: aliases(),
     deps: deps()]
  end

  defp aliases do
    ["ecto.setup": ["ecto.create", "ecto.migrate", "run priv/repo/seeds.exs"],
     "ecto.reset": ["ecto.drop", "ecto.setup"],
     "test": ["ecto.create --quiet", "ecto.migrate", "test"]]
  end

Here they are using aliases to create convenience tasks like ecto.setup that creates, migrate and seeds the database.

But you can also use aliases to override existing Mix tasks. So given that we have created a Mix task in the ‘Mix.Tasks.Compile.Elm’ module you could create the following alias:

defp aliases do
   ["compile": ["compile", "compile.elm"]]
end

We are in essence telling mix to first run its normal compile and afterwards run the Elm compile task. Mix aliases are awesome and can be really helpful in smoothing out your work flow. But overriding the compile task is perhaps not the best use of aliases.

Adding a compiler

The other alternative is to place you task in the Mix.Tasks.Compile.Elm namespace and then adding it to the list of Mix compilers.

Again: Take a look at the mix.exs file from a Phoenix project. Under the project definition you’ll spot the following: compilers: [:phoenix, :gettext] ++ Mix.compilers.

They are adding :phoenix and :gettext to the compilers run by mix compile. Modify that to compilers: Mix.compilers ++ [:elm] and you’re good to go.

Building your own GenServer

Mon, 24 Oct 2016 20:00:00 +0000

Sometimes it’s a good learning experience to try and reimplement some of the functionality that a standard library provides. So how much does it take to reimplement the simple parts of a GenServer?

The simple parts

A GenServer provides you with a lot of functionality out of the box:

The goal of a GenServer is to abstract the “receive” loop for developers, automatically handling system messages, support code change, synchronous calls and more
– The GenServer docs

Let’s focus on implementing the “receive” loop, synchronous calls and making sure that it fits in Supervision tree.

We’re going to more or less implement the Stack example from the GenServer docs but without using a GenServer.

Creating the project and the first Stack module

We need a project to get us started and we’re going to need a supervisor. So mix new stack --sup is is.

Start out simple by defining the interface we want. We’ll need push and pop functions to be a stack. And if we want to start the stack as worker child of the appliction Supervisor we’ll also need a start_link function.

defmodule SimpleStack do
  def push(item) do
  end

  def pop() do
  end

  def start_link() do
  end
end

The `start_link` function

We’re not going for anything like a full re-implementation of GenServer so let’s try and keep our start_link as simple as possible. Reading over the start_link/3 docs and looking at the possible return values we see that we´re supposed to return a {:ok, pid} tuple with the pid of the server.

Start out by adding SimpleStack to the Supervisor in Stack

  children = [
      # Starts a worker by calling: Stack.Worker.start_link(arg1, arg2, arg3)
      # worker(Stack.Worker, [arg1, arg2, arg3]),
      worker(SimpleStack, [])
    ]

Running the application now (mix run) will result in a error:

** (Mix) Could not start application stack: Stack.start(:normal, []) returned an error: shutdown: failed to start child: SimpleStack
** (EXIT) nil

So let’s start working on that start_link function. Should be simple enough: We just need to start a new process with spawn_link and return it’s pid.

defmodule SimpleStack do
  require Logger

  def start_link() do
    pid = spawn_link(fn () -> Logger.debug "Started" end)
    {:ok, pid}
  end

  def push(item) do end
  def pop() do end
end

Let’s try running that again: mix run --no-halt

21:14:16.783 [debug] Started

21:14:16.783 [debug] Started

21:14:16.783 [debug] Started

21:14:16.783 [debug] Started

Well, that didn’t work. But what is going on? Try adding max_restarts: 10 to the options sent to the supervisor like so:

# See http://elixir-lang.org/docs/stable/elixir/Supervisor.html
# for other strategies and supported options
opts = [strategy: :one_for_one, name: Stack.Supervisor, max_restarts: 10]
Supervisor.start_link(children, opts)

Then run the application again. Now we get our [debug] Started ten times. So the supervisor keeps calling start_link.

Time to fire up iex and do some digging:

Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Interactive Elixir (1.3.1) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> pid = spawn_link(fn () -> IO.puts "Started" end)
Started
#PID<0.82.0>
iex(2)> Process.alive? pid
false
iex(3)>

Ah! Out process stops as soon as its done its work. The supervisor sees this and then restarts the process and keeps doing it until it reaches its max_restarts limit.

The receive loop

Processes communicate via messages and each process has its own mailbox. We can use the receive do end statement to wait for messages to arrive at our mailbox. First let’s start out by listening for a :pop message that we’ll use to support the pop functionality of our stack.

I’ve changed the spaw_link statement so that we can move the receive loop into its own function:

def start_link() do
    pid = spawn_link(__MODULE__, :receiver, [[]])

    {:ok, pid}
  end

  def push(item) do
    #Add something here...
  end

  def pop() do end


  def receiver(state) do
    state = receive do
      {:push, element} ->
        [element | state]
      _ ->
        Logger.warn("Unexpected messaged received")
        state
    end
    receiver(state)
  end

The receiver function implements the receive loop. It gets initialized with an empty list. The receive statement lets us patterne match on incomming messages. We handle a {:push, element} message by adding the new element to the front of the list and after doing so we call the receiver recursively to wait for the next message.

Pushing

Next up is implementing the push/1 function in the SimpleStack module. We know that we need to send a {:push, element} message to the process we created in start_link. But how do we find the pid of that process again?

You know how you can register a GenServer with a name by calling something like this: GenServer.start_link(Stack, [:hello], name: MyStack).

Turns out that giving a process a name is a pretty useful feature, and a feature that is provided by Process.register/2.

Great, well extend the start_link function to use that.

def start_link() do
  pid = spawn_link(__MODULE__, :receiver, [[]])
  Process.register(pid, :stack)
  {:ok, pid}
end

def push(element) do
  send(:stack, {:push, element})
  :ok
end

And now the push/1 function can easily send a message to the process with: send(:stack, {:push, element}).

And so we have a simple form of GenServer.cast.

Returning a response

The last thing we need to be able handle synchronous calls. Again we do this by sending messages. We’ll define a new message: {:pop, pid}. So besides telling that we want to perform a pop on the stack we’ll also send along the PID of the process to send the result to.

def pop() do
  # Send the pop message with the PID of the current process.
  send(:stack, {:pop, self()})

  # And wait for the response
  receive do
    {:ok, element} ->
      element
    _ ->
      Logger.warn("Unexpected messaged received")
      :error
  end
end


def receiver(state) do
  state = receive do
    {:push, element} ->
      [element | state]
    {:pop, caller} ->

      [element | tail] = state
      send(caller, {:ok, element})

      tail
    _ ->
      Logger.warn("Unexpected messaged received")
      state
  end
  receiver(state)
end

The calling function pop/0 starts by sending the :pop message. It finds its own PID by calling self(). Then it waits for the response message with receive.

The :pop message is handled we take the first element, send it to the calling process and then recursively call the receiver/1 function with the popped stack.

And there we have the synchronous call we wanted. You can find the completed code here if you want to play around with it.

Unexpected message

Mon, 24 Oct 2016 20:00:00 +0000

Do you know on which process the start_link function of a GenServer in Elixir is run?
GenServer is one of the central building blocks often turned to when building something in Elixir but the above question is not something I had ever really given any thought.

I knew that you had to be mindful of which process is doing the work when working with agents but had somehow missed that the same might be the case with GenServers.

Background

I’ve been playing around with nerves, a Raspberry Pi and an Arduino board. One the things I was trying to do was to read messages sent from the Arduino to the Pi over USB.
The nerves_uart makes it simple to read lines from USB and to receive those lines as as messages. So the plan was to setup a GenServer to start nerves_uart and to receive the messages from it.

Unexpected messages

The first attempt at the code to start nerves_uart and receive the messages looked a bit like this:

defmodule HelloNerves.Messages do
  use GenServer
  require Logger

  def start_link do
    # Start the Nerves.UART GenServer
    {:ok, _} = Nerves.UART.start_link

    # We want to listen to the USB plugged in at "ttyACM0"
    # and use newlines as a separator for the data.
    Nerves.UART.open(pid, "ttyACM0", speed: 9600, active: true,
      framing: {Nerves.UART.Framing.Line, separator: "\r\n"})

    GenServer.start_link(__MODULE__, :ok, [])
  end

  def handle_info({:nerves_uart, _serial_port_name, data}, state) do
    Logger.debug "Got info: #{data}"
    {:noreply, state}
  end
end

The HelloNerves.Messages is then started with as a worker with worker(HelloNerves.Messages, []).

Compile, upload to the Raspberry PI, sit back and watch the data flow.

Unintended recipient

Well, not quite as it turns out. Because instead of the loging of the data I expected I started seeing the following error message instead:
Supervisor received unexpected message {:nerves_uart, "ttyACM0", "400"}

Let’s dig into the nerves_uart source and see if we can find the source of our troubles. The source for the Nerves.UART GenServer can be found here: Github.

If we look at the function that handles the open call that the pid of the calling function gets stored in the state as controlling_process. And messages are sent to the controlling_process, but why do these messages end up at the Supervisor?

The call sequence

HelloNerves.Messages is started with the worker() statement causing HelloNerves.Messages.start_link to be called which then in turn calls Nerves.UART.start_link.

In overview something like this:

HelloNerves.ex -> HelloNerves.Messages.start_link -> Nerves.UART.start_link

So Nerves.UART.start_link was in fact called from the Supervisor process. Which in then makes it the recipient of messages from Nerves.UART.

The solution

Digging trough first the Elixir and then the Erlang the GenServer documentation revealed the following in the description of init/1 function:

this function is called by the new process to initialize

– STDLIB Reference Manual

The point here being that while the HelloNerves.Messages.start_link function is run on the Supervisors process the init/1 function is called on the newly spawned process for the HelloNerves.Messages GenServer.

So if I want to Nerves.UART.start_link from process of the HelloNerves.Messages GenServer it has to be done from the init/1 callback.

This leads to the following rewrite:

defmodule HelloNerves.Messages do
  use GenServer
  require Logger

  def init(port) do
    {:ok, pid} = Nerves.UART.start_link
    Nerves.UART.open(pid, port, speed: 9600, active: true, framing: {Nerves.UART.Framing.Line, separator: "\r\n"})

    {:ok, %{nerves_pid: pid}}
  end

  def start_link do
    IO.inspect(Nerves.UART.enumerate)

    GenServer.start_link(__MODULE__, "ttyACM0", [])
  end

  def handle_info({:nerves_uart, _serial_port_name, data}, state) do
    Logger.debug "Got info: #{data}"
    {:noreply, state}
  end
end

Conclusion

In hindsight it seems obvious that start_link would be run on the callers process. But it took me a fair while to figure out that it was init/1 I was looking for.
Coming from a mainly OOP background I think that there is a lot of hidden complexity lurking here not stemming from anything but lack of experience.

REST services in Elixir with Cowboy

Sun, 16 Oct 2016 14:00:00 +0000

cowboy_rest makes it easy to build REST services with cowboy.

It is inspired by webmachine¹, services in Elixir. In this blog I’ll give a quick example of how to create a couple of simple resources and lastly touch on a few of the downsides of using it.

Where plug might be seen as a way to describe how to do a transformation of the request into the response, cowboy_rest lets your describe the characteristics of your resource and then let cowboy_rest decide on how to reply to the request.

In short:

the request is handled as a state machine with many optional callbacks describing the resource and modifying the machine’s behavior

– The cowboy docs

This makes for a nicely declarative way of describing your resources as you’ll hopefully see below. It provides a lot of help (or at least, opinions) in providing correct replies.

You can find the code used in this blog here (alongside parallel implementations in Phoenix and Webmachine).

The first resource

Lets start by creating a new Elixir project using the --sup flag to get a supervisor for our project:

mix new cowboy_rest_example --sup

For the rest of this blog I’ll assume you ran the command above and thus named the project cowboy_rest_example.

Getting :cowboy up and running

We now need to get :cowboy up and running. Open the file containing the start/2 Application callback: lib/cowboy_rest_example.ex. Add the following:

#Dispatcher ~ router
dispatch = CowboyRestExample.Dispatch.dispatch()
{:ok, _} = :cowboy.start_http(:http, 100,
                              [port: 8000],
                              [
                                env: [dispatch: dispatch]
                              ])

The first two arguments :http and 100 tells cowboy to listen to http connections (the other option being :https) and to start 100 ranch acceptors. The choice of 100 was more or less arbitrary and I can’t offer much advice as to this setting.

env: [dispatch: dispatch] sets the dispatcher (aka router) for our resources, we’ll get to that in a moment.
[port: 8000] tells cowboy to listen to port 8000.

Dispatching

One of the options we supplied to cowboy was a dispatch table that we’ll define in CowboyRestExample.Dispatch:

Routes in cowboy are defined as a list of {host, [routes]} tuples. Most likely you’ll want to set the host to :_ meaning match everything. But you could use the host part to match on different subdomains (see: the routing section of the cowboy documentation for more information).

We’ll start out with a dispatch table that looks like this:

defmodule CowboyRestExample.Dispatch do
  def dispatch() do
    # Routes in cowboy are {host, [paths]} tuples
    #The :_ atom causes cowboy_router to match any host.
    :cowboy_router.compile([{:_, routes()}])
  end

  def routes() do
    [
      {"/", CowboyRestExample.Hello, []},
    ]
  end
end

I’ve moved the routes/paths out into the routes function. Paths are defined as a three element tuple {path_match, handler, options}. So in the above example we have told cowboy to let CowboyRestExample.Hello handle any requests to "/" (and we’ve supplied no options to CowboyRestExample.Hello).

The “Hello, world” handler

Time to write our first handler. Create a file for the CowboyRestExample.Hello module.

defmodule CowboyRestExample.Hello do
  def init(_protocol, _request, _options), do: {:upgrade, :protocol, :cowboy_rest}
  def to_html(request, state), do: {"<h1>Hello, World</h1>", request, state}
end

The init method is the first callback we get from cowboy when handling requests to "/". Here we tell cowboy that we want to perform a protocol upgrade to :cowboy_rest. In essence we’re telling cowboy that our handler will respond to the callbacks used by cowboy_rest.

As stated in the introduction introduction request will be handled as a state-machine and the callbacks we respond to change the behavior alters the behaviour of that state-machine. But we don’t have to respond to every call! The callbacks all have reasonable defaults.
So for now we can get away with just implementing the to_html function:

def to_html(request, state), do: {"<h1>Hello, World</h1>", request, state}

Every callback from :cowboy_rest will take the same form. The function will receive request and state arguments to the function and it’ll respond with a tuple consisting of its reply and the request and state tuples.

If you start your project now with mix run --no-halt and open localhost:8000 in a browser you should see the traditional “Hell, World” greeting.

But what if we wanted a json response? Try curl -I --request GET --url http://localhost:8000/ --header 'accept: application/json' in terminal. You’ll get the following response:

HTTP/1.1 406 Not Acceptable
server: Cowboy
date: Sat, 15 Oct 2016 10:05:14 GMT
content-length: 0

So our resource does not support replying with application/json, lets fix that.

The content_types_provided callback is how we define what content-types our resource supports.

def content_types_provided(request, state) do
    {[{"text/html", :to_html},
    {"application/json", :to_json}], request, state}
  end

Our response here (aside from request and state which we’ll always have to return) is a list of tuples, each tuple containing a content-type and the name of the function that can generate the response in that content-type. Lets add the to_json function:

def to_json(request, state), do: {Poison.encode!(%{hello: "world"}), request, state}

Restart cowboy after making the changes (take a look at remix if you find it annoying restarting cowboy after each change). Run that curl command again: curl --request GET --url http://localhost:8000/ --header 'accept: application/json'. Note that the -I argument has been removed.
Sucess! We now have our JSON response. We can now also understand why earlier it was sufficient to just implement the to_html function. cowboy_rest defaults to serving html by calling the to_html function.

What happens if you don’t set an accept header? And what happens if you change the order of the tuples in our reply from content_types_provided?
Answer: If no accept header is present in the request cowboy_rest chooses to reply with the first content-type in the content_types_provided list.

The “Todo” handler

Lets extend the example with new resource for getting a list of todos and creating a todos.
First the dispatcher is extended and then the resource gets defined in its own module.

Extending the dispatcher

Add {"/todo", CowboyRestExample.TodoResource, []} the list of routes in CowboyRestExample.Dispatch.routes.

Next create the CowboyRestExample.TodoResource module as below:

defmodule CowboyRestExample.TodoResource do
def init(_protocol, _request, options), do: {:upgrade, :protocol, :cowboy_rest}

def rest_init(request, _options), do: {:ok, request, %{}}
end

There is a new callback here: rest_init. Its the first callback the resource gets after doing a protocol upgrade to :cowboy_rest. It allows us to set the initial state and return it as the last element of the response tuple, here its initialized as a empty map.

We only want to respond with json so add a content_types_accepted that makes that true:

{[{"application/json", :from_json}], request, state}

Now we want to be able to get the list Todos. These will be fetched from a MySql database using ecto. I wont show the setup here but take a look at the github repo here if you want the details.

We now have a choice to make: What reply do we want if there are no todos? Empty list or a 404 status code? Here we’ll use the 404 status code because it allows me to introduce the next callback: resource_exists. Returning false from this function will yield a 404 reply. It also makes for a nice place to actually fetch the Todos from the database.

def resource_exists(request, state) do

  case Repo.all(CowboyRestExample.Todo) do
    nil -> {false, request, state}
    todos -> {true, request, Map.put(state, :todos, todos)}
  end

end

In essence we’re pulling all the todos from the database and adding it to our state map under the :todos key. Now that we have a list of Todos in our state we can transform those into JSON in the to_json function:

def to_json(request, state) do
    todos = state.todos
    |> Enum.map(&(%{
              todo: &1.subject,
              created: &1.inserted_at
              }))
    |> Poison.encode!()

    {todos, request, state}
  end

And with that we have our list Todo resource done.

Posting a Todo

But a Todo example isn’t worth its salt if you can’t add todos. In order to do that we need define two additional callbacks and add a function saving the new todo to the database.

If you try and send a POST at /todo right now you’ll get a 405 Method not allowed response back. To fix that add the following:

 def allowed_methods(request, state) do
      {["GET", "HEAD", "OPTIONS", "POST"], request, state}
  end

We now supports POST. Trying another POST at /todo will result in a 415 Unsupported Media Type response because we haven’t told :cowboy_rest that we want to accept application/json.

The content_types_accepted callback allows us to state what content-types we accept.

def content_types_accepted(request, state) do
 {[{'application/json', :from_json}], request, state}
end

The :from_json function also needs defining:

def from_json(request, state) do
    {:ok, body, request} = :cowboy_req.body(request)

    todo = Poison.decode!(body)
    Repo.insert!(%CowboyRestExample.Todo{subject: todo["todo"]})

    {true, request, state}
  end

The first line of that function reads the body of the request. And here we stumble onto one of the quirks of cowboy all functions on :cowboy_req will return a request tuple which may or may not differ from the one passed to it. As the :cowboy docs state:

Whenever Req is returned, you must use this returned value and ignore any previous you may have had. This value contains various state informations which are necessary for Cowboy to do some lazy evaluation or cache results where appropriate.

– The cowboy docs

The the second line uses Poison to decode the request body into a map.
The third line inserts a new Todo into the database.
And finally the fourth line tells :cowboy_rest that we succeeded.

If you try posting now you’ll get at 204 No Content response. Huh? I was expecting a 201 Created response.
Well it turns out you should set a location header in the response when sending a 201 status code.

Luckily that is easily fixed:

def from_json(request, state) do
    {:ok, body, request} = :cowboy_req.body(request)

    todo = Poison.decode!(body)
    todo = Repo.insert!(%CowboyRestExample.Todo{subject: todo["todo"]})

    {host, request} = :cowboy_req.host(request)

    { {true, "#{host}/todo/#{todo.id}"}, request, state}
  end

We need to know the host and the id so that we can construct a link for the location header. Then we can extend our reply to {true, "#{host}/todo/#{todo.id}"}. That is: “Yes, we could understand the body you sent us and you can find the result over there”.

Try posting again. What? 303 See other? 303 is what you would send if your service prevented duplicates by pointing the client at the preexisting version of what they tried to create.
And sure enough our resource_exists did return true indicating that the resource did already exist.

Ok, time to fix that.

def resource_exists(request, state) do
  {method, request} = :cowboy_req.method(request)
  todos = Repo.all(CowboyRestExample.Todo)

  case {method, todos} do
    {"POST", _} -> {false, request, state}
    {_, nil} -> {false, request, state}
    {_, todos} -> {true, request, Map.put(state, :todos, todos)}
  end
end

So if the request is a POST or there are no Todos we’ll return false otherwise return true. You might be tempted to try and use pattern matching you match different resource_exists implementations based on the request method. But unfortunately the request argument is a humongous tuple that does not lend itself easily to pattern matching.

Lets try that POST again. Bingo! A nice and shiny 201 Created response!

Getting a single Todo

One final resource: Getting a single Todo.
Again extend the dispatcher add {"/todo/:id", CowboyRestExample.TodoResource, []} to the routes.

If you have done any projects in Phoenix the :id syntax should seem familiar. Its tells :cowboy to bind any value after /todo/ to the id :id so that we may fetch is in our resource.

Now create the CowboyRestExample.SingleTodoResource module:

defmodule CowboyRestExample.SingleTodoResource do
  alias CowboyRestExample.Repo
  def init(_protocol, _request, _options), do: {:upgrade, :protocol, :cowboy_rest}

  def rest_init(request, _options), do: {:ok, request, %{}}

  def allowed_methods(request, state) do
      {["GET", "HEAD", "OPTIONS"], request, state}
  end

  def resource_exists(request, state) do
    {id, request} = :cowboy_req.binding(:id, request)
    case Repo.get(CowboyRestExample.Todo, String.to_integer(id)) do
      nil -> {false, request, state}
      todo ->
        {true, request, Map.put(state, :todo, todo)}
    end
  end

  def content_types_provided(request, state) do
    {[{"application/json", :to_json}], request, state}
  end

  def to_json(request, state) do
    todo = Poison.encode!(%{
              todo: state.todo.subject,
              created: state.todo.inserted_at
              })

    {todo, request, state}
  end

end

The formula should seem familiar now: Upgrade the protocol to :cowboy_rest, fetch the todo in resource_exists and transform it to JSON in to_json. But aren’t things getting a bit repetitive and thus drawing focus away from the differences between the two Todo resources?

Lets define a module with some defaults:

defmodule CowboyRestExample.Defaults do
  defmacro __using__(_) do
    quote do
      @behaviour CowboyRestExample.DefaultBehaviour

      alias CowboyRestExample.Repo

      def init(_protocol, _request, _options), do: {:upgrade, :protocol, :cowboy_rest}

      def rest_init(request, _options), do: {:ok, request, %{}}

      def content_types_provided(request, state) do
        {[{"application/json", :to_json}], request, state}
      end

      def is_authorized(request, state) do
        {auth_header, request} = :cowboy_req.header("authorization", request)
        IO.inspect auth_header
        case auth_header do
          :undefined -> { {false, "Basic"}, request, state}
          _ -> {true, request, Map.put(state, :rights, "Everything")}
        end
      end

      def forbidden(request, state), do: {true, request, state}

      defoverridable [forbidden: 2]
    end
  end
end

defmodule CowboyRestExample.DefaultBehaviour do
  @callback to_json(Tuple.t, Tuple.t) :: Tuple.t
end

If we now add use CowboyRestExample.Defaults we now skip implementing the callbacks for the functions that won’t differ across resources thus drawing out the differences between our resources. We have also defined some safe defaults for the security related is_authorized and forbidden callbacks. In effect denying access to a resource unless we explicitly grant access.
defoverridable [forbidden: 2] allows the different resources to override the forbidden callback (which would not be possible if we did not add defoverridable).

As a final touch we add a very simple behaviour: CowboyRestExample.DefaultBehaviour which just defines our own to_json callback. This is here to help our future selves. If we forget to implement a to_json function in a resource that uses CowboyRestExample.Defaults we’ll get a nice compiler warning reminding us to do so.

You can find the final code on github.

And that brings us to the end of the example.

Why not just use Phoenix?

Phoenix is a valid choice for doing REST service. At work we have our first two Phoenix service moving into production just about now). And we really, really like Phoenix. But experience of building our first two services has given us the chance to evaluate if Phoenix is right for us.

Is the plug model right for us?

They way we handle requests seems like an ill fit for the plug model. We have to do authentication, authorization and JSON schema validation of every request received to our APIs.
Authorization is the same across resources so thats not the problem, but authentication differs across resource and method (for instance one user might have access creating a give resource but not altering it). So we end up with constructs like this:

plug :authorize, [permissions: ["Read: Sessions"]] when action in [:list, :show, :list_by_team]
 plug :authorize, [permissions: ["Create: Sessions"]] when action in [:create]
 plug :authorize, [permissions: ["Update: Sessions"]] when action in [:update]
 plug :authorize, [permissions: ["Delete: Sessions"]] when action in [:delete]

We have much the same situation with performing JSON schema validations.

We don’t use templates, channels nor i18n

We’re building APIs (that we then build web apps on top of) and as a consequence we’re not using templates, channels nor i18n. So there’s a lot of Phoenix we’re not using.

Batteries not included

cowboy_rest is not without its drawbacks among which are:

No logging: By default cowboy performs no logging, so you’ll have to figure out your own
Erlang stacktraces: You’ll have to learn to read Erlang stacktraces. Which takes some getting used to.
Way less creature comforts: Phoenix does a lot to help the developer along. Automatic recompilation, great getting started guides, and excellently helpful error messages. Cowboy has none of that.

Performance?

Performance has not been a factor for looking beyond Phoenix. Phoenix is plenty fast.
But below you’ll find a graph comparing Posting a Todo, getting a single Todo, getting a list of 10 todos and getting the “Hello, World” resource across cowboy, Webmachine and Phoenix.
Test were conducted using wrk running with 300 connections.
Wrk, MySql and phoenix/cowboy/webmachine are all running on the same machine (a Mid 2015 MacBook Pro).
Don’t read to much into the results - in real life most likely everything but phoenix/cowboy/webmachine is going to dominate the response times.

I highly recommend reading the blog series by Sean Cribbs on webmachine found here if you’re unfamiliar with webmachine) ↩

Elixir, Phoenix and workers, part 2: Why use workers

Thu, 18 Feb 2016 14:00:00 +0000

Before I start diving into different worker/job-scheduler libraries it might be wise to take a look at why I think workers are needed.

Just to recap: Clients send JSON to the server. That JSON has to be transformed into XML and sent to a third party system (while maintaining the correct order of the messages).

But why not just do the work directly in the controller or in an agent? First and foremost we need to ensure that the message order is preserved. The consequence is that we can’t send a reply to the client before we’re done processing the message. Otherwise we run the risk of messages overtaking each other.

But how much of an impact does it have on performance if we were to do the work in the controller or an agent? I tried adding between 100 and 0 milliseconds of delay to the controller and the agent. Without any delay in the controller the server is able to handle around 450,000 requests/min dropping down to just over 50,000 requests/min for 100 ms of extra delay. That isn’t too surprising.

Ok - but handing the work of to an Agent would do the trick right? It can keep state across all the requests and ensure messages are handled in order and because we’re handing it off to the agent performance shouldn’t suffer.

Well no performance wont just suffer - the server will keel over and die.

But what is happening with the agent? Even adding 1 ms of extra delay has a significant impact on the performance of the server. What is going on? If you take a look at some of the documentation for agents you’ll run across this sentence:

“Everything that is inside the function we passed to the agent happens in the agent process”.

Agents are for keeping state - which means that messages passed to an agent are processed in order. So any delay caused by any of the requests to the server will hurt all following requests. And the consequences are clear.

So using an Agent won’t work. Doing the work in the controller runs the risk of not preserving message order (the same would be true of Tasks).

So workers it is. Next time.

Elixir, Phoenix and workers, part 1: The setup

Sun, 14 Feb 2016 16:38:00 +0000

This is the first post in a series looking at Elixir, Phoenix and a few different worker/job-scheduler libraries for a very specific use-case.

The use-case is not made out of whole cloth but a lot of details have been abstracted way. The use-case is as follows: A bunch of clients send JSON messages to a web server. The messages a converted into XML and forwarded to a third party system (over which we have no control).
There is one rule we absolutely cannot break: The order of the messages is important. One message from a given client cannot overtake another and skipping messages is also not allowed.

The idea of this blog series is to first build something to act as the third party system and a simple Phoenix based web server to receive JSON messages from the clients. The next parts of the series will then look at different worker/job-scheduler libraries and see if they can be made to work for this specific use-case.

Before we start looking at the parts we’ll need before looking at the first worker there is one point I’d like to stress: Im just starting out with Elixir and Phoenix. So this is going to be a learning experience for me and you should take everything with a grain of salt¹.

The necessary parts

This time I’ll focus on the bits need before we can start testing out different workers. The bits are:

A web server for receiving messages from the clients
Something to act as the third party system
Something to act as a bunch of clients

The web server (Phoenix)

I’ll be using Phoenix for building the simple web services used by the clients. The server will receive JSON messages that looks like this:

{
  "client": "Some client",
  "data": {
    "sequenceNumber": 0,
    "sample": [1,2,3,4]
    }
  }

There will be a simple controller for receiving these messages and handing them off to a appropriate worker. If the client remembers to set its content-type header to application/json the JSON will be parsed and be made available to the relevant controller method in its params that it receives as its second argument.

Now it makes good sense to map the received JSON into Elixir structs as a way to sanitize received data. In this case the structs looks like this:

defmodule ExternalSystemMock.Message do
  alias ExternalSystemMock.MessageData, as: MessageData
  defstruct client: "", data: %MessageData{}
end


defmodule ExternalSystemMock.MessageData do
  defstruct sequenceNumber: 0, data: []
end

The newly constructed struct is then sent to workers that will handle forwarding the message to the Agent acting as the third party system.

One thing I ran into was mapping from JSON to these structs proved harder than expected. Elixir has the Kernel.struct!/2 function that can take a map and map it into a specified struct. But there is one caveat that is not immediately apparent from the docs: struct!/2 only maps from keys that are atoms.

Phoenix uses Poison for parsing JSON into maps and Poison does have an option us create maps where the keys are atoms. So why doesn’t Phoenix utilize that option? Turns out theres a very good reason for that: Atoms in Elixir are not garbage collected. So if Poison was setup to provide maps with keys that were atoms it would open up the server a denial of service attack aiming to exhaust memory.

The third party system (An Agent)

The primary characteristic of the third party system is that it does not like it when the messages it receives from any given client are out of order. In order to emulate this behavior we need something that keep state across several requests. The Elixir docs state that “Agents are a simple abstraction around state.”. So an Agent would seem to fit the bill perfectly.

Agent is seeded with an empty map as its initial state and usage of the Agent is wrapped in module hiding away the details. The full code for the modules is:

defmodule ExternalSystemMock do
  alias ExternalSystemMock.Message, as: Message
  alias ExternalSystemMock.MessageData, as: MessageData

  def start do
    Agent.start fn -> Map.new() end
  end

  def receive_message(agent, %Message{client: client, data: %MessageData{sequenceNumber: sequenceNumber, data: _}}) do
    updater = &update(sequenceNumber, client, &1)

    Agent.update(agent, updater)
  end

  defp update(newSequenceNumber, client, state) do
    try do
      %{^client => [seq | _]} = state
    rescue
      MatchError -> IO.puts "match error"
    end

    case state do
      %{^client => [lastSeenSequenceNumber | _]} when lastSeenSequenceNumber + 1 == newSequenceNumber ->
        Map.update!(state, client, fn list -> [newSequenceNumber | list] end)

      %{^client => [lastSeenSequenceNumber | _]} when lastSeenSequenceNumber < newSequenceNumber ->
        raise "Missed message"

      %{^client => [lastSeenSequenceNumber | _]} when lastSeenSequenceNumber > newSequenceNumber ->
        raise "Skipped message"

      %{^client => [lastSeenSequenceNumber | _]} when lastSeenSequenceNumber == newSequenceNumber ->
        raise "Message repeat"

      %{} -> #new client
        Map.put(state, client, [newSequenceNumber])
    end
  end
end

The real meat is in the update/3 function. Guard expressions ensure that if any sequence is repeated, skipped or received out of order the Agent raises an exception. Raising an unhanded exception from a Agent will cause it to die. This is important because it will make it much easier do detect if any of the worker libraries we look at later on make any changes to the order of the messages.

The clients (wrk)

We need something to act as a bunch of clients. I had intended to use Gatling which I’ve used before with some success. But Gatling gave out before the server did and reported response times what seemed way to high.

So I turned to wrk instead (and seemed up to the job). The small Lua script below sets up a small test. Each client will send fifty messages with increasing sequence numbers and then sends final halt messages (and then the message repeats).

local counter = 1
local threads = {}

function setup(thread)
   thread:set("threadCounter", counter)
   table.insert(threads, thread)
   counter = counter + 1
end

local requests = 0
function request()
    requests = requests + 1

    local requestHeaders = {}
    requestHeaders["Content-Type"] = "application/json"

    local r = {}
    local clientName = threadCounter .. "-" .. requests
    for i = 0, 50 do
      r[i] = wrk.format("POST", "/api/data", requestHeaders, "{\"client\":\"" ..  clientName .. "\",\"data\": {\"sequenceNumber\":" .. i .. ", \"data\": [1,2,3,4]}}")
    end

    r[50] = wrk.format("POST", "/api/data", requestHeaders, "{\"halt\": true, \"client\":\"" ..  clientName .. "\"")

    return table.concat(r)
end

Just as a side node: Before adding any workers and running both wrk and Phoenix on the same machine Phoenix seems to top out at around 440000 requests per minute. I’ll use this as the baseline going forward.

Next time

Next time I’ll take a look at the first worker library and give an explanation as to why a worker is necessary at all.

But if you spot something stupid please let me know. ↩

How to do support if you don’t value your customers

Wed, 27 Jan 2016 19:39:48 +0000

With helpful examples from Oracle and Philips.
I see customer support as having a polarizing effect: It will either loose you customers or win you fans. But winning fans via customer support takes time and effort and will only work if you actually care about your customers.

I’ve been on the receiving end of an uncaring support organization more times than I care to remember and I think I’ve spotted a few patterns emerge from those experiences that I’d like to share.

First I’ll give to examples of experiences with uncaring support. You can skip the next two sections and just jump right ahead to the conclusion if you wish (it’s mostly just me blowing of steam anyway)

#Example 1: Oracle (circa: 2010)

A few years back I was working on a project that was mostly just SOAP messages passing trough an Oracle Enterprise Service Bus. Basically the Service Bus would receive XML documents where parts of the document where signed. The service bus would pull out the signed parts stick them into other xml documents and forward them to the recipient. Bread and butter stuff for a service bus.

But then the recipients started complaining of broken XML signatures. And true enough it did seem that the Service bus was breaking signatures. Specifically a component in the service bus called XMLBeans. It was breaking the signatures by moving around namespace declarations in a way that broke the signatures.

Problem spotted and likely cause found we turned to Oracle support. Since we could point to the exact component that was causing problem and had build a small bit of code reproducing the problem then it shouldn’t be to hard get support to accept the bug and provide a solution.

Well… turns out…

When submitting a support request you had to fill around 20 fields with information about product versions, OS version and all kinds semi relevant information. Ok, that does make some sense. But you had to fill out those fields every single time you wrote a comment or answered at questions. Every. Single. Time.

Pattern number one: Make your customer jump trough hoops.
In order to teeming masses at bay make sure it takes a lot of effort to get support. This is a war of attrition and the more hoops you have the quicker the customer will surrender.

Hoops sufficiently jumped we got the first reply from Oracle support. “Hi. Please install this patch and see if it fixes the problem”. The release note for that patch does not mention anything about our problem but we’ll give a shot. Didn’t fix the problem.
Pattern number two: Try and look like you’re doing something.

But make sure the customer has to do the actual work. Customers just love it when it seems like you’re helping. Just make sure you don’t actually have to do any work. Supplying random software updates work great!

So we got back to Oracle and told them that the patch did not work. And so they suggested that we try installing another patch. We did. It did not work.

Back to Oracle that then suggested we install the first patch we gave them.
Pattern number three: Never read the full context. Only ever read the newest comment – remember you have 23746 other tickets to get trough today.

Back to Oracle which then told us that we we’re using xml signatures wrongly.
Pattern number four: Make sure supporters can ask questions that take a lot of effort to answer.

So we reread the XML signature specs and the XML canonicalization specs. We went and got a second opinion. We then spend hours explaining why it should work to a host of different supporters.
Pattern number five: Don’t become personally involved. You might end up actually caring.

Oracle then suggested we stop using XML signatures.
Pattern number six: Try to convince the customer that they don’t really need that feature.

Oracle then said: The error is not in a component we made.

Pattern number seven: Shift the blame. If you can get the customer to go elsewhere with his problem then you won. Sure it’s in a component that is part of our product – but we did not write that component. Go talk to those guys.

But then after two years we finally got the patch we needed.

#Example 2: Philips (2015-2016)

This second example is from the consumer space. I went and bought a Philips TV (Model: 50/PUS6809/12. Do not by that TV.). It quickly turned out to have a host of problems:

Randomly turning on 3D mode
Randomly turn down the contrast by 50%
Show “Certificate expired” error messages that cannot be dismissed
Unable to wake up from Stand-By
Suddenly Hobbit stopped working.

And the dummy I am I turned to support instead of just returning the TV.

Philips: Please install this update (that has nothing to do with any of the errors you are experiencing)
Pattern number two: Try and look like you’re doing something

Philips: Please “reinstall your TV” loosing all your settings in the process.
Pattern number one: Make your customer jump trough hoops.

Philips: The TV stations changed the way they are doing HbbTv.
Pattern number seven: Shift the blame.

Philips: Please provide screenshots of the errors and please provide information on signal strength for all your channels.
Pattern number four: Make sure supporters can ask questions that take a lot of effort to answer.

Philips: Try turning off HbbTV
Pattern number six: Try to convince the customer that they don’t really need that feature.

Philips: Ok, just one more question.
Pattern number eight: Feign progress. At some point the customer is going to get angry. That means you’re close to winning the war of attrition. So pretend to have only a few more questions. But make sure you have plenty of hoops left for the customer to jump tough.

Eventually I gave up after dealing with Philips support for three months.

#Conclusion

In case you skipped the last two sections I’ll summarize the patterns for doing horrible support:

Make your customer jump trough hoops
Try and look like you’re doing something
Never read the full context. Only ever read the newest comment
Make sure supporters can ask questions that take a lot of effort to answer
Don’t become personally involved. You might end up actually caring.
Try to convince the customer that they don’t really need that feature.
Shift the blame.
Feign progress.

Neither Oracle nor Philips are dumb organizations. They have just figured out that you only need their support after they have made their sale. That means support is just something that eats into their margins.

Obviously if people were to ask me if they should use an Oracle Enterprise Service Bus or by a Philips TV I would advice against it. And equally obviously is not hurting them enough or they would have changed their tactics.

But if you’re a small business and each individual customer actually has an impact on your business I would suggest you reverse the patterns outlined above:

Make getting support as easy as possible
Do something to help the customer
Know the full context before answering the customer.
Make sure the customer has to work as little as possible
Do become personally involved. Actually care.
Take responsibility for your features.
Own up to your mistakes.
Provide honest status updates.

Ending on a up note: It’s been 4 years since I last used a service bus and I now watch way less TV.