Investigating GStreamer pipeline latency

April 27th, 2017

Just wanted to very briefly post a method I’ve used to investigate latency in complex GStreamer pipelines. This method is useful for showing how long each element is taking to process a frame.


This prints a lot of useful data, but far too much to read and interpret. Cut it down, by only displaying one message each time a buffer enters an element. Pipe stderr to the following grep:

grep "calling chainfunction"

If you only want to look at certain elements, this grep can be useful:

egrep "\<(element1name|element2name):sink\>"

Pipe the output to a file. Completed example:

GST_DEBUG=GST_SCHEDULING:6 ./application 2>&1 | grep "calling chainfunction" | egrep "\<(element1name|element2name):sink\>" > log.txt

Now, to time an individual frame flowing through the pipeline, open the log and pick one. Make a note of its pts. Grep for it and process the output for readability. eg:

grep 0:00:06.573958668 log.txt |sed "s/ .*</ /"|sed "s/:sink.*$//"

Example output:

0:00:02.593278667 element1:sink
0:00:02.593474334 element2:sink
0:00:02.595585334 element3:sink
0:00:02.595757667 element4:sink
0:00:02.603250000 element5:sink
0:00:02.603501334 element6:sink
0:00:02.603622334 element7:sink

The times are the time the frame entered that element. You may like to use a spreadsheet to calculate the time spent in each element.


Using British Geological Survey Data in Openlayers

June 3rd, 2014

Just a quick post to share something that took me hours of frustrated tinkering to figure out.

There are several sets of data available by WMS. Two were of interest to me.

The first was “British geology onshore digital maps (1:50 000 scale)”. I ended up giving up on this one for the time being as it was being incredibly picky about the size and shape of BBOX in the WMS request and returns a white square with no error if not happy. I have not yet found a way to persuade openlayers to give it what it wants.

The second was “UK onshore and offshore bedrock and superficial geology (1:625 000 scale). This one seems a bit more reasonable, although not without problems.

It simply refused to work with a projection of EPSG:900913 which was what my Openlayers map was configured as. Changing to EPSG:27700 made it work, but was incompatible with OpenStreetMap which was my other base layer.

Apparently OpenStreetMap 2.x does not properly support base layers of differing projections and everything hinges on which one gets loaded first. I was able to have one working and the other broken, not very helpful.

Eventually the solution was extremely simple. The projection of EPSG:3857 turns out to be identical to EPSG:900913 and was supported by the server.

Working code snippet below:

var bgslayer = new OpenLayers.Layer.WMS( "British Geological Survey",
"", {layers: 'BGS_EN_Bedrock_and_Superficial_Geology', CRS: 'EPSG:3857', version: '1.3.0'}, {projection: 'EPSG:3857'} );

Enjoy. If I ever get the detailed map working, I’ll share here. Do let me know if you do (BBOX strategy parameters look interesting).

UPDATE: The hydrogeological map also works with these parameters. URL is and layer name is Hydrogeology.


Pthread mutex debugging method

July 12th, 2012

On several occasions, while working on multi-threaded applications, I have experienced cases where a thread is waiting forever for a mutex owned by another thread.

Causes of this include deadlocks (where each thread attempts to lock a mutex held by the other, preventing the other from ever getting a chance to release it, resulting in both threads waiting forever) and issues as simple as forgetting to unlock the mutex.

In these cases, I’ve wished for a way to determine which thread holds the mutex. In the past. I’ve used various methods to discover this information, including:

  • Adding wrappers around the lock and unlock functions to print each lock/release.
  • Using GDB to view the owner field of the mutex and then checking each thread to see if it matches (when switching threads, GDB prints the thread ID).
  • Giving up and getting drunk instead.

Although each of these methods were successful, performing them again and again to debug issues proved too time-consuming (in addition to the need to either add the wrappers or remember to enable core-dumps). A better, easier method was needed.

The problem is that the owner field of the mutex may mean any number of things, depending on the environment (whether threads are handled in the kernel or not), so the code below is not particularly portable. However, it works fine on both my desktop machine and ARM development board, so it’s good enough for my debugging purposes (and may or may not be of any use to you).

A call is added at the start of each new thread to a function that stores the thread’s name and ID. This function must be called WITHIN the thread in order to get the thread’s ID.

A signal handler is installed for SIGINT and SIGUSR1 which examines each mutex and if it’s locked then it checks the thread list to determine the owner and prints the owner’s name and lock count. For SIGINT, the signal handler then calls abort() to create a core dump for further debugging, and for SIGUSR1 just reinstalls the signal handler.

The code may be downloaded here:


  • As mentioned before, not portable (but good enough for my purposes).
  • Requires a function to be called at the start of each thread (I cannot find any way to automate this with macros, etc).
  • Fixed number of threads can be tracked (I’ve never been in the situation of not being able to guess, to within an order of magnitude, the number of threads my application will use. It wouldn’t be hard to make the allocation dynamic, but it would be unlikely to be useful).
  • There’s currently no dynamic adding of mutexes, you need to modify the source (mutex-debug.c:dump_mutexes()).

Summary: hackily put together, unportable and completely unsuitable for production code. Provided in the hope that it might be useful to you for debugging purposes.


  1. Download the source and build mutex-debug.c into your project. Include mutex-debug.h. (The code may be downloaded here:
  2. Add calls to register_thread to the initial function of each thread.
  3. You can use dump_mutex() to show the status of a particular mutex at any given time.
  4. OR you may also modify the dump_mutexes function to dump each of your mutexes (example included) and call mutex_debug_init() to implement the most common use-case.


The code is poor, don’t bother telling me so ;) .

But if you’d like to provide patches to make it better, feel free.


Qemu and openvpn: Secure and convenient remote access to virtual servers

July 30th, 2011


This article describes a method for remotely accessing virtual servers. The aim is to be able to have multiple clients access one or more virtual servers across one or more physical hosts as if everyone was on the same LAN. It is also necessary to do this without compromising on security and without ANY extra configuration for the virtual servers.
We will achieve this using Qemu and Openvpn.
Read the rest of this entry »


Using Ctags with Nano (and other editors)

May 12th, 2011


Ctags is a pretty useful utility to aid navigating source code. It creates a database of function/variable/type definitions within your code.
Vi and Emacs support using a Ctags database, Nano (along with many other editors) does not.
I have implemented a script which reads the database and finds the appropriate file and line to open the editor at.
This post describes the installation and operation of this utility.

Read the rest of this entry »


Porting the uIP tcp/ip stack to the msp430 (easyweb3)

May 22nd, 2010


I recently purchased an olimex easyweb 3 and wanted to get it working.

However, I had no luck whatsoever with any of the software listed on the olimex site. Only 1 builds with mspgcc, and that didn’t work.

I had heard good things about uIP and was happy to find that there was a port for my board by Paul Curtis of Rowley Associates.

There were 2 problems with this.
1. It was for an old version of uIP (0.9).
2. It was for some expensive IDE that Rowley Associates sell.

So I set about modifying Curtis’ port to work with my compiler and uIP version.
Read the rest of this entry »