by Mike Bennett
What to know before debating type systems is a really good article discussing static and dynamic type systems and type inference. I found it via Dry Sellers CodeBetter.com blog post Sunday Thought: Embracing the Type System.
Related to that, if you’d like to learn more about type inference algorithms, have a read of Code Commit’s blog post What is Hindley-Milner? (and why is it cool?). The post gives an easy to understand overview of the Hindley-Milner type inference algorithm.
From a practical software development perspective, I see that according to Slashdot the speed of Javascript in Firefox has been improved by 20% to 30%. Firefox uses the SpiderMonkey Javascript Engine. Apparently the speed improvement is due to enabling type inference in the Javascript engine.
I wonder will SpiderMonkey’s speed improvements have a significant impact on Thwonk’s speed, esp. when I finish off the Thwonk code for caching and reusing JIT compiled Javascript?
by Mike Bennett
On and off over the years I’ve developed various software projects for fun. One of those projects is called Thwonk.
Today, I’m releasing the C, SQL and Javascript code for the backend of Thwonk and placing it under an MIT License (means you can take the code and reuse it in both open and closed software projects).
Thwonk is a bit like node.js, except it’s for messaging and online communication systems. The purpose of Thwonk is to enable developers to easily and quickly create and prototype new kinds of Internet messaging and communication systems.
A key aspect of Thwonk is that you program the messaging system rules and constraints in Javascript. I’ve taken the Spidermonkey Javascript engine, extended it, and built it into some custom written C daemons that handle and hide the complexity of messaging systems.
For example, with Thwonk you can easily develop a mailing list that trims all mailing list posts to 140 characters (a Twitter-like mailing list!). In Thwonk, this Twitter-like mailing list works as follows:
1. A message (in this case an email) arrives at a Thwonk email address, such as play@thwonk.com
2. Thwonk daemons take the message, look at the destination address, and based on the address find the Javascript associated with that destination address
4. Thwonk daemons then run the Javascript associated with the address (in a sandboxed environment)
3. The Javascript does whatever you’ve programmed it to do with the incoming message, e.g. trim the message to 140 characters
4. Your Javascript code then sends outgoing messages, e.g. send trimmed message to all play@thwonk.com subscribers
You can easily imagine other kinds of messaging rules, e.g. a mailing list where everyone on the list can only post once a day, or where all mailing list subscribers HAVE to post once a day (otherwise automatically unsubscribe them). As Thwonk matures I’ll extend it so you’ll be able to play with the rules of other online communication systems, such as web forums.
With this first version of Thwonk you can create new kinds of mailing lists in Javascript. You don’t have to worry about handling multiple messages arriving at once, nor worry about message queues, managing message delivery protocols, handling high and low loads, resource management (Javascript in an infinite loop is automatically stopped), etc. As Thwonk develops I’ll add support for other messaging protocols and communication frameworks, such as Web Services, XMPP, Twitter and Facebook.
The source code for Thwonk is up on Github:
www.github.com/mikemb/thwonk (see backend dir)
This alpha version 2 code compiles to a set of daemons that run on Unix servers, talk to SQL databases, and run Javascript (in a sandboxed environment on a backend server).
Thwonk provides multiple message queues, both incoming and outgoing – so you can easily have many types of parallel messaging systems running on a single Thwonk server. There’s also a virtual file system (VFS) for storing and publishing files, esp. handy for building messaging archives and publishing them on web. One great benefit of the VFS is you can serialize and deserialize Javascript objects to it, which makes storing state information easy, e.g. store a list of the mailing list subscribers in a Javascript Object or as a JSON file. There’s also security models, user rights and synchronization management (you don’t need to worry about handling multiple messages arriving at once) – but I won’t get into that today.
There is a fronted for Thwonk, which I’ve written in Django and Python. I want to tidy the frontend up a bit before publishing the code. With the frontend, anyone can create their own Thwonks and write the Javascript for the Thwonk rules in a simple online web IDE. In the next few weeks I’ll setup a public Thwonk server you can play with it.
This version of Thwonk compiles against the 1.6 branch of Spidermonkey on Linux. I’ll be shortly updating the code to support the recent 1.8 branch of Spidermonkey and recent versions of the mailutils dev libraries. There are automated build scripts for the backend, i.e. configure; make. Thwonk currently depends on spider monkey 1.6, libmailutils 1.2 and mysql dev libraries.
A few immediate todos are:
1. Setup Thwonk on a server for other’s to play with, i.e. enable people to create a Thwonk and program the Javascript for it.
2. Add in documentation explaining how to get Thwonk up and running on your own server.
3. Simplify the setup and configuration (there’s a bunch of hardcoded configuration options that absolutely should be stored in the database and external configuration file – naughty me!).
4. Provide examples of writing messaging systems in Thwonk Javascript.
5. Publish architecture documents, i.e. how do the various components and dameons in Thwonk fit together and communicate.
6. Add unit testing.
7. Add support for web services, i.e. both accept and publish message via a web API
8. Refactor Virtual File System
Over the next few weeks I’ll go into more details of Thwonk and provide examples of how to program with it. If you’re wondering how fast Thwonk is, as in how many messages it can handle per second – the non-specific answer is lots and its got a distributed architecture so it should scale across multiple cores and computers. I’ll re-run my evaluation scripts and graph the results of the number of messages it can handle per second. IIRC from the last time I tested Thwonk it was able to process messages much faster than the mail server could accept and deliver them.
Thwonk was inspired by a previous project called BumpList, which I co-created with Jonah Brucker-Cohen. Thwonk is our new project – you’re welcome to get involved!
by Mike Bennett
Recently I noticed that a newly configured Lighttpd 1.4.28 web server with PHP and mod_fastcgi was logging to the error.log every second. An error wasn’t reported, instead each error.log entry was something like:
2XXX-12-12 12:00:00: (mod_fastcgi.c.2736) proc: unix:/tmp/php.socket-1 1 1 0 4766
I suspected that I’d left a logging flag turned on in the Lighttpd config files, but nothing immediately popped out. So I grabbed the Lighttpd 1.4.28 source code to have a look at the code for line 2736 in mod_fastcgi.c. The C code was:
if (p->conf.debug > 2) {
log_error_write(srv, __FILE__, __LINE__, "sbdddd",
"proc:",
proc->connection_name,
proc->state,
proc->is_local,
proc->load,
proc->pid);
}
Based on the code I could see that a debug option was turned on, so I went back to the Lighttpd config files. This time I spotted in the mod_fastcgi configuration that I’d set fastcgi.debug = 3 – this was causing the excessive logging. So the solution is to set fastcgi.debug = 0 for no debug logging, or to 1 for debug logging.
Interestingly, when you read the related Lighttpd docs, they say that only fastcgi.debug values of 0 and 1 are used. Yet, based on the if condition at line 2735 above, you can see that all values above 2 lead to even more debug logs (this is a fairly common behavior in other daemons).
Mistakenly, I’d assumed that any fastcgi.debug value 1 and above are equal. This is not the case. Out of curiosity I tested what happens when fastcgi.debug = 1. A value of 1 does turn on debugging output, but it is a lot less debugging output than having mod_fastcgi writing to the logs every second.
by Mike Bennett
On and off I’ve been working away on my second iPhone / iPad app. I haven’t touched the code in a while, so this evening I was testing the code on iOS 5 – it had worked fine in iOS 4 and earlier.
Most of the code was behaving itself, except for some reason my modal views were no longer properly dismissed in iOS 5? They stayed on screen, even after the following code, which I’d previously tested in iOS 4:
/*
* This delegate is called when user picks a camera photo
*/
- (void)imagePickerController:(UIImagePickerController *)picker
didFinishPickingMediaWithInfo:(NSDictionary *)info
{
[[picker parentViewController] dismissModalViewControllerAnimated:YES];
[picker release];
}
After doing some digging around, I found that the error is caused by an iOS 5 change. In iOS 5 [picker parentViewController] returns nil.
To fix the problem you don’t need to change much code. Just tweak your code so it directly calls the dismissModelViewControllerAnimated method on picker, as shown below:
- (void)imagePickerController:(UIImagePickerController *)picker
didFinishPickingMediaWithInfo:(NSDictionary *)info
{
[picker dismissModalViewControllerAnimated:YES];
[picker release];
}
This fix also works with iOS 4, though so far I’ve only confirmed that in the iPhone emulators.
BTW is this a bug in iOS 5? The previous approach I used to dismiss modal views is based on Apple’s Developer code examples.
by Mike Bennett
I’m about to getting hacking on a new eyetracking project, that’ll probably use a Microsoft Kinect for input. So recently I’ve been refreshing my memory of C#, as it’s been a while since I developed and shipped products in it. One of the best set of C# tutorials I’ve come across can be found at C# Station. The tutorials are nicely detailed and a quick read, perfect if you’ve existing development and Object-Oriented experience.
Here’s an interesting talk on RabbitMQ – an open source message broker / message platform. Looks powerful and useful! I may have a need for it in a few months.
I think I need a C ORM (Object-Relational Mapping) – or more accurately a C Struct Relational Mapping (SRM). I wonder does such a thing exist?
by Mike Bennett
I often crunch large amounts of numerical data with Matlab. This involves writing Matlab code to automate the analysis. Often I need to search an array of numbers for the closest matching number. For example in the Matlab code below, I want to find the position of integer k in array a.
k = 18
a = [19 13 24 18 15]
This is trivial to do in Matlab:
i = find(a == k)
On return i is set to 4.
So far, this is a simple search problem, i.e. search an array for a matching value and if a match is found return the position of the match. Unfortunately, this doesn’t solve my problem, as most of the time I don’t have an exact match. In those cases, I want to find the position of the closest match. For example, I want to find the position of the closest match to m in a.
m = 22
If I use the previous find() I’ll get a null result. Below is the code that’ll find the closet matching value:
[d p] = min(abs(a - m))
On return p is set to position 3, which corresponds to value 24 in a.
Neat isn’t it!
Do you see how the code works? Lets quickly step through what the code does, while breaking it down into intermediate steps:
% An intermediate array b is generated from (a - m)
b = a - m
% The resulting array that is assigned to b
% b = [-3 -9 2 -4 -7]
% Now get rid of the negative signs
d = abs(b)
% d = [3 9 2 4 7]
% Then with min(d) we get the position of the smallest value in d
[d p] = min(d)
% p = 3
Have you spotted the limitation of the min(abs()) code? Multiple matches are ignored. If there are two values in a that are equally close, then only the position of the first closest value in the array is returned.
A fix to return multiple matches is easy to code in Matlab, so I’ll leave it as an exercise for you – enjoy.
BTW I spotted this trick on another blog a few years ago – so all credit to that blog…if I can find it again I’ll update this post with a link back.
by Mike Bennett
Let’s launch this blog with an issue I recently ran into. This problem popped up after setting up Lighttpd 1.4.28 on an OpenVZ Virtual Private Server (VPS) running Linux Debian Squeeze 6.0. With this new setup, for some reason my existing Server Side Includes (SSI) wouldn’t run on the new Lighttpd install.
I’d setup Lighttpd with Server Side Includes enabled (mod_ssi) – then I restarted the server. After poking around at the problem for a few minutes I wasn’t spotting anything obviously wrong with either the SSI or the Lighttpd configuration files.
I knew my SSI code worked correctly in webpages, as the code had worked on another of my older Lighttpd (1.4.19) installs. The SSI code was like:
<!--#include virtual="/path/to/file.txt">
After more digging around I discovered that the reason the SSI wasn’t working was because of a documented bug in Lighttpd. In some versions of Lighttpd there’s a bug where SSI won’t work correctly when mod_ssi is enabled AFTER mod_compress.
The solution is simple – in your Lighttpd configuration you need to enable mod_ssi BEFORE mod_compress.
