Planet Samfundet

Takk for 14 fine år.

Woop woop! Det er nytt år. Vi gjør ikke som Adele-elskerne her over. Ikke stiller vi opp i ledertrøyer i den europeiske pressen heller. Bare så det er sagt. Ekte ledere trenger ikke dress-to-impress, så lenge vi bare dress-to-avoid-uncomfortable-attention.

Derimot så bruker vi året på internettet sammen med alle de andre figurene:

Vi liker spesielt godt at man poster med fullt navn og fullt bilde i riksmedia. Når man atpåtil er så dårlig i elementær fysikk så er det vel greiest at man kringkaster det. Sånn til skrekk og advarsel.

For ordensskyld så er volumet av en slik “laptop-veske” på 0,032 m^3. Fylt med vann skulle den vel da veie omlag – ja jommen – 32 kilo. Hadde det vært gull så ville kofferten veid 608 kilo. Reiser man med Norwegian, som er litt mer smalhans, så kan det lønne seg å pakke gullet i flytende form, da vil det bare veie 544 kilo. Det blir jo litt mindre gull med, men det får en bare bite i seg hvis en absolutt skal spare på flybillettene.

Så ikke bare beste fall teoretisk, men høyst praktisk også. Hurra. Godt at vi er så glupe at vi kan fortelle internettet om det.

Vel nå må vi tilbake til komfyren, vi skal smelte om gullreservene til carry-on size.

Hello, everyone. Just made this: The thirty second gift suggestor – released now for the UK, it allows you to find gift ideas in a fast and fun interface. Hope people check it out, and that you'll tell me if you find any errors or have other feedback! =)

…selv om vår halvdel fra det mørke kontinent ikke er fremmed for litt lyssky handling i bazarens bakgater, så er den alfabetisk-overlegne trebokstaveren av oss to litt fremmed for den nye formen for anskaffelser av næringsmiddler etter at det ble innført Военный коммунизм her i landet.

Det at man må i mørke bakgater og handle med tvilsomme herrer for å få smør på brødet er for oss fremmed, men hva gjør en ikke for å sette mat på bordet?

Vi har det. Dere andre Rema-slitere, suck on it. Nå blir det bacon&smør!

…derfor er det ironisk at Norsk Dagkirurgisk Forenings vintermøte går over to dager.

Så, hvem ville ikke kjøpt telefondeksel til sin iPhone 4 fra en butikk med navn “iPhone foreskin”?

Twitterstyling it. Vi har troa.

A little python snippet for creating dot-representation of the dependencies in a java app.

Let this little baby chew on your target/classes dir and you will get a dot-file on stdout.

 #!/usr/bin/env python

import re
import os
import sys
from os.path import join, basename

LIB_REG_EXP = re.compile('[,+L#][a-z][A-Za-z0-9/]+')
PACKAGE_FILTER = ''
CLASS_PREFIX_REMOVE = ''

if len(sys.argv) > 2:
    PACKAGE_FILTER = sys.argv[2]

if len(sys.argv) > 3:
    CLASS_PREFIX_REMOVE = sys.argv[3]

def get_class_dependecies(class_file_contents):
    libraries = set(map(lambda x: x.replace('/','.'), [l[1:] for l in LIB_REG_EXP.findall(class_file_contents)]))
    return filter(lambda x: PACKAGE_FILTER in x, libraries)

class_dependecies = {}
for root, dirs, files in os.walk(sys.argv[1]):
    if '.svn' in dirs:
        dirs.remove('.svn')
    class_files = [f for f in files if f.endswith('.class') and '$' not in f]
    for class_file in class_files:
        class_content = open(join(root, class_file)).read()
        class_name = join(root, class_file).replace('/','.')[len(CLASS_PREFIX_REMOVE):-6]
        if PACKAGE_FILTER not in class_name:
            continue
        class_dependecies[class_name] = []
        for dep in get_class_dependecies(class_content):
            if dep == class_name:
                continue
            class_dependecies[class_name].append(dep)

print "digraph G {"
for key, deps in class_dependecies.items():
    for dep in deps:
        print '"%s" -> "%s";' % (key, dep)
print "}"

The second argument creates a basic filter for the class-names you want. The third parameter is used for removing the folder-names (target.classes. in usual cases)

Example:

python dotclass.py target/classes us.klette.myproject target.classes. > deps.dot

Have fun

Ja, det er en fuckings bloggpost. Det handler om mitt plystrende vindu.

Jeg er så heldig at jeg bor på Bjølsen studentby. Åh, der er det fint, sier folk. Ja, jævla fint. Så fint at vinduene plystrer av lykke.

Jeg la inn melding til vaktmester. Vinduet mitt plystrer, skrev jeg. Det plystrer og jeg får ikke sove. Vær så snill, jeg trenger hjelp. Jeg trenger nattesøvn. Eksamen kommer, og vinduet mitt gir faen.

Vaktmesteren kom. Det er trykk i hele huset, sier han. Han åpner og lukker vinduet fire ganger. Vi må fikse hele huset, sier han. Så går han, og vinduet mitt plystrer fortsatt. Det gir fortsatt faen.

VEL. På et eller annet tidspunkt må nok være nok. Er man gaffagenerasjon, så er man det. Gaffa fikser alt. Jeg har gaffet vinduet (med så stor iver at jeg i skrivende stund har en bit gaffa mellom fortennene). Jeg har gaffet vinduet både på innsiden og på utsiden. OG VINDUET HAR SLUTTET Å PLYSTRE HERREGUD KAN DERE TRO DET DET ER HELT STILLE PÅ HYBELEN.

(Noe intens hamring på tastaturet kan dog høres, og nattesøvnen har ennå ikke kommet, men jeg har trua. Det blir bedre.)

Jeg teipet ventilgreia. Det hjalp ikke. Ikke tro at det hjelper å gaffe ventilen. Vinduet gir faen.

Det er her det gjelder. Det er her det må gaffes.

Bare hyggelig.

Before I start, I'll admit that I'm not a real RPM packager. Maype I'm approaching this from completely the wrong direction, what do I know?

I'm in the process of packaging Varnish 3.0.2 which includes mangling the spec file. The top of the spec file reads:

%define v_rc
%define vd_rc %{?v_rc:-%{?v_rc}}

Apparently, this is not legal, since we're trying to define v_rc as a macro with no body. It's however not possible to directly define it as an empty string which can later be tested on, you have to do something like:

%define v_rc %{nil}
%define vd_rc %{?v_rc:-%{?v_rc}}

Now, this doesn't work correctly either. %{?macro} tests if macro is defined, not whether it's an empty string so instead of two lines, we have to write:

%define v_rc %{nil}
%if 0%{?v_rc} != 0
%define vd_rc %{?v_rc:-%{?v_rc}}
%endif

The 0{?v_rc} != 0 workaround is there so that we don't accidentially end up with == 0 which would be a syntax error.

I think having four lines like that is pretty ugly, so I looked for a workaround and figured that, ok, I'll just rewrite every use of %{vd_rc} to %{?v_rc:-%{?v_rc}}. There are only a couple, so the damage is limited. Also, I'd then just comment out the v_rc definition, since that makes it clear what you should uncomment to have a release candidate version.

In my naivety, I tried:

# %define v_rc ""

# is used as a comment character in spec files, but apparently not for defines. The define was still processed and the build process stopped pretty quickly.

Luckily, doing # % define "" seems to work fine and is not processed. I have no idea how people put up with this or if I'm doing something very wrong. Feel free to point me at a better way of doing this, of course.

Hvordan kan campusskjermer gjøres interessante? Bidra gjerne med dine tanker, dette er et slags førsteutkast for min del.

I dag har NTNU flere skjermer rundt på campus, i hovedsak for å pushe informasjon på studentene som de ikke har bedt om. Det må på alle måter kunne kalles reklame, bortsett fra at formålet ikke er salg, men at NTNU ønsker studenten skal ha informasjonen. Fordi det ikke gis informasjon ut fra hva studenten synes er interessant, blir tavlen uinteressant – og ubrukt.

Måten vi gjør slike skjermer mer brukt, er å se på NTNUs informasjon som reklame.Hvordan får vi folk til å se på reklame? Selvsagt ved å pakke reklamen inn i “innhold”, enten det er Idol eller en reportasje i Morgenbladet. Innhold kjennetegnes ved at det skal være interessant ut fra studentens egen vurdering – innhold som ingen har lyst på, er dårlig innhold. Dette i motsetning til reklame/informasjon.

Så, hvilke behov kan “innhold” på infoskjermer dekke?

Først litt om situasjonen og folks atferd folk når de bruker infoskjermene

• Demografi: de er studenter (lite penger, mye frihet, mellom 19 og 25 år).
• De er på campus
• Klokken er mellom 0800 og 1500, men vanligvis mellom hel og kvart over fordi dette er pausene mellom forelesninger
• De er på vei fra ett sted til et annet, og vet derfor hvor de skal og hva de skal gjøre den neste timen. De ser seg ikke mye omkring.
• De går forbi skjermen og har i beste fall noen få sekunder til rådighet (mellom to og fire?) – innholdet skal være helt ekstraordinært for å få noen til å stoppe opp.
• De går der sannsynligvis ofte, og gjør det de pleier gjøre av gammel vane. Hvis de er vant med å se på en infoskjerm, gjør de det. Hvis de ikke er vant med det fordi skjermen er full av reklame, gjør de det ikke. Suksesskriteriet er ikke om de kjenner til skjermen, men om de bruker den.

Med det i mente, hvilke behov kan vi dekke med innholdet? Her er noe jeg kommer på i farta:

• Underhold meg 1! – gi studentrelevante ferske nyheter i twitterformat, kort nok til at man kan få alt med seg på to til fire sekunder
• Underhold meg 2! – vis fotografier, gjerne fra studentrelevante deler av nyhetsbildet eller kulturlivet (hva med et bilde av gårsdagens David Guetta-konsert?)
• Hva er det til middag i dag? – vis frem dagens middagsmeny i kantinene på samme campus
• Hva sier værmeldingen for resten av dagen og i morgen? – man kan spørre seg om verdien av dette i et Trondheim hvor studenter går med allværsjakke uansett vær, men været er noe som angår alle.
• Hva kan jeg gjøre på campus i dag? – gi informasjon om studentrelevante events som er åpne for studenter på campus senere i dag, så man kan få fylt den ledige timen mellom forelesninger
• (Hva skal jeg finne på i kveld? – gi informasjon om studentrelevante arrangementer som foregår samme dag/uke)

I tillegg til dette kommer selvsagt “reklamen” som NTNU i første omgang ønsker å få ut med skjermene; dels for NTNU-events og generell informasjon (antar jeg), og ikke minst informasjon fra studentorganisasjonene (“kom på arrangementet vårt!” eller “bli med i organisasjonen vår!”).

Det er en forutsetning her at all informasjonen er bearbeidet slik at den er fordøyelig for de forbipasserende (husk – de har bare to til fire sekunder på seg).

1. Alltid noe nytt. Hvis informasjonen er den samme når du går til som fra kantina, slutter du etter hvert å se på skjermen. Mens plakater byttes ut en gang i uken, vil en slik skjerm “alltid” kunne ha noe nytt å vise deg.
2. Kan det ikke twitres, kan det ikke vises. Informasjonen presenteres i så kort format at studentene rekker lese den på sine to til fire sekunder. Jeg gjetter at lister eller setninger ikke kan være mer enn 100-140 tegn (15-25 ord).
3. Konsistens er vesentlig. For hver aktør man slipper til med dårlig prosessert informasjon (dvs for mange ord), synker skjermens troverdighet. En kurator/redaktør er derfor helt nødvendig for at skjermene skal brukes og reklamen svelges.

I’ve recently been tweaking my server setup for a Django 1.3 web site with the goal of making it a bit faster. Of course, there is a lot of speed to gain by improving e.g. the number of database queries needed to render a web page, but the server setup also has an effect on the web site performance. This is a log of my findings.

All measurements have been done using the ab tool from Apache using the arguments -n 200 -c 20, which means that each case have been tested with 20 concurrent requests up to 200 requests in total. The tests was run from another machine than the web server, with around 45ms RTT to the server. This is not a scientific measurement, but good enough to let me quickly test my assumptions on what increases or decreases performance.

The Django app isn’t particularly optimized in itself, so I don’t care much about the low number of requests per second (req/s) that it manages to process. The main point here is the relative improvement with each change to the server setup.

The baseline setup is a Linode 1024 VPS (Referral link: I get USD 20 off my bill if you sign up and remain a customer for 90 days), running Apache 2.2.14 with mpm-itk, mod_wsgi in daemon mode with maximum 50 threads and restart every 10000 requests, SSL using mod_ssl, and PostgreSQL 8.4.8 as the database. For the given Django app and hardware, this setup is strolling along at 4.0 req/s.

With this blog post as reference, I switched from Apache+mod_wsgi to using nginx 0.7.5 as SSL terminator, for serving static media, and as a proxy in front of Gunicorn 0.13.4. Gunicorn is a WSGI HTTP server, hosting the Django site. The Linode VPS got access to four CPU cores (n=4), so I set up nginx with 4 workers (n) and Gunicorn with 9 workers (2n+1). Different values for these settings are sometimes recommended, but this is what I’m currently using. This setup resulted in an increase to 9.0 req/s.

A nice improvement, but I changed multiple components here, so I don’t know exactly what helped. It would be interesting to test e.g. Apache with mod_proxy in front of Gunicorn, as well as different number of nginx and Gunicorn workers. The nginx version is also a bit old, because I used the one packaged in Ubuntu 10.04 LTS. I should give nginx 1.0.x a spin.

Next up, I added pgbouncer 1.3.1 (as packaged in Ubuntu 10.04 LTS, latest is 1.4.2) as a PostgreSQL connection pooler. I let pgbouncer do session pooling, which is the safest choice and the default. Then I changed the Django app settings to use pgbouncer at port 6432, instead of connecting directly to PostgreSQL’s port 5432. This increased the performance further to 10.5 req/s.

Then, I started looking at SSL performance, without this being the bottleneck at all. I learned a lot about SSL performance, but didn’t improve the test results at all. Some key points was:

• nginx defaults to offering Diffie-Hellman Ephemeral (DHE) which takes a lot of resources. Notably, the SSL terminators stud and stunnel does not use DHE. See this blog post for more details and how to turn off DHE in nginx.

• If you’re using AES, you can process five times as many requests with a 1024 bit key compared to a 2048 bit key. I use a 2048 bit key.

• 64-bit OS and userland doubles the connections per second compared to 32-bit. My VPS is stuck at 32-bit for historical reasons.

• SSL session reuse eliminates one round-trip for subsequent connections. I set this up, but my test setup only use fresh connections, so this improvement isn’t visible in the test results.

• Browsers will go a long way to get hold of missing certificates in the certificate chain between known CA certificates and the site’s certificate. To avoid having the browser doing requests to other sites to find missing certificates, make sure all certificates in the chain are provided by your server.

  If you’re switching from Apache to Nginx, note that Apache uses separate files for your SSL certificate and the SSL certificate chain, while Nginx wants these two files to be concatenated to a single file, with your SSL certificate first.

Next, I read about transaction management and the use of autocommit in Django. The Django site I’m testing is read-heavy, with almost no database writes at all. It doesn’t use Django’s transaction middleware, which means that each select/update/insert happens in its own transaction instead of having one database transaction spanning the entire Django view function.

Since I’m using PostgreSQL >= 8.2, which supports INSERT ... RETURNING, I can turn on autocommit in the Django settings, and keep the transaction semantics of a default Django setup without the transaction middleware. Turning on autocommit makes PostgreSQL wrap each query with a transaction, instead of Django adding explicit BEGIN, and COMMIT or ROLLBACK statements around each and every query. Somewhat surprisingly, this reduced the performance to 9.2 req/s. Explanations as to why this reduced the performance are welcome.

Reverting the autocommit change, I got back to 10.5 req/s. Then I tried tuning the PostgreSQL configuration using the pgtune tool. I went for the web profile, with autodetection of the amount of memory (1024 MB):

pgtune -i /etc/postgresql/8.4/main/postgresql.conf -o postgresql-tuned.conf -T Web
mv postgresql-tuned.conf /etc/postgresql/8.4/main/postgresql.conf

pgtune changed the following settings:

maintenance_work_mem = 60MB        # From default 16MB
checkpoint_completion_target = 0.7 # From default 0.5
effective_cache_size = 704MB       # From default 128MB
work_mem = 5MB                     # From default 1MB
wal_buffers = 4MB                  # From default 64kB
checkpoint_segments = 8            # From default 3
shared_buffers = 240MB             # From default 28MB
max_connections = 200              # From default 100

After restarting PostgreSQL with the updated settings, the performance increased to 11.7 req/s.

To summarize: in a few hours, I’ve learned a lot about SSL performance tuning, and–without touching any application code–I’ve almost tripled the amount of requests that the site can handle. The performance still isn’t great, but it’s a lot better than what I started with, and the setup is still far from perfect.

To get further speed improvements, I would mainly look into three areas: adding page (or block) caching where appropriate, log database queries and tweak the numerous or slow ones, and look further into tweaking the PostgreSQL settings. But, that’s for another time.

If you have suggestions for other server setup tweaks, please share them in the comments, and I’ll try them out.

Updated: Removed the “mean response time” numbers, which simply is (time of full test run) / (number of requests). It just told us the same as req/s in a less intuitive way. The other interesting number here is the perceived latency for a single user/request. I’ll make sure to include it in future posts.

New project; a hosted version of the munin (http://munin-monitoring.org/) server, running on: http://hostedmunin.com/

Munin is an open source solution for collecting metrics from servers and plotting them on trend graphs. It has a simplicity that beats all other monitoring solutions I’ve seen, including super easy plugin creation for adding new charts for your application.

This is for you if you:

• have a couple of  machines / instances here and there (for v1.0), and just want monitoring to work.
• want proper notifications and trending, without doing any configuring yourself.

The idea is that you can spend all your precious hours awake on making your application run supersmooth, instead of tuning monitoring systems.

Pricing looks like will be subscription based with a basic free package (1-3 servers, limited #plugins, but quite usable) and premium package with more nodes and extra features.

We’re in early beta, but have room for some more trial customers.If you want in, drop me an email.

I’m quite fascinated by how quickly code rots and becomes a mess, despite the programmers best intentions. A lot could be fixed by testing and and refactoring more often, but it always seems to come second to getting shit done, and does sometimes have unintended effects.

For some reason I see this more often in Java code than in other languages. Maybe it’s because of its class-madness creating deep nested code (try to trace a call through Jersey, I dare you.).

Jersey is actually a nice example of this. The code is in itself clean, documented and well tested, but it is still (in my opinion) a mess. Seems to me that abstraction does not only have an performance penalty, but a readability issue as well. And in this case, it seems the code has been tested and refactored multiple times, but still it’s hard to understand.

Seems like there is a fine line between reusability and loss of understanding for what happens inside the black box.

Photo by Will Sculling

In Pykka 0.13–which was released almost two weeks ago–traversing the attributes of an actor is about 8.3 times faster than it used to be. To paraphrase Apple: “8.3X faster. That’s amazing!” (Update: This was written a couple of hours before the news of Jobs’ passing arrived. May he continue to inspire us.)

So, what is “traversable attributes”? Let’s take a few steps back.

If we were a conservative actor adhering strictly to the actor model, we surely wouldn’t share our attributes with anybody else. We would expect other actors to send serializable messages to us, asking nicely to get the value of the attribute, or maybe asking for something else. Of course, the other actors wouldn’t even know the attribute existed unless we told them, and even then they wouldn’t ever dream of requesting a reference to our attribute or altering the attribute directly. It would be indecent. It would break the rules of the actor model. It would be unsafe.

When using Pykka, you can keep to the traditional way of passing messages back and forth between the actors. You start the actor by calling the Actor.start() class method, which returns an ActorRef object. This object can safely be passed around and even shared between threads. The ActorRef object got two methods for sending messages to the actor, called send_one_way() and send_request_reply(). This is nice enough by itself, and it gives you a way to build concurrent applications which is easier to reason about–just as promised by advocates of the actor model–than when you do the thread and lock management dance. You can quickly hack together a simple actor implementation like this from scratch for each and every application you make using e.g. Thread and Queue. I’ve done this a couple of times, and it works.

But, I wanted a bit more, so I created Pykka.

First, I wanted to get rid of verbose dict messages all over my code base. I just wanted to call regular methods and access regular attributes on regular objects. Pykka provides a safe way of doing this, called ActorProxy. An ActorProxy is nothing more than a wrapper around an ActorRef. It does all it’s magic by sending messages to the actor, just like you used to do yourself.

from pykka.actor import ThreadingActor

class X(ThreadingActor):
    y = 1

>>> x = X.start().proxy()
>>> x.y.get()
1
>>> x.stop()

Second, I wanted to be able to organize actors like regular code, e.g. by splitting them into multiple classes. Imagine a running actor a which have the attribute b. The “subobject” b have the method c().

from pykka.actor import ThreadingActor

class B(object):
    def c(self):
        return 1 + 1

class A(ThreadingActor):
    b = B()

If you call a.b.c(), the following happens:

1. We send a message to actor a requesting attribute b, and immediately get a future object back which is our handle to the result which will be available in the future.

2. Actor a gets the message, looks up attribute b, and returns a copy of the object referenced by the b attribute.

3. We call c() on the future, but the Future class doesn’t have an attribute called c, so it fails. Alternatively, we use the future correctly and call get() on the future to get the real result, a copy of b. Then we call c() on the copy of b. The method c() is now running, but it is running in the caller’s thread, and not in the actor a like I wanted it to do.

>>> a = A.start().proxy()
>>> a.b.c()
AttributeError: 'ThreadingFuture' object has no attribute 'c'
>>> a.b.get().c()
2   # Result calculated in the caller's thread
>>> a.stop()

The simple attribute access that the ActorProxy provides isn’t enough to make this work.

To make the a.b.c() method call be executed in the actor a instead of the caller’s thread, we need to traverse attribute b without having it returned to us, so that we can get to c() while still inside the actor a, and call its method c(). We need what we in Pykka call traversable attributes.

To make an attribute traversable, the only thing we need to do is to mark it as such by adding the attribute pykka_traversable to the traversable attribute:

from pykka.actor import ThreadingActor

class B(object):
    pykka_traversable = True
    def c(self):
        return 1 + 1

class A(ThreadingActor):
    b = B()

>>> a = A.start().proxy()
>>> a.b.c().get()
2   # Result calculated by the actor `a`
>>> a.stop()

When you access a regular attribute of a Pykka actor, you just get a future object, which, when you call get() on it, will return a copy of the attribute. When you access a traversable attribute of a Pykka actor, you get a brand new ActorProxy which wraps the same ActorRef, but method calls and attribute accesses on the new proxy object will work on the actor’s attribute instead of the actor itself.

Speeding up access to traversible attributes

If you’re still following, you’re maybe wondering how we sped up access to traversable attributes with a factor of 8.3. The answer is a few lines up: “you get a brand new ActorProxy.”

So, why should that matter?

If you split your actor into multiple classes using traversable attributes, you’re probably going to use each traversable attribute more than once. Maybe really often. Turns out, creating brand new ActorProxy objects for the same attribute over and over again is kind of wasteful.

How did you find out?

John Bäckstrand was irritated by Mopidy being almost unusable on his slow system, and attacked the problem in the scientific way: by measuring where the bottleneck was. John quickly pointed out that access to second-level attributes, which required the traversal of a traversable attribute, was five times slower than access to first-level attributes, which didn’t involve traversable attributes. This observation made it obvious that the creation of new ActorProxy objects whenever we accessed traversable attributes–even though the proxy objects didn’t contain any state and was fully reusable–probably needed refinement.

To be sure we fixed the issue, we started by writing a performance test which compared attribute access with and without the traversal of a traversable attribute.

# Using Pykka 0.12.4
test_direct_plain_attribute_access took 0.958s
test_direct_callable_attribute_access took 0.977s
test_traversible_plain_attribute_access took 8.259s
test_traversible_callable_attribute_access took 8.344s

Then, the fix was short and easy: Cache and reuse ActorProxy objects.

diff --git a/pykka/proxy.py b/pykka/proxy.py
index 27c075b..4c6b908 100644
--- a/pykka/proxy.py
+++ b/pykka/proxy.py
@@ -58,6 +58,7 @@ class ActorProxy(object):
         self.actor_ref = actor_ref
         self._attr_path = attr_path or tuple()
         self._known_attrs = None
+        self._actor_proxies = {}
 
     def _update_attrs(self):
         self._known_attrs = self.actor_ref.send_request_reply(
@@ -88,7 +89,10 @@ class ActorProxy(object):
         if attr_info['callable']:
             return _CallableProxy(self.actor_ref, attr_path)
         elif attr_info['traversable']:
-            return ActorProxy(self.actor_ref, attr_path)
+            if attr_path not in self._actor_proxies:
+                self._actor_proxies[attr_path] = ActorProxy(
+                    self.actor_ref, attr_path)
+            return self._actor_proxies[attr_path]
         else:
             message = {
                 'command': 'pykka_getattr',

The result was immediate: The performance test for traversable attribute access showed an 8.3X improvement.

# Using Pykka 0.13
test_direct_plain_attribute_access took 0.953s
test_direct_callable_attribute_access took 0.988s
test_traversible_plain_attribute_access took 0.984s
test_traversible_callable_attribute_access took 1.006s

Mopidy use Pykka’s traversable attributes heavily to organize its backend code. Obviously, we try to avoid wiring up lots of actors in Mopidy’s unit tests, but we’ve been lazy and use some actors in the tests. These five lines of code inserted at the right place in a dependency made Mopidy’s test suite run 20% faster, and made John’s use case run 166% faster.

We could use more of five-line patches like that :-)

We use SugarCRM at work and I've complained about its not-very-RESTy REST interface. John Mertic a (the?) SugarCRM Community Manager asked me about what problems I'd had (apart from its lack of RESTfulness) and I said I'd write a blog post about it.

In our case, the REST interface is used to integrate Sugar and RT so we get a link in both interfaces to jump from opportunities to the corresponding RT ticket (and back again). This should be a fairly trivial exercise or so you would think.

The problems, as I see it are:

• Not REST-y.
• Exposes the database tables all the way through the REST interface
• Lack of useful documentation forcing the developer to cargo cult and guess
• Annoying data structures
• Forced pagination

My first gripe is the complete lack of REST in the URLs. Everything is just sent to https://sugar/service/v2/rest.php. Usually a POST, but sometimes a GET. It's not documented what to use where.

The POST parameters we send when logging in are:

method=>"login"
input_type=>"JSON"
response_type=>"JSON"
rest_data=>json($params)

$params is a hash as follows:

user_auth => {
            user_name => $USERNAME,
            password => $PW,
            version => "1.2",
},
application => "foo",

Nothing seems to actually care about the value of application, nor about the user_auth.version value. The password is the md5 of the actual password, hex encoded. I'm not sure why it is, as this adds absolutely no security, but it is. This is also not properly documented.

This gives us a JSON object back with a somewhat haphazard selection of attributes (reformatted here for readability):

{
     "id":"<hex session id>,
     "module_name":"Users",
     "name_value_list": {
             "user_id": {
                     "name":"user_id",
                     "value":"1"
             },
             "user_name": {
                     "name":"user_name",
                     "value":"<username>"
             },
             "user_language": {
                     "name":"user_language",
                     "value":"en_us"
             },
             "user_currency_id": {
                     "name":"user_currency_id",
                 "value":"-99"
             },
             "user_currency_name": {
                     "name":"user_currency_name",
                     "value":"Euro"
             }
     }
}

What is the module_name? No real idea. In general, when you get back an id and a module_name field, it tells you that the id exists is an object that exists in the context of the given module. Not here, since the session id is not a user.

The worst here is the name_value_list concept which is used all over the REST interface. First, it's not a list, it's a hash. Secondly, I have no idea what would be wrong by just using keys directly in the top level object, so the object would have looked somewhat like:

{
     "id":"<hex session id>,
     "user_id": 1,
     "user_name": "<username>,
     "user_language":"en_us",
     "user_currency_id": "-99",
     "user_currency_name": "Euro"
}

Some people might argue that since you can have custom field names this can cause clashes. Except, it can't, since they're all suffixed with _c.

So we're now logged in and can fetch all opportunities. This we do by posting:

method=>"get_entry_list",
input_type=>"JSON",
response_type=>"JSON",
rest_data=>to_json([
            $sid,
            $module,
            $where,
            "",
            $next,
            $fields,
            $links,
            1000
])

• $sid is our session id from the login
• $module is "Opportunities"
• $where is opportunities_cstm.rt_id_c IS NOT NULL. Yes, that's right. An SQL fragment right there and you have to know that you'll join the opportunities_cstm and opportunities tables because we are using a custom field. I find this completely crazy.
• $next starts out at 0 and we're limited to 1000 entries at a time. There is, apparently, no way to say "just give me all you have".
• $fields is an array, in our case consisting of id, name, description, rt_id_c and rt_status_c. To find out the field names, look at the database schema or poke around in the SugarCRM studio.
• $links is to link records together. I still haven't been able to make this work properly and just do multiple queries.
• 1000 is the maximum number of records. No, you can't say -1 and get everything.

Why this is a list rather than a hash? Again, I don't know. A hash would make more sense to me.

The resulting JSON looks like:

{
    "result_count" : 16,
    "relationship_list" : [],
    "entry_list" : [
       {
          "name_value_list" : {
             "rt_status_c" : {
                "value" : "resolved",
                "name" : "rt_status_c"
             },
             […]
          },
          "module_name" : "Opportunities",
          "id" : "<entry_uuid>"
       },
       […]
    ],
    "next_offset" : 16
}

Now, entry_list actually is a list here, which is good and all, but there's still the annoying name_value_list concept.

Last, we want to update the record in Sugar, to do this we do:

method=>"set_entry",
input_type=>"JSON",
response_type=>"JSON",
rest_data=>to_json([
    $sid,
    "Opportunities",
    $fields
])

$fields is not a name_value_list, but instead is:

{
    "rt_status_c" : "resolved",
    "id" : "<status text>"
}

Why this works and my attempts at using a proper name_value_list didn't work? I have no idea.

I think that pretty much sums it up. I'm sure there are other problems in there (such as the over 100 lines of support code for the about 20 lines of actual code that does useful work), though.