Speeding up JSON decoder in PyPy

So, a couple of days ago I read a blogpost on the PyPy Status Blog written by Maciej Fijalkowski about speeding up JSON encoding in PyPy. Fijal is one of the core developers of PyPy, and in his blogpost he explains how he edited the JSON encoder to better fit the PyPy style (remember that many python files in PyPy are still tuned for running on cPython, and can be improved with simple changes).

I don’t have any particular suggestions different from the ones presented by Fijal, just want to point out that JITViewer is really useful for understanding where the code could be improved, and that sometimes just being smart can save quite a bit of computation time.

I spent some time trying to improve the string concatenation of decoder.py. The current version builds a list of chunks that are eventually joined together to build the concatenated string. This appears to be actually the fastest way (in this instance, at least): I tried the __pypy__.builders.StringBuilder and the classic += concatenation, but they turn out to be less efficient (so, mission failed, here).

I instead changed the way JSONObject builds its list of key-value elements: from Python 2.7 the decoder supports a new parameter, object_pairs_hook, for preserving ordering and multiple instances of keys. For supporting it, JSONObject builds a list of tuples storing each key-value parsed, converting it in a dictionary if no object_pairs_hook is given; the point here is that building a list of tuples is much slower than building a dictionary, and we would probably prefer to avoid this if we actually don’t need a list of pairs. This simple improvements helped speeding up the decoder a bit, achieving some good results.

I tested the new code on PyPy 1.8.1-dev0 using two benchmarks: a modification of pypy/benchmarks/json_bench (which currently evaluates only encoding, and not decoding) and kracekumar/cerealization, here the results:

PyPy 1.8.1-dev0 PyPy 1.8.1-dev0 MOD Speedup
Run #1 37.3687298298 32.1885719299 +16.09%
Run #2 37.5971119404 31.1700959206 +20.62%
Run #3 37.4318089485 31.1591031551 +20.13%
Run #4 37.3458600044 31.0966379642 +20.10%
Run #5 37.3631129265 31.117978096 +20.07%
Run #6 37.3080999851 31.0478279591 +20.16%
Run #7 37.2737519741 31.0621140003 +20.00%
Run #8 37.4270458221 30.9801039696 +20.81%
Run #9 37.3104338646 31.004087925 +20.34%
Average 37.3806616995 31.2029467689 +19.80%
Variance 0.0093633203 0,1407598879
Results on pypy/benchmarks/json_bench


cPython 2.6.1 cPython 2.7.2 PyPy 1.8.1-dev0 PyPy 1.8.1-dev0 MOD
Dump: json 26.6461949348 9.7420539856 2.9267168045 2.90510392189
Dump: simplejson 13.1301739216 12.8199129105 13.8134858608 13.8025200367
Dump: cPickle 10.0216720104 9.97437906265 25.7030470371 26.0591058731
Dump: tnetstrings 31.7227909565 30.9078440666 2.97724795341 2.96262979507
Load: json 99.1293950081 13.0056598186 10.1231429577 7.314661026 (+38.39%)
Load: simplejson 6.15336108208 5.84075093269 10.8849618435 10.9000110626
Load: cPickle 4.93965697289 4.70230102539 12.822371006 13.0902838707
Load: tnetstrings 38.5776751041 43.1720809937 6.17516303062 6.17008113861
Results on kracekumar/celeralization

Code is available on BitBucket.


Google and OAuth problems

Okay, I’ve lost like three days on this, so I really have to write it down.

It happened I found myself trying to connect to Google using the Google Data API for retrieving the user contacts (in my case I was using Python and Django, but it doesn’t really matter). I was – after a while – able to get an access token, but I was unable to actually download data: Google always returned an error page saying:

401. That’s an error.
There was an error in your request. That’s all we know.

Thanks, Google.

Well, playing around with the OAuth Playground (very bugged, but useful) I realized the problem was with my access token, so I debugged a bit. It just happens that Google returns you an escaped access token, in my case:


While the actual (unescaped) token was


After running a simple


on it, I was finally able to download data.


How to compile iPhone 3.1.3 applications with Xcode 3.2.1 without Provisioning Profile

Some days ago I started to develop an iPhone application with the latest version of Xcode. When I realized that I need to pay 99 dollars to Apple in order to test my application (that, by the way, I would share freely), I started to search on the web a solution for this problem. Unfortunately all the proposed solutions where about iPhone 2.0, and these don’t work the same with iPhone 3.1.3. Today I found an how-to that worked for me, but with some passages that could be a bit tricky for non-hackers, so I decided to share it here, with some more passages needed to get our final goal: develop an iPhone application with Xcode 3.2.1 without being registered to any Apple Development Service.

Nothing from this page comes from me, everything was written by KennyTM on his blog: networkpx Project Blog so don’t thank me – if it works – but go to his blog and visit him!

Only compile the executable

If you just want to compile the executable, maybe for including it in a .ipa file, or for copying it with scp on your device, just follow the instructions on KennyTM’s blog, under the “I want to compile.” title. It’s simple, clear and understandable (thanks, KennyTM!).

Also install it on your device

Well, this part is more tricky, but it’s not so difficult if you know how to use the Terminal (and you know how to use it, right?).

  • The point 1 is quiet simple, just download the executable by following the link, and place it in /usr/local/bin as KennyTM said. Remember to make it executable running
    chmod +x /usr/local/bin/ldid.
  • The point 2 is also simple, but remember again to make it executable, as shown above.
  • Points 3 and 4 are easy.
  • In point 5, in order to copy installd from my device to my mac, I used ssh. I have it since I have a Jailbroken iPod Touch, and I have been able to install OpenSsh with Cydia; if you have an original OS on your device (good for you) check if ssh is available looking in the Preference Panel of the device which is its IP address (you need to connect it at the same network of your mac, obviously) and executing
    ssh root@IP_ADDRESS

    from a Terminal of your mac. The default password for your device is “alpine” (remember to change it with passwd, since it’s the same for all iPhones and iPods). If it works, just run

    scp root@IP_ADDRESS:/usr/libexec/installd ~/Desktop/

    and you have copied installd on your mac.

  • Points 6 and 7 are simple, just follow them.
  • In order to complete point 8, you need to use the iPhone gcc. Looking at some comments on the blog you can find how to do this, but here it is the command you have to execute:
    /Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/gcc-4.2 libmiss.c \
    -dynamiclib \
    -install_name /usr/lib/libmiss.dylib \
    -current_version 1 \
    -compatibility_version 1 \
    -Wl,-reexport-lmis \
    -flat_namespace \
    -isysroot /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS3.0.sdk \
    -arch armv6 \
    -o libmiss.dylib
  • Points from 9 to 12 are simple, just use scp again to copy the modified files from your mac to your device (scp SOURCE DEST).

Remember to add your device to Xcode. To do that, run it and from the menu select “Window / Organizer”. Connect your device, select it and enable it!
Now you are able to compile your applications on your device! Happy working!


How to use emule collection files in mlDonkey

Well, this is not a plugin, neither a standalone or cross-platform application, it’s only a, well, I suppose we can call it workaround…

Emule collections are files used to store the ed2k link of one or (usually) more files. They can be in two different format, text or byte.

In the lucky case we have a text based emule collection file and we want to start downloading with mlDonkey all the files linked in it, we have only to copy the content of the file and paste it in the “Direct Link” window of mlDonkey (the one that is shown clicking on “DL” on the top menu, yep, that one). In fact this window let us to start downloading all the links we write in it, one for line.

Unfortunately, in most cases the files we find on the net are not text-based. In fact when we open them with a text editor we see a lot of strange characters. These files store more information about each file (have a look here if you are curious). We don’t need so much data, so the idea is to extract from the file the only one thing we need: the ed2k link of each file. We can do that with the source code of the amule-emc plugin. This plugin is used by amule exactly for reading emule collection files and using them in the application.

First of all we need the amule plugin code, we can download it from here: amule-emc download list. I used with the version 0.5.2, the last at the moment. Unpack the archive somewhere and open the plugin folder. We need only some source files, so enter the source folder and compile them with g++ as follows:

cd amule-emc-x.x.x/src/
g++ MuleCollection.cpp amule-emc.cpp -o ed2k_exporter

The amule-emc.cpp file has a main function that reads the file passed as first argument and lists all the links. So now we can execute our new application simply with

./ed2k_exporter filename.emulecollection

where obviously we have to replace “filename.emulecollection” with our file. The result is that we will have all the links printed in our terminal, so the only thing we have to do now is to copy them and use the “Direct Link” window of mlDonkey, as before.


MindGapper changes into MooGraph

Yep, it is, MindGapper has been renamed into MooGraph.

Why? Well, it’s pretty simple, MindGapper was a funny name based on GapMinder, but now MindGapper – MooGraph is a complete software for statistical data, so we decided to change it’s name.

Sooner or later also the project on SourceForce will be renamed (we hope!), so keep on following us on our trac! And if you have questions, don’t heistate writing them down as comments!


Ruby on Rails: remote functions do not work

Your remote_function or remote_form_tag could stop working if you use jQuery into your Ruby on Rails project. This is because the jQuery $ function is different from the prototype one, and RoR uses some particular function that is not available with jQuery.

For example, you can get an error like this submitting your remote_form_tag:

$(form).getElementsByTagName is not a function
Line 3483

To solve this kind of problem, you have to use the jQuery no-conflict method, like explaned on the jQuery site. Simply add theese lines of code in your default layout between into a <script> tag:

jQuery(document).ready(function($) {

Now, RoR can use prototype calling the default function $ and you can use jQuery using the jQuery function:



Ruby on Rails: adding custom elements in generated xml

If you create a scaffold for your rails project, it will create for you a lot of things; for example, a way to show your object in xml format. But what about adding custom elements to that xml?

I solved in this way:

class MyObject < ActiveRecord::Base
  def to_xml(options={})
    cp = attributes.clone
    cp["something"] = "some value"
    cp["another thing"] = self.some_relation.length


AS3 – Flex: TileList with ItemRenderer scrolling problem

When using wx:TileList with your own ItemRenderer, you have to remember two important things to avoid scrolling problems:

  1. If you override the set data method, remember to call super.data = value.
  2. Remember that the TileList reuses the ItemRenderer instances when scrolling the component! So, if you have a TileList with 5 rows and 5 cols, and an array with 500 elements as DataProvider, it will create about 25 instances of your ItemRenderer, calling the set data method on them when scrolling. So, if you add some child to your ItemRenderer under some data condition (like I did), remember to remove them if the data condition is not valid.


AS3: write a simple undefined loader

I love to write my own visual classes in actionscript3. Some days ago I was searching for an undefined loader, like a rotating animated gif. But I thought that it would be great to have something wrote in as3 to do that, so I code it.

I wrote a simple as3 class that shows some circles that moves from a point to the external of the loader, creating a simple bubble effect.

Here you can see it in action: UndefinedLoader.swf

What do we need? Well, I usually use Caurina.Tweener for the animations, you can download it on google code. Download the as3 version and copy it in your source directory.

After that create a new class extending Sprite and create a simple costructor like this:

package myControls
    import caurina.transitions.Tweener;
    import flash.display.*;
    public class UndefinedLoader extends Sprite
        private var w:Number;
        private var h:Number;
        private var colors:Array = [0xFFFFFF];
        public function UndefinedLoader(w:Number, h:Number, colors:Array = null)
            this.w = w;
            this.h = h;
            this.visible = false;
            if (colors != null)
                this.colors = colors;

We set the width and height attributes, and if the user want to change the color of the loader we set it.
We will use the ‘visible’ attribute to indicate if the loader is running or not. So we have to write a get function for that:

        public function get loading(): Boolean {
           return this.visible;

After that, we need some constants in the class, indicating the minimum and the maximum radius of the circles, the time for the animation, how many circles we want and the type of transition (see the Tweener documentation for the possible values)

        private const NCIRCLES: int = 25;
        private const MIN_DURATION: Number = 1;
        private const MAX_DURATION: Number = 2.5;
        private const MIN_RADIUS:int = 2;
        private const MAX_RADIUS:int = 5;
        private const TRANSITION: String = "linear";

Ok, now we need the start and the stop functions. When we start the loader, we want to create the circles, storing them into a private array. After that we will call the animation function on each of them. Stopping the loader will only hide it. The animation will automatically stop by itself (see later).

        public function start(): void
            if (this.visible)
                return; // animazione gia` partita!
            this.visible = true;
            if (circles.length == 0)
                for (var i:int = 0; i &lt; NCIRCLES; i++) {
                    circles[i] = new Sprite();
            for (i = 0; i &lt; circles.length; i++)
        public function stop(): void
            this.visible = false;

Now let we write the tween function: first of all we need to check if the loader is still shown. If it’s not, we will return, doing nothing. Otherwise, we generate randomly the color of the circle, the duration of the animation, the radius and the angle to move on. After that, reset the circles attribute, redraw it, and start the animation. When it will be completed, it will be recalled the tween function, checking if the loader is still working or not.

        private function tween(c:Sprite): void
            if (!this.visible) {
                return; // fermiamo l'animazione.
            var color:int = (colors.length == 1) ? colors[0] : Math.random()*(Math.abs(colors[1]-colors[0]));
            var time:Number = Math.random()*(MAX_DURATION - MIN_DURATION);
            var radius:Number = Math.random()*(MAX_RADIUS - MIN_RADIUS);
            var beta:Number = Math.random()*(2*Math.PI);
            c.alpha = 1;
            c.x = 0;
            c.y = 0;
            c.graphics.lineStyle(1, color, 1);
            c.graphics.drawCircle(w/2, h/2, radius);
            Tweener.addTween(c, {x:w/2*Math.cos(beta), y:h/2*Math.sin(beta),
                                 alpha:0, time:time, transition:TRANSITION,
                                 onComplete: tween, onCompleteParams:[c]} );

You can download the source code here: undefinedloader.as


Ubuntu 8.10 and MacBook Pro: audio problem

I have a MacBook Pro 4th generation (Penryn) and using Ubuntu 8.04 all was  perfect!

Two days ago I upgraded the operating system to Intrepid Ibex (8.10), and after some hour downloading the packages and installing them, the new Ubuntu was running on my MacBook… But also some problems war running as well! Ubuntu has some audio problem (no sound at all!!) on MacBook Pro…

The sound card is the Hda-Intel and the kernel module is the snd-hda-intel. The solution? After some search on the net, I found that: help.ubuntu.com

Here the solution (very easy, really):

1. Open the file /etc/modprobe.d/options (you need to be root)

sudo gedit /etc/modprobe.d/options

2. Add the following line at the end of the file

options snd_hda_intel model=mbp3

3. If you have changed /etc/modprobe.d/alsa-base previously, adding something similar to the previous line, comment it.

4. Reboot (mute your microphone before doing that, or your ears will implore pity!)

Older posts «