Metisse / Façades : did I fail promoting it ?

Before I forget, thanks to Luis, I have a new hackergotchi. If you are aggregating myblog in a planet, please update my hackergotchi. Thanks.

Back on the main subject : I'm both happy and sad to see Jim talking about Façades. I'm happy because I'm no longer the only one talking about the various features of Metisse. But I'm sad, because Jim seems to discover Métisse and Façades, while we have been promoting Metisse for more than two years in Mandriva Linux and I've created screencast and doing demonstrations in various conférences. Our goal was to try to raise awareness of those kind of technologies (resulting from a lot of research and user testing done by Metisse team) in Free software desktop environment, hoping interested people would help Metisse team in develop their software or integrate some of Metisse ideas in "common" desktop environments.

I have mixed feeling about the results : we succeed because Metisse team was able to get a funding for hiring someone to work on Metisse code (Rémi Cardona) and helping to either port it to last Xorg technologies or get relevant Métisse technologies merged in Xorg platform (input redirection is handled in Métisse for a long time, for instance). But I also have the feeling I failed on promoting Metisse, since people are only discovering it now, instead of two years ago :(

So, if you want to refresh your views about Metisse, you can view screencasts I did two years ago (created directly from Metisse), either on Dailymotion (now available in high quality) or download them there. I also suggest you read their various research papers, they are extremely interesting. And of course, you can grab Mandriva Linux 2009 (or earlier versions), where Metisse is available as an option, just like Compiz Fusion.

I'm weak

Two weeks ago, I was invited in a Playstation event in a nice loft near Trocadero to test upcoming games on both PS3 and PSP, so I mostly tested Little Big Planet (on PS3), Locoroco 2 on PSP (I'm a fan of the first Locoroco on PSP) and Resistance 2 (I'm really bad at this kind of game). I mostly wanted to test Little Big Planet, since I first saw some videos from the game and next Prince of Persia (but it wasn't there to be tested unfortunately). And I wasn't disappointed (compared to two years ago where I was invited to a similar event, before PS3 official launch in France and almost no game was incentive enough for buying a PS3, unlike Wii). So, I was just weak : I waited until official launch of Little Big Planet in France (last Friday), available in a PS3 bundle and I bought it. And I really don't regret my decision, this game is great.

Since Friday, I have innoculated Little Big Planet virus to 6 other players (4 who had already a PS3 and now want to buy the game, and a couple who were waiting for good games on PS3 before buying one and plan to now ;), and I also contamined another friend, by just talking to him on Friday after over irc (and I must confess I was sure he already had a PS3 when I discussed with him, but he only had a PSP :D).

I'm weak, I know that now. And if you want to be weak with me on Little Big Planet, just add "UnFred" as your PSN contact...

Mandriva Linux 2009 released

We have just released Mandriva Linux 2009, with, among many features, GNOME 2.24 and all the boot optimizations I talked about last time (and new ones too).

You can download it using bittorrent from http://torrent.mandriva.com/public/.

Enjoy !

Improving boot time on a general Linux distribution, not an easy task

We have just just released Mandriva Linux 2009 RC2 (with GNOME 2.24 final version, among new features ;), as well as reduce boot time on a lot of systems.

I thought it would be interesting to explain the various things we tried to save some seconds when booting, since it is a hot topic these days, with impressive results from various people, including Arjan Van de Ven 5s boot on a EEE 901 PC, even if I don't agree with all Arjan conclusions, mostly because it is not always possible to achieve the same kind of tuning with a flexible distribution which can run on many hardware platform, in contrast of a stripped installation and on a single (and now underpowered) hardware platform (Unfortunately, Mandriva folks couldn't attend to LPC this year, because we were busy working on Mandriva 2009 release ; let's hope new year LPC schedule won't conflict with our own schedule).

Before continuing, boot time covers three different aspects (and timing) :
- full boot time : from kernel startup to text login being available
- perceived boot time : from kernel startup to graphical login being available (you'll understand why I separate both later ;)
- graphical desktop startup time : from graphical login to desktop environment up and running (all apps from the session running and correctly rendered)

Over the years, at Mandriva, we have worked on improving boot time but causing regressions in our distribution :

• In 2002 (yes, 6 years ago), we started reducing perceived boot time by starting X server (and display manager) before the entire boot was completed. This was the premise of "parallel boot" and gave good results, after some tuning (you must be sure all services needed when display manager starts have already been started).

• In 2006, Couriousous (from Mandriva community) developed a parallel init implementation, called PrcSys, which was based on initscript LSB headers to handle dependencies across services. This created a vertuous circle : by ensuring initscripts were LSB compliant, we had parallel init support as a bonus. When done properly, it allows full boot time reduction up to 12s (compared to standard boot), reduction of perceived boot time is often not as big.

For Mandriva Linux 2009.0, we decided to try to improve boot time again (mostly full boot time and perceived boot time) by trying to fix bottlenecks in boot time when we found some, after testing many different systems.

First fix was to no longer wait for network up for dbus or for display manager if user authentication was not using network (LDAP / NIS / Samba). 1s gain in perceived boot time.

Then, we looked into "udev is slow" complain. Despite what most people think, udev by itself is not slow. What is usually slow is "coldplug" ie ensuring all modules for hardware on the system are loaded at startup and waiting for those modules to settle.

After decyphering strace logs, we discovered about 256 legacy ptys were created by kernel, which are no longer needed for most usage. With help from our kernel team, we reduce the default value of those ptys to 0 (it can be increased dynamically). Gain in for full boot time and perceived boot time : 2s.

We also had reports of "udev takes forever" when people had usb storage devices plugged on their system. We did some tests and it was adding about 5s to boot, mostly because of "usb-storage" settle delay (which is 5s), when udev coldplug starts. To try to reduce this, we are now loading usb-storage module before udev is started, if an usb mass storage device is detected, to make sure the 5s "usb-storage" settle delay is done in parallel with udev. Average gain : 3s (there is still a penalty of about 2s when usb mass storage is plugged but we can't really do anything about it ATM).

We also found some hardware specific issues (Asus EEE 701 and also a Core2 duo laptop) where initrd was extremely long. After investigation, our installer was adding usb-storage support in initrd for those platforms, even when it was not needed. And since initrd was waiting for usb devices to "settle", we were loosing between 6 to 15s. Yes, 15s on EEE PC ! (BTW, you can check if you have the issue on your Mandriva system, by checking /etc/modprobe.conf for scsi_hostadapter line ; if modprobe usb-storage is there, remove the call and regenerate your initrd).

Another issue was floppy support (you know, those old plastic squares). floppy module was still trying to be loaded by default on all systems at startup, even if there was no floppy drive present, delaying the entire boot for about 2s in that case.

Unfortunately, not all our experiments were successful in reducing udev startup : since coldplug is causing a lot of modprobe calls and modprobe is not very smart (parsing kernel alias for each calls), blino wrote a modprobe daemon to try to reduce those calls but it didn't gave us any improvements (and sometime even, we got regressions). Same result when using "modprobe --all" instead of several calls to modprobe for each module.


Still on module loading subject, we tried to reduce such modprobe calls in initscripts (better to handle this with modprobe configuration file). Alsa startup script got fixed (it is much less costly now than on 2008.1 and we will probably nuke it completely on 2009 Spring, by only using udev to handle alsa support), as well as iptables script (it was doing a lot of modprobe for some optional features which were not enabled on 99% of case). This allowed 3 to 5s gain, but since those scripts were run in parallel, it didn't really reduce full boot time, but at least, cpu is not spending precious cycles doing useless work.

Then, we checked what was spending too much time in the early part of boot (preventing display manager startup) and we found two bottlenecks : harddrake and DKMS :

• harddrake (our hardware autoconfiguration tool, which reconfigure on the fly your system if your hardware has changed since previous boot or if you change kernel version and no proprietary driver are available, harddrake will reconfigure X to use free driver) was quite long and since it was blocking display manager start (you want to be sure X is properly configured), it was directly impacting perceived boot time. 
• dkms (it was integrated in our distro since 2004), is handling kernel module automatic rebuild, mostly for driver not included in Mandriva kernel or for proprietary modules. Unfortunately, even when module were correctly build for current kernel, dkms script was still very long, even for precompiled dkms module (for this particular feature, it was clearly a bug, since dkms script was not needed).

For those two bottlenecks, we discovered they were much faster to run when we timed execution after boot (after ensuring disk caches were flushed). They were impacted badly by parallel init. So, we moved dkms and harddrake startup outside parallel init to rc.sysinit and we were able to gain 3s for harddrake and 2 to 4s with dkms.

So far, we got good results but you might wonder why I didn't talk about readahead, since it is used on other distros. Well, we did experiment readahead in the past, and each time, we had regression in both full boot time and perceived boot time. Why ? Because parallel init is already doing a pretty good job and readahead was not causing regression when we disabled parallel init.
When using default readahead setup, additional IO were done when other services were also trying to start, causing bad performance. And even ensuring readahead is started before all other services caused regressions in boot time.

Does this mean readahead is a dead-end ? Not really, when you look at bootchart closely. The idea was to find time slots where no IO were done and cause readahead at that time (thanks to Arjan idea from his 5s boot talk). And there is a big slot with low IO usage : udev coldplug ! First trial was to start readhead very early in the boot but it was still causing boot time regression. Then, we tried to call readahead directly in start_udev script, just after coldplug is initiated, before all udev triggers are settled. And things started to improved (yay !).

So, we tried to check with "custom" readahead list (not using our default stripped list of files but the real list of files used on the test system). And then, regression came back. Back to drawing board. Discussion with other collegues (and Arjan hints) lead us to try something else : scheduling readahead IO as idle, to make sure readahead is not impacting other processes if readahead file list is large. And guess what ? it worked ! no more regression in boot time and even better, improved boot time, both full boot time and perceived boot time, and we were able to move readahead call back early in the boot process, before starting udev. But since test was only done on one system, we checked this change was also working on a lot of different systems, with powerful or low-end CPU, slow harddrive, fast harddrive, SSD (which are still quite slow these days). And results were quite positive : we never had any regression in boot time. Either timing were the same (EEE 701 is a good example, CPU is always at 100%, so IO are not bottleneck), or both full and perceived boot time were improved. On my home system, I got a 2s improved, in both full and perceived boot time.

So, we plugged IO idle readahead (for testers, don't search it in Mdv 2009 RC2, it was not part of it), as well as automatic readahead file list creation (based on work from Fedora folks). What does it mean ? On first boot, readahead will not be improve boot time but instead will monitor which files were used for boot. Then, the list will be optimized based on storage device and will be used on the second boot. Moreover, this list will be refreshed automatically every month (after reboot of course), to ensure optimisations are still relevant to the system.

So far, so good, but what about desktop login timing ? Kudos to Bedhad for his work on preload daemon : it preloads files used at login by desktop environment, using the idle time in display manager (gdm / kdm), when it is waiting for user to input his login and password. This daemon is monitor system to learn which programs are being used at login and preload them automatically. This is great because it is not based on static file list (difficult to do for a desktop agnostic distro like us) and doesn't require a specific mode for file monitoring, like readahead. And even better, if user changes his habbits (switching from KDE to GNOME for instance), after several logins, preload will preload GNOME files instead of KDE files. Even better, if autologin is enabled (so there is no idle time to preload), there is no regression in desktop login time, since IO preload is done in idle. After some measures, preload gave us about 5s improvement in desktop login.

In conclusion, as you can see, improving boot time is not an easy task, but we worked hard to improve it for upcoming Mandriva 2009. It requires a lot of measures (thanks bootchart), a lot of experiments and a wide range of systems (you can find some of the bootcharts used during our tests here).

Des podcasts à écouter toute l'année

Ca fait un moment que je voulais parler des différents podcasts que j'écoute régulièrement, mais étant tous francophone, j'attendais d'avoir une section française sur mon blog pour en parler (ne voyant pas trop l'intérêt sur un post en anglais) :

• Arte Radio : attention, c'est un bijou de radio. Très eclectique, pionner en France. C'est bourré de "sons" (comme on dit en radio) sur pleins de sujets divers et variés. Je conseille en particulier les différentes chroniques de Martin Winckler, ainsi que différents reportages de Claire Hauter (qui a officié avant sur France Inter dans La-Bas si j'y suis). N'hésitez pas à farfouiller le site (malheureusement en Flash) et à consulter les archives. Vous ne le regretterez pas.
• Déclencheur : toute personne un tant soit peu intéressé par la photographie se doit d'écouter Déclencheur et apprécier le professionnaliste et la passion de Benoit Marchal. J'ai découvert Déclencheur il y a un peu plus d'un an et je suis accro. Je vous conseille de télécharger tous les podcasts et de les écouter dans l'ordre, ça permet de se faire une idée de la progression du podcast, en qualité et en quantité. 
• Utopod : Fan de SF et de littérature de l'imaginaire francophone, Utopod est fait pour vous. Vous pourrez y écouter des nouvelles de SF et fantastique. Cerise sur le gateau, l'équipe d'Utopod cherche à rémunérer les auteurs des oeuvres lues. On ne peut qu'encourager de telle initiative.
• Rendez-Vous Création : C'est le podcast de Nicolas Esposito, l'un des comparses de Benoit Marchal (de Déclencheur) pour des interviews assez intéressants avec différents "créateurs", pas uniquement dans le monde de la photo.

Voilà, n'hésitez pas à vous abonnez à ces podcasts et à en faire la promotion autour de vous, ils le méritent vraiment.

Et vous pouvez aussi faire un petit post sur vos blogs pour dire quels podcasts vous écoutez..

USB Guadec keys arrived !!!

Yay, Mandriva USB  keys, Guadec edition arrived in Istanbul. If you didn't get yours yet, go to the bag stand with your badge.

Remember to fix your USB key for the live install following these instructions.

Note : it seems a subset of the USB keys were not duplicated correctly by our manufacturer and are completely empty (when mounted, label is Mandriva instead of Mdv_Flash). If you have such a key, come and see me, I'll dump flash image on your USB key.

Mandriva 2008 Spring Flash key, GUADEC edition

For those fortunate people who are attending GUADEC this year, you'll get a nice 4GB USB key featuring Mandriva Linux 2008 Spring Flash, in a special GUADEC edition, with your favorite GNOME 2.22 desktop on it :

Compared to last year Guadec key, it is now possible to use the key to install a new system with Mandriva Linux, just like you would do with a Live CD.

Currently, keys are being processed by Turkish customs at Istanbul, so I don't know yet if they'll be available for Guadec opening on Monday but if not, they should be available during the week. Guadec team will tell everybody when they get theirs hands on them.

However, we discovered (after sending keys to production, of course :( a small bug which will prevent this new install system to work properly (only Guadec keys are concerned, not standard Mandriva 2008 Spring Flash key). To fix your key and be able to use the installer, just follow this procedure :

• download this initrd file 
• if you are downloading the file from a running Mandriva Flash system, copy it to  /live/media/.boot/usb directory and restart Mandriva Flash system. 
• if you have just mounted the key on any other system, copy downloaded file to the mounted key, in .boot/usb directory.
• Here you are, you can now use Live Install feature any time you want.

Even if you don't plan to use the USB Guadec key immediatly (or if you want to give to somebody else, etc..), I strongly advise people to do the procedure so they have a fixed key which can be use whenever they want.

We also found a gdm crash when you boot the key for the first time, after filling configuration steps : you go back to text login, instead of getting autologin. If this happens, as root, just run : service dm restart. You can get the same issue the first time you'll boot a system installed with the key. It appear to be system dependent (it doesn't crash on all systems), probably a race condition where gdm doesn't always like to be told to reload his configuration.

Of course, if you have any problem with your Mandriva USB Guadec key, just grab me in GUADEC venue, I'll be attending the entire conference.

Enjoy !