Hell Oh Entropy!

The title pretty much summarizes what most of my day looked like on day 2 of FUDCon. Well, not exacly, but it comes quite close. I had three sessions lined up in a single day and I was worried that I might lose my voice by the end of it. Ankur Sinha had all of 4 talks in the single day, so I was definitely better off that him.

The day started with Harish Pillay’s keynote on the community architecture team. The turnout on day 2 was less than that on day 1, which was a little surprising. Most of them trickled later in the day, so it meant that a large number of the attendees in Harish’s talk were Red Hatters and the CoEP volunteers. We probably started a little too early for a Saturday.

Immediately following that was my session on qpid messaging. The attendance in the session was modest (about 8-10 people), but the best part was that they were very involved in the session and that made the session worthwhile. Mrugesh Karnik also joined the session mid-way and asked some really good questions that actually helped my session. We ended up doing a queue design for a fictitious stock trading system and I was able to show how the design could scale very easily with a qpid messaging broker in place. Unfortunately, most of the attendees did not have laptops, so I could not engage them in a hands-on session. In fact, that was my story of the day to a large extent. I had intended all my sessions to be hands-on, but most of it never really materialized because most of the audience did not have laptops.

After the qpid session, I spent some time chatting with Sankarshan, Mrugesh, Anurag and Nisha over lunch. After that I decided to double-check my exploit code samples because it was the one session that I had never done before and it was something that is not my area of expertise. The only aspect of the exploits that I was really comfortable with was how they worked and how I could explain that using the usual tools like gdb, objdump, etc.

I was sitting in the speakers lounge cleaning up my examples when Aditya Patawari came in and asked me about my session. That reminded me that I had to actually go into the session :D We quickly left for the classroom and found pjp finishing up his python session, which had a packed audience. Once he was done, a lot of people left, which led me to think that even this talk was going to have a modest audience. However, people trickled in as I was about to begin and by the time I did begin, the room was full.

The exploits session was probably one of the best sessions I have done so far, mainly because I personally enjoyed it. The audience also consisted of people who were interested (exploits are sexy, as someone said later) and I got a lot of questions during and after the session. The talk also seemed to give some people from the audience the impression that I am a security expert, which is flattering but incorrect.

Then came the awesome part where Pai and Yogesh Babar followed up my session with impromptu sessions, which the audience lapped up eagerly as well. Pai talked about extensibility of postgresql by making it call routines in perl (typical dinosaur stuff ;) ) and Yogesh did a talk on kdump. I learnt later that Rahul Sundaram did something similar in one of the seminar halls by asking the audience to “ask him anything about Fedora and Open Source”. Pretty cool stuff.

After Pai and Yogesh were done, it was again time for me to get on to the platform for another session, this time on autotools. This was something I had done multiple times with the same examples, so it was pretty uneventful.

Day 2 was probably awaited by a lot of people for another reason -- the FUDPub! We went to Park Estique near Vimaan Nagar for dinner. There was loud music and bling bling lights and food and drink. I enjoyed the food and drink; the lights gave my a headache and the loud music was, well, too loud. In any case it was fun chatting with people and having the really good food.

Like the first day, I did not get to attend any other sessions, this time for a different reason. I’ll probably submit less sessions in the next conference so that I actually get to attend other sessions and meet and talk to more people. I did meet a lot of interesting people on day 2, so all of that hectic schedule was completely worth it.

So we’re on the final day of FUDCon Pune now and I realized that I hadn’t written about my experience at the event at all. The last two days were very hectic and I got only about 6-7 hours of sleep on both days altogether.

I arrived at the venue at 7:30 AM to help out with registrations. Satya was in charge of that and we did not have any hiccups there. People trickled in fairly slowly and hence it was easier than we had thought it would be. We had a good crowd in; going by the fact that the auditorium was about half full, thee may have been over 250-300 people in there. The auditorium capacity is 900.

After a quick welcome note from the CoEP management, we kicked off the event with Jared Smith’s keynote on Fedora. Jared came up with the familiar analogy of streams to explain the way Fedora and upstream communities interact. He talked about how community members could transition from users of Fedora to contributors in Fedora. It was a great talk in general. I had one personal bone to pick in that however, which is the kind of options conveyed to students as avenues to contribute to Fedora.

I have been interacting with students on and off for about 2 years or so now and also been noticing the avenues that students seem to prefer to contribute to Fedora. One of the things I have always been concerned about in this regard is the fact that most students seem to prefer being ambassadors and volunteers and generally doing as less code as possible. In that context, stressing about non-technical ways to contribute was not something I agree with. Of course, it is not something that is wrong with Jared’s message in the talk; it is the fact that the message may be received incorrectly that I am concerned about. This is also why I am generally more inclined towards doing technical sessions instead of broad talks especially if students are the audience.

Anyway, so that was pretty much the only talk I attended, since I was the designated hall monitor for the speakers lounge. My job was to make sure that things run smoothly there and help speakers if they needed anything. While sitting there, I decided to work on the code for my exploits demo. I had wasted about 4 hours last night trying to get the shellcode exploit to work. I had found out in the end that the gold linker builds binaries without execute permissions for the stack and to et an executable stack, I would have to provide ‘-z execstack’ as an extra argument to the linker.

Huzaifa and Eugene Teo came in while I was working on this and Huzaifa introduced me to Eugene. This gave me a chance to pick their brains about the stuff I was about to do in the exploits workshop. Huzaifa pointed me to a few resources and also gave tips about various things I could add to my session to make it more complete. I gave up working on this after the crowd in the lounge increased and returned to chatting with people, some who I was meeting after a long time and others who I had met for the very first time.

I occasionally walked over to the sessions happening nearby to see how things were going and was pleasantly surprised that these sessions had good attendance too. I was concerned about the fact that we may find it difficult to direct attendees to the classrooms since they were a good 5-7 minute walk from the main auditorium and the seminar halls. On the contrary Arun Sag’s introductory session on Linux and Linux commands was hugely successful with the classroom being packed full.

So the first day of FUDCon was a good day of meeting friends for me. The second day was very hectic too and I will write a separate post about it since now it is time for me to work on Kushal’s libgqpid.

Firstly,

and I’ve got a lot of things to do once I’m there!

FUDCon Pune 2011 has a wonderful line-up of talks and sessions going for it this weekend. I have added three workshops of my own too:

• Getting started with autotools: This is intended to be a hands-on session on autotools. I have done this session multiple times before and it has been quite successful with beginners looking to understand the GNU build system.
• Security exploits, Live!: This was originally Amit Shah’s idea, since he wanted someone to do a demo of buffer overflows and similar stuff. You don’t really have to be an aspiring security professional to attend this, since most of the ideas are based on basic programming fundamentals
• Apache Qpid messaging: “Enterprise Applications” are not typically things that you deal with when you’re in college. Quite often one ends up thinking of these things as inaccessible due to high costs and high perceived difficulty level. In this session, I intend to demonstrate how easy it is to write really complicated applications with mind-blowing ease.

I finally got involved in preparations for FUDCon this week to try and help wherever needed. Rahul suggested I help out with the schedule arrangement, since I had some ideas on the layout of some sessions. We slugged it out for about 4 hours yesterday and another 3 hours today at the Red Hat office, trying to make sure we schedule talks in a manner that attendees can follow tracks (virtualization, security, web apps, embedded, etc.) and at the same time, have enough flexibility to add talks and sessions without having to shuffle things around all the time. Saleem has become a pro at copy-pasting across cells in spreadsheets as a result.

The end result is pretty impressive and we’re expecting even more submissions to keep people busy. There were concerns on whether there were too many sessions running in parallel, but I don’t think that matters. The event attendance is expected to be quite large, so there will e enough audience to keep all speakers busy. Besides, I don’t know a single conference where one gets to attend each and every talk in the conference. The real fun of the conference is to meet friends, collaborators and fellow geeks and not just sitting in rooms and listening to people talk.

Oh, and we have hackfests throughout the day on Sunday. The one I am particularly psyched about is Kushal Das’ libgqpid. It was an idea we had brainstormed about earlier and he wrote a lot of the code in it. Hopefully we can get a release out on Sunday with the core qpid client features ready. For the uninitiated (which is pretty much everyone I guess, since Kushal has not published the code yet), libgqpid is a glib based C wrapper around the qpid C++ client.

Looking forward to having a great time at FUDCon!

I have finally moved my website content over to dreamhost. The Net4 account has been utter nonsense, with its idiotic control panel and silly pricing. For more or less the same price, I am hosting two websites on dreamhost with unlimited email aliases, subdomains, mysql database, host of apps and a really good control panel. Livejournal has been fairly sucky too lately with its random downtimes, ads and loads of comment spam.

Everything looks pretty much the same right now, but I’ll soon start making changes. I will send requests to update planet links (at least wherever I know I am syndicated) once I have everything in place. I need to write up a bit on the couple of sessions I’ll be doing at FudCon Pune. this coming weekend, hopefully by tomorrow.

September has been a very busy month for me with Siddhi going to UK for her studies, Mom moving in with us and my role change at work. In all of this confusion, a Fedora Activity Day (FAD) was planned for 10th September. I had to skip it because I had to go to Mumbai to spend the week with Siddhi before she left for Manchester. (Un)luckily, that FAD was rescheduled because of some last moment issues at COEP and it finally happened today at our office.

This FAD was planned to be something of an initiation for students so that they could do much more at the FUDCon than just attend talks. Going with that theme, I pitched in to do a rerun of my autotools demo that I had done last year, and even in the Fedora classroom. It is one of the basic things that a Fedora packager ought to know and if we're doing sessions on packaging, we might as well throw in some autotools foo.

There were a bunch of talks planned, mostly by Rahul Sundaram and Shakthi Kannan and then some by me, Prasad, Kashyap and Shreyank. But only a few of those talks actually happened. Essentially, since I came in, only I, Shakthi and Rahul spoke at all.

To begin with, I was late. According to the schedule, the talks were to happen in two meeting rooms in parallel and me and Shakthi were to do the autotools demo in parallel in both rooms. But when I reached office, I saw that there were enough to fit into one large meeting room, so scheduling talks was going to be easier.I had missed Rahul's first talk and he had already begun talking about packaging. Rahul walked everyone through the process of packaging the hello world app. After joking around a bit with Pai and Kushal, I joined Kushal and Shakthi in helping those who got stuck in the packaging demo.

Lunch followed the rpm walkthrough and we literally destroyed the pizzas that came in - there weren't enough in the end and we had to order a few more pizzas.

While the pizzas were being delivered, Shakthi walked everyone through version control using git. The main theme was managing love letters to various bollywood superstars using git. Pretty entertaining :)

After the second lunch break (where more pizzas were quickly destroyed), I was to start my demo on autotools. Before it started, Pai and Kashyap had a couple of quick 2 minute talks. Kashyap because he was asked by a number of attendees on how upstream, Fedora and RHEL were related. He showed a quick 1 minute video in which Paul Frields explained how bits come in from upstream into Fedora and finally into RHEL and how engineers from Red Hat are involved at every stage with the community. Pai wanted to gauge audience interest in databases and their role in ERPs -- there were a fair number of people who were interested, so we might see a talk from him on those lines at the next FAD.

My autotools walkthrough ended up being the last thing on the agenda because I (just like the talks before me) took more time than was allotted to me to finish. We finished with some good feedback and suggestions for improvement from the audience.

The best takeaway from this session was the involvement of all of the students who attended today's FAD. The schedule was quite gruelling and all of the walkthroughs were very intense. Despite that it was refreshing to see all of them putting their heads down and giving it their best, asking good questions and not giving up at any stage. I am hoping that we will have a similar if not better response in our next FAD in October.

No, it is not a java program, it is in assembly, which was possible because I recently flashed Cyanogenmod on my phone. Here it is:

.text
.global _start
_start:
    mov r7, #1
    mov r0, #42
    swi #0

The program simply returns 42 and exits. I used my own cross-built binutils (--target=armv-android-eabi) to build this and simply copied it over to my phone, chmod and executed it. You need to put it outside your sdcard environment however, since you cannot execute anything in /sdcard. A Stackoverflow thread helped figure out the right opcode for the syscall.

It took me all of 6 hours to figure this out. 5 minutes to figure out the above code and 5 hours and 55 minutes to realize and fix the last line from

swi 0

to

swi #0

I was following an email thread in the libc-help mailing list, where the OP asked how one could force a program to load with a different loader, i.e. a loader from a different build of glibc. The answer, which is available if you follow the thread, is that it is necessary to rebuild the program to do this, with the --dynamic-linker switch to the linker. The best answer for the thread was to do this without a rebuild by making a wrapper script that invokes the program with an alternate loader, like:

/path/to/other/ld-linux.so program_name

So the story should have ended there. But it got me thinking; how hard could it be to edit the program to change the hard-coded loader value in the program? So I decided to find out.

The first try was as simple as replacing the string with my new loader string in a test program, which is basically just a Hell Oh World program. To do this, I opened the binary in emacs and changed to hex mode using M-x and then hexl-mode. Next I just wrote over the string with the absolute path of the new loader. I saved my changes and ran the program:

[siddhesh@localhost ~]$ ./a.out
bash: ./a.out: cannot execute binary file

Obviously I had done something wrong. So I had to do a bit of reading to understand what all was to change. One very obvious thing about this was that the length of the loader string was being recorded somewhere -- I was prety sure I had not overwritten anything and I had not changed the offset of the string at all. So I started reading the System V ABI documentation, which includes information on the ELF header. The ELF header documents where the program header table is, and the size of each program header table entry. one could read this very easily using the elfutils readelf command. So here's what I got:

[siddhesh@localhost ~]$ readelf -h ./a.out
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x4003e0
  Start of program headers:          64 (bytes into file)
  Start of section headers:          2472 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         8
  Size of section headers:           64 (bytes)
  Number of section headers:         30
  Section header string table index: 27

So the important information here is the start of program headers and the size of each program header. In the case above, the program header starts 64 bytes into the file and then every 56 bytes after that was a program header. You can also read the program headers using elfutils:

[siddhesh@localhost ~]$ readelf -l ./a.out

Elf file type is EXEC (Executable file)
Entry point 0x4003e0
There are 8 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x0000000000400040 0x0000000000400040
                 0x00000000000001c0 0x00000000000001c0  R E    8
  INTERP         0x0000000000000200 0x0000000000400200 0x0000000000400200
                 0x000000000000001c 0x000000000000001c  R      1
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
  LOAD           0x0000000000000000 0x0000000000400000 0x0000000000400000
                 0x000000000000068c 0x000000000000068c  R E    200000
  LOAD           0x0000000000000690 0x0000000000600690 0x0000000000600690
                 0x00000000000001ec 0x0000000000000200  RW     200000
  DYNAMIC        0x00000000000006b8 0x00000000006006b8 0x00000000006006b8
                 0x0000000000000190 0x0000000000000190  RW     8
  NOTE           0x000000000000021c 0x000000000040021c 0x000000000040021c
                 0x0000000000000044 0x0000000000000044  R      4
  GNU_EH_FRAME   0x00000000000005e8 0x00000000004005e8 0x00000000004005e8
                 0x0000000000000024 0x0000000000000024  R      4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     8

 Section to Segment mapping:
  Segment Sections...
   00     
   01     .interp 
   02     .interp .note.ABI-tag .note.gnu.build-id .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn .rela.plt .init .plt .text .fini .rodata .eh_frame_hdr .eh_frame 
   03     .ctors .dtors .jcr .dynamic .got .got.plt .data .bss 
   04     .dynamic 
   05     .note.ABI-tag .note.gnu.build-id 
   06     .eh_frame_hdr 
   07     

The relevant section here is the INTERP section. Each program header is basically this struct:

typedef struct {
        Elf32_Word p_type;
        Elf32_Off p_offset;
        Elf32_Addr p_vaddr;
        Elf32_Addr p_paddr;
        Elf32_Word p_filesz;
        Elf32_Word p_memsz;
        Elf32_Word p_flags;
        Elf32_Word p_align;
} Elf32_Phdr;

Here, the relevant values of the string size are p_filesz and p_memsz. If you look into the values from the readelf, you'll see that the size is 0x1c, which is the length of the string, including the trailing null character. So that is what I had to change. Now I had to find this header in the hex editor. The header is the one beginning with PT_INTERP, which has value 0x03:

87654321  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789abcdef                                                                                                               
00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000  .ELF............
00000010: 0200 3e00 0100 0000 e003 4000 0000 0000  ..>.......@.....
00000020: 4000 0000 0000 0000 a809 0000 0000 0000  @...............
00000030: 0000 0000 4000 3800 0800 4000 1e00 1b00  ....@.8...@.....
00000040: 0600 0000 0500 0000 4000 0000 0000 0000  ........@.......
00000050: 4000 4000 0000 0000 4000 4000 0000 0000  @.@.....@.@.....
00000060: c001 0000 0000 0000 c001 0000 0000 0000  ................
00000070: 0800 0000 0000 0000 0300 0000 0400 0000  ................
00000080: 0002 0000 0000 0000 0002 4000 0000 0000  ..........@.....
00000090: 0002 4000 0000 0000 1c00 0000 0000 0000  ..@.....(.......
000000a0: 1c00 0000 0000 0000 0100 0000 0000 0000  (...............

I just changed the highlighted 1c to 28, which is the hex value of the string length of my interpreter (/home/siddhesh/sandbox/lib/ld-2.8.90.so). This and the change in the string was all that was needed to change the interpreter for the program. This has one really basic flaw, which is that if the interpreter string is longer than the current available space in the header, I will end up overwriting other data, thus corrupting the binary. So if you're being adventurous, be sure that your interpreter path is small enough to fit into the available space.

Lately I have been tinkering around with code in shotwell, which is the photo manager I use to sort pictures I click. As a result I built an svn snapshot and set about making my changes. On firing up the freshly built app, I saw an "Updating libraries..." progress bar pulsing away for quite some time. it just would not go away. I didn't wait long enough for it and ended the app to modify my code further. I started the app again. The progress bar comes up again and does not leave.

This had been happening for a few days and I kept ignoring it because I was working on a different part of the code (porting shotwell to cairo). I finally decided to find out what was going on today and I started shotwell in debug mode, which is simply:

SHOTWELL_LOG=1 SHOTWELL_LOG_FILE=:console: ./shotwell

... and it struck me that I had been lazy the very first time and did not bother to set the default directory for shotwell to scan. Shotwell simply assumed it to be my home directory and that was it. Scanning my home directory obviously was taking very long since it has a little over 30GB of data. Anyway, I moved my photos to a different location but it turns out that shotwell cannot read them anymore despite pointing it to the new location. I will have to modify its database to update the file paths.

So the unique thing about shotwell is that it does not modify the photo files. It stores all edits into a database correlated with the file path. So all I had to do was to edit this database (which is $HOME/.shotwell/data/photos.db, an sqlite3 database) and change the file paths to the new location.

Now here's the big warning: NEVER do such changes when you're having a conversation with someone else or are distracted by anything else.

I ended up writing an incorrect query... and then cancelling it midway, thus leaving the database in a pretty messy state. So in the end the only reliable information I had was the name of the image files. I then hacked up a little script to correct this:

#!/bin/sh

# Set your photo path here
photopath=$HOME/photos

sqlite3 photo.db "select filename from PhotoTable" |
    while read file; do
        basename=$(basename "$file")
        fullname=$(find $photopath -name "$basename")

        if [ -n "$fullname" ]; then
            echo "Updating $fullname"
            sqlite3 photo.db "update PhotoTable set filename='$fullname' where filename='$file'"
        fi
    done

So those who want to move their shotwell photo databases to another location, here's one (fairly roundabout) way to do this. Enjoy!

I concluded my previous post yesterday, on my way to FOSS.in to attend day two. Before I left, I saw Gopal saying w00t! to Pradeepto on twitter and wondered what was going on. Turns out, Pradeepto was coming to FOSS.in!

When I arrived at the venue, I found that we had (once again) captured a table in the Expo area. We spread out the DVDs and buttons and waited for people to come over. Saleem was obsessed about spreading the DVDs in a perfect line and arranging the buttons to form the 'f' of Fedora. of course, people were constantly picking the buttons from the 'f' and ruining his design. Fedora goodies were a hit on the first day and we had quite a few people picking them up on the second day too. There were a few people asking for 64-bit DVDs; we might have to note that in future conferences and hopefully till then the flash nonsense will sort itself out. to me, that seems to be the only hurdle right now for a completely trouble-free installation of a 64 bit system.

We had a few people coming over asking for help with their installations and some of us helped out till lunch time. I then met Pradeepto and Sujith along with a bunch of KDE folks for lunch. Post-lunch, we had the Fedora miniconf.

The miniconf was sort of not very well timed for me, since I wanted to attend artagnon's talk on the git object store. Rahul and Amit Shah kicked off the proceedings and Amit gave the first talk of the session. Rahul announced the workouts -- the Fedora for Kids spin by Aditya Patawari and Fedbura Bugzapping by /me. I went up to the first floor and waited for prospective contributors. Two people came in, fish_sticks and jijo. I walked them through setting up their Fedora account, bugzilla account and the sign-up for bugzapping. I also walked them through triaging bugs and the different scenarios they would face. They then started setting up virtual machines so that they could reproduce bugs in emacs. Finally, we could not get much done at all, since neither of them could get their virtual machines up and running in the time we had. Both were running Ubuntu -- it's kinda hard reproducing Fedora bugs in Ubuntu ;)

While this was going on, I was also discussing packaging with some college students from Vellore. Their college had a CMS that students had designed and they were presenting it in the FOSS.in expo. I think it was Pragyaan CMS; I can't remember the name. I volunteered to help thm get started in packaging it for Fedora. I did not want to do it myself since I personally do not have any use for a CMS. They were again trying to set things up to try out a small test rpm package using a tutorial. That was kinda hard going though since they too were using Ubuntu.

In the end the best I could do was point them to some documentation and give some tips on how they could get started and where they could get help. I think they finally joined Aditya's workout, which seemed to be a very crowded affair -- possibly the most popular workout in FOSS.in. I'm not sure how it went though, since I was busy talking to Philip and artagnon later. Oh, I had another person approach me for bugzapping when I had wrapped up and picked up my stuff -- he was a Fedora user for a change ;) We could not really sit down and work on something at that point since it was a little late, so we talked about how he could sign up and start contributing. I hope he finally does sign up for bugzapping.

The day ended with a security talk by James Morris. I made a quick exit after that since I had promised my wife that I will have dinner with her.

I will not be attending day 3, since I have a few things to wrap up before I leave for a week long holiday in north India tomorrow. It's unfortunate that I will be missing Aanjhan's closing keynote as a result. Like the last FOSS.in, I will remember this event for friends -- all the people you get to meet only once a year as well as new people you meet and exchange ideas with. It was definitely not as crowded as last year though. Too bad this will probably be the last FOSS.in, but I hope there is a similar conference somewhere similarly accessible next year.

So I made it to what apparently is the last FOSS.in. It was late as usual and so I did not miss much by reaching late. There was a bit of confusion about big grown men like myself coming in as student delegates, but that was cleared out fairly quickly when one of the organizers explained to the volunteers that miniconf speakers were to be admitted as student delegates. I am to conduct a workout today on bugzapping in Fedora today (day 2).

The beginning of FOSS.in was definitely less emphatic than before. Kishore Bhargava had replaced Atul Chitnis with the kick-off ceremony. That was quickly compensated however, by a really interesting talk by Danese on Wikipdia and its technical architecture. There was talk of an Indian mirror. That will be a very cool thing if it materializes. I'm sure one of the IITs can set aside bandwidth and hardware for them.

The wikipedia talk was followed by two completely technical talks; on page cache optimization in VMs by balbir Singh and the other by Lennart Poettering on his latest interest -- systemd. Balbir Singh's talk was quite interesting, where he ex[plained the problem of page cache duplication between guests and hosts in a VM environment. He went on to explain a few possible approaches to solving this. Lennart's talk didn't give anything new for me since I had been following the Fedora devel discussions.

Me, Shreyank and Rahul Sundaram then spent some time pulling artagnon's leg. We went back to the main hall in time to see Philip Tellis finish his talk. Rahul then followed that with his talk on failures in Fedora and what we learned from them.

We finally moved on for our Fedora dinner. Dimitris Glezos wanted to have kababs and we were told of a place where we could get good kababs. That was good, except for the fact that we didn't know where it was. After running around Forum mall looking for the place, we finally settled for Firangi Pani inside Forum mall. All that most of us ended up having was drinks and starters (kababs of course). Lots of them. That accompanied with Lennart and Dimitris discussing how they could get translations from Transifex to systemd with all of the commit log intact. We'll probably hear more about it once they actually reach a conclusion on that.

Now I'm sitting in the bus writing this post and preparing for the workout that I will be doing today. I hope I get a few bugzappers interested today.