Setting up patchwork on Dreamhost
Posted: Mar 07, 2014, 06:53We have been talking about having a patch review system in place for glibc for some time now since the volume of patches has been steadily increasing and we don’t have enough reviewers to go through them in time, leading to missed patches and general contributor unhappiness. Due to the fact that we’re a primarily email driven project, we needed something that fits into our current workflow and patchwork was the obvious choice to start with. I decided to do a setup on my domain first to get a feel of things before I made a request to set up a patchwork instance on sourceware. The instance is live now on patchwork.siddhesh.in for glibc contributors to get a feel of it.
The hard part about patchwork is that the documentation is a myth. There is an INSTALL file that sort of works, except that it doesn’t the moment you decide to use slightly different settings. Additionally, the instructions are targeted at dedicated hosting providers, so they almost completely don’t apply to someone trying to set up patchwork on Dreamhost on their shared hosting account. Of course, figuring out what to do with your patchwork installation once it is done is an adventure as well, since there seems to be no user documentation at all. Anyway, here’s how I did it:
Setting up the server and getting sources
I assume you have a shell account for the user that would be administering the subdomain, since you’d be doing a fair bit of sysadminy stuff on it. Also, it’s assumed that you’re hosted on a Linux based server; I don’t really care about how it works for Windows based hosting.
Create your Dreamhost subdomain using the control panel and make sure you have Passenger support enabled. The Passenger support creates a directory called public in your subdomain directory. Don’t bother setting up django at this stage.
Now get patchwork from their git repo:
$ git clone git://ozlabs.org/home/jk/git/patchwork
and copy the contents (i.e. whatever is inside patchwork, not the directory itself) into your subdomain directory. The contents of your subdomain directory would then be something like this:
$ ls apps docs htdocs lib public templates tools
Now remove the public directory and create a symlink from htdocs.
$ ln -s htdocs public
We could technically just copy things over from htdocs to public, but it’s easier to update this way.
Next, we need django. patchwork needs django 1.5.x, so if your server doesn’t have it, you’ll need to download the sources yourself. To check the installed django version:
$ python Python 2.6.6 (r266:84292, Dec 26 2010, 22:31:48) [GCC 4.4.5] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import django >>> django.VERSION (1, 2, 3, 'final', 0) >>>
Since we don’t have 1.5.x, we install django from sources in the lib/packages in your subdomain directory:
git clone https://github.com/django/django.git -b stable/1.5.x
Create a directory lib/python in your subdomain directory and symlink the django installation in it:
$ ln -s ../packages/django/django ./django
Configuring django and patchwork sources
The first thing to configure is the database. From your dreamhost control panel, create a mysql database and user. Have that information handy to put in your django/patchwork configuration.
The default settings for patchwork (and django) are in apps/settings.py. We need to override those by creating our own file called apps/local_settings.py. The first thing to go in our local_settings.py is our database configuration:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'db_name',
'USER': 'db_username',
'PASSWORD': 'super_secret_password',
'HOST': 'mysql.myhost.name',
'PORT': ''
},
}
The instructions in the patchwork documentation mention using DATABASE_* style variables, but they didn’t work for me, probably because I figured out that I had an older version of django after I was done with the initial configuration.
Next you need set the following variables in local_settings.py:
SECRET_KEY = 'a random generated long string'
ADMINS = (
('Super Admin', 'super@foo.com'),
)
TIME_ZONE = 'Asia/Kolkata'
LANGUAGE_CODE = 'en-us'
DEFAULT_FROM_EMAIL = 'Patchwork (foo.com) '
NOTIFICATION_FROM_EMAIL = DEFAULT_FROM_EMAIL
# If you change the ROOT_DIR setting in your local_settings.py, you'll need to
# re-define the variables that use this (MEDIA_ROOT and TEMPLATE_DIRS) too.
ROOT_DIR = '/path/to/patchwork.foo.com'
TEMPLATE_DIRS = (
# Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
# Always use forward slashes, even on Windows.
# Don't forget to use absolute paths, not relative paths.
os.path.join(ROOT_DIR, 'templates')
)
# Absolute path to the directory that holds media.
# Example: "/home/media/media.lawrence.com/"
MEDIA_ROOT = os.path.join(
ROOT_DIR, 'lib', 'python', 'django', 'contrib', 'admin', 'media')
The SECRET_KEY can be generated using the following python snippet:
import string, random
chars = string.letters + string.digits + string.punctuation
print repr("".join([random.choice(chars) for i in range(0,50)]))
Other options are obvious from their names and values, so adjust them to your taste. ROOT_DIR is set to the directory where patchwork is, i.e. your subdomain directory. TEMPLATE_DIRS and MEDIA_ROOT are derived from ROOT_DIR, so I’ve just copied those over from settings.py.
Next up, we need to get static files for admin sessions into a place where django can find and serve it. They’re present in contrib/admin/static in your django installation and we need to copy them over to htdocs/static. Once this is done, we need to tell django that it can find the static files by adding the following configuration snippet to local_settings.py:
PROJECT_ROOT = os.path.normpath(os.path.dirname(__file__))
STATIC_ROOT = os.path.join(PROJECT_ROOT, 'static')
STATIC_URL='/static/'
ADMIN_MEDIA_PREFIX='/static/admin/'
INSTALLED_APPS = (
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.admin',
'django.contrib.staticfiles',
'patchwork',
)
Finally, we don’t want to spew out debugging messages to the server and we want to be able to debug problems at the same time, so we need logging support to be enabled. Add the following snippet to local_settings.py:
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
'file': {
'level': 'DEBUG',
'class': 'logging.FileHandler',
'filename': '/path/to/patchwork.foo.com/django-debug.log',
},
},
'loggers': {
'django.request': {
'handlers': ['file'],
'level': 'DEBUG',
'propagate': True,
},
},
}
making sure that the path in the ‘filename’ is writable by django. We can now disable debugging, so add this:
DEBUG=False
Now here’s the fun part, disabling debugging changes the behaviour of django, in that it suddenly starts doing extra checks, due to which you’ll start seeing failures like below:
SuspiciousOperation: Invalid HTTP_HOST header (you may need to set ALLOWED_HOSTS)
so fix this by adding the following:
ALLOWED_HOSTS = ['your.subdomain']
These are all the settings you would normally need to get patchwork up. Now we run manage.py from the apps directory (note that the instructions in INSTALL are slightly wrong here):
$ PYTHONPATH=../lib/python ./manage.py syncdb
This should initialize the database for patchwork and django. Follow whatever prompts that come up till you come back to the shell. If the command throws errors, read up and fix your configuration. The django documentation is surprisingly good once you get used to the layout, so don’t despair if patchwork documentation doesn’t help (it won’t).
Getting patchwork up
With the database set up and the sources in place, one needs to tell apache how to serve content through django. We had set up passenger for precisely this, so we just need to add a python script in our subdomain directory to tell passenger what to do. The script should be named passenger_wsgi.py and it’s contents should be:
import sys, os basedir = os.getcwd() sys.path.insert(0, os.path.join(basedir, 'lib/python')) sys.path.append(basedir) sys.path.append(os.path.join(basedir, 'apps')) os.environ['DJANGO_SETTINGS_MODULE'] = "apps.settings" import django.core.handlers.wsgi application = django.core.handlers.wsgi.WSGIHandler()
This tells passenger where to find the app and what paths to use to find our custom django and any additional libraries and also our settings. At this point, browsing to your site should get you to a working patchwork installation. Login to the admin page (which should be http://patchwork.foo.com/admin/) with the superuser name and password you had created while running manage.py.
The first thing you’ll need to do is create a Site. Make sure you delete the example.com there since it will otherwise be used first and you won’t like it. After the site change, if you start getting a server error, then look at your django log and look for this error:
DoesNotExist: Site matching query does not exist.
This likely means that there was some data that referred to example.com and broke. You’ll have to edit the row for your subdomain in the django_sites table and change the id to 1.
Next, you add a project of your choice. It took me a while to understand what the names of the fields in the new project form meant, but maybe that was because it was getting very late. Anyway, Linkname is any name you wish to give to the project, which appears in the URL – I used glibc here. Name is a free form name for the project – I used GNU C Library. Listid is the List-ID header in the mailing list email that it should look for. Listemail is the email address for the mailing list. I just ignored the rest of the fields.
Getting emails from the project
Patchwork is useless without emails, so subscribe to the mailing list whose patches you want to monitor. You obviously need an email address for it, so create an email address first. There is usually a confirmation step involved in mailing list subscription, which should also be completed.
Next step is to forward emails from this account to the shell account where the site is running. To do this, dreamhost has a wiki document that describes how to link an email address to a shell account email. Follow that and send test emails to make sure that emails are reaching the shell user.
Now we need a mechanism to forward the mails received by the shell user to the parsing script in patchwork. This is easy to set with a forwarder file for postfix, called .forward.postfix in your home directory with the following contents:
"|/path/to/subdomain/apps/patchwork/bin/parsemail.sh"
Now you should start seeing patches in your project view on the website.
Sending emails from patchwork
This bit is fairly straightforward – patchwork needs to be able to send emails via SMTP for various events or for registration confirmation. Django needs the following information in local_settings.py to be able to send these emails, or else it uses the local SMTP, which won’t work on Dreamhost:
EMAIL_HOST = 'mail.patchwork.foo.com' EMAIL_HOST_USER = 'from@patchwork.foo.com' EMAIL_HOST_PASSWORD = 'supersecret' EMAIL_USE_TLS = True
Getting started with patch reviews
When browsing, you’ll quickly figure out that you can’t really do anything with the patches you see there, unless you have sent a patch, i.e. your email address configured in patchwork is the same as the email address of the patch sender. This is not really very useful as far as peer reviews are concerned and working through the UI doesn’t tell you anything. One would obviously gravitate towards the admin UI to see if there are any settings there in the Users or Groups or Projects or People sections, but there are none.
The option is effectively hidden in plain sight in the User Profiles section. Click through to your user (or any user you're interested in updating) and come to the page that shows the projects that you're part of (Primary Projects) and projects that you're maintainer of (Maintainer Projects). The list of projects that show up next to Maintainer Projects aren't projects you're maintainer of, unless they're selected! So select the project(s) you are maintainer of and save your changes. Now when you login as that user, you'll see options to change patch review state and even delegate to other reviewers.
Final thoughts
It was quite an exhausting experience to get patchwork working and it wasn't just because it was on dreamhost. The project is fairly poorly documented and the usual clumsiness associated with web apps didn't help things. My relative inexperience with django may have compounded the problems I had getting this up, but I would again blame that on clumsy webapp syndrome.
I have written this down with the hope that someone else looking to do this would be able to get patchwork up in a bit less time than I took, and also because we may have to do it again on sourceware when there is consensus on using it. Given my wonderful memory, I'll probably end up making all the mistakes once again when I try it out the next time.
nullcon 2014
Posted: Feb 16, 2014, 06:31I have always had a peripheral interest in application security, so when some folks at work were discussing about attending nullCon (considered India’s premiere security conference), I decided to join them too. As usual, I submitted a talk because if selected, it pays for your attendance and it makes it easier to interact with more people.
I demoed and spoke about the recent pt_chown vulnerability in Linux and glibc, slides are here. Special thanks to Martin Carpenter for finding this vulnerability and later being available for advice and help when I was preparing for this talk. It was a fairly short talk (I had 20 minutes and I finished in about 15, including the one question) and it was the first one of the night talks, so I was quickly into attendee mode for the rest of the conference. There was an interesting talk on browser extension security in the night talks track, given by a student, Abhay Rana. It gave an overview of the javascript context model of Firefox and Chrome, and then went on to talk about the issue of extension writers asking for more permissions from the framework. Not exactly my primary interest (which is system software and applications security as opposed to web-based stuff), but was interesting nevertheless.
The main conference did not have a lot of things that interested me greatly, because heuristic analysis, penetration testing and fuzzing seemed like the primary focus and also the fact that there was little presented in the Free Software space, i.e. security research on Linux and/or BSD systems and software. I was even more disappointed when I found out that Chris Evans could not make it and was told that another Google engineer would give a replacement talk. Replacement talks are usually very high level, templated and not a lot of fun as a result, but I was in for a surprise. Sumit Gwalani talked about Chrome browser and OS security and for me that was the best talk of the conference. I had a very useful chat with Sumit later about some aspects of glibc and memory allocation tweaks that Chrome does.
Other than that, there were a number of hallway talks and discussions with attendees and speakers over interesting topics like reversing programs, binary patching and malware unpacking. The Bogmallo beach was probably the most beautiful Goan beach I have been to till date, with friendly people and great food. The Bogmallo beach resort is good, but overpriced a bit.
Yerwada, birding in the heart of the city
Posted: Dec 01, 2013, 07:43There’s something exciting about finding a secret spot in the heart of a busy city. Pune is not quite Mumbai in terms of being busy, but it’s not very far. So finding a silent spot right in the middle is quite amazing.
The mention of Yerwada would bring images of dusty wide roads into my mind - the mental hospital and jail don’t figure since I have never seen them. So I was initially a bit surprised when I saw mention of Yerwada as a birding spot in Pune. Looking at the map, it seemed obvious - the Mula-Mutha river flows right next to it, on to Kalyani Nagar. Since it’s not more than a 20-minute bike ride from my place, I decided to go there on a weekday morning.
Finding the birding spot is not very difficult. Look for the crematorium in Yerwada and drive there. You cross the Yerwada garbage collection depot on the left before you see a narrow road go down towards the river on the left. There’s a board pointing at the Yerwada Smashan Bhumi. Take that road, but don’t go right down to the crematorium. Just before the crematorium there’s a narrow road on the left alongside the garbage depot. The road quickly becomes a narrow trail big enough for just a bike or for walking. There, you’ve entered the birding zone!
The area is quite dry, due to which the sun penetrates through rather early. This gives sufficient light to capture inland birds. One could find a trail and walk down to the bank of the river too. There are plenty of spaces to park yourself and let the birds come closer. It’s quite dirty though, with bones and skeleton of small animals strewn about.
I had a couple of hours to explore as much as I could before going home to go to work. Here are some pictures for your viewing pleasure.
Watching Birds
Posted: Dec 01, 2013, 07:28So this is my first non-technical post since I migrated all my stuff over to siddhesh.in. It’s not that there’s nothing interesting happening in my life - it’s just that I never felt the urge to write for a very long time. Now I do and I hope it continues for some time.
In the last few years, I developed a new hobby of photographing birds. I would collect bird feathers as a child with Milind and we would try to spot birds, without much success. In fact, without any success. I remember this one time we saw a large bird fly down from a cliff at National Park - we were on top of the cliff. We imagined that it was a rare golden bird - well, I was about 12 and Milind was 14. In hindsight, I guess the bird may have been a Black Kite with the sun right on it, but I’m going to stick to the description of a golden rare bird to preserve the memory of those days.
It’s not easy to spot birds, but all it takes though is one good birding trip and all of a sudden you see birds all over the place - out the office window, outside your home, while you’re walking, while you’re driving - everywhere. It’s not that they suddenly appeared - they were always there. It’s just that your eyes open up to this amazing new world, which is really not new at all.
So despite being able to see birds everywhere, why do we go birding to specific places? I guess the big reason is to get away from other humans and be with the birds - just the birds and you. It’s like they do all those special things just for you. I know it’s not true - that they really don’t even care about me unless I try to get too close - but it feels good to imagine that they like your company as much as you theirs.
It’s funny how this post has developed - I was going to write about a specific birding spot that doesn’t seem to be very popular and yet is in the heart of Pune. I think that will have to wait for another post…
Hotplugging monitors the old-fashioned way
Posted: Dec 17, 2012, 10:49Kushal saw me run the update-monitors script that I had written to autodetect and start monitors using xrandr and wanted it, so here it is. Thanks to this request, I now also have a repo to put some of the more interesting scripts that I use regularly. I’ll eventually slap a Free license on them. There are only two scripts now, the other one being something I use to format my glibc patches and are only of use to glibc contributors.
New look for siddhesh.in
Posted: Dec 17, 2012, 03:50I noticed recently that the comments section of my journal had a very tiny font. I quickly hacked up the stylesheet to increase the font size, but was dissatisfied in general with the look and decided to do something about it this weekend. If you’re reading this, you’ve already seen the result. I used the Titan theme from the Theme Foundry as the base for my theme code. I wrote the stylesheet completely from scratch. Here are a few things I did that were new to me.
border-radius - Rounded corners everywhere!
The curved underline in the top navigation menu, on the sidebar and the blockquotes (one of my old posts has this) are not images. They’re the result of a new css property I found out about called border-radius. I liked it and I have hence used it all over the place! The interesting thing about this property is what happens when you use percentages to denote the radius instead of the usual px. The curves at the end take a slightly different shape, which I liked. However I didn’t keep it because I didn’t know if they were standard or unintended side-effects.
Spam-free Wordpress
I realized that the plugin didn’t really work or worked badly since I was still getting spam comments to moderate. So I replaced it with a text based captcha. I hate the image captchas since I cannot decipher most of them most of the times.
ARIA and the Wordpress doctype
Wordpress uses the aria accessibility attributes for input tags (specifically, aria-required) and sets the doctype as XHTML 1.1. The result of this is that W3.org validator complains of aria-required being an invalid attribute. I looked around online and saw suggestions to either ignore the error or remove the attribute. Both are wrong. The correct resolution is to set your doctype to this:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML+ARIA 1.0//EN"
"http://www.w3.org/WAI/ARIA/schemata/xhtml-aria-1.dtd">
Ayttm and libyahoo2: So what's going on with them?
Posted: Apr 27, 2012, 02:26It’s been a while since I even looked at the code for ayttm and libyahoo2, as is evident from the fact that there have been no releases for a couple of years now. My current interests are centred around more low level OS stuff and I haven’t found the motivation to get back to these projects. I wrote my first real C code in these projects and a lot of my current knowledge will find its roots in these projects, so I’m really grateful to have gotten a chance to contribute.
So this is pretty much a (very late) confession that I am no longer writing code for these projects and they are likely dead once again. I am sure new contributors/maintainers will be welcome if they show their interest in the form of patches. Thanks Philip for giving me the chance to write some real and amazing code!
My muttrc
Posted: Nov 30, 2011, 07:43Ankur asked me to put out a blog post describing my mutt setup since I had been using mutt for a while now. Most of my muttrc is borrowed from various resources on the internet, so what I’ll do here is dump my muttrc so that anyone interested can try out snippets and options they haven’t seen before. It has a couple of unique quirks like ldap lookups and adding zimbra invites from emails. I haven’t written those either; I got them from a couple of colleagues at work.
So here goes:
set abort_nosubject=ask-yes set abort_unmodified=ask-yes set copy=yes set delete=yes #set forward_edit=yes set include=yesset timeout=10 set mail_check=5 set sort=threads set sort_aux=date set sort_re set mark_old=no unset wait_key
###########################
Account stuff
########################### set spoolfile=imaps://mail.domain.com/Inbox set folder=imaps://mail.domain.com/ set imap_user=siddhesh@domain.com set imap_pass=super_secret_password set imap_check_subscribed=yes
set record=imaps://mail.domain.com/Sent set postponed=imaps://mail.domain.com/Drafts
set header_cache=/home/siddhesh/Mail/
set smtp_url=smtp://smtp.domain.com/ set from=“siddhesh@domain.com” set realname=“Siddhesh Poyarekar”
set mime_forward=yes set mime_forward_rest=yes
##########################
Address Book
##########################
source ~/.mutt-alias set alias_file=~/.mutt-alias set query_command = “/home/siddhesh/.bin/mutt-ldap.py ‘%s’”
#############################################################
Contents of mutt-ldap.py
############################################################# ##!/usr/bin/env python # #import ldap, sys #myname = sys.argv[0] # #try: # search = sys.argv[1] #except: # print ‘Usage: %s <name> [<name> …]’ % myname # sys.exit(1) # #search = sys.argv #users = [] #l = ldap.initialize(‘ldap://ldap.domain.com:389’) # #for s in search: # if s == myname: # continue # # x = l.search_s(‘ou=Users,dc=domain,dc=com’,ldap.SCOPE_SUBTREE,‘(|(cn=’ + s + ‘)(sn=’ + s + ‘)(uid=’ + s + ‘))‘,[‘cn’,‘uid’,‘sn’,‘mail’]) # for user in x: # try: # uid = user[1][‘uid’][0] # mail = user[1][‘mail’][0] # cn = user[1][‘cn’][0] # except: # continue # users.append(’<%s>\t%s’ % (mail, cn)) # #total = len(users) #print ‘LDAP query: found %d’ % total #if total == 0: # sys.exit(1) # #for i in users: # print i ##############################################################
##############################
Run imapfilter on F5
############################## macro generic <f5> “<shell-escape> imapfilter<Enter>$” “Filter messages using imapfilter” macro index <f5> “<shell-escape> imapfilter<Enter>$” “Filter messages using imapfilter” macro pager <f5> “<shell-escape> imapfilter<Enter>$” “Filter messages using imapfilter”
macro index <f3> “T~N<Enter>;N;t$” “Mark all as Read”
##############################
The look
############################## #
Pretty Colors
color status black yellow # The status line is barely visible to my eyes otherwise :( color index brightgreen default ~N color index green default ~O color index red default ~D # deleted color index brightmagenta default ~T # tagged color index brightyellow default ~F # flagged color header green default “^Subject:” color header yellow default “^Date:” color header yellow default “^To:” color header yellow default “^Cc:” color header yellow default “^Bcc:” color header yellow default “^From:” color header red default “^X-.*:”
set index_format=“%Z %{%d-%b-%Y %X} %-15.15L (%?l?%4l&%4c?) %s” set folder_format=“%N %F %2l %-8.8u %-8.8g %8s %d %f”
auto_view text/html
set markers=no set smart_wrap=yes set wrap=80
#############################
zimbra invites
############################# macro pager Z ‘<view-attachments><search>text/calendar<enter><pipe-entry>curl -k -u siddhesh --data-binary @- https://domain.com/calendar?fmt=ics<enter>q'