Hotplugging monitors the old-fashioned way
Kushal saw me run the update-monitors script that I had written to autodetect and start monitors using xrandr and wanted it, so here it is. Thanks to this request, I now also have a repo to put some of the more interesting scripts that I use regularly. I’ll eventually slap a Free license on them. There are only two scripts now, the other one being something I use to format my glibc patches and are only of use to glibc contributors.
New look for siddhesh.in
I noticed recently that the comments section of my journal had a very tiny font. I quickly hacked up the stylesheet to increase the font size, but was dissatisfied in general with the look and decided to do something about it this weekend. If you’re reading this, you’ve already seen the result. I used the Titan theme from the Theme Foundry as the base for my theme code. I wrote the stylesheet completely from scratch. Here are a few things I did that were new to me.
border-radius - Rounded corners everywhere!
The curved underline in the top navigation menu, on the sidebar and the blockquotes (one of my old posts has this) are not images. They’re the result of a new css property I found out about called border-radius. I liked it and I have hence used it all over the place! The interesting thing about this property is what happens when you use percentages to denote the radius instead of the usual px. The curves at the end take a slightly different shape, which I liked. However I didn’t keep it because I didn’t know if they were standard or unintended side-effects.
Spam-free Wordpress
I realized that the plugin didn’t really work or worked badly since I was still getting spam comments to moderate. So I replaced it with a text based captcha. I hate the image captchas since I cannot decipher most of them most of the times.
ARIA and the Wordpress doctype
Wordpress uses the aria accessibility attributes for input tags (specifically, aria-required) and sets the doctype as XHTML 1.1. The result of this is that W3.org validator complains of aria-required being an invalid attribute. I looked around online and saw suggestions to either ignore the error or remove the attribute. Both are wrong. The correct resolution is to set your doctype to this:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML+ARIA 1.0//EN"
"http://www.w3.org/WAI/ARIA/schemata/xhtml-aria-1.dtd">
Observations on glibc libm
One of my recent (ongoing) projects has been to figure out a way to make performance characteristics of some transcendental functions (exp, log, sin, cos, etc.) in libm a bit more uniform. All of the functions have been implemented with two computation phases. The first phase uses a table of precomputed values and a polynomial approximation of the function. If the result is not accurate enough, the second phase is triggered, which descends into multiprecision computation. These multiprecision computations are dog-slow and that’s what I’m trying to avoid.
As expected, this has turned out to be more of a mathematics project than programming. I haven’t done any serious mathematics (beyond filing tax returns) in recent years, so it has been quite an amazing ride so far with the real fun still to come. I’ve started a page on the glibc wiki to document my observations as I go along. Currently it has notes on pow and exp functions since that’s where I started. Read up and enjoy!
Making mutt send attached patches inline by default
So I moved back to mutt after I got my new laptop because I wanted to rip out as much of the desktop cruft as I possibly could. There was one feature from claws that I missed sorely though, which is the ability to send plain text attachments inline by default. If you add an attachment using the ‘a’ key in the compose window, it is set to be sent as an attachment by default. It is possible to do a ^D to change the disposition to inline, but I would keep forgetting to do that now and then, leading to pain for patch reviewers.
So after forgetting to press ^D before sending for the nth time, I have now ‘configured’ mutt to send text attachments inline with the following one-liner patch:
diff -pruN mutt-1.5.21/sendlib.c mutt-1.5.21.patched/sendlib.c
--- mutt-1.5.21/sendlib.c 2012-12-07 14:55:30.066959607 +0530
+++ mutt-1.5.21.patched/sendlib.c 2012-12-07 15:11:53.443348151 +0530
@@ -1392,6 +1392,9 @@ BODY *mutt_make_file_attach (const char
*/
att->type = TYPETEXT;
att->subtype = safe_strdup ("plain");
+
+ /* HACK! make text attachments inline by default. */
+ att->disposition = 0;
}
else
{
As the comment clarifies, it is a hack and is not suitable for mutt upstream. The right way to do this would be to add a muttrc clause that toggles this behaviour. Doesn’t matter to me though - I have what I need.
Malloc per-thread arenas in glibc
The malloc subsystem in glibc has had the feature of per-thread arenas for quite some time now and based on my experience, it seems to be the source of a lot of confusion. This is especially for enterprise users who move from RHEL-5 to RHEL-6 to see their apps taking up a lot more ‘memory’ and you’ll see a fair amount of content in this regard in the Red Hat Customer Portal if you’re a RHEL customer. This post is an attempt to get more perspective on this from the internal design perspective. I have not written any of the code in this implementation (barring a tiny improvement), so all of the talk about ‘original intention’ of any design is speculation on my behalf.
Background
The glibc malloc function allocates blocks of address space to callers by requesting memory from the kernel. I have written about the two syscalls glibc uses to do this, so I won’t repeat that here beyond mentioning that they two ‘places’ from where the address space is obtained are ‘arenas’ and ‘anonymous memory maps’ (referred to as anon maps in the rest of the post). The concept of interest here is the arena, which is nothing but a contiguous block of memory obtained from the kernel. The difference from the anon maps is that one anon map fulfills only one malloc request while an arena is a scratchpad that glibc maintains to return smaller blocks to the requestor. As you may have guessed, the data area (or ‘heap’) created by extending the process break (using the brk syscall) is also an arena - it is referred to as the main arena. The ‘heap’ keyword has a different meaning in relation to the glibc malloc implementation as we’ll find out later.
The Arenas
In addition to the main arena, glibc malloc allocates additional arenas. The reason for creation of arenas always seems to have been to improve performance of multithreaded processes. A malloc call needs to lock an arena to get a block from it and contention for this lock among threads is quite a performance hit. So the multiple arenas implementation did the following to reduce this contention:
- Firstly, the malloc call always tries to always stick to the arena it accessed the last time
- If the earlier arena is not available immediately (as tested by a pthread_mutex_trylock), try the next arena
- If we don't have uncontended access on any of the current arenas, then create a new arena and use it.
We obviously start with just the main arena. The main arena is extended using the brk() syscall and the other arenas are extended by mapping new ‘heaps’ (using mmap) and linking them. So an arena can generally be seen as a linked list of heaps.
An Arena for a Thread
The arena implementation was faster than the earlier model of using just a single arena with an on-demand model for reduction of contention in the general case. The process of detecting contention however was sufficiently slow and hence a better idea was needed. This is where the idea of having a thread stick to one arena comes in.
With the arena per-thread model, if malloc is called for the first time within a thread, a new arena is created without looking at whether the earlier locks would be contended or not. As a result, for a sane number of threads*, one can expect zero contention among threads when locking arenas since they’re all working on their own arenas.
There is a limit to the number of arenas that are created in this manner and that limit is determined based on the number of cores the system has. 32-bit systems get twice the number of cores and 64-bit systems get 8 times the number of cores. This can also be controlled using the MALLOC_ARENA_MAX environment variable.
- A sane number of threads is usually not more than twice the number of cores. Anything more and you’ve have a different set of performance problems to deal with.
The Problem (or not)
The whole issue around the concept of arenas is the amount of ‘memory’ it tends to waste. The common complaint is that the virtual memory footprint of programs increase due to use of arenas and definitely more so due to using a different arena for each thread. The complaint is mostly bogus because of a couple of very important features that have been there for years - loads and loads of address space and demand paging.
The linux virtual memory model can allow a process to access most of its virtual memory address space (provided it is requested and mapped in). A 64-bit address space is massive and is hence not a resource that is likely to run out for any reasonable process. Add to it the fact that the kernel has mechanisms to use physical memory only for pages that are needed by the process at that time and you can rest assured that even if you map in terabytes of memory in your process, only the pages that are used will ever be accounted against you.
Both arena models rely on these facts. A mapped in arena is explicitly requested without any permissions (PROT_NONE) so that the kernel knows that it does not need any physical memory to back it. Block requests are then fulfilled by giving read+write permissions in parts. Freed blocks near the end of heaps in an arena are given back to the system either by using madvise(MADV_DONTNEED) or by explicitly unmapping.
So in the light of all of this, as an administrator or programmer, one needs to shift focus from the virtual memory column in your top output to the RSS column, since that is where the real resource usage is. Address space usage is not the same thing as memory usage. It is of course a problem if RSS is also high and in such cases one should look at the possibility of of a memory leak within to process and if that is not the problem, then fragmentation within the arenas. There are tuning options available to reduce fragmentation. Setting resource limits on address space usage is passe and it is time that better alternatives such as cgroups are given serious consideration.
Higher order functions in C?
The Structure and Interpretation of Computer Programs course has a class on higher order functions, which is the ability of a function to accept a function and/or returns another function that uses the input function. The C programming language has very limit capability to do this and it is limited to being able to accept function pointers or return function pointers. I wanted to explore this limitation further to figure out exactly how far I can stretch this, so I wrote a small C program that tries to emulate this concept. This is just random tinkering, so if the reader is looking for a specific lesson, then let me clarify that there is none.
I wrote a program to get square roots using the Newton-Raphson method -- the same one that was used in the course with an attempt to keep the structure as close as possible to the scheme example in the class video. The Lisp-based listing for the course can be found on the internet and here is what I wrote:
#include <stdio.h>
#include <stdlib.h>
typedef double (*generic_func_t) (double);
#define dx 0.000001
#define absolute(n) ((n) < 0 ? -(n) : (n))
generic_func_t deriv(generic_func_t f)
{
double deriv_func(double x)
{
return (f(x + dx) - f(x)) / dx;
}
return deriv_func;
}
double fixed_point(generic_func_t f, double x)
{
while (absolute(f(x) - x) > dx)
x = f(x);
return f(x);
}
double newton(generic_func_t f, double guess)
{
double newton_func(double y)
{
generic_func_t df = deriv(f);
return (y - f(y) / df (y));
}
return fixed_point(newton_func, guess);
}
double sqrt(double x)
{
double sqrt_func(double y)
{
return (x - y*y);
}
return newton(sqrt_func, 1);
}
int main(int argc, char **argv)
{
if (argc < 2) {
printf(“Usage: %s <number>\n”, argv[0]);
exit (1);
}
printf (“%lf\n”, sqrt(strtod(argv[1], NULL)));
return 0;
}
The first thing that is evident in the above is that the syntax is non-standard. I have declared some functions within another function and that is not allowed by the C standard. GCC however implements this concept of nested functions as an extension, so this compiled cleanly for me. Running this however is a completely different thing.
Executing the above program for any argument results in segfaults and analysis of the program under gdb shows that the generated code is quite ridiculous. This is expected however, since we have pushed the gcc nested function support too far. The nested functions within gcc are OK as long as they remain within the following limits:
- They are only used within the function that nests them, either directly or indirectly via some function calls. Essentially, control from the nested function should return before it’s nesting function returns.
- If they’re used outside the nesting function and called after the nesting function returns, then they don’t use any local variables or arguments of the nesting function
So I modified the deriv function and came up with the listing below:
#include <stdio.h>
#include <stdlib.h>
typedef double (*generic_func_t) (double);
#define dx 0.000001
#define absolute(n) ((n) < 0 ? -(n) : (n))
double deriv(double x, generic_func_t f)
{
return (f(x + dx) - f(x)) / dx;
}
double fixed_point(generic_func_t f, double x)
{
while (absolute(f(x) - x) > dx)
x = f(x);
return f(x);
}
double newton(generic_func_t f, double guess)
{
double newton_func(double y)
{
double dy = deriv(y, f);
return (y - f(y) / dy);
}
return fixed_point(newton_func, guess);
}
double sqrt(double x)
{
double sqrt_func(double y)
{
return (x - y*y);
}
return newton(sqrt_func, 1);
}
int main(int argc, char **argv)
{
if (argc < 2) {
printf(“Usage: %s <number>\n”, argv[0]);
exit (1);
}
printf (“%lf\n”, sqrt(strtod(argv[1], NULL)));
return 0;
}
The only change above is that instead of accepting a function and returning another in deriv, I now accept a function and return the evaluation of the derived function for the specified value. This is highlighted in bold above. This program now builds with gcc and also runs correctly. It should similarly be possible to eliminate the other nested functions to get a much more compact and standards-compliant program.
Ayttm and libyahoo2: So what's going on with them?
It’s been a while since I even looked at the code for ayttm and libyahoo2, as is evident from the fact that there have been no releases for a couple of years now. My current interests are centred around more low level OS stuff and I haven’t found the motivation to get back to these projects. I wrote my first real C code in these projects and a lot of my current knowledge will find its roots in these projects, so I’m really grateful to have gotten a chance to contribute.
So this is pretty much a (very late) confession that I am no longer writing code for these projects and they are likely dead once again. I am sure new contributors/maintainers will be welcome if they show their interest in the form of patches. Thanks Philip for giving me the chance to write some real and amazing code!
My muttrc
Ankur asked me to put out a blog post describing my mutt setup since I had been using mutt for a while now. Most of my muttrc is borrowed from various resources on the internet, so what I’ll do here is dump my muttrc so that anyone interested can try out snippets and options they haven’t seen before. It has a couple of unique quirks like ldap lookups and adding zimbra invites from emails. I haven’t written those either; I got them from a couple of colleagues at work.
So here goes:
set abort_nosubject=ask-yes set abort_unmodified=ask-yes set copy=yes set delete=yes #set forward_edit=yes set include=yesset timeout=10 set mail_check=5 set sort=threads set sort_aux=date set sort_re set mark_old=no unset wait_key
###########################
Account stuff
########################### set spoolfile=imaps://mail.domain.com/Inbox set folder=imaps://mail.domain.com/ set imap_user=siddhesh@domain.com set imap_pass=super_secret_password set imap_check_subscribed=yes
set record=imaps://mail.domain.com/Sent set postponed=imaps://mail.domain.com/Drafts
set header_cache=/home/siddhesh/Mail/
set smtp_url=smtp://smtp.domain.com/ set from=“siddhesh@domain.com” set realname=“Siddhesh Poyarekar”
set mime_forward=yes set mime_forward_rest=yes
##########################
Address Book
##########################
source ~/.mutt-alias set alias_file=~/.mutt-alias set query_command = “/home/siddhesh/.bin/mutt-ldap.py ‘%s’”
#############################################################
Contents of mutt-ldap.py
############################################################# ##!/usr/bin/env python # #import ldap, sys #myname = sys.argv[0] # #try: # search = sys.argv[1] #except: # print ‘Usage: %s <name> [<name> …]’ % myname # sys.exit(1) # #search = sys.argv #users = [] #l = ldap.initialize(‘ldap://ldap.domain.com:389’) # #for s in search: # if s == myname: # continue # # x = l.search_s(‘ou=Users,dc=domain,dc=com’,ldap.SCOPE_SUBTREE,‘(|(cn=’ + s + ‘)(sn=’ + s + ‘)(uid=’ + s + ‘))‘,[‘cn’,‘uid’,‘sn’,‘mail’]) # for user in x: # try: # uid = user[1][‘uid’][0] # mail = user[1][‘mail’][0] # cn = user[1][‘cn’][0] # except: # continue # users.append(’<%s>\t%s’ % (mail, cn)) # #total = len(users) #print ‘LDAP query: found %d’ % total #if total == 0: # sys.exit(1) # #for i in users: # print i ##############################################################
##############################
Run imapfilter on F5
############################## macro generic <f5> “<shell-escape> imapfilter<Enter>$” “Filter messages using imapfilter” macro index <f5> “<shell-escape> imapfilter<Enter>$” “Filter messages using imapfilter” macro pager <f5> “<shell-escape> imapfilter<Enter>$” “Filter messages using imapfilter”
macro index <f3> “T~N<Enter>;N;t$” “Mark all as Read”
##############################
The look
############################## #
Pretty Colors
color status black yellow # The status line is barely visible to my eyes otherwise :( color index brightgreen default ~N color index green default ~O color index red default ~D # deleted color index brightmagenta default ~T # tagged color index brightyellow default ~F # flagged color header green default “^Subject:” color header yellow default “^Date:” color header yellow default “^To:” color header yellow default “^Cc:” color header yellow default “^Bcc:” color header yellow default “^From:” color header red default “^X-.*:”
set index_format=“%Z %{%d-%b-%Y %X} %-15.15L (%?l?%4l&%4c?) %s” set folder_format=“%N %F %2l %-8.8u %-8.8g %8s %d %f”
auto_view text/html
set markers=no set smart_wrap=yes set wrap=80
#############################
zimbra invites
############################# macro pager Z ‘<view-attachments><search>text/calendar<enter><pipe-entry>curl -k -u siddhesh --data-binary @- https://domain.com/calendar?fmt=ics<enter>q'
FUDCon Notes on my Security Exploits Session
I was asked by a couple of people for notes on the security exploits session that I conducted at FUDCon. I had posted the code samples on the talk page, but that is probably a little terse, so here’s a little write-up to support the code samples. To repeat what I had said multiple times earlier; I am not a security researcher, not even a security freak. This topic was suggested to me by Amit Shah, and I developed an interest in it due to my original interest, which is operating systems tools. The preparation of this talk got me interested in security, but only through the perspective of operating systems tools and programs, so I am still relatively indifferent to the subject of web-based security.
I started preparing for the session fairly late; i.e. 2 days before FUDCon. I am a little familiar with glibc code and with the way the compiler, linker, loader, etc. work on Linux, so that helped me understand a lot of the concepts behind exploits fairly easily. But concepts != working code and getting exploit code to work was the real challenge, especially when I had just about 3 evenings+nights for it. I had started with an idea of showing stack smashing and privilege escalation examples, but given the time constraint, audience level (college students) and also the constraint of my knowledge, I decided to restrict it to stack based attacks. All of the examples have a buffer which is being written to without checking for bounds of that buffer, typically with an strcpy.
The shellcode sample:
The shellcode sample as well as the final vulnerability demo (smash.c and vulnerable.c) were derived from the article Stack Smashing for Fun and Profit. That is a great article that explains in much more detail than I went into in my session, as to how the shellcode exploit can be developed.The core idea of this is:
- Obtain the binary equivalent of execve (name[0], name, NULL ) where name = {“/bin/sh”, NULL}
- Append additional data to the binary data and pass it to the buffer copy routine (strcpy) in such a way that, it writes the location of the first instruction in the above binary equivalent into the location of the return address
I spent a lot of time trying to figure out where this was set and finally found the -z option of the linker. So to write out a binary that sets up an executable stack, I had to call the linker with -z execstack. This finally enabled me to get the shellcode working.
The actual exploit
Once the shellcode was done, I could get the final vulnerability working and I immediately set about trying it. The exploit is based on the above shellcode example, except for one difference. The shellcode example is just that, an example. It is not an actual exploit; it is just a roundabout way to get a shell. The exploit I was about to do was a real crack. The idea now is to accept a string as input, which is then fed in to make a regular and buggy program provide you with a shell.
To imagine how this would work, think of the program that gives you a login prompt. In the context of this exploit, you should be able to input a crafted string into this login prompt and have it give you a shell without actually knowing the password! This is what the actual exploit ought to look like.
Again, writing the exploit was the easy part; getting it to run was quite another thing altogether. The exploit works as follows:
- Pages of memory are generally mapped at the same addresses for processes, so the top of stack for a process can be a good starting estimate for the top of stack for another process. From that point on, you need a finite number of guesses to get to the point in stack where the instructions have been injected
- The smash program records its own top of stack and prepares a string such that it the return address will be overwritten by the top of stack address. The string obviously begins with shell code. To improve the chances of hitting valid instructions in the code, the buffer is written over by a lot of single-byte NOP instructions just before the shellcode instructions
- The smash program executes a shell with an environment variable exported ($EGG). This environment variable can now be used to execute the vulnerable program within that shell. This could have been done differently too, like storing the output to a file and executing the vulnerable program with input from that file. So the way input is given doesn’t really matter here
echo 0 > /proc/sys/kernel/randomize_va_space
Even with this, my example would not execute by itself and would end with a SIGILL. I suspect this has something to do with the fact that my systm is x86_64 while the samples are all 32-bit. Our overflow string does not seem to agree with the instruction set on my system. In any case, it seems to run just fine inside a debugger. So if you run smash to get a shell, run gdb vulnerable and then run it with $EGG, you get the shell! At least I had a demo now.
Jump to libc
While I was trying out the shellcode example, I continued thinking about various other ways in which I could get a shell. One of the methods I thought of was to overwrite the return address with the address of the system() glibc call and pass the string via stack. I later found out via Huzaifa that this is in fact a documented way to exploit unchecked buffers on stack. Huzaifa also said that I may be missing out on something there and gave me some tips on finding the right resources for this. I still could not get this working, but at least I found out why the exploit did not work.
This exploit seemed attractive to me because it does not require an executable stack. The instructions I want to execute are already there in memory. So I only have to overwrite the return address and continue writing “/bin/sh” on the stack. I first tried with x86_64 in this case, because I was going by my own idea at that time. I soon figured out that the system() function on x86_64 did not take function arguments from stack. It took the argument from the %rdi register. My devious plan had been foiled! I did not give up however and looked at the system() implementation on i686. This retained the old behaviour of popping arguments from the stack, so my exploit was still possible here.
Not. My code was correct, but every time I run the program, the address of system() had just 3 bytes set. So it would always look something like: 0x00aabbcc. This was bad news because this meant that I cannot continue writing the shell string into the stack (strcpy stops copying when it encounters a 0x0). This means that I can call system() (like I was able to on x86_64 too), but I cannot pass it an argument. After trying enough number of times, I concluded that this must be a security feature. This was backed up by the tip Huzaifa had shared with me to (ironically) get the exploit to work. This was perhaps the first documentation of a return to libc exploit by Solar Designer. In his explanation, Solar designer mentioned that a way to fix this would be to ensure a 0x00 in the address, which is precisely what is happening here.
This obviously does not deny the fact that such an exploit can be carried out if you want to call functions that do not have arguments. Think for example, of a function that executes a shell ;)
Conclusion
The last modify example was a simple little trick I wrote on the last day to demonstrate how buffer overflows work and how they can be used to alter program flow. That again is not an exploit at all. At most, it can be called… a buffer overflow ;)
I had even more fun preparing for this session than actually presenting it because it taught me a lot more than I could ever have done by just reading literature. I hope those who attended my session at FUDCon enjoyed the session too.
FUDCon Pune 2011 Day 3: Hack and eat
The last day of FUDCon. I had not slept much the last couple of nights, so I slept in a little late. Due to this I reached the venue late too and found that a lot of the speakers had reached late and that seemed to have got the volunteers (understandably) a little annoyed. All of the action was to happen just in the auditorium and the seminar halls this time and I stuck mostly to the auditorium for most of the day. I had planned to work on the libgqpid library, but I had not decided what it is that I was going to do. I started off by writing my day 1 post. That was quickly followed by lunch, where we hogged on pizzas.
I told Kushal by that time that I will work on autotoolizing libgqpid since he was busy working on his book. I started working on that after lunch, quickly finding out that there are a few things that wouldn’t work very easily there.
libgqpid is a wrapper around the apache c++ client library and the library check for the qpid client library would have to be in c++. autotools does not have any macros for c++ library checks, so I had to write a check that looked like this:
AC_LANG_PUSH([C++])After this one little hurdle it was pretty much smooth sailing and the result was a pull request to Kushal for the patch. By early evening, everyone was done and ready to go. The volunteers, especially the girls were very excited about getting their pictures clicked and were calling all of the major organizers (Rahul, Amit, Satya, etc.) to get their pictures clicked with them. It was pretty entertaining to watch.AC_MSG_CHECKING([if qpid c++ client libraries are present]) AC_LINK_IFELSE( [AC_LANG_PROGRAM([#include <qpid/messaging/Connection.h>], [qpid::messaging::Connection con])], [QPID_LIBS=“$LIBS”], [AC_MSG_ERROR([qpid c++ client development libraries not found])])
AC_LANG_POP
This was followed by a cake, which was mostly eaten and the rest of it smeared on Rahul and Jared’s faces. To end the event, we had a feedback session with the volunteers and they gave us a few good tips on how we could have done some things better.
By the time we were back at Magarpatta City, everyone looked tired. Rahul had organized a parting dinner at the Cocoon for all the speakers and organizers, which was a lot of fun. I had a great time chatting with Heherson, Izhar, Arun S A G, Srishti Sethi, Anurag and Eugene Teo. After dinner it was time to head back home.
We did a lot of things right in this event and kudos to Rahul, Amit Shah and all of the core team for getting together a really great event. I hope that at least some of all of those college students make the transition from being users to being contributors, especially contributing code to Fedora and upstream projects.