Moy Blog

It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the age of bug-hunting!

Recently I fixed 2 bugs, yeah, I know I spent a lot of time fixing bugs but this 2 were quite interesting to me, not because of the bugs itself, but rather because of some stuff I learned in the process like the implementation of variadic functions and how the C++ compiler optimizes certain stuff unveiling odd bugs.

Bug 1

Let’s analyze the first one, it was a bug I had with some Unicall R2 installation in 64 bits. The problem was simple, as soon as I loaded chan_unicall.so Asterisk crashed

After running Asterisk with gdb I found the crash happened inside libc function strlen that was being called by uc_log(), the Unicall logging function. As most logging C functions, uc_log is a variadic function. uc_log does not do any complicated stuff, is mostly just a wrapper to vsnprintf and the variable arguments were just passed on to vsnprintf and there the crash was occurring, so, how can one see the arguments a variadic function receives using gdb? First, one must know how variadic functions are implemented by the compiler and platform you are working on.

Most common implementation of variadic functions in C is just to define va_list as an unsigned char* pointing to the last argument of the function and each call to va_arg() retrieves the next chunk of memory of the specified size and increment va_list to point to the start of the next argument, therefore, displaying arguments is just matter of printing the memory area after the last argument. However, AMD64 has a different implementation, va_list is an array of 1 structure with members:

.gp_offset
.fp_offset
.overflow_arg_area
.reg_save_area

gp_offset is how many bytes after reg_save_area the first argument is. To print the first variable argument that we know is an “int” we do:

(gdb) p *(int *)(((char *)arg_ptr[0].reg_save_area)+arg_ptr[0].gp_offset)

however, gp_offset will be only incremented after calling va_arg() macro, if you want to see more arguments you must increment reg_save_area by the number of bytes you know arguments take, in the case of uc_log, initial value of gp_offset is 24, probably because it receive 3 fixed arguments (8 bytes * 3). So, the first variable argument starts at .reg_save_area + 24, the second at .reg_save_area + 32 (we’re in a 64 bit machine).

So, what about .fp_offset and .overflow_arg_area?, well, it seems .reg_save_area is quite limited (possibly limited by the number of the processor registers) and you can never go beyond .gp_offset == 40, therefore that will only work for up to 6 arguments (including the fixed ones). .overflow_arg_area is used for any subsequent argument and .fp_offset is the pointer to the next argument on that memory area. Well, that’s enough, let’s get straight to the point, the crash was caused because unicall.h include the following prototype:

extern const char *uc_statet2str(int state);

That function returned the value passed to uc_log(…., uc_state2str()) … so what’s the issue? well, read once again the prototype and how uc_log used it. Is not a typo here in my blog, the prototype really is uc_statet2str, and the function call is uc_state2str, indeed there is a typo in the header file causing the compiler to default to the return value “int” and not const char* when compiling libmfcr2, for 64 bit platform there is 4 bytes of difference between char* and int causing a crash due to invalid memory read.

Bug 2

This one is easier to explain with a chunk of code, can you tell what’s wrong with it and what possible outputs will have when running it as “./test t”?

#include <stdio.h>
#include <string.h>

#define SIZE 100

int main(int argc, char *argv[])
{

        char *bufptr = NULL;
        if (argc == 2) {

                char inblock_buff[SIZE];
                bufptr = inblock_buff;
                strcpy(bufptr, "some buffer");
        }

        printf("buffer: %s\n", bufptr);
        if (argc == 2 && argv[1][0] == 't') {

                char otherbuff[SIZE];
                otherbuff[0] = 0;
        }

        printf("buffer: %s\n", bufptr);
        return 0;
}

Indeed, the output will depend on how you compile it and even probably will depend on the compiler implementation? The thing is, that if you compile this code in Linux with gcc 4.1.2 as gcc -O2 bug.c -o bug, and then run it as ./bug t

The output is:

buffer: some buffer
buffer: some buffer

But, compiling without optimizations gcc -O0 bug.c -o bug the output is:

buffer: some buffer
buffer:

When the second if() block is optimized-out the value of the block variable inblock_buff is not overwritten and therefore bufptr remains pointing to “some buffer” and the code seems to “work”, but when -O0 the second if() block is not optimized and the bug arise, bufptr will point to char 0 printing nothing. In my particular case this buffer was the input of the keyboard of a 5250 session, hence, in some cases the keyboard input was just ignored.

Heck, I hate when things stop working suddenly without apparent reason ( and who doesn’t ? ). But this time I have learned some stuff and finding a workaround has been fun. Some weeks ago my touchpad just stopped working as soon as I logged into Gnome session. The touchpad even worked at the login screen, but it stopped working as soon as I typed my password and the login process started. To add some oddity to this problem, if I typed “CTRL + ALT + F1″ to get a console, my touchpad worked again!, but when I typed “CTRL + ALT + F7″ to get back to my gnome session it stopped working again, WTF???. Fortunately the trackpoint still worked at all times, so, even when I liked more the touchpad I got used to the trackpoint quickly and even I felt it had some advantages. However, I was still puzzled because once in a while I wanted to scroll down without having to press the down arrow in my keyboard. So, as soon as I had some time, I started investigating the issue.

The first thing I did was open the xorg.conf configuration and X logs, probably some IBM Open Client ( IBM Red Hat based distro ) update broke my config or something like that. Did not find errors in the logs but I found that the mouse input device was configured as “/dev/input/mice”, as usual. After googling a couple of minutes I got tired of searching. So I decided to do more low level investigation instead of just blindly tweaking X configuration file and end user stuff like that. I did a small test program a bit like this:

fd = open("/dev/input/mice", O_RDONLY);

FD_ZERO(&rfds);
FD_SET(fd);
while ( 1 ) {
    res = select(fd + 1, &rfds, NULL, NULL, NULL);

    if ( FD_ISSET(fd, &rfds) ) {
        res = read(fd, read_buffer, sizeof(read_buffer));

        printf("got some data\\n");
    }
}

So, this just served the purpose of confirming no event was generated when I moved my finger in the touchpad, and events were generated with the trackpoint. So, was this a driver problem? I did a quick ‘ls -la’ to find out the major and minor of “/dev/input/mice”, however I did not find a way to map the major/minor to the kernel driver that takes care of the file operations ( read(), write(), blah() ). I asked several people and nobody knew how to find out the driver/kernel module that takes care of creating a /dev/ entry given the major and minor. Obviously the kernel can, but I did not find any user space tool for that. In order to try to find the driver I downloaded the kernel sources that closely matched my installed kernel ( 2.6.18 ). After some searching in the kernel tree I found “drivers/input/” directory. I was sure somewhere there it was the answer to my problem. Since every time I entered the graphics mode touchpad stopped working, I thought X was doing some nasty stuff, may be some ioctl() to disable the touchpad if trackpoint was found as well? I found some posts in forums saying some people want to disable touchpad because they accidentally touch it when writing and that disturbs them, so I started considering the possibility of X disabling intentionally my touchpad. I did a “grep” searching for ioctl definitions in the “drivers/input” directory and found several ones, particularly evdev.c was interesting, since events were not being received in user space after all. I did not understand shit of what the ioctl’s were doing there, so I searched for documentation in Documentation/input/input.txt , and voilá!, things started to make sense. I did some googling on the ioctl’s defined in evdev.c because input.txt did not say a word about them. I found this interesting articles:

http://www.linuxjournal.com/article/6396
http://www.linuxjournal.com/article/6429

Interesting, but, still did not say anything about why possibly events could not be received in user space. So, back to the code in evdev.c I paid more attention at the ioctl’s and started experimenting with some of them like EVIOCGVERSION and EVIOCGNAME. I found some documentation in include/linux/input.h . I found a comment about EVIOCGRAB saying “Grab/Release device”. Hum, that sounded interesting, what implications has grabbing a device? The articles did not mention that ioctl, so, Let’s try it!

/* fd is /dev/input/event0 ( my keyboard )*/
ioret = ioctl(fd, EVIOCGVERSION, &version);

ioret = ioctl(fd, EVIOCGNAME(sizeof(device_name)), device_name);

ioret = ioctl(fd, EVIOCGRAB, &grab);
if ( -1 == ioret ) {
  perror("ioctl()");
}

printf("ver: %d, ret = %d\n", version, ioret);
printf("device name is: %s\n", device_name);

I paid the price of my stupidity, testing with my keyboard was not a smart thing, really. My keyboard stopped working and I had to brute force reboot. So, it seems that EVIOCGRAB ioctl is something like getting exclusive rights over the input events of the specified device, thus, X was not able to get my keyboard events, the only process that was able to receive them was my program, and it was not doing anything useful with them except printing “got some data” message.

I tried next with /dev/input/event1 ( my touchpad ) and got a “Device or resource busy” and tried with /dev/input/event2 ( my trackpoint ) and it succeeded. I connected an USB mouse and it created /dev/input/event3. Tried with that one and it also worked. So, it was clear to me that someone ( X of course ) was grabbing my touchpad for some reason and ignoring events, error or feature?, I still don’t know. So, I did a “touchpad-takeover” program that will not allow X to take control of my touchpad and then, when already logged in, my program will release the device. That way events will start flowing again to any application waiting on /dev/input/event1 and /dev/input/mice ( this device is a mix of all the mouse devices events ), may be that way X will learn to play nice with me

Here is how the touchpad-takeover look like:

http://www.moythreads.com/htmlcode/touchpad-takeover.html

Now, everytime my laptop boots, I launch that program, with /dev/input/event1 as argument, in the background. X report an error saying “Synaptics can’t grab event device, errno=16″, aha!, how does that feels X??! hu? … As soon as I start my gnome session I send SIGINT to my program (kill -2) so it will release /dev/input/event1 and I am able to use my touchpad

Is not that odd? If X is able to grab my touchpad device, touchpad does not work. If I grab the device first so X cannot grab it, and then, once logged in, I release the device, I can work with my touchpad. Bug or feature? I think some X configuration might alter that behaviour, but I have not found it so far. May be with some more time in my hands I will dig into the X server code to find out what is going on. Even when I am able to use my touchpad, the scroll down feature is missing in my touchpad . Before this strange stuff happened, I could move my finger on the right edge of the touchpad and it had the effect of scrolling down my screen/documents, now it does not.

Long post … time to go back to Guadalajara. I arrived at Puebla yesterday to give an Asterisk talk at ENLi, I was planning to get back yesterday at night, however, Tato did not show up to give his Asterisk presentation, fortunately I talked with Sandino this morning and he said Tato is OK, so I gave the presentation this morning in his place. Gotta go, I’m quite tired …