Find the presentations below:

- Elastix World 2011, Mexico D.F. Nov 3-4 - ”Negociación de codecs en Asterisk”

- FSL, Vallarta, México. Nov 5 – “FreeSWITCH – Asterisk con esteroides”

- 4K Conference, Buenos Aires, Argentina, Nov 24  - “Manejo de medios en FreeSWITCH“

What is a core dump?

Sometimes problems with FreeSWITCH, Asterisk or just about any other program in Linux are hard to debug by just looking at the logs. Sometimes the process crashes and you don’t have a chance to use the CLI to look at stats. The logs may not reveal anything particularly interesting, or just some minimal information. Sometimes you need to dig deeper into the state of the program to poke around and see what is going on inside. Linux process core dumps are meant for that.

The most typical case for a core dump is when a process dies violently and unexpectedly. For example, if a programmer does something like this:

*(int *)0 = 0;

Is usually not that straight forward, it may be the that the programmer did not expect certain variable to contain a NULL (0) value. The process is killed by the Linux kernel because by default, a Linux process does not map the memory address 0 to anything. This causes a page fault in the processor which is trapped by the kernel, the kernel then sees that the given process does not have anything mapped at address 0 and then sends the SIGSEGV (Unix signal) to the process. This is called a segmentation fault.

The default signal handler for SIGSEGV dumps the memory of the process and kills the process. This memory dump contains all the memory for this process (and all the threads belonging to that process) and that is what is used to determine what went wrong with the program that attempted to reference an invalid address (0 in this case, but any invalid address can cause it).

You can learn more about it here: http://en.wikipedia.org/wiki/Segmentation_fault

How can I make sure the core dump will be saved?

Each process has a limit for how big this core can be. If the limit is exceeded no core dump will be saved. By default this limit is 0!, that means no core will be dumped by default.

Before starting the process you must use “ulimit“. The “ulimit” command sets various limits for the current process. If you execute it from the bash shell, that means the limits are applied to your bash shell process. This also means any processes that you start from bash will inherit those limits (because they are child processes from your bash shell).

ulimit -a

That shows you all the limits for your bash shell. In order to guarantee that a core will be dumped you must set the “core file size” limit to “unlimited”.

ulimit -c unlimited

If you are starting the process from an init script or something like that, the init script has to do it. Some programs are smart enough to raise their limits themselves, but is always better to make sure you have unlimited core file size for your bash shell. You may then want to add those ulimit instructions inside your $HOME/.bashrc file.

Where is the core dump saved?

Each process has a “working directory”. See http://en.wikipedia.org/wiki/Working_directory

That is where the process core dump will be saved by default. However, some system-wide settings affect where the core is dumped.

“/proc/sys/kernel/core_pattern” and “/proc/sys/kernel/core_uses_pid” are 2 files that control the base file name pattern for the core, and whether the core name will be appended with the PID (process ID).

The recommended settings are:

mkdir -p /var/core
echo "/var/core/core" > /proc/sys/kernel/core_pattern
echo 1 > /proc/sys/kernel/core_uses_pid

You can confirm what you just did with:

cat /proc/sys/kernel/core_pattern
 
cat /proc/sys/kernel/core_uses_pid

This settings will cause any process in the system that crashes, to dump the core at:

/var/core/core.<pid>

What if I just want a core dump without killing the process?

In some situations, if a process becomes unresponsive or the response times are not ideal. For example, you try to execute CLI commands in Asterisk or FreeSWITCH but there is no output, or worst, your command line prompt gets stuck. The process is still there, it may be even processing calls, but some things are taking lot of time or just don’t get done. You can use “gdb” (The GNU debugger) to dump a core of the process without killing the process and almost with no disruption of the service. I say almost because for a large process, dumping a core may take a second or two, in that time the process is freezed by the kernel, so active calls may drop some audio (if you’re debugging some real time audio system like Asterisk or FreeSWITCH).

The trick to do it fast is to first create a file with the GDB commands required to dump the core.

Latest versions of CentOS include (with the gdb RPM package) the “gcore” command to do everything for you. You only need to execute:

gcore $(pidof freeswitch)

To dump the core of the running process.

If you are in a system that does not include gcore, you can do the following:

echo -ne "generate-core-file\ndetach\nquit" > gdb-instructions.txt

The 3 instructions added to the file are:

generate-core-file
detach
quit

This will do exactly what we want. Generate the core file for the attached process, then detach from the process (to let it continue) and then quit gdb.

You then use GDB (you may need to install it with “yum install gdb”) to attach to the running process, dump the core, and get out as fast as possible.

gdb /usr/local/freeswitch/bin/freeswitch $(pidof freeswitch) -x gdb-instructions.txt

The arguments to attach to the process include the original binary that was used to start it and the PID of the running process. The -x switch tells GDB to execute the commands in the file after attaching.

The core will be named core.<pid> by default and the path is not affected by the /proc/sys/kernel settings of the system.

This core can now be used by the developer to troubleshoot the problem.

Sometimes though, the developer will be more interested in a full back trace (stack trace), because the core dump itself can’t be easily examined in any other box than the one where was generated, therefore it might be up to you to provide that stack trace, which you can do with:

gdb /usr/local/freeswitch/bin/freeswitch core.<pid>
(gdb) set logging file my_back_trace.txt
(gdb) thread apply all bt full
(gdb) quit

Then send the file my_back_trace.txt to the developer, or analyze it yourself, sometimes is easy to spot the problems even without development experience!

Anyways, the topic of my session this time was media handling in FreeSWITCH. We’ll be doing some work in in that area at Sangoma in the following months so I thought I’d kill two birds with one stone.

Presentation here:
PPT: http://www.moythreads.com/congresos/cluecon2011/media-handling-freeswitch-final.ppt

PDF: http://www.moythreads.com/congresos/cluecon2011/media-handling-freeswitch-final.pdf

http://svn.openzap.org/svn/openzap/trunk/

I also created some basic documentation about how to set it up: http://wiki.freeswitch.org/wiki/OpenZAP_OpenR2

This means that from now on FreeSWITCH will support MFC-R2 signaling with the same stack that Asterisk is using since 1.6.2

I still need to do some work on the documentation and lots of stress testing, but you can start playing with it and bugging me if it does not work

The power point presentation can be found here: FreeSWITCH modules for Asterisk Developers

The presentation will show you the key internal data structures and call flow when making a two-leg call in FreeSWITCH and Asterisk between SIP and PRI protocols. And of course, how to write modules for both telephony engines.

After the talk some people approached me to ask some particular questions, it seems the interest in creating FreeSWITCH modules and applications on top of it is increasing, it’s definitely going to be interesting what happens in the next years, the future of FreeSWITCH really looks promising.

http://lists.digium.com/pipermail/asterisk-dev/2009-March/037262.html (English)

http://www.saghul.net/blog/2009/03/18/sobre-el-modelo-de-desarrollo-de-asterisk/ (Spanish)

For those not involved in the Asterisk users and/or developers community, it all comes down to users complaining about Asterisk reliability and the new (1.6) development model that will allow new features to be introduced in dot releases which (they say) will make it worst. Telephony systems are damn critical. Users are used to see their computer crash (yeah, even Linux users, not that often but it happens). But telephony lines are very reliable circuits (basically because they’re very simple in nature and have been around a long time). Asterisk started a revolution and I don’t think anyone can deny that, it has brought a lot of flexibility to telephony systems, but that revolution comes of course with a cost.

I started playing with Asterisk at the beginning of 2004, I’ve seen segfaults, deadlocks and all kind of funny behaviours here and there. Sometimes a brand new release of Asterisk has basic functionality broken (like originate calls from the manager). That’s true as well, it can be said that Asterisk, out of the box, is not reliable (scalability is another beast I don’t want to talk about now), and naturally users don’t like that.

Having said that, I need to clarify what I mean by “reliable”. In this context, by reliable I mean, if you take 1.4.N release, deploy applications on top of it and then you blindly upgrade from 1.4.N to 1.4.(N+1) WITHOUT TESTING and put it in production expecting it to just work, well, good luck with that, and enjoy your new job flipping burgers at McDonalds. That is, Asterisk is not reliable for users who are not willing to put some effort when upgrading. Asterisk is a complex beast (and built on top of somewhat still shaky core), the Asterisk developers had been doing a great job improving the core, however, nasty hacks like masquerade are still there.

I don’t have problems with Asterisk being criticized, that’s what it will make it better in the end. I do have a problem though with all that people that are nothing but leeches of the community. Here is the kind of user I have a problem with:

1. They typically just ask questions in the mailing list, never spend time helping other users.
2. They download for free Asterisk and expect it to work out-of-the-box for their particular (profit) purpose without spending time not even doing their homework, testing their particular scenarios etc.
3. They bitch about Digium not fixing bugs in the bug-tracker, bugs that according to them are so damn critical to their business that they not even put a bounty on the voip-info wiki or in the asterisk-biz mailing list for someone to fix it.
4. They do not download beta releases or release candidates, they just wait for the “stable” release and again, expect things to magically work for them to profit.

In short, they don’t want to spend a single cent, nor spend some time out of their busy life, all they expect is their life to be free of problems and cash big bucks. One common argument for these users is that they are not developers and cannot help, that’s just bs, there is other ways to help and in the end you can always spend some of your free-asterisk-based business revenue to place bounties for the development of test beds or whatever you feel is needed to improve Asterisk.

At the end, I agree Asterisk development has to improve. But Digium does not has unlimited resources, and is already paying for a big team of development creating Asterisk and you can download it for free. Being free is not an excuse for lacking quality, but just think, there is limited resources and Digium is allocating those resources where it makes sense for its business. Type “core show warranty” in your Asterisk CLI and tell me what your warranty is.

In the other hand, those leech-users should know that not only users, but also some developers are unhappy with the development model (but different arguments than leech-users had). That’s why CallWeaver was born, and not only that. The biggest example I like to use of one of the foundations of open source (if you don’t like it, then fix it) is the FreeSwitch project.

The FreeSwitch project was born out of the discontent (to put it nicely) of one of the top developers of Asterisk: Anthony Minessale II, he complains a lot about Asterisk, yeah, but he also did a lot more for Asterisk than anyone else I know of beyond Mark himself and just a couple of the top developers of Asterisk. So, yes, from my perspective in some way he has earned the right to talk shit about Asterisk, because he knows it, he has proposed solutions and he has actually brought solutions: FreeSwitch. FreeSwitch has brought some serious competition to Asterisk (sorry, but let’s be serious, CallWeaver and Yate had never been close to match Asterisk functionality and usage, let alone GNU Bayonne).

FreeSwitch, from my perspective has solved many of the core problems that Asterisk has, it also solves the licensing issues (it’s MPL) and it’s developer friendly (since there is no business driving force yet behind it and you can get an svn branch right away). Probably FreeSwitch is not a short-term solution for those who are already hooked into Asterisk business, but in any case, at this point I don’t think there is short-term solutions for the problems Asterisk users complain about, it’s gonna cost ya baby, one way or another, and from my perspective that is the way is supposed to be.

In the end, Olle Johansson did quite good being the first in reply to that post and resumed the situation, which is not simple, but one thing is for sure, the community has to stop bitching and start doing. Well, I really don’t care if you keep bitching as long as you fucking do something else to make the situation better.

The full schedule is still being prepared so be sure to check the site later when it’s done.

I could not decide which banner to use here, so I decided to put 3 I really liked … if you use any open source telephony application, support developers by attending or at least putting one of these banners on your site.

From the ClueCon site:
ClueCon – is an annual 3-Day Telephony User and Developer Conference bringing together the entire spectrum of Telephony from TDM circuits to VoIP and everything in between. The presentations and discussions will cover several open source telephony applications such as Asterisk/Callweaver, OpenSIPS/Kamailio (formerly OpenSER), Bayonne, YATE and FreeSWITCH. Other great projects that will be discussed include OPAL and Woomera.

More funny banners at: http://files.freeswitch.org/cluecon_2009/

The past weekend I spent some time writing a module for the FreeSwitch project in order to support the G729A codec in it. This codec is patent encumbered, however, my target was not to do this in software, but just create the software interface for FreeSwitch to talk with a PCI card manufactured by Digium that does the transcoding for G729A and G723.1 .

This is the data sheet for the TC400B board. The programming interfaces to access the encoders and decoders is not documented (or at least I could not find any documentation), but it’s enough to have available the source code for the module in Asterisk that uses that very same board.

This board expose its available encoders to the DAHDI / Zaptel core driver, which in turn exposes all transcoders registered by the boards through the Linux filesystem in /dev/dahdi/transcode or /dev/zap/transcode, depending on whether you have Zaptel or DAHDI drivers.

Here I want to explain the few interfaces required to use this board.

The first thing you usually want to do is verify that there is encoders and decoders available. This is done through an ioctl to request the information about the availability of these transcoders.

        struct dahdi_transcoder_info info = {0};

        fd = open("/dev/dahdi/transcode", O_RDWR);
        if (fd < 0) {

                fprintf(stderr, "Failed to open dahdi transcode device\n");
                exit(1);
        }
        for (info.tcnum = 0; !(res = ioctl(fd, DAHDI_TC_GETINFO, &info)); info.tcnum++) {

                printf("Found transcoder '%s', numchannels = %d, dstfmts = %d, srcfmts = %d.\n", info.name,
                                info.numchannels, info.dstfmts, info.srcfmts);
        }

        close(fd);

The driver will let you know the number of encoders, decoders and the source and destiny formats. The formats masks can be found in /usr/include/dahdi/kernel.h. In future versions that may change, I discussed this with one of the DAHDI developers, since I think that this should be in dahdi/user.h and not in kernel.h given that is a user space interface.

Once you know which transcoders are available you can request an encoder or decoder, or both.

        int encoder_fd, decoder_fd;

        struct dahdi_transcoder_formats g729_encoder;
        struct dahdi_transcoder_formats g729_decoder;

        g729_encoder.srcfmt = DAHDI_FORMAT_ULAW;

        g729_encoder.dstfmt = DAHDI_FORMAT_G729A;

        g729_decoder.srcfmt = DAHDI_FORMAT_G729A;

        g729_decoder.dstfmt = DAHDI_FORMAT_ULAW;

        encoder_fd = open("/dev/dahdi/transcode", O_RDWR);

        if (encoder_fd < 0) {
                printf("Failed to open transcode device\n");
                exit(1);
        }
        if (ioctl(fd, DAHDI_TC_ALLOCATE, &g729_encoder)) {

                printf("Failed to allocate encoder\n");
                close(encoder_fd);
                exit(1);
        }

        decoder_fd = open("/dev/dahdi/transcode", O_RDWR);
        if (decoder_fd < 0) {

                printf("Failed to open transcode device\n");
                exit(1);
        }
        if (ioctl(fd, DAHDI_TC_ALLOCATE, &g729_decoder)) {

                printf("Failed to allocate decoder\n");
                close(fd);
                exit(1);
        }

Finally, once allocated, you just have to write chunks of ulaw data and read chunks of decoded g729 data and viceversa. You can choose whether the device will accept ulaw or alaw to g729 or g723 manipulating the srcfmt and dstfmt members of dahdi_transcoder_formats. Just remember that ulaw or alaw encoded data requires 8 times more bytes than g729 encoded data, therefore if you write a frame of 20ms of alaw (160 bytes for a sampling rate of 8000hz) you will read just 20 bytes of g729 encoded data, of course, the same apply when you decode a g729 frame or for g723 (for which alaw and ulaw requires 12 times more space).

The module is available here under the MPL license. Also, the module was just commited to trunk yesterday in the Freeswitch SVN repository. I want to thank to Voiceway for sponsoring the module and Neocenter for providing the hardware to test it.

The module works by registering 4 zaptel spans. The first 2 are connected, by software, in loop. Which means anything you write to channel 1 (span 1) it will be available for reading in channel 32 (span 2). It does this by soft-wiring the tx and rx buffers of each span channel and registering a timer callback with the kernel, which will call it at a 1024Hz rate (just like ztdummy) and calling zt_transmit() and zt_receive() from within this callback (since there is no E1/T1 hardware interrupts, this timer is required).

The other 2 spans are for line tapping. All channels in span 3 tap span 1 and span 4 taps span 2. If you open a channel in span 3 you will be able to eavesdrop the data over the corresponding channel in span 1 (this tapping functionality can also be accomplished by zaptel pseudo devices, that is the way ztmonitor works).

The number of loop and tapping spans is configurable passing parameters to the kernel module.

Even though this module works fine for PRI lines, until today, it did not work for MFC/R2 lines, because it was lacking CAS support. After reading for some time Zaptel drivers code it started to make sense and I added a few lines of code required to loop R2 lines.

I basically added the span handler (a function pointer in the zt_span structure) to handle the transmission of CAS bits. This handler is called each time some process from user space calls ioclt(fd, ZT_SETTXBITS), most drivers would talk to the E1/T1 hardware to ask it to change the bits position, but, what I did is to call zt_rbsbits() function in the span that I know this span is looped to, that way the other span will “see” the bits and wake up any user space process that is waiting on bits changes.

It’s kind of cool that now I can test my Asterisk MFC/R2 development in my laptop w/o needing any hardware at all

The source code for this modified ztloop kernel module can be found here: http://www.moythreads.com/ztloop.c. I will send the patch to the DruidSoftware guys to see if they include it in their next release.

http://www.sofaswitch.org/eg/pix/mfp_small.jpg

Enjoy,