Core

The FreeSWITCH core is contained in the src/*.c files, everything that you see there is considered “core”. The FreeSWITCH core provides common services and protocols that are re-used among many different call control or media protocols. For example, the core provides several useful API abstractions:

• OS abstraction based on top of the APR project (Apache Portable Runtime). There is a thin layer of abstraction (src/switch_apr.c) that prevents introducing a direct dependency between the rest of the system and the APR library. This means the rest of the system *DOES NOT* use APR directly, they use the API exposed in src/switch_apr.c for operating system abstraction primitives to create sockets, threads, manage memory etc.
• RTP stack (src/switch_rtp.c) and an API to create rtp streams, read/write to them etc. This provides access to RTP as RTP is used by many other signaling protocols such as SIP, H.323, Megaco etc, so it makes sense to have it in the core.
• Call switching (src/switch_ivr.c, src/switch_ivr_originate.c) API routines to create a new call (session) towards a destination without explicitly knowing the low-level protocol details. A module can therefore use switch_ivr_originate() API to place a call in SIP, H.323, SS7, PRI etc, using the same consistent API provided by the core.
• Media playing API (src/switch_ivr_play_say.c, src/switch_ivr_say.c) that allows you to play a file, insert tones etc etc on a given session (regardless of the signaling protocol)
• Session and channel management (src/switch_core_session.c, src/switch_channel.c) that allows to perform call control operations (answer a channel, hangup a channel) and low-level I/O operations (write a frame of media, read a frame of media) on sessions (regardless of the underlying signaling and media protocols).

(Image taken from the FreeSWITCH wiki long ago, can’t recall the URL anymore)

Sessions and Channels

The core abstraction for a call leg is the switch_core_session_t opaque structure. This contains all the necessary information for a given call leg. The session contains a channel. The channel is simply a lower-level representation of the call leg that contains information about the underlying protocols and data structures used to interface with that call leg (ie, contains function pointers to functions that allow the call to be answered, hangup, provide media etc) and the state of the session.

Sessions in FreeSWITCH live in their own thread. A typical call involves 2 sessions (the inbound session and then an outbound session, both of them “bridged”). Each session lives in its own thread, looping through a finite state machine (the session channel states go from CS_INIT, last state is CS_DESTROY, there are other states such as CS_ROUTING, CS_EXCHANGE_MEDIA etc etc. These states determine what the session is doing, each of the state handlers to perform the operation (ie, hangup) is executed in the session thread. It is possible however that other threads want to send a message to indicate something, or read/write information about a session in it. In order to do this in a safe way, there are several APIs that can be used to get a locked pointer to a session that allows you to peek into the session data safely without fearing the session thread will terminate and destroy the session while you’re looking at it or doing something with it from another thread, the main core function for this is:

switch_core_session_t *switch_core_session_locate(const char *uuid);

Every session has a UUID (universal identifier) that is unique across computers and across time (at least for practical purposes). You can use that UUID to ask the core to find a session in the system with that UUID. The core will attempt to find the session (in an internal hash table) and if it’s found, it will lock it and return it to you. You can then use the session to retrieve data, set data or perform operations on it or in its underlying switch_channel_t (which is obtained through switch_core_session_get_channel()) without fear of the session thread to destroy the session (for example if someone sets the state to CS_DESTROY). You must remember to unlock the session when done, and you should do that fast enough (not more than a few milliseconds) otherwise you’re blocking other threads from accessing that session if they need to do something with it. Unlock the session with:

switch_core_session_rwunlock();

This function unlocks the session and this releases it so other threads can use it (including destroy it).

Be aware that there are other core functions that return a session locked, such as switch_core_session_get_partner(). It is your responsibility to verify if the function you are using returns a locked session and then use switch_core_session_rwunlock() to unlock it when done.
Modules

FreeSWITCH is highly modular. The FreeSWITCH engine/core source code is contained in the src/ directory. The plugins or modules source code are contained in the src/module/ directory (the only exception is mod_freetdm, which is contained at libs/freetdm/mod_freetdm, there are some legacy reasons why this was done this way, but there is talks in the FreeSWITCH open source community about moving it to the right place in src/mod/endpoints/ folder).

There are different types of modules/plugins (in the future, I’ll refer to them simply as modules). There are endpoints, formats, loggers, event handlers, dialplans, TTS engines etc etc. Pretty much every directory under src/mod/ is a type of module. Each type of module registers an “interface” with the FreeSWITCH core. Great care has been taken to make proper abstractions and do not expose “module-specific” data into the FreeSWITCH core (src/*.c) files. The modules make use of FreeSWITCH core API primitives to request core services, the FreeSWITCH core uses the abstract interface exposed by different type of modules for performing operations. For example, the module mod_sofia is an endpoint module. Therefore mod_sofia code is contained at src/mod/endpoints/mod_sofia/

FreeTDM

The FreeTDM library is a modular API that allows applications to initialize different types of TDM/Analog signaling stacks, place/receive calls and read/write media. It supports signaling modules for SS7, ISDN(PRI/BRI), Analog, MFCR2 etc. It also supports a I/O modules to support multiple different hardware manufacturers. The 2 most prominent I/O modules are src/ftmod/ftdm_wanpipe/ftmod_wanpipe.c (For Sangoma Wanpipe cards) and the src/ftmod/ftmod_zt/ftmod_zt.c (For DAHDI-enabled cards such as Sangoma and Digium).

The FreeTDM project is part of the same source tree as FreeSWITCH, however FreeTDM does not depend on FreeSWITCH. FreeSWITCH also does not depend on FreeTDM, the glue that links them together is mod_freetdm, which is an endpoint module for FreeSWITCH that allows FreeSWITCH to place calls in SS7, PRI, MFC-R2 and Analog telephony networks. Note that the mod_freetdm module is just a plugin/extension to FreeSWITCH and it is a “user” of the freetdm library. As such, you should never expose internal opaque details of freetdm to mod_freetdm or worst, to FreeSWITCH. The freetdm.h header is the main public header that is used by freetdm API users. There are other headers under a private/ folder in freetdm that are not meant to be used by the API users, only for internal use of other freetdm C files/components.

Memory Allocation

When allocating memory within FreeSWITCH you must ask yourself where to allocate from. FreeSWITCH uses APR memory pools (wrapped on the switch_memory_pool_t) data structure. You can request a completely new memory pool for yosuelf, but that may be a waste, instead ask yourself whether the memory you’re allocating will be associated to a particular session (call leg) and whether needs to persist. For example, memory for a data structure to keep track of RTCP statistics of an RTP stream, will always be associated to the RTP stream, which in turn is associated to the session. In such cases it is recommended to use the session memory pool, which is automatically destroyed at the end of the call.

In the other hand, if the memory you are allocating must persist beyond the call life cycle, you will be better off requesting a new memory pool and then allocating from it, but now you’re responsible to destroy the pool when you’re done with your object life cycle. Remember it is not possible to free memory allocated from a pool until you destroy the whole pool completely.

Some functions to remember (from src/include/switch_core.h and src/switch_core_memory.c):

switch_core_session_get_pool() – Returns the memory pool associated with a given session
switch_core_session_alloc() – Allocate x amount of bytes from a given session memory pool
switch_core_session_sprintf() – Allocates a new string with the provided format using the session memory pool
switch_core_session_strdup() – Duplicates a new string using the session memory pool
switch_core_new_memory_pool() – Creates a new pool ready to start allocating objects
switch_core_destroy_memory_pool() – Destroy the given pool, you better be sure no one will be needing any of the objects allocated from that pool
switch_core_alloc() – Allocate memory from a given memory pool
switch_safe_free() – Free memory allocated directly using malloc()/calloc(), testing for NULL pointer
For all the session-related allocation functions, you must be sure the allocated object will NOT outlive the session.

Remember all switch allocation functions already initialize memory, you do not need to call memset() after allocation

There are a few cases where it’s justifiable to use standard memory allocation functions such as strdup()

strdup() is ok when you need a quick duplicate of another string but you have no session pointer or it’s something called often and short-lived, and therefore you do not want to use the session pointer with switch_core_session_strdup() because that would leave the memory allocated until the end of the call

String Manipulation

switch_copy_string() – Safely copy a string, use this instead of strcpy() or strncpy(). It will guarantee the output is null-terminated.
switch_set_tring() – Use this to initialize a string buffer of a fixed length. DO NOT USE THIS with char* pointers as the implementation uses sizeof() and sizeof(char*) is only 4 or 8!!
switch_toupper() – Convert a string to uppercase
switch_strstr() – Find a string in a substr
switch_safe_atoi() – Turn a string into a number, testing for NULL and defaulting to a given value in case of NULL
More gems can be found in src/include/switch_utils.h, please check there before hand-coding your own function or check if you can use one from switch_utils.h instead of using direct libc functions!!

Booleans

switch_true() – Use this to test if a string is true (ie, “true”, “enabled”, “on”, etc, all are considered true), this is useful when parsing configuration files
switch_false() – Use this to test if a string is false (ie “false”, “disabled” etc), this is useful when parsing configuration files
SWITCH_TRUE / SWITCH_FALSE – Use this along with switch_bool_t instead of using integers

ClueCon 2015

I just got back from attending ClueCon 2015 in Chicago. This year the FreeSWITCH team presented their video conferencing support in version 1.6. Pretty neat stuff!

This year my presentation was about “Scaling FreeSWITCH Performance”. One of the highlights is that using an alternative memory allocator may (or may not) improve drastically the performance of your FreeSWITCH instance.

Astricon 10 just finished. It was nice to be back after missing it for the last 2 years.  This time I shared my experiences with the Asterisk WebRTC implementation.

Find the presentation here: https://moythreads.com/congresos/astricon2013/

Also available on SlideShare: https://www.slideshare.net/MoisesSilva6/implementation-lessons-using-webrtc

One of the highlights of the presentation is that if you’re trying to use Asterisk for WebRTC using secure WebSockets (TLS) you may notice that the connection is not reliable (may not work, hangs, etc). This is now a known problem and I’ve posted some patches/branches that address that issue, follow the activity in the Asterisk bug tracker: https://issues.asterisk.org/jira/browse/ASTERISK-21930

Starting with Asterisk 12 you also need to install the pjproject stack to use WebRTC at all, otherwise, no errors are printed on calls but simply you may end up without audio (due to lack of ICE support if pjproject libraries are not instlalled/compiled and linked to Asterisk)

I’ve udpated the Asterisk wiki WebRTC instructions to add this very same warning.

https://wiki.asterisk.org/wiki/display/AST/Installing+pjproject

As every year, I attended in August 2012  and August 2013 to this telephony developer conference  in Chicago (organized by the FreeSWITCH developers).

In 2012 I presented about using FreeSWITCH as an SBC (Session Border Controller)

Find the presentation here: http://www.moythreads.com/congresos/cluecon2012/

You can also view it in SlideShare: http://www.slideshare.net/MoisesSilva6/cluecon-2012kickasssbc

In 2013 I presented about SIP Testing

Find the presentation here: http://www.moythreads.com/congresos/cluecon2013/

You can also view it in SlideShare: http://www.slideshare.net/MoisesSilva6/sip-testing-with-freeswi

Having said that, still showed some stability issues and it became a burden to maintain because it is a patch to Asterisk.

Something you may find interesting is that a bit later after I wrote the tapping feature for Asterisk, I also did it for FreeSWITCH

It has been more stable in FreeSWITCH, mostly due to the fact that FreeSWITCH TDM abstraction is modular and it has been much more easy to maintain a tapping module rather than a patch. You can find the module in FreeSWITCH’s tree at libs/freetdm/src/ftmod/ftmod_pritap/ftmod_pritap.c

In addition to the TDM tapping module, I also wrote an RTP tapping module called mod_oreka that can be used for tapping any media stream on FreeSWITCH (SIP, TDM, H.323 etc)

See press release from OrecX

Find the presentations below:

– Elastix World 2011, Mexico D.F. Nov 3-4 – “Negociación de codecs en Asterisk”

– FSL, Vallarta, México. Nov 5 – “FreeSWITCH – Asterisk con esteroides”

– 4K Conference, Buenos Aires, Argentina, Nov 24  – “Manejo de medios en FreeSWITCH“

What is a core dump?

Sometimes problems with FreeSWITCH, Asterisk or just about any other program in Linux are hard to debug by just looking at the logs. Sometimes the process crashes and you don’t have a chance to use the CLI to look at stats. The logs may not reveal anything particularly interesting, or just some minimal information. Sometimes you need to dig deeper into the state of the program to poke around and see what is going on inside. Linux process core dumps are meant for that.

The most typical case for a core dump is when a process dies violently and unexpectedly. For example, if a programmer does something like this:

*(int *)0 = 0;

Is usually not that straight forward, it may be the that the programmer did not expect certain variable to contain a NULL (0) value. The process is killed by the Linux kernel because by default, a Linux process does not map the memory address 0 to anything. This causes a page fault in the processor which is trapped by the kernel, the kernel then sees that the given process does not have anything mapped at address 0 and then sends the SIGSEGV (Unix signal) to the process. This is called a segmentation fault.

The default signal handler for SIGSEGV dumps the memory of the process and kills the process. This memory dump contains all the memory for this process (and all the threads belonging to that process) and that is what is used to determine what went wrong with the program that attempted to reference an invalid address (0 in this case, but any invalid address can cause it).

You can learn more about it here: http://en.wikipedia.org/wiki/Segmentation_fault

How can I make sure the core dump will be saved?

Each process has a limit for how big this core can be. If the limit is exceeded no core dump will be saved. By default this limit is 0!, that means no core will be dumped by default.

Before starting the process you must use “ulimit“. The “ulimit” command sets various limits for the current process. If you execute it from the bash shell, that means the limits are applied to your bash shell process. This also means any processes that you start from bash will inherit those limits (because they are child processes from your bash shell).

ulimit -a

That shows you all the limits for your bash shell. In order to guarantee that a core will be dumped you must set the “core file size” limit to “unlimited”.

ulimit -c unlimited

If you are starting the process from an init script or something like that, the init script has to do it. Some programs are smart enough to raise their limits themselves, but is always better to make sure you have unlimited core file size for your bash shell. You may then want to add those ulimit instructions inside your $HOME/.bashrc file.

Where is the core dump saved?

Each process has a “working directory”. See http://en.wikipedia.org/wiki/Working_directory

That is where the process core dump will be saved by default. However, some system-wide settings affect where the core is dumped.

“/proc/sys/kernel/core_pattern” and “/proc/sys/kernel/core_uses_pid” are 2 files that control the base file name pattern for the core, and whether the core name will be appended with the PID (process ID).

The recommended settings are:

mkdir -p /var/core
echo "/var/core/core" > /proc/sys/kernel/core_pattern
echo 1 > /proc/sys/kernel/core_uses_pid

You can confirm what you just did with:

cat /proc/sys/kernel/core_pattern

cat /proc/sys/kernel/core_uses_pid

This settings will cause any process in the system that crashes, to dump the core at:

/var/core/core.

What if I just want a core dump without killing the process?

In some situations, if a process becomes unresponsive or the response times are not ideal. For example, you try to execute CLI commands in Asterisk or FreeSWITCH but there is no output, or worst, your command line prompt gets stuck. The process is still there, it may be even processing calls, but some things are taking lot of time or just don’t get done. You can use “gdb” (The GNU debugger) to dump a core of the process without killing the process and almost with no disruption of the service. I say almost because for a large process, dumping a core may take a second or two, in that time the process is freezed by the kernel, so active calls may drop some audio (if you’re debugging some real time audio system like Asterisk or FreeSWITCH).

The trick to do it fast is to first create a file with the GDB commands required to dump the core.

Latest versions of CentOS include (with the gdb RPM package) the “gcore” command to do everything for you. You only need to execute:

gcore $(pidof freeswitch)

To dump the core of the running process.

If you are in a system that does not include gcore, you can do the following:

echo -ne "generate-core-file\ndetach\nquit" > gdb-instructions.txt

The 3 instructions added to the file are:

generate-core-file
detach
quit

This will do exactly what we want. Generate the core file for the attached process, then detach from the process (to let it continue) and then quit gdb.

You then use GDB (you may need to install it with “yum install gdb”) to attach to the running process, dump the core, and get out as fast as possible.

gdb /usr/local/freeswitch/bin/freeswitch $(pidof freeswitch) -x gdb-instructions.txt

The arguments to attach to the process include the original binary that was used to start it and the PID of the running process. The -x switch tells GDB to execute the commands in the file after attaching.

The core will be named core.<pid> by default and the path is not affected by the /proc/sys/kernel settings of the system.

This core can now be used by the developer to troubleshoot the problem.

Sometimes though, the developer will be more interested in a full back trace (stack trace), because the core dump itself can’t be easily examined in any other box than the one where was generated, therefore it might be up to you to provide that stack trace, which you can do with:

gdb /usr/local/freeswitch/bin/freeswitch core.
(gdb) set logging file my_back_trace.txt
(gdb) thread apply all bt full
(gdb) quit

Then send the file my_back_trace.txt to the developer, or analyze it yourself, sometimes is easy to spot the problems even without development experience!

Anyways, the topic of my session this time was media handling in FreeSWITCH. We’ll be doing some work in in that area at Sangoma in the following months so I thought I’d kill two birds with one stone.

Presentation here:
PPT: http://www.moythreads.com/congresos/cluecon2011/media-handling-freeswitch-final.ppt

PDF: http://www.moythreads.com/congresos/cluecon2011/media-handling-freeswitch-final.pdf

http://area51.stackexchange.com/proposals/12932/telephony

Presentation in PDF and PPT available here:

http://www.moythreads.com/congresos/astricon2010/asterisk-pri-passive-recording.pdf

http://www.moythreads.com/congresos/astricon2010/asterisk-pri-passive-recording.ppt