Warning: main(/www/www/htdocs/style/globals.php) [function.main]: failed to open stream: No such file or directory in /www/www/docs/6.4.1/ide_en/user_guide/memory.html on line 1

Warning: main() [function.include]: Failed opening '/www/www/htdocs/style/globals.php' for inclusion (include_path='.:/www/www/common:/www/www/php/lib/php') in /www/www/docs/6.4.1/ide_en/user_guide/memory.html on line 1

Warning: main(/www/www/htdocs/style/header.php) [function.main]: failed to open stream: No such file or directory in /www/www/docs/6.4.1/ide_en/user_guide/memory.html on line 8

Warning: main() [function.include]: Failed opening '/www/www/htdocs/style/header.php' for inclusion (include_path='.:/www/www/common:/www/www/php/lib/php') in /www/www/docs/6.4.1/ide_en/user_guide/memory.html on line 8

Finding Memory Errors

In this chapter:

Introduction

Have you ever had a customer say, “The program was working fine for days, then it just crashed”? If so, chances are good that your program had a memory error — somewhere.

Debugging memory errors can be frustrating; by the time a problem appears, often by crashing your program, the corruption may already be widespread, making the source of the problem difficult to trace.

Memory analysis is a key function to ensuring the quality of your systems. The QNX Memory Analysis perspective shows you how your program uses memory, and can help ensure that your program won't cause problems. The perspective helps you quickly pinpoint memory errors in your development and testing environments before your customers get your product.


Note: The QNX Memory Analysis perspective may produce incorrect results when more than one IDE is communicating with the same target system. To use this perspective, make sure that only one IDE is connected to the target system.

Memory management in QNX Neutrino

By design, Neutrino's architecture helps ensure that faults, including memory errors, are confined to the program that caused them. Programs are less likely to cause a cascade of faults because processes are isolated from each other and from the microkernel. Even device drivers behave like regular debuggable processes:


Introduction; Neutrino architecture


The microkernel architecture.

This robust architecture ensures that crashing one program has little or no effect on other programs throughout the system. If a program faults, you can be sure that the error is restricted to that process's operation.

Neutrino's full memory protection means that almost all the memory addresses your program encounters are virtual addresses. The process manager maps your program's virtual memory addresses to the actual physical memory; memory that is contiguous in your program may be transparently split up in your system's physical memory:


Introduction; Contiguous memory


How the process manager allocates memory into pages.

The process manager allocates memory in small pages (typically 4 KB each). To determine the size for your system, use the sysconf(_SC_PAGESIZE) function.

Virtual memory

As you'll see when you use the Memory Information view of the QNX System Information perspective, the IDE categorizes your program's virtual address space as follows:


Introduction: Process memory


Process memory layout on an x86.

The Memory Information and Malloc Information views of the QNX System Information perspective provide detailed, live views of a process's memory. For more information, see the Getting System Information chapter.

Program memory

Program memory holds the executable contents of your program. The code section contains the read-only execution instructions (i.e. your actual compiled code); the data section contains all the values of the global and static variables used during your program's lifetime:


Introduction: Memory, program


The program memory.

Stack memory

Stack memory holds the local variables and parameters your program's functions use. Each process in Neutrino contains at least the main thread; each of the process's threads has an associated stack. When the program creates a new thread, the program can either allocate the stack and pass it into the thread-creation call, or let the system allocate a default stack size and address:


Introduction: Memory, stack 2


The stack memory.

When your program runs, the process manager reserves the full stack in virtual memory, but not in physical memory. Instead, the process manager requests additional blocks of physical memory only when your program actually needs more stack memory. As one function calls another, the state of the calling function is pushed onto the stack. When the function returns, the local variables and parameters are popped off the stack.

The used portion of the stack holds your thread's state information and takes up physical memory. The unused portion of the stack is initially allocated in virtual address space, but not physical memory:


Introduction: Memory, stack 1


Stack memory: virtual and physical.

At the end of each virtual stack is a guard page that the microkernel uses to detect stack overflows. If your program writes to an address within the guard page, the microkernel detects the error and sends the process a SIGSEGV signal.

As with other types of memory, the stack memory appears to be contiguous in virtual process memory, but isn't necessarily so in physical memory.

Shared-library memory

Shared-library memory stores the libraries you require for your process. Like program memory, library memory consists of both code and data sections. In the case of shared libraries, all the processes map to the same physical location for the code section and to unique locations for the data section:


Introduction: Memory, library


The shared library memory.

Object memory

Object memory represents the areas that map into a program's virtual memory space, but this memory may be associated with a physical device. For example, the graphics driver may map the video card's memory to an area of the program's address space:


Introduction: Memory, object


The object memory.

Heap memory

Heap memory represents the dynamic memory used by programs at runtime. Typically, processes allocate this memory using the malloc(), realloc(), and free() functions. These calls ultimately rely on the mmap() function to reserve memory that the malloc library distributes.

The process manager usually allocates memory in 4 KB blocks, but allocations are typically much smaller. Since it would be wasteful to use 4 KB of physical memory when your program wants only 17 bytes, the malloc library manages the heap. The library dispenses the paged memory in smaller chunks and keeps track of the allocated and unused portions of the page:


Introduction: Memory, heap1


The heap memory

Each allocation uses a small amount of fixed overhead to store internal data structures. Since there's a fixed overhead with respect to block size, the ratio of allocator overhead to data payload is larger for smaller allocation requests.

When your program uses the malloc() function to request a block of memory, the malloc library returns the address of an appropriately sized block. To maintain constant-time allocations, the malloc library may break some memory into fixed blocks. For example, the library may return a 20-byte block to fulfill a request for 17 bytes, a 1088-byte block for a 1088-byte request, and so on.

When the malloc library receives an allocation request that it can't meet with its existing heap, the library requests additional physical memory from the process manager. These allocations are done in chunks called “arenas”. By default, the arena allocations are performed in 32 KB chunks. The value must be a multiple of 4 KB, and currently is limited to less than 256 KB. When memory is freed, the library merges adjacent free blocks within arenas and may, when appropriate, release an arena back to the system.

For detailed information about arenas, see Arena allocations in the QNX Neutrino System Architecture guide.


Introduction: Memory, heap2


How virtual memory is mapped to physical memory.

For more information about the heap, see Heap corruption in the QNX Neutrino System Architecture guide.

Process memory

Typically, virtual memory occupied by a process can be separated into the following categories:


Note: It is important to know how much memory each individual process uses, otherwise you can spend considerable time trying to optimize the heap (i.e., if a process uses only 5% of the total process memory, is it unlikely to return any noticeable result). Techniques for optimizing a particular type of memory are also dramatically different.

For information about obtaining process memory distribution details, see Inspecting your process memory distribution.”

What the Memory Analysis perspective can reveal

The main system allocator has been instrumented to keep track of statistics associated with allocating and freeing memory. This lets the memory statistics module unobtrusively inspect any process's memory usage.

When you launch your program with the Memory Analysis tool, your program uses the debug version of the malloc library (libmalloc_g.so). Besides the normal statistics, this library also tracks the history of every allocation and deallocation, and provides cover functions for the string and memory functions (e.g. strcmp(), memcpy(), memmove()). Each cover function validates the corresponding function's arguments before using them. For example, if you allocate 16 bytes, then forget the terminating NULL character and attempt to copy a 16-byte string into the block using the strcpy() function, the library detects the error.

The debug version of the malloc library uses more memory than the nondebug version. When tracing all calls to malloc() and free(), the library requires additional CPU overhead to process and store the memory-trace events.


Note: Be sure to occasionally check the Downloads area on our website for updated versions of the debug malloc library.

The QNX Memory Analysis perspective can help you pinpoint and solve various kinds of problems, including:

Memory leaks

A memory leak is a portion of heap memory that was allocated but not freed, and the reference to that area of memory cannot be used by the application any longer. Typically, the elimination of a memory leak is critical for applications that run continuously because even a single byte leak can crash a mission critical application that runs over time.

Memory leaks can occur if your program allocates memory and then forgets to free it later. Over time, your program consumes more memory than it actually needs.

In its mildest form, a memory leak means that your program uses more memory than it should. QNX Neutrino keeps track of the exact memory your program uses, so once your program terminates, the system recovers all the memory, including the lost memory.

If your program has a severe leak, or leaks slowly but never terminates, it could consume all memory, perhaps even causing certain system services to fail.

The following tabs in the Memory Analysis editor can help you find and fix memory leaks:

For more information about the memory analysis editor, see Viewing memory analysis data (memory analysis editor) later in this chapter. For detailed descriptions about memory errors, see Interpreting errors during memory analysis. For more information about the heap, see Dynamic memory management.

Memory errors

Memory errors can occur if your program tries to free the same memory twice or uses a stale or invalid pointer. These “silent” errors can cause surprising, random application crashes. The source of the error can be extremely difficult to find, because the incorrect operation could have happened in a different section of code long before an innocent operation triggered a crash. For more information about how to interpret memory errors during memory analysis, see the topic Interpreting errors during memory analysis later in this chapter.

In the event of a memory error, the IDE can:

The resulting action that the IDE takes depends on the setting that you choose in the Memory Analysis Tool area of the launch configuration (see Launching your program with Memory Analysis,” later in this chapter).

The Memory Analysis editor's Errors and Statistics tabs show memory errors and — if possible — the exact line of source code that generated each error. The Trace tab lets you find the prior call that accessed the same memory address, even if your program made the call days earlier. For more information about the memory analysis editor, see Viewing memory analysis data (memory analysis editor) later in this chapter. For detailed descriptions about memory errors, see Interpreting errors during memory analysis.


Note: To learn more about the common causes of memory problems, see the topic Heap Analysis: Making Memory Errors a Thing of the Past chapter of the QNX Neutrino Programmer's Guide.

Launching your program with Memory Analysis

To extract the most information from your program, you should launch it with the Memory Analysis tool enabled:


Note: If your binary is instrumented with Mudflap, you can't run Memory Analysis on it because there will be a conflict (trying to overload the same functions), and it will cause the program to crash.

  1. Create a Run or Debug type of QNX Application launch configuration as you normally would, but don't click Run or Debug.
  2. In the Create, manage, and run configurations dialog, click the Tools tab.
  3. Click Add/Delete Tool.
  4. In the Tools Selection dialog, check Memory Analysis:

    Adding a tool

  5. Click OK.
  6. Click the Memory Analysis tab.

    Memory Analysis tool

  7. To configure the Memory Analysis settings for your program, expand disclosure to view the appropriate set of options:
  8. If you want the IDE to automatically change to the QNX Memory Analysis perspective when you run or debug, check Switch to this tool's perspective on launch.
  9. Click Apply to save your changes.
  10. Click Run, Debug, or Profile. The IDE starts your program and lets you analyze your program's memory.

Note: Don't run more than one Memory Analysis session on a given target at a time, because the results may not be accurate.

Analyzing a running program

You can perform memory analysis on a running program, if that program was started using the debug malloc library and the proper environment variables. Once the program is running, you can attach the Memory Analysis perspective and gather your data.

For more information, see Attaching to a running process in this chapter.


Note:

If a program uses fork, the control thread of the Memory Analysis tool must be disabled. fork works only with single threaded programs.

To disable the control thread option for memory analysis:

  1. From an existing launch configuration, select the Tools tab.
  2. If the Memory Analysis tool is not currently enabled, select Add/Delete Tool, select Memory Analysis, then click OK.
  3. Expand Advanced Settings.
  4. Disable the Create control thread option if it is currently enabled.
  5. Click Apply.
  6. Click Run.

For information about the Create control thread option, see Launching your program with Memory Analysis in this chapter.


Interpreting errors during memory analysis

Although the QNX Memory Analysis perspective shows you how your program uses memory, and can quickly direct you to memory errors in your development and testing environments, you need to understand the types of memory errors that you might run into.

During memory analysis, you may encounter the following types of memory errors:

When memory errors occur, the IDE can:

To enable memory analysis:

  1. From an existing launch configuration, select the Tools tab.
  2. Select Add/Delete Tool.
  3. Select Memory Analysis and click OK.
  4. Select any required memory tools and specify any desired options for that tool.

After you configure the IDE for memory analysis, you can begin to use the results to identify the types of memory errors in your programs, and then trace them back to your code.

To view the memory errors identified by the IDE:

  1. After enabling Memory Analysis in a launch configuration, run that configuration.
  2. In the Session view, double-click your desired launch configuration.

    A dialog with the same name as your application will contain a list of memory errors that the IDE encountered in your program.

    Memory Analysis Tool

    In addition, an editor with multiple tabs will open at the bottom of the Workbench window.

    Memory Analysis Tool

  3. Click the Errors tab.
  4. In the window that lists the types of errors, select an error. Notice that the information in the Errors tab dynamically updates to reflect the error that you've selected.
  5. On the Errors tab, double-click on an error to navigate to that error in the code editor.
  6. Modify the code, as required, to correct the memory error for the selected error.

Illegal deallocation of memory

The illegal deallocation of memory occurs when a free() operation is performed on a pointer that doesn't point to an appropriate heap memory segment. This type of error can occur when you attempt to do any of the following activities:

Consequences

The illegal deallocation of memory can generate the following runtime errors:

Detecting the error

In the QNX IDE, the Memory Analysis tool detects this error (if error detection is enabled), and it traps the illegal deallocation error when any of the following functions are called:


Note: For instructions about enabling error detection in the IDE, see Enabling error detection for the illegal deallocation of memory.

Enabling error detection for the illegal deallocation of memory

To enable error detection for the illegal deallocation of memory:

  1. In the Launch Configuration window, select the Tools tab.
  2. Expand Memory Errors and select the Enable error detection checkbox.
  3. Select the Enable check on realloc()/free() argument checkbox.
  4. Click OK.

Message returned to the QNX IDE

In the IDE, you can expect the message for this type of memory error to include the following types of information and detail:

For a list of error messages returned by the Memory Analysis tool, see Error message summary (memory analysis).

How to address the illegal deallocation of memory

To help address this memory problem, try the following:

Example

The following code shows an example of the illegal deallocation of memory:

int main(int argc, char ** argv){
  char * str = "";
  if (argc>1) {
     str = malloc(10);
     // ...
  }
  printf("Str: %s\n",str);
  free(str);
  return 0;
}

NULL pointer dereference

A NULL pointer dereference is a sub type of an error causing a segmentation fault. It occurs when a program attempts to read or write to memory with a NULL pointer.

Consequences

Running a program that contains a NULL pointer dereference generates an immediate segmentation fault error.


Note: For instructions about enabling error detection in the IDE, see Enabling error detection for a NULL pointer dereference.

When the memory analysis feature detects this type of error, it traps these errors for any of the following functions (if error detection is enabled) when they are called within your program:

strcat()
strdup()
strncat()
strcmp()
strncmp()
strcpy()
strncpy()
strlen()
strchr()
strrchr()
index()
rindex()
strpbrk()
strspn() (only the first argument)
strcspn()
strstr()
strtok()

The memory analysis feature doesn't trap errors for the following functions when they are called:

memccpy()
memchrv()
memmove()
memcpy()
memcmp()
memset()
bcopy()
bzero()
memccpy()
memchrv()
memmove()
memcpy()
memcmp()
memset()
bcopy()
bzero()
bcmp()
bcmp()

Enabling error detection for a NULL pointer dereference

To enable error detection for the NULL pointer dereference:

  1. In the Launch Configuration window, select the Tools tab.
  2. Expand Memory Errors and select the Enable error detection checkbox.
  3. To detect the passing of a zero (0) pointer to string and memory functions, select Verify parameters in string and memory functions.
  4. To detect the freeing of a zero (0) pointer, select Enable check on realloc()/free() argument.

Message returned to the QNX IDE

In the IDE, you can expect the message for this type of memory error to include the following types of information and detail:

For a list of error messages returned by the Memory Analysis tool, see Error message summary (memory analysis).

How to address a NULL pointer dereference

You can perform an explicit check for NULL for all pointers returned by functions that can return NULL, and when parameters are passed to the function.

Example

The following code shows an example of a NULL pointer dereference:

int main(int argc, char ** argv){
  char buf[255];
  char * ptr = NULL;
  if (argc>1) {
    ptr = argv[1];
  }
  strcpy(str,ptr);
  return 0;
}

Buffer overflow

A buffer overflow error occurs when a program unintentionally writes to a memory area that's out of bounds for the buffer it intended to write to.

Consequences

A buffer overflow generates the following runtime errors:

Detecting the error

The Memory Analysis tool can detect a limited number of possible buffer overflows with following conditions:

strcat()
strdup()
strncat()
strcmp()
strncmp()
strcpy()
strncpy()
strlen()
strchr()
strrchr()
index()
rindex()
strpbrk()
strspn()
strcspn()
strstr()
strtok()
memccpy()
memchr()
memmove()
memcpy()
memcmp()
memset()
bcopy()
bzero()
bcmp()

Enabling error detection

To enable error detection for a buffer overflow or underflow:

  1. In the Launch Configuration window, select the Tools tab.
  2. Select Enable error detection checkbox.
  3. To detect an immediate overflow, select Verify parameters in string and memory functions.
  4. To detect a small overflow in block's memory overhead area, select Enabled bounds checking (where possible).
  5. To detect a corrupted heap, caused by overflowing other regions, select Perform full heap integrity check on every allocation/deallocation.

Message returned to the QNX IDE

In the IDE, you can expect the message for this type of memory error to include the following types of information and detail:

For a list of error messages returned by the Memory Analysis tool, see Error message summary (memory analysis).

How to address buffer overflow errors

Locate the code where the actual overflow occurred. Ensure that the size of the memory region is always accompanied by the pointer itself, verify all unsafe operations, and that the memory region is large enough to accommodate the data going into that location.

Example

The following code shows an example of a buffer overflow trapped by a library function:

int main(int argc, char ** argv){
  char * ptr = NULL;
  ptr = malloc(12);
  strcpy(ptr,"Hello World!");
  return 0;
}

The following code shows an example of a buffer overflow trapped by a post-heap check in a free() function:

int main(int argc, char ** argv){
  char * ptr = NULL;
  ptr = malloc(12);
  ptr[12]=0;
  free(pre);
  return 0;
}

Using freed memory

If you attempt to read or write to memory that was previously freed, the result will be a conflict and the program will generate a memory error. For example, if a program calls the free() function for a particular block and then continues to use that block, it will create a reuse problem when a malloc() call is made.

Consequences

Using freed memory generates the following runtime errors:

Detecting the error

The Memory Analysis tool can detect only a limited number of situations where free memory is read/written with following conditions:

strcat()
strdup()
strncat()
strcmp()
strncmp()
strcpy()
strncpy()
strlen()
strchr()
strrchr()
index()
rindex()
strpbrk()
strspn()
strcspn()
strstr()
strtok()
memccpy()
memchr()
memmove()
memcpy()
memcmp()
memset()
bcopy()
bzero()
bcmp()

malloc()
calloc()
realloc()
free()

Enabling error detection

To enable error detection when using freed memory:

  1. In the Launch Configuration window, select the Tools tab.
  2. Expand Memory Errors and select the Enable error detection checkbox.
  3. To detect usage of freed memory, select Verify parameters in string and memory functions.
  4. To detect writing to a freed memory area, select Enabled bounds checking (where possible).

Message returned to the IDE

In the IDE, you can expect the message for this type of memory error to include the following types of information and detail:

For a list of error messages returned by the Memory Analysis tool, see Error message summary (memory analysis).

How to address freed memory usage

Set the pointer of the freed memory to NULL immediately after the call to free(), unless it is a local variable that goes out of the scope in the next line of the program.

Example

The following code shows an example using already freed memory:

int main(int argc, char ** argv){
  char * ptr = NULL;
  ptr = malloc(13);
  free(ptr);
  strcpy(ptr,"Hello World!");
  return 0;
}

Reading uninitialized memory

If you attempt to read or write to memory that was previously freed, the result will be a conflict and the program will generate a memory error because the memory is not initialized.

Consequences

Using an uninitialized memory read generates a random data read runtime error.

Detecting the error

Typically, the IDE does not detect this type of error; however, the Memory Analysis tool does trap the condition of reading uninitialized data from a recently allocated memory region.

For a list of error messages returned by the Memory Analysis tool, see Error message summary (memory analysis).

How to address random data read issues

Use the calloc() function, which always initializes data with zeros (0).

Example

The following code shows an example of an uninitialized memory read:

int main(int argc, char ** argv){
  char * ptr = NULL;
  ptr = malloc(13);
  if (argc>1)
     strcpy(ptr,"Hello World!");
  ptr[12]=0;
  printf("%s\n",ptr);
  return 0;
}

Resource (memory) leaks

Memory leaks can occur if your program allocates memory and then does not free it. For example, a resource leak can occur in a memory region that no longer has references from a process.

Consequences

Resource leaks generate the following runtime errors:

Detecting the error

This error would be trapped during the following circumstances:

Enabling error detection

In the IDE, you can expect the message for this type of memory error to include the following types of information and detail:

  1. In the Launch Configuration window, select the Tools tab.
  2. Expand Memory Errors and select the Perform leak check when process exits checkbox.
  3. Optional: Specify how often to check for leaks in the Perform leak check every (ms) field. The minimum depends on target speed; however, on average, it should be no less than 100 ms.

Message returned to the QNX IDE

In the IDE, you can expect the message for this type of memory error to include the following types of information and detail:

For a list of error messages returned by the Memory Analysis tool, see Error message summary (memory analysis).

How to address resource (memory) leaks

To address resource leaks in your program, ensure that memory is deallocated on all paths, including error paths.

Example

The following code shows an example of a memory leak:

int main(int argc, char ** argv){
  char * str = malloc(10);
  if (argc>1) {
     str = malloc(20);
     // ...
  }
  printf("Str: %s\n",str);
  free(str);
  return 0;
}

Functions checked for memory errors during memory analysis

During memory analysis, the following functions are checked for memory errors:

strcat()
strdup()
strncat()
strcmp()
strncmp()
strcpy()
strncpy()
strlen()
strchr()
strrchr()
index()
rindex()
strpbrk()
strspn()
strcspn()
strstr()
strtok()

memccpy()
memchr()
memmove()
memcpy()
memcmp()
memset()
bcopy()
bzero()
bcmp()

malloc()
calloc()
realloc()
free()

Managing Memory Analysis sessions

The Session view lets you manage your memory analysis sessions. You can load these sessions into the Memory Analysis perspective, so you can search for memory management errors in your application.

Session view

The view lists all of the memory analysis sessions that you've created in your workspace while running programs with the Memory Analysis tool active. Each session is identified by a name, date stamp, and an icon that indicates its current state.

The icons indicate:

[Connected]
This memory analysis session is open and can be viewed in the Memory Analysis editor.
[Disconnected]
This session is closed and cannot currently be viewed.
[Running]
This session is still running on the target; you can select the session and view its incoming traces.

Note: If the session is running, you may need to close and reopen the tabs at the top of the editor periodically to refresh the information in the Errors and Statistics panes.

[Indexing]
The traces and events are being indexed. This icon appears only if you stop the memory analysis session or your process terminates. If your process terminates, the running icon may still be shown while the database is registering the events and traces; when this is done, the indexing icon appears. Wait until indexing is finished, or the information might be incomplete.

Right-clicking on a connected session ([Connected]) shows a menu with several options:

Right-clicking on a disconnected session ([Disconnected]) shows a menu with several options:

Connecting to a session

Memory Analysis sessions must be “connected” before they can be viewed in the Memory Analysis editor.

To connect to a session:

  1. Right-click the session in the Session view.
  2. Choose Connect from the context menu.

After a moment, the session is connected ([Connected]).


Note: You can also double-click on the session in the Session view to connect to it.

Deleting a session

To delete a session:

  1. Right-click the session in the Session view.
  2. Choose Delete from the context menu.

The IDE deletes the memory analysis session.

Disconnecting from a session

To disconnect a session and recover the resources it uses while connected:

  1. Right-click the session in the Session view.
  2. Choose Disconnect from the context menu.

After a moment, the session is disconnected ([Disconnected]).

Exporting trace data

You'll use this option to export your trace data session information from a Memory Analysis session view. When exporting memory analysis information, the IDE lets you export the event-specific results in .csv format, and the other session trace data in .xml format. Later, you can import the event-specific results into a spreadsheet, or you can choose to import the other session trace data into a Memory Analysis session view.

For more information about exporting session information in CSV or XML format, see Exporting memory analysis data.

Filtering information for a session

Occasionally, there may be too much information in a Memory Analysis session, and you might want to filter some of this information to narrow down your search for memory errors, events, and traces.

To filter out Memory Analysis session information:

  1. Expand your Memory Analysis session in the session view.
  2. Select specific session components, such as a library, thread, or both, that you want to filter on. You can double-click any of the session components to open a corresponding Memory Analysis Allocations pane containing memory events and traces that belong to the selected component.

Importing session information

You can import trace data session information from a Memory Analysis session view. When importing memory analysis session information, the IDE lets you import the event-specific results for libmalloc events in .csv format, and the other session trace data in .xml format. You'll use this item after you've logged trace events to a file on the target system and copied the file to your host system.

For more information about importing memory analysis event data or XML data see Importing memory analysis data.

Properties: Showing information about a session

To view information about a session:

  1. Right-click the session in the Session view.
  2. Choose Properties from the context menu.

The IDE shows a Properties dialog for that memory analysis session:

Memory Analysis session properties

Renaming a session

To rename a memory analysis session:

  1. Right-click the session in the Session view.
  2. Choose Rename from the pop-up menu.

    The IDE shows the Rename Session dialog.

    Rename Session dialog

  3. Enter a new name for the session, then click OK to change the session's name.

Viewing a session

To view a connected session in the Memory Analysis editor:
Double-click the session in the Session view.

The IDE opens the memory analysis session in the Memory Analysis editor.

Viewing Memory Analysis data (Memory Analysis editor)

When you view a connected Memory Analysis session, the Memory Analysis perspective opens that session in the main editor area of the IDE:

Memory Analysis editor

The top half of the window shows details for the data that is selected in the bottom half. The bottom half shows an overview of the entire memory analysis session data set:

Details of selected allocations

The details include a table of information about the allocations. If you select an allocation, a vertical line indicates its position in the Details chart, and the backtrace (if available) is shown below the table. If you click on a backtrace, the editor shows the associated source code (if available) in another tab.

The icons in the table indicate the type of allocation or deallocation:

Matched allocation
An allocation with a matching deallocation.
Matched deallocation
A deallocation with a matching allocation.
Unmatched allocation
An allocation without a matching deallocation.
Unmatched deallocation
A deallocation without a matching allocation.

The Allocations Overview can be very wide, so it could be divided into pages. You can use the Page field to move from one page to another, and you can specify the number of points to show on each page.


Note: If the process does many allocations and deallocations, it could take some time for the traces and events to be registered, indexed, and shown.

The tabs at the bottom let you switch between several different data views:

Selecting data

To select data in the overview:
Click and drag over the region you're interested in.

The Memory Analysis perspective updates the details to reflect the data region you've selected.

Controlling the view

The Memory Analysis editor has several icons that you can use to control the view:

Use this icon: To:
[Horizontal layout] Set the Chart and Detail Pane to a horizontal layout, one beside the other
[Vertical layout] Set the Chart and Detail Pane to a vertical layout, one above the other
[Restore] Shows the Detail Pane if it's currently hidden
[Chart] Hide the Detail Pane so the Chart pane has more display room
[Detail Pane] Hide the Chart pane so the Detail Pane has more display room
[Toggle Chart Overview] Toggle the Overview pane on and off

Controlling the overview

Right-click on the Overview pane to change the view options.

Overview pane's context menu

This menu includes:

By Timestamp
Show the events sorted by their timestamp. Because several memory events can occur with the same time stamp, this might present the events in a confusing order (for example, a buffer's allocation and deallocation events could be shown in the wrong order if they happen during the sampling interval).
By Count
Show events sorted by their event index. This is the default ordering in the Overview pane.
Filters...
Filter the events that are shown by size, type, or both. You can also hide the matching allocations and deallocations, so that you see only the unmatched ones:

Memory Analysis Filters

Zoom In
Zoom in on the selected range of events.
Zoom Out
Zoom out to the set of memory events that you previously zoomed in on.

Controlling the detail pane

You can control the Detail pane through its context menu:

  1. Right-click on the Detail pane.
  2. Choose a graph from the Chart Types menu:

Allocations tab

As described above, the Allocations pane shows you allocation and deallocation events over time. Select a range of events to show a chart and details for those events.

Errors tab

The Errors pane shows any memory errors (in red) or leaks (in blue) detected while collecting statistics. Select a line in the top pane to see a function backtrace in the lower pane.

Errors

Bins tab

The allocator keeps counters for allocations of various sizes to help gather statistics about how your application is using memory. Blocks up to each power of two (2, 4, 8, 16, etc. up to 4096) and “large” blocks (anything over 4 KB) are tracked by these counters.

The Bins pane shows the values for these counters over time:

Bins

The counters are listed at the top of the Bins pane. Click the circle to the left of each counter to enable or disable the counter in the current view.

When the Bins pane is shown, the Chart pane shows allocations and deallocations for each bin at the time selected in the Detail pane. The Detail pane lists memory events for the selected region of the Bins pane.

The Bins pane includes these additional buttons:

Play icon
Play the selected range of the Use Bins; the Bins Statistics chart shows the usage dynamically.
Play icon
Stop.

Because of the logging that's done for each allocation and deallocation, tracing can be slow, and it may change the timing of the application. You might want to do a first pass with the bins snapshots enabled to determine the “hot spots or ranges”, and on the second pass reduce the tracing to a certain range (minimum, maximum) to filter and reduce the log set.

Bands tab

For efficiency, the QNX allocator preallocates “bands” of memory (small buffers) for satisfying requests for small allocations. This saves you a trip through the kernel's memory manager for small blocks, thus improving your performance.

The bands handle allocations of up to 16, 24, 32, 48, 64, 80, 96, and 128 bytes in size, any activity in these bands is shown on the Bands pane:

Bands

Usage tab

The Usage pane shows your application's overall memory usage over time.

Usage

Trace Details tab

This tab plays back the allocations, synchronized with the bins, bands, or usage, depending on what you last selected. You can use this display to look for memory leaks (e.g. bins where the number of allocations is much greater than the number of deallocations):

Trace details

The Trace Details tab shows the trace event information (malloc() and dealloc()) for the timestamps selected in other view. You can select a region from either Bins, Bands, or Usage Tabs, right-click and then select Show in Trace Details to play back the events specified in the selection. For example, if you select Bins, the IDE would show the allocations and deallocations up to last timestamp selected. You can then click Playback to review the allocations.

In addition, you can select a region in the Trace Details tab, right-click and then select Trace Details… to return to the Allocations tab to see the selected area in the Details pane.

Statistics tab

The Statistics pane gives you several different statistics views for the Memory Analysis session.

The Allocations pane shows the number of calls to different kinds of allocations, plus a count of each allocation for a given number of bytes:

Statistics - Allocations

The Backtraces pane shows you a list of memory event points in your application. Select one to show a function backtrace for that event:

Statistics - Backtraces

The Outstanding traces pane shows allocations that weren't deallocated when the trace ended. These aren't necessarily errors.

Statistics - Outstanding traces

The Errors pane shows you a list of memory errors, grouped by type, that were encountered while running your application.

Statistics - Errors

Memory Errors tab

This tab lets you change the settings for a running process.

Settings

For information about these memory settings, see Launching your program with Memory Analysis in the Finding Memory Errors chapter.

The additional icons (from left to right) let you:

Error message summary (memory analysis)

The following table shows a summary of potential error messages you might encounter during memory analysis:

Message Caused by Description
no errors
allocator inconsistency - Malloc chain is corrupted, pointers out of order A buffer overflow occurred in the heap. The heap memory is corrupted.
allocator inconsistency - Malloc chain is corrupted, end before end pointer A buffer overflow occurred in the heap. The heap memory is corrupted.
pointer does not point to heap area The illegal deallocation of memory. You attempted to free non-heap memory.
possible overwrite - Malloc block header corrupted A buffer overflow occurred in the heap. The heap memory is corrupted.
allocator inconsistency - Pointers between this segment and adjoining segments are invalid A buffer overflow occurred in the heap. The heap memory is corrupted.
data has been written outside allocated memory block A buffer overflow occurred in the heap. The program attempted to write data to a region beyond allocated memory.
data in free'd memory block has been modified Attempting to use memory that was previously freed. The program is attempting to write to a memory region that was previously freed.
data area is not in use (can't be freed or realloced) A buffer overflow occurred in the heap. The heap memory is corrupted.
unable to get additional memory from the system All memory resources are exhausted. There are no more memory resources to allocate.
pointer points to the heap but not to a user writable area A buffer overflow occurred in the heap. The heap memory is corrupted.
allocator inconsistency - Malloc segment in free list is in-use A buffer overflow occurred in the heap. The heap memory is corrupted.
malloc region doesn't have a valid CRC in header A buffer overflow occurred in the heap. The heap memory is corrupted.
free'd pointer isn't at start of allocated memory block An illegal deallocation of memory. An attempt was made to deallocate the pointer that shifted from its original value when it was returned by the allocator.

Memory Analysis GUI flags and corresponding environment variables

The following table shows a summary of Memory Analysis Tool (MAT) graphical user interface options (flags) and their corresponding environment variables:

Environment variable Where to find in Memory Analysis Tool GUI What option to set Additional information
LD_PRELOAD= librcheck.so Memory Analysis-->Advanced Settings Runtime library The newer library file is librcheck.so; the older file is libmalloc_g.so. A supported option for rcheck library.
MALLOC_ACTION=<number> Memory Analysis-->Memory Errors When an error is detected: Set the error action behavior to: 0 to ignore, 1 to abort, 2 to exit, 3 for core, and 4 to stop. A supported option for rcheck library.
MALLOC_BTDEPTH=10 Memory Analysis-->Memory Errors Limit back-trace depth to: 5
MALLOC_CKACCESS=1 Memory Analysis-->Memory Errors Limit back-trace depth to: Check strings and memory functions for errors. A supported option for rcheck library.
MALLOC_CKBOUNDS=1 Memory Analysis-->Memory Errors Enable bounds checking (where possible) Check for out of bounds errors. A supported option for rcheck library.
MALLOC_CKALLOC=1 Memory Analysis-->Memory Errors Enable check on realloc()/free() argument Check alloc() and free() functions for errors. A supported option for rcheck library.
MALLOC_CKCHAIN=1 Memory Analysis-->Memory Errors Perform a full heap integrity check on every allocation/deallocation Check the allocator chain integrity for every allocation/deallocation. A supported option for rcheck library.
MALLOC_CTHREAD=1 Memory Analysis-->Advanced Settings Create control thread Start a control thread. A supported option for rcheck library.
MALLOC_DUMP_ LEAKS=1 Memory Analysis-->Memory Errors Perform leak check when process exits Enable the dumping of leaks on exit. A supported option for rcheck library.
MALLOC_ERRFILE= /dev/null N/A N/A Error file location.
MALLOC_EVENTBTDEPTH=<number> Memory Analysis-->Memory Tracing Limit back-trace depth to: 5 Set the error traces depth to a specific number. A supported option for rcheck library.
MALLOC_FATAL=0 Memory Analysis-->Memory Errors When an error is detected: report the error and continue
MALLOC_FILE=<file> Memory Analysis-->Advanced Settings Target output file or device: Re-direct output to a file. You can use ${pid} in the filename to replace with process Id and escape $ if running from the shell. A supported option for rcheck library.
MALLOC_HANDLE_SIGNALS=0 N/A N/A When the value is 0, don't install signal handlers. A supported option for the rcheck library.
MALLOC_START_TRACING=1 Memory Analysis-->Memory Tracing Enable memory allocation/deallocation tracing Enable memory tracing (0 to disable). A supported option for rcheck library.
MALLOC_STAT_BINS=<bin1>,<bin2>,... Memory Analysis-->Memory Snapshots Bin counters (comma separated) e.g. 8, 16, 32, 1024 Set custom bins. A supported option for rcheck library.
MALLOC_TRACEBTDEPTH=<number> Memory Analysis-->Memory Tracing Limit back-trace depth to: Set the allocation traces depth to a specific number. A supported option for rcheck library.
MALLOC_TRACEMAX=<number> Memory Analysis-->Memory Tracing Maximum allocation to trace: Only trace the allocation for the <= <number> of bytes. A supported option for rcheck library.
MALLOC_TRACEMIN=<number> Memory Analysis-->Memory Tracing Minimum allocation to trace: Only trace the allocation for the >= <number> of bytes. A supported option for rcheck library.
MALLOC_TRUNCATE=1 N/A N/A Truncate the output files before writing. A supported option for rcheck library.
MALLOC_USE_CACHE=<number> N/A N/A Set to 0 to disable optimization. The default is 32. A supported option for rcheck library.
MALLOC_VERBOSE=1 Memory Analysis-->Advanced Settings Show debug output on console When set to 1, it enables the debug output. A supported option for the rcheck library.
MALLOC_WARN=0 Memory Analysis-->Memory Errors When an error is detected: report the error and continue

Using a file to log the trace

You can perform memory analysis on a running program, or you can log the trace to a file on the target system. The advantage of logging the trace is that doing so frees up qconn resources; you run the process now, and perform the analysis later.


Note: When analyzing the data from a log file, you can't do any backtracing.

To log the trace to a file:

  1. Open the Launch configuration and select the Memory Analysis tab.
  2. Expand Target Settings.
  3. By default, the Send traces to field is set to /dev/dbgmem, which the qconn agent reads. Specify the name of the file on the target system (e.g. /tmp/log.memory) where you'd like to send the traces instead.
  4. Run your application on the target.
  5. Copy the file back to the host, then right-click inside the Session view and click Import.

    A dialog appears:

    Importing trace events

  6. Choose an existing session or click Create Session to create a new one.
  7. Browse to the file that you copied from the target, and then click OK. The IDE reparses the file for viewing.

You can also run this from the command line by setting the appropriate environment variables. For example:


Note: For the supported options of the rcheck library, see the summary of Memory Analysis Tool (MAT) graphical user interface options (flags) and their corresponding environment variables at Memory Analysis GUI flags and corresponding environment variables.”

Launching with debug malloc

To start a program using the debug malloc library:
Launch the program using the LD_PRELOAD (set to the debug malloc library) and MALLOC_CTHREAD (set to 1) environment variables:
LD_PRELOAD=librcheck.so MALLOC_CTHREAD=1 ./my_app

Attaching to a running process

As mentioned earlier, you can analyze memory events and traces for a running process. To do this, you need to create a launch profile, as follows:

  1. If the Run menu doesn't include a Profile entry, add it like this:
    1. Select Customize Perspective ... from the Window menu.
    2. Select the Commands tab.
    3. In the list of checkboxes, ensure that the Profile checkbox is enabled.
    4. Click OK.
  2. Choose Run-->Profile....
  3. Set up the launch configuration.

After launching, a dialog appears with a list of all the running processes on the target. Choose the process you want to attach to; the Session view then lists it.

When you're done, disconnect from the process and let it continue.

Analyzing shared objects

To analyze shared objects, you must set up the Memory Analysis tab in your launch configuration like this:

In the Session View, you can expand your session, expand your process, and then select a shared object to view its memory events and traces in a new tab in the editor.

Associated views

The QNX Memory Analysis perspective includes the following views:

Testing an application for memory leaks

To test an application for memory leaks:

  1. Build and then run an application with the Memory Analysis tool enabled.
  2. In the Memory Analysis perspective, in the Target Navigation view, select the process to examine.
  3. Use the Malloc Information view to compare memory usage at specific times.
  4. Watch the Allocated column (which is the Outstanding allocations in memory) and observe the value to see if it increases.

    In the example below, notice the steady growth in the chart. If memory usage continues to increase over time, then the process is not returning some of the allocated memory.

Observed trend in the steady increase in allocated memory

Since memory leaks can be apparent or hidden, to know exactly what's occurring in your application, use the Memory Analysis tool to automatically find the apparent memory leaks type. A memory leak is considered apparent when the binary address of that heap block (marked as allocated) is not stored in any of the process memory and current CPU registers any longer.

Finding memory leaks using the QNX Memory Analysis tool

There are three ways of finding memory leaks using the QNX Memory Analysis tool:

Performing an automatic leak check options

To perform an automatic leak check, use the Perform leak check when process exits option on the Memory Analysis tab of the launch configuration. Use the Perform leak check every (ms) to specify your desired interval options. If these options are enabled, the Memory Analysis tool has automatically checks for memory leaks in the currently running program.

The following illustration shows a list of memory leaks that the memory Analysis tool identified in the example application:

Malloc information

For this example, the Allocated column (which refers to the outstanding allocations in memory), shows the steady growth for its value in the chart. The Byte range from 9 - 16 increases each time the view refreshes; however the free count doesn't. In addition, the histogram chart below shows a clear indication that there's a problem. The number of malloc()s continue to steadily increase over time. This trend shows that the selected process is not returning some of the allocated memory.

Histogram dististribution of history information

Get Leaks button

To find a memory leak, click the Get Leaks button on the Settings page of the Memory Analysis Session viewer while the application runs (which can also be used by the debugger).

Get Leaks icon

Enabling memory leak detection

In a continuously running application, the following procedure enables memory leak detection at any particular point during program execution:

  1. Find a location in the code where you want to check for memory leaks, and insert a breakpoint.
  2. Launch the application in Debug mode with the Memory Analysis tool enabled.
  3. Change to the Memory Analysis perspective.
  4. Open the Debug view so it is available in the current perspective.
  5. When the application encounters the breakpoint you specified, open the Memory Analysis session from the Session View (by double-clicking) and select the Setting page for the Session Viewer.
  6. Click the Get Leaks button.

    Before you resume the process, take note that no new data will be available in the Session Viewer because the memory analysis thread and application threads are stopped while the process is suspended by the debugger.

  7. Click Resume in the Debug view to resume the process' threads.

    If leaks did not appear on the Errors tab of the Session Viewer, either there were no leaks, or the time given to the control thread (a special memory analysis thread that processes dynamic requests) to collect the leaks was not long enough to perform the command; and was suspended before the operation completed.

  8. Switch to the Errors page of the viewer, to review information about collected memory leaks.

Note: Besides apparent memory leaks, an application can have other types of leaks that the memory Analysis tool cannot detect. These leaks include objects with cyclic references, accidental point matches, and left-over heap references (which can be converted to apparent leaks by nullifying objects that refer to the heap). If you continue to see the heap grow after eliminating apparent leaks, you should manually inspect some of the allocations. You can do this after the program terminates (completes), or you can stop the program and inspect the current heap state at any time using the debugger.

Optimizing static and stack memory

In the previous section, we explained tool-assisted techniques for optimizing heap memory, and now we will describe some tips for optimizing static and stack memory:

Code

In embedded systems, it is particularly important to optimize the size of a binary, not only because it takes RAM memory, but also because it uses expensive flash memory. Below are some tips you can use to optimize the size of an executable:

Data

Stack

In some cases, it is worth the effort to optimize the stack, particularly when the application has some frequent picks of stack activity (meaning that a huge stack segment would be constantly mapped to physical memory). You can watch the Memory Information view for stack allocation and inspect code that uses the stack heavily. This usually occurs in two cases: recursive calls (which should be avoided in embedded systems), and heavy usage of local variables (keeping arrays on the stack).


Note: Tasks such as finding unused objects, structures that are not optimal, and code clones, are not automated in the QNX Momentics IDE. You can search for static analysis tools with given keywords to find an appropriate tool for this task.

Optimizing heap memory

You can use the following techniques to optimize memory usage:

Another optimization technique is to shorten the life cycle of the heap object. This technique lets the allocator reclaim memory faster, and allows it to be immediately used for new heap objects, which, over time, reduces the maximum amount of memory required.

Always attempt to free objects in the same function as they are allocated, unless it is an allocation function. An allocation function is a function that returns or stores a heap object to be used after this function exits. A good pattern of local memory allocation will look like this:

p=(type *)malloc(sizeof(type));
do_something(p);
free(p);
p=NULL;
do_something_else();

After the pointer us used, it is freed, then nullified. The memory is then free to be used by other processes. In addition, try to avoid creating aliases for heap variables because it usually makes code less readable, more error prone, and difficult to analyze.

Types of allocation overhead

Another large source of memory usage occurs from the following types of allocation overhead:

User overhead
The actual data occupies less memory when requested by the user
Padding overhead
The fields in a structure are arranged in a way that the sizeof of a structure is larger than the sum of the sizeof of all of its fields.
Heap fragmentation
The application takes more memory than it needs, because it requires contiguous memory blocks, which are bigger than chunks that allocator has
Block overhead
The allocator actually takes a larger portion of memory than required for each block
Free blocks
All free blocks continue to be mapped to physical memory

User overhead usually comes from predictive allocations (usually by realloc()), which allocate more memory than required. You can either tune it by estimating the average data size, or - if your data model allows it - after the growth of data stops, you can truncate the memory to fit into the actual size of the object.

Estimating the average allocation size

To estimate the average allocation size for a particular function call, find the backtrace of a call on the Backtraces tab on the Statistics page of the Session Viewer. The Count column represents the number of allocations for a particular stack trace, and the Total Allocated column shows the total size of the allocated memory. To calculate an average, divide the Total Allocated value by the Count value.

Padding overhead

Padding overhead affects the struct type on processors with alignment restrictions. The fields in a structure are arranged in a way that the sizeof of a structure is larger than the sum of the sizeof of all of its fields. You can save some space by re-arranging the fields of the structure. Usually, it is better to group fields of the same type together. You can measure the result by writing a sizeof test. Typically, it is worth performing this task if the resulting sizeof matches with the allocator block size (see below).

Heap fragmentation

Heap fragmentation occurs when a process uses a lot of heap allocation and deallocation of different sizes. When this occurs, the allocator divides large blocks of memory into smaller ones, which later can't be used for larger blocks because the address space isn't contiguous. In this case, the process will allocate another physical page even if it looks like it has enough free memory. The QNX memory allocator is a “bands” allocator, which already solves most of this problem by allocating blocks of memory of constant sizes of 16, 24, 32, 48, 64, 80, 96 and 128 bytes. Having only a limited number of possible sizes lets the allocator choose the free block faster, and keeps the fragmentation to a minimum. If a block is more that 128 bytes, it's allocated in a general heap list, which means a slower allocation and more fragmentation. You can inspect the heap fragmentation by reviewing the Bins or Bands graphs. An indication of an unhealthy fragmentation occurs when there is growth of free blocks of a smaller size over a period of time.

Block overhead

Block overhead is a side effect of combating heap fragmentation. Block overhead occurs when there is extra space in the heap block; it is the difference between the user requested allocation size and actual block size. You can inspect Block overhead using the Memory Analysis tool:

In the allocation table, you can see the size of the requested block (11) and the actual size allocated (16). You can also estimate the overall impact of the block overhead by switching to the Usage page:

You can see in this example that current overhead is larger than the actual memory currently in use. Some techniques to avoid block overhead are:

You should consider allocator band numbers, when choosing allocation size, particularly for predictive realloc(). This is the algorithm that can provide you with the next highest power or two for a given number m if it is less than 128, or a 128 divider if it is more than 128:

int n;
if (m > 256) {
n = ((m + 127) >> 7) << 7;
} else {
n = m - 1;
n = n | (n >> 1);
n = n | (n >> 2);
n = n | (n >> 4);
n = n + 1;
}

It will generate the following size sequence: 1,2,4,8,16,32,64,128,256,384,512,640, and so on.

You can attempt to optimize data structures to align with values of the allocator blocks (unless they are in an array). For example, if you have a linked list in memory, and a data structure does not fit within 128 bytes, you should consider dividing it into smaller chunks (which may require an additional allocation call), but it will improve both performance (since band allocation is generally faster), and memory usage (since there is no need to worry about fragmentation). You can run the program with the Memory Analysis tool enabled once again (using the same options), and compare the Usage chart to see if you've achieved the desired results. You can observe how memory objects were distributed per block range using Bands page:

Bands allocation to show memory usage

This chart shows, for example, that at the end there were 85 used blocks of 128 bytes in a block list. You also can see the number of free blocks by selecting a time range.

Free blocks overhead

When you free memory using the free() function, memory is returned to the process pool, but it does not mean that the process will free it. When the process allocates pages of physical memory, they are almost never returned. However, a page can be deallocated when the ratio of used pages reaches the low water mark. Even in this case, a virtual page can be freed only if it consists entirely of free blocks.

Analyzing allocation patterns

After you have prepared a memory analysis (profiling) session, double-click on a session to open the Memory Analysis Session viewer. The Allocations page shows the Overview: Requested Allocations chart. For example, let's take a closer look at this chart.

Overview of Requested Allocations chart for memory allocations and deallocations

This example chart shows memory allocation and deallocation events that are generated by the malloc() and free() functions and their derivatives. The X-axis represents the event number (which can change to a timestamp), and the Y-axis represents the size (in bytes) of the allocation (if a positive value), or the deallocation (if a negative value).

Let's take a closer look at the bottom portion of the chart. The Page field shows the scrollable page number, the Total Points field shows how many recorded events there are, the Points per page field shows how many events can fit onto this page, and the Total Pages field shows how many chart pages there are in total.

For this example, there are 202 events that fit within the chart; however for some larger charts, all of them would not likely fit on this single chart. If that were the case, there are several choices available. First, you can attempt to reduce the value in the Points per page field to 50, for example.

Overview of modified Requested Allocations chart for memory allocations and deallocations

However, in the case where the number of events is large (the X-axis value is a large number, 1482 events), changing the value of Points per page field might not significantly improve the visual appearance of the data in the chart. For this example, there are 1482 events, and all of these events don't fit on a single chart:

Overview of modified Requested Allocations chart for memory allocations and deallocations

If you reduce the value in the Points per page field to 500, the graphical representation in the following illustration is better; however, it's still not very useful.

Overview of modified Requested Allocations chart for memory allocations and deallocations

Alternatively, you can use filters to exclude data from the chart. If you look at the Y-axis of the following chart, notice some large allocations at the beginning. To see this area more closely, select this region with the mouse. The chart and table at the top change to populate with the data from the selected region.

Now, locate the large allocation and check its stack trace. Notice that this allocation belongs to the function called monstartup(), which isn't part of the user defined code; meaning that it can't be optimized, and it can probably be excluded from the events of interest.

Detail allocations

You can use a filter to exclude this function. Right-click on the Overview chart's canvas area and select Filters... from the menu. Type 1000 in the To allocation size field. The overview will look like this:

Requested allocations

From the filtered view, there is a pattern: the allocation is followed by a deallocation, and the size of the allocations grows over time. Typically, this growth is the result of the realloc() pattern. To confirm the speculation, return to the Filters... menu option, and disable (un-check) all of the allocation functions, except for the realloc-alloc option. Notice that the growth occurs with a very small increment.

Requested allocations with filters

Next, select a region of the Overview chart and explore the event table. Notice the events with the same stack trace; this is an example of a realloc() call with a bad (too small) increment (the pattern for a shortsighted realloc()).

realloc call with a bad increment

Notice that the string in the example was re-allocated approximately 400 times (from 11 bytes to 889 bytes). Based on that information, you can optimize this particular call (for performance) by either adding some constant overhead to each realloc() call, or by double allocating the size. In this particular example, if you double allocate the size, re-compile and re-run the application, and then open the editor and filter all but the realloc() events, you'll obtain the following:

Requested allocation with only realloc events

The figure above shows only 12 realloc() events instead of the original 400. This would significantly improve the performance; however, the maximum allocated size is 1452 bytes (600 bytes in excess of what is required). You can adjust the realloc() code to better tune it for a typical application run. Normally, you should make realloc() sizes similar to the allocator block sizes.

To check other events, in the Filters menu, enable all functions, except for realloc(). Select a region in the overview:

Requested allocation with focused region

In the Details chart, the alloc/free events have the same size. This is the typical pattern for a short-lived object.

Allocations: pattern for a short-lived object

To navigate to the source code from the stack trace view, double-click on a row for the stack trace.

Allocations: pattern for a short-lived object

This code has an object that allocates 11 bytes, and then it is freed at the end of the function. This is a good candidate to put a value on the stack. However, if the object has a variable size, and originates from the user, using stack buffers should be done carefully. As a compromise between performance and security, you can perform a size verification, and if the length of the object is less than the buffer size, it is safe to use the stack buffer; otherwise, if it is more than the buffer size, the heap can be allocated. The buffer size can be chosen based on the average size of allocated objects for this particular stack trace.

Shortsighted realloc() functions and short-lived objects are memory allocation patterns which can improve performance of the application, but not the memory usage.

Tuning the allocator

Occasionally, application driven data structures have a specific size, and memory usage can be greatly improved by customizing block sizes. In this case, you either have to write your own allocator, or contact QNX to obtain a customizable memory allocator.

Use the Bin page to estimate the benefits of a custom block size. First, enter the bin size in the Launch Configuration of the Memory Analysis tool, run the application, and then open the Bins page to explore the results. The resulting graph shows the distribution of the heap object per imaginary blocks, based on the sizes that you selected.

Observing changes in memory usage (allocations and deallocations)

It is important for you to know when and where memory is being consumed within an application. The Memory Analysis tool includes several views that use the trace data from the Memory Analysis session to help extract and visually show this information to determine memory usage (allocation and deallocation metrics). Showing this information using various charts helps you observe the changes in memory usage

The IDE includes the following tabs to help you observe changes in memory over time:

To access these tabs:

  1. Select Window-->Show View-->Other.
  2. Select QNX System Information-->Malloc Information.
  3. Click OK.

Note: To begin to view data on your graphs, you need to set logging for the target, and you need to select an initial process from the Target Navigator view.

These charts show the memory usage and the volume of memory events over time (the allocation and deallocation of memory). These views reflect the current state of the active editor and active editor pane. You can select an area of interest in any of the charts; then, using the right-click menu, zoom in to show only that range of events to quickly isolate areas of interest due to abnormal system activity.


Note:

The vertical scale always changes to reflect the maximum range of data currently being shown. To lock the current view of the data on the graph so that you can see the results without constant change, click the Lock icon (Icon Lock ) located at the bottom right of each tab.

Depending on the time interval you specify, an indicator on the lines of the graphs is used to mark the time measurement intervals. However, if you select a different process for a period of time and then return, or if there is no change in the data for the selected process, these indicator marks aren't shown on the graph line for the specified period where there is no activity. To identify where no deltas in memory have occurred (for whatever reason), the line changes to a solid line.

For the first three byte ranges (1-2, 3-4, and 5-8) the values are zero because the allocator starts at byte 16 (byte range 9-16).


Outstanding allocations

This graph shows the total allocation of memory within your program over time for the selected process.

Outstanding Allocations

If you compare it to the Overview History tab, you can see the trend of how memory is being allocated within your program.

Outstanding Allocations

Allocation deltas

This graph shows the changes to the allocation of memory within your program over time for the selected process. From this type of graph, you can observe which band(s) has the most activity.

Allocation Deltas

Deallocation deltas

This graph shows the changes to the deallocation of memory within your program over time for the selected process from the Target Navigator view. From this type of graph, you can observe which band(s) has the least activity.

Deallocation Deltas

Outstanding allocation deltas

This graph shows the differences between the memory that was allocated and deallocated for the selected process; it shows a summary of the free memory. From this graph, you can observe which band(s) might be leaking memory, and by how much.

Outstanding Allocation Deltas

Manually inspecting outstanding allocations

To manually inspect outstanding allocations:

  1. Open the Allocations page of the Session Viewer.
  2. In the Overview chart, select the Filters... option from the menu.
  3. Enable the Hide matching allocation/deallocation pair option and click Ok.
  4. Review the results (only those allocations that remain in memory, or were in memory at the moment of the exit).
  5. Select the allocations using the mouse.

    The Details view and table become populated with the data.

  6. Select one allocation from the table. The Trace view becomes populated with the current stack trace for the selected event.
  7. Inspect the stack trace of the allocation using source code navigation, and determine whether it is a leak.

    Inspecting the allocations

    The same data that is grouped by backtraces is available from the Statistics page on the Outstanding traces tab.

    Inspecting the Outstanding Traces information

    Alternatively, you can inspect memory allocations that occurred during a particular time frame of interest by opening the Usage page, which will be populated with snapshots of memory usage (if you enabled this option).

    memory usage trends

  8. Select the region you are interested in, and the statistics for the memory usage will populate the table above.
  9. Right-click the Allocations page and change the X-axis mode to be timestamp (changes to label to be timestamp).

    Specifying by timestamp

  10. Select an area of interest from the chart.

    You will notice the allocations and deallocations that occurred during this period in the Details chart and event table. Review the event table results.

    Event table results

    All malloc() calls that do not have a paired free() would have the following icon in the left column that doesn't have a checkmark beside it:

    A deallocation that doesn't have a matching allocation

Importing and exporting memory analysis trace data

The Memory Analysis Tool can import and export trace data from a Memory Analysis session view. With the Memory Analysis Tool, you can:

Exporting memory analysis data

In the IDE, you can export your trace data session information from a Memory Analysis session view. When exporting memory analysis information, the IDE lets you export the event-specific results in .csv format, and the other session trace data in .xml format. Later, you can import the event-specific results into a spreadsheet, or you can choose to import the other session trace data into a Memory Analysis session view to review the results.


Note:

For the IDE to begin to trace and collect memory analysis session information, the Memory Analysis tool must be enabled.

To enable the Memory Analysis tool:

  1. From an existing launch configuration, select the Tools tab.
  2. Select Add/Delete Tool.
  3. Select Memory Analysis and click OK.
  4. Select any required memory tools, and specify any desired options for that tool.

Exporting a data file

To export a memory analysis data file:

  1. Click File-->Export.
  2. Select QNX-->Memory Analysis Data, and click Next.

    The list shows all of the memory analysis trace data that you accumulated for the current view.

    Exporting Memory Analysis Data wizard

  3. You can choose to export the data in the following two formats:
  4. In the Output File field, click Browse to select the output file you want to save the XML or CSV results in, or specify a new location and file name.

    Note: If you select an output file that currently exists, you're prompted during the export process to click Yes to overwrite this file.

  5. To begin the export process, click Finish.

The resulting output file contains all of the memory analysis data for the selected session(s), based on your selected options.


Note: When you export session information and then import it into a Memory Analysis session view to review the results, the session is the same; however, the name, date, and some other properties that are unique to a session will be different.

CSV file format

When the IDE exports the event data in CSV format, the resulting data for the exported file contains different information depending on the type of event selected for export. The types of event formats you can expect in an exported CSV file are:


Note: To include column headers for the event data in the exported CSV file, select the Generate header row checkbox in the Exporting Memory Analysis Data wizard.

Memory event results format

For a memory event (allocation/deallocation events), the data in the results file appears in the following order:

Bin event results format

For a bin event, the data in the results file appears in the following order:

Runtime error event results format

For a runtime error event, the data in the results file appears in the following order:

Band event results format

For a band event, the data in the results file appears in the following order:

Importing memory analysis data

In the IDE, you can import trace data session information from a Memory Analysis session view. When importing memory analysis session information, the IDE lets you import the event-specific results for libmalloc events in .csv format, and the other session trace data in .xml format.

Importing from an XML file

To import data from an XML file:

  1. Click File-->Import.
  2. Select QNX-->Memory Analysis Data, and click Next.

    Importing Memory Analysis Data wizard

  3. You can choose to import data from the following two formats:

Importing session information from an XML file

To import data from an XML file:

  1. For the Input File field, click Browse to select an .xml input file.

    Note: You don't need to select any sessions from the list because they were automatically created (using the same names) when they were exported with the prefix “imported:”.

  2. Click Finish.

Note:

When the import process completes, you can open a Memory Analysis session and view the results.

After importing, you can rename the session by right-clicking in the session and selecting Rename.


Importing event information


Note: If it isn't possible to start a Memory Analysis session using the IDE, for example when there is no network connection between target and host machines, or qconn is not running on target machine, you can import memory analysis trace data for libmalloc events.

To import memory analysis trace data for libmalloc events:

  1. Click File-->Import.
  2. Select QNX-->Memory Analysis Data, and click Next.

    Importing Memory Analysis Data wizard

  3. For the Input File field, click Browse to select an input file.
  4. Select a session file from the Session to import list.
  5. Select an events file.

    For information about exporting session information to create an events file, see Exporting memory analysis data.”

  6. Choose a session from which to import events from the Session to import list, or click Create New Session to create a new session.

    You can select only one session for the import process.

  7. Click Next.

    Importing Memory Analysis Data wizard

  8. On this page, you can select an executable for the application. Click From Workspace or From File System to select an executable file.

    Although this step is optional, you should select a binary and source search folder for the application; otherwise, reported events will not have a connection to the source code, and traces will not have navigation data.

    The executable you select should be exactly the same as the one running on the target machine.

  9. Optional: Add locations for the source folders. This step is required only if you intend to navigate to the editor from the memory analysis tables. Click Add from File System or Add From Workspace to add a source lookup path to the list.
  10. Click Finish, to begin importing.

When the importing process completes, you can open the Memory Analysis session to view the results.


Warning: main(/www/www/htdocs/style/footer.php) [function.main]: failed to open stream: No such file or directory in /www/www/docs/6.4.1/ide_en/user_guide/memory.html on line 3703

Warning: main() [function.include]: Failed opening '/www/www/htdocs/style/footer.php' for inclusion (include_path='.:/www/www/common:/www/www/php/lib/php') in /www/www/docs/6.4.1/ide_en/user_guide/memory.html on line 3703