Friday, December 19, 2014

RAD Telemetry 2.1A Released!

This release brings major changes to the runtime implementation.


  • Change: the 2.1 runtime breaks compatibility with Telemetry 1.x servers.
  • Change: (PS3) SPU Telemetry runtime removed.
  • Change: TMO_RECORD_TELEMETRY_STALLS support dummied out - it had too much overhead to be useful.
  • Change: TMOF_DONT_KILL_OTHER_SESSIONS is now ignored; Telemetry always behaves as if it was on, i.e. server never kills existing live sessions from an app when the same app reconnects.
  • Change: Mac users, the Telemetry 2.1A runtime now requires OS X 10.7
  • Enhancement: The runtime now supports 64-bit iOS.
  • Enhancement: Significantly reduced Telemetry overhead - details depend on usage, but 2x-4x CPU time reduction is not uncommon.
  • Enhancement: checked builds of Telemetry are now faster and un-checked builds now allow enabling TMO_OUTPUT_DEBUG_INFO.
  • Enhancement: user-defined IO (via callbacks) is now officially supported.
  • Enhancement: Telemetry now fully supports closing sessions mid-stream (even while other threads are using the context) and re-opening them later - see $ug_context_lifecycle in the docs for details.
  • Enhancement: new TMO_BREAK_ON_WARNING to simplify tracking down causes of Telemetry API warnings.
  • Enhancement: much lower CPU overhead and network bandwidth usage for context switch tracking on all platforms that support it.
  • Enhancement: significantly reduced cost of call stack collection on Linux, OS X and iOS.
  • Enhancement: increased size and improved hit rate of unique call stack cache, reducing network bandwidth.
  • Enhancement: reduced size of zone events in network stream; typically 15-25% smaller.
  • Enhancement: filtered zones are now cheaper.
  • Enhancement: improved network IO patterns: Telemetry now frequently sends data in medium-sized blocks, instead of rarely sending very large ones.
  • Enhancement: explicitly detect and warn about incorrect use of format strings.
  • Enhancement: reduced processing time on IO ("telemetry") thread.
  • Bugfix: gracefully handle the case when Telemetry context string table overflows (don't just report all further strings as "unknown")
  • Bugfix: fix race condition in code that determines loaded modules/shared objects (triggered by opening two Telemetry contexts at once)


  • Change: status bar is now always shown.
  • Enhancement: width of track headers is now resizeable.
  • Enhancement: highlight long frames in zone view ruler.
  • Enhancement: zone view ruler shows length of shortest/longest frame when multiple frames are grouped.
  • Enhancement: improved text rendering quality.
  • Enhancement: add "centered" windows for generated plots.
  • Bugfix: fix Visualizer refusing to open sessions with less than 2 ticks in them.
  • Bugfix: fix handling of sessions with no available frametime plot.
  • Bugfix: don't draw thread tracks in zone view if there are no events for them.
  • Bugfix: fix visually jarring disappearance of timespan tracks when there's no visible events for them in current view.
  • Bugfix: fix zone sublabels (TMMF_ZONE_SUBLABEL) not being displayed.
  • Bugfix: fix positioning of very wide or very tall tooltips.
  • Bugfix: (OS X) fix frequent Visualizer crashes that started in 2.0P3 (long-standing bug exposed by switching from build with libstdc++ to libc++)
  • Bugfix: (OS X) fix checkboxes not being toggled on double-clicks.
  • Bugfix: (OS X) fix several symbolization bugs.


  • Change: update to SQLite, which brings performance improvements.
  • Enhancement: lots of internal optimizations. Datasets typically process ~2.3x faster and take up 30% less space on disk.
  • Enhancement: live mode updates views more regularly now.
  • Bugfix: fix server crashes on leaked tmEnter without matching tmLeave.
  • Bugfix: (Linux) fix a bug where some sessions stored on a Linux server would not show in the Visualizer.
  • Bugfix: fix bug where incorrect plots far in the future would be generated from TMZF_PLOT_TIME zones that were open across tick boundaries.
  • Bugfix: fix double-enumeration of active live sessions in certain cases.


  • Enhancement: improved tmxgpu implementation for PS4 and Wii U.

Tuesday, November 11, 2014

RAD Telemetry 2.0P3 Released!

This is a bugfix release.


  • Bugfix: use actual path of executable, not argv[0], to figure out where to look for


  • Bugfix: fix data from live sessions only showing up with significant delay.
  • Bugfix: fix underscores not being visible in text edit fields.
  • Bugfix: fix "profiler" panel computing totals over incorrect range.
  • Bugfix: fix a crash on Linux.

Wednesday, September 10, 2014

RAD Telemetry 2.0P Released!

This release significantly improves the server- and visualizer-side handling of plots, locks and context switches.


  • Bugfix: (Linux) Remove 'lsof scan' debug spew.
  • Bugfix: (PS4) Fix broken hostname resolution.


  • Enhancement: redesigned, more intuitive display of summary information for fast-changing plots.
  • Enhancement: show "density" of summarized zones using opacity.
  • Enhancement: significantly improved rendering of lock events in zone view.
  • Enhancement: context switch and lock tracks now display way more information when zoomed out.
  • Enhancement: display summary information about timespans too short to be visible at the current zoom level. 
  • Enhancement: new "plot per-frame totals" option for zones! (Right-click on zones in zone view.)
  • Enhancement: zone search now has "before selection" and "after selection" modes. 
  • Enhancement: added "count" aggregation function and leading windows to generated plots.
  • Bugfix: fixed numerous serious usability issues in text edit controls.
  • Bugfix: fix double-clicking on a range of frames in timeline view jumping to the wrong place in the zone view. 
  • Bugfix: panning the zone view didn't work in selection region unless mouse cursor was over a zone. 
  • Bugfix: draw profiler pane on top of selection, not the other way round. 
  • Bugfix: fix 60fps / 30fps markers in timeline view being drawn in the wrong position sometimes. 
  • Bugfix: modify 60Hz / 30Hz vertical scale modes to actually affect axis scaling again. 
  • Bugfix: fix plot scale factor in timeline view resetting on scroll. 
  • Bugfix: fix incorrect rendering/display of TMPT_PERCENTAGE_COMPUTED and TMPT_TIME plot values. 
  • Bugfix: fix caching bug where high-resolution plot information wouldn't get fetched in certain cases. 
  • Bugfix: fix buggy zone find "visible" behavior (would use wrong search range in certain cases) 
  • Bugfix: fix numerous small UI glitches. 
  • Change: cleaned up session view UI.


  • Change: database schema is changed! If you have scripts/tools that read TDBs directly, please refer to the docs for a description of the new schema. 
  • Bugfix: fixed a per-connection memory leak.
  • Enhancement: reduced traffic between Server and Visualizer for plots. 
  • Enhancement: processed datasets with lots of plots, locks or context switches are typically 25-50% smaller. 
  • Enhancement: processing time of datasets with context switches or plots is significantly reduced, by up to 30% in our tests.


  • Change: now writes CSV files in a slightly different format and produces one CSV file per plot instead of one CSV with all plots.

Friday, July 18, 2014

RAD Telemetry 2.0N Released!

This release focuses on making the Visualizer more responsive.


  • Bugfix: fixed a bug where printf format strings containing 64-bit integers followed by strings would not work in certain cases.


  • Bugfix: fixed UI flickering on cursor-key scrolling when hitting edges of the dataset.
  • Bugfix: fixed a bug where semaphore lock counts would not get displayed in the visualizer.
  • Bugfix: fixed a bug where plot highlight state would flicker when scrolling.
  • Bugfix: fixed a bug where LOD plots wouldn't display min/max sometimes.
  • Bugfix: fixed bug where free location was incorrectly reported for long-lived allocations.
  • Bugfix: (OS X) fixed incorrect positioning of server status and search bars.
  • Bugfix: (OS X) dragging with middle mouse button to scroll now works the same way as on other platforms.
  • Enhancement: revised the protocol between Server and Visualizer. When using a dedicated server, network traffic to the visualizer is typically reduced by 5-6x!
  • Enhancement: significantly reduced data fetch times from server.
  • Enhancement: improved prefetching logic in the Visualizer. Waiting for data to load should happen much less often now.
  • Enhancement: UI is now much more responsive when lots of zones are visible.


  • Enhancement: substantially smaller databases when context switch tracking is enabled.
  • Enhancement: dataset processing time is reduced further.


  • Enhancement: (Linux) Telemetry now officially supports CentOS 6 and RHEL 6.

Thursday, June 26, 2014

RAD Telemetry 2.0M Released!

Telemetry 2.0M has just been released.  This release focuses on reducing server-side processing time.


  • Change: On PS4, allocate context switch trace buffer directly from the kernel instead of using new[] so no large libc heap is required.


  • Bugfix: fixed missing/flickering zones in presence of thread name changes.
  • Bugfix: fixed incorrect messages appearing in "Log" view in certain cases.
  • Enhancement: reduced fetch times for zones and plots.
  • Enhancement: importing recorded runs from a file is now much faster and displays progress information.


  • Change: updated to SQLite 1.8.5.
  • Enhancement: significant reduction in dataset processing time; in our tests, processing is now typically between 2 and 4 times faster.
  • Enhancement: real-time indexing mode is now much faster - typically 30%-50% longer processing time than late indexing, instead of the 3x and more it was previously.
  • Enhancement: processed datasets are now approximately 25% smaller due to a revised index structure.

Wednesday, May 28, 2014

RAD Telemetry 2.0L3 Released!

Telemetry 2.0L3 has just been released.  This is strictly a minor maintenance release to deal with some small bugs.


  • Bugfix: fixed plot LOD rendering issues
  • Bugfix: file import now works on file names with spaces
  • Bugfix: path names with leading spaces now work
  • Change: warn if we fail to launch tmimport32 or tm2sym
  • Change: remove unused search and memory tabs
  • Enhancement: added TDB double click support on windows


  • Enhancement: added docs on TMO_NULL_NETWORK

Tuesday, April 22, 2014

RAD Telemetry 2.0L released!

Telemetry 2.0L has just been released.  This is strictly a minor maintenance release to deal with some small bugs.

NOTE: The ABI has changed, so be sure to recompile/relink everything instead of just swapping out libraries!

Run Time

  • Bugfix: varargs with float types were not handled properly at times on OS X (64-bit), Linux/x64 and PS4
  • Bugfix: TM_VA_LIST parameters were incorrectly warned as 'stack variables' by checked build
  • Change: Android runtime is now built with -funwind-tables by default
  • Enhancement: checked build now does extra verification to watch for 'wrapper' related bugs


  • Bugfix: unknown string pointers are silently handled instead of generating an exception and associated disconnection
  • Bugfix: locks/timespans could accidentally start with a negative ref count


  • Enhancement: 'edit source code' is now a right-click zone option
  • Bugfix: backgrounds were drawn in green instead of black of any timeline sections were present but the current visible area did not have one defined
  • Bugfix: tooltip was not showing file name and line properly in zone view
  • Bugfix: in some situations Visualizer on Windows would leave clipboard open, locking out other applications from copy/paste


  • Change: added documentation discussing the perils of wrapping calls to Telemetry vs. just using Telemetry in place

Monday, March 31, 2014

RAD Telemetry 2.0K is out!

Telemetry 2.0K is out the door!  The biggest new feature is context switch trapping support on PS4, putting it on a par with Windows and 360.  Also, the directory/file naming structure is no longer so rigid (so you can rename directories and TDB files).  Finally, fetch times for very dense sets of plots, zones, and messages are way faster now.  Some fetches that used to take 15 seconds now take < two seconds!

Note that the server/Visualizer/run-time need to be simultaneously upgraded since there is a small change in the protocol.

Run Time

  • Enhancement: context switch capture on PS4
  • Enhancement: add server configuration API (tmSetServerConfiguration)
  • Bugfix: fixed bug where timespans could hang the run-time if tmOpen was called late
  • Change: linux (and other BSD-ish) network implementation now spew debug on wouldblock (checked build)
  • Change: disabled network policy on PS4 telemetry socket
  • Enhancement: added TM_VAR macro and no longer exclude tmtypes.h on NTELEMETRY.  This means you can wrap Telemetry declarations and variables in a way that is compatible with NTELEMETRY, e.g TM_VAR(TmU64 x);


  • Change: unknown strings are a bit more known (null vs. unrecognized)
  • Enhancement: decoupled sessions from their filenames, so you can now rename files/directories (yay!)
  • Bugfix: fixed crash bug with oversized messages
  • Bugfix: generated mem usage plots (via TMZF_MEM_REGION_EXPERIMENTAL) were using incorrect paths


  • Enhancement: mousewheel now works in message view
  • Enhancement: greatly sped up fetch times for zones, messages, plots, and mem events
  • Enhancement: session notes are now on a separate line and green, and if then completely skipped
  • Change: disable 'copy' button in message pane until all have arrived
  • Change: messages without newlines now have newlines appended automatically
  • Bugfix: fixed file/line missing from zone tooltips

Thursday, February 20, 2014

RAD Telemetry 2.0j released!

This is a minor maintenance and bugfix release, but it does require a recompile/relink.

Run Time

  • Enhancement: tmMessage now has a tmMessageEx variant. This was added so you can change zone label/sublabels in virtual threads.
  • Enhancement: (Windows) you can now specify full path for TMDATA files when using TMCT_FILE captures
  • Bugfix: fixed benign AppVerifier warning about dangling CriticalSection


  • Bugfix: in certain situations long zones would not be recorded properly
  • Bugfix: fixed pathological edge case in zone garbage collector leading to N^2 processing times


  • Change: (Windows) Visualizer now uses system default text editor instead of Notepad when editing notes
  • Enhancement: plot scale with mousewheel should work now
  • Bugfix: double clicking a frame in plot view would sometimes take you to the wrong location
  • Bugfix: right-click 'zoom' on plot would sometimes take you to the wrong location


  • Enhancement: Mac/Linux/Win now have gpu_opengl.cpp sample file included in distributions
  • Enhancement: Python SQLite export script has been greatly enhanced
  • Change: 'tools' only distributions now included exporter script
  • Change: updated zone filtering documentation
  • Change: updated timespan document to note the ordering constraints of *TimespanAt APIs

Tuesday, February 11, 2014

Dynamic Strings: Solution to a Problem and a Problematic Solution Combined

Telemetry's run-time markup system relies on the assumption that string identifiers are both constant and (hopefully) string pooled.  This allows us to send over a string only once when we encounter it, e.g.:

void foo( void )
   tmZone( cx, TMZF_NONE, "hello world" ); // "hello world" should be sent only the first time we see it

   // do a bunch of foo stuff

This works great for the general case where you're statically marking up a lot of code, which is common when using compiler provided constants like __FUNCTION__.

However sometimes the description you pass to tmEnter or tmZone isn't a const string (e.g. it may be copied out of string table, etc.).  Or (more likely) one of the string parameters you're passing to your zone markup isn't actually a const string and instead a string on the stack.

void foo( char const *name )
   // if 'name' points to something on the stack, this may be bad...
   tmZone( cx, TMZF_NONE, "foo: %s", name );
   // do a bunch of foo stuff

Here's where the problem occurs.  Since Telemetry assumes that the string pointers passed to it are const, it will try to read from name in its background processing thread probably long after name is out of scope and gone.  In addition, once it sees name's address, it won't send it again.  This is to save a lot of bandwidth, but if the contents of name are constantly changing (which is highly likely) you'll see a lot of the same string repeating in the Telemetry Visualizer (since it's never resent with the new content) instead of the updated string.

(Instead of just using the pointer we could hash the contents, but that has a surprising amount of overhead when working with lots of strings, particularly on lower end devices)

Telemetry's solution to this problem is to allow you to tag certain strings as dynamic, i.e. volatile memory that may change or disappear almost immediately.

void foo( char const *name )
   tmZone( cx, TMZF_NONE, "foo: %s", tmDynamicString( cx, name ) );
   // do a bunch of foo stuff

Now the full contents of name are sent over immediately every time they're encountered.  In exchange for correctness we've now incurred a potentially massive amount of network overhead.  This is fine as a first pass integration (to make sure your markup is working), but if you leave it in and have a lot of markup you may find that Telemetry is suddenly eating a tremendous amount of network bandwidth due to redundant string sends.  This is really a problem on devices with limited bandwidth to begin with such as WiiU, XBOX 360, and mobile.

Figuring all this out can be a headache, so thankfully Telemetry's server automatically generates plots indicating the amount of dynamic string activity in a session.  If you enable the plots you should be able to see if you're hammering the dynamic string system or not.  A few dozen dynamic strings should be fine, but if you're finding that you're sending over hundreds or thousands of strings every frame, that is likely bad.

If you find yourself in this situation then you'll want to invest some time doing a string pooling/interning system that provides a const mapping from a volatile string to a static one.  There's no easy way for Telemetry to provide this functionality since it depends heavily on your app's underlying string management architecture, but it's a worthwhile investment for the performance.

Tuesday, February 4, 2014

RAD Telemetry 2.0i released!

Whoops, that was quick.  This is an interim bug fix release with some minor new features.  The big issues are that plot generation was crashing, so that's been fixed, and we've allowed you to prefetch a lot more data per server data fetch which reduces latency when just scrolling around in the zone view.


  • Change: QNX/ARM is now built with -fpic
  • Change: removed dependency on dbghelp.dll for Windows
  • Change: 'processing time' plot is now 'process time(ms)'


  • Change: default zone pool is larger now, reducing need to resize continuously


  • Bugfix: plot generation was crashing
  • Change: generated plot names are shortened and now include 'by frame' or 'by time' unit
  • Enhancement: prefetch window can be altered to reduce latency when scrolling around.  This is a slider in the Options page.
  • Enhancement: state tag backgrounds in zone view are more prominent
  • Bugfix: mem track button would disappear and stay gone
  • Bugfix: time scale calculations were off in certain rare instances due to quantization artifacts


  • Change: updated docs on using regsrv32 to fix MS DIA SDK registration issues

Tuesday, January 28, 2014

RAD Telemetry 2.0h released!

Telemetry 2.0h has been released!  This is a collection of small fixes/updates from December and through January.  Updated to support the latest console SDKs, implemented a bunch of customer requests, updated and clarified some documentation, etc.  This will hopefully be the basis for the latest 2.0 stable build so I can start working on 2.1.

NOTE: This requires a full rebuild!  The ABI has changed slightly to support the tmPlotAt functionality, so please remember to update your headers and libraries simultaneously and perform a full rebuild.

The only really significant gotcha is that TMZF_PROFILER_ONLY zones are now effectively discarded by the server, since they were a Telemetry 1.x specific feature.  A similar feature may be forthcoming in Telemetry 2 but because some of the constraints have changed (we no longer have monolithic frames like we did in Telemetry 1.x) it'll have to be designed a little differently.


  • Enhancement: added tmPlotAt APIs so you can retroactively place a plot in time.  For example, you might want to accumulate some information but then plot it back where it's most relevant, not where it's available.
  • Enhancement: updated to latest platform SDKs.
  • Bugfix: fixed XB1 checked build link errors
  • Change: 64-bit Linux static lib is now built with -fPIC


  • Enhancement: server now does periodic flushing every 100ms instead of only flushing on tmTick processing
  • Change: TMZF_PROFILER_ONLY zones are now explicitly discarded


  • Enhancement: user defined zone, timespan, and plot colors.  By putting "color: rrggbb" in your description the Visualizer will color accordingly.
  • Enhancement: added sort-by option in profiler results pane so you can sort by time or count
  • Enhancement: plot tooltip now shows time offset from start of session (so like T1 you can now see how far in a frame is, not just its duration)
  • Enhancement: TMZF_IDLE zones are now more obvious
  • Enhancement: 'show idle zones' checkbox is back
  • Enhancement: selected region in zone view will display its duration in tooltip
  • Enhancement: 1ms tick intervals are now drawn in the zone view
  • Bugfix: profiler pane now shows correct counts for accumulation zones


  • Change: updated docs to clarify that TMZF_PROFILER_ONLY is a T1 specific feature