Telemetry 1.1e Released!

Even though it's been only a couple weeks since our last release we wanted to get another big one out before the holidays. We've made a lot of little enhancements, added support for a new platform, and, best of all, have implemented a very handy feature -- off-line performance capture!

As long time Telemetry users know, Telemetry's architecture is based on streaming data over the network to a server which acquires, processes, stores, and eventually serves up the data. Unfortunately this isn't practical in certain situations. For example, a testing lab may not have network access for security reasons; some consoles have notoriously poor networking performance (even while wired); or a test machine may not have access to a Telemetry server.

To address these concerns we've implemented off-line data capture. The only change you have to make is a parameter to tmOpen -- specify TMCT_FILE instead of TMCT_TCP. Now instead of writing to the network Telemetry will write to disk. When done you take the "tmdata" files and then import them into the Visualizer (or play them back to the server). It's that simple, and in some cases can make a dramatic (5-10x) performance difference.

Run-Time

• Enhancement: Performance improvements for 360 and PS3
• Enhancement: Added TMPI_START_TIME parameter to tmGetPlatformInformation
• Enhancement: Win32: Changed context switch trace management code in an effort to be more robust across a wider range of systems
• Enhancement: Duration of TCP send now has a plot automatically generated
• Enhancement: Added more send buffers to increase performance and prevent stalls while writing to the network
• Enhancement: TMCT_FILE for local mirror of network traffic (i.e. disconnected captures are now possible!)
• Enhancement: BETA support for new platform
• Change: PS3: Telemetry thread now defaults to hwthread 1
• Change: 360: Telemetry thread now defaults to hwthread 5

Server

• Enhancement: Greatly enhanced performance under certain conditions with lots of frames
• Bugfix: Occasional server crashes when open zones spanned a very large number of frames
• Change: removed TM_APP_STRING and TM_APP_TICK console spam

Visualizer

• Enhancement: "Import Data Files" feature added (related to TMCT_FILE support)

Other

• Bugfix: Some documentation was using tmDynamicString improperly
• Enhancement: Cleaned up example files
• Enhancement: Added 'tmimport' command to playback raw net files captured with TMCT_FILE sessions

Telemetry 1.1d Released!

This is another maintenance release for Telemetry. Thanks to everyone who submitted bug reports and feature requests!

Runtime

• Bugfix: calling tmInitializeContext on an active context would result in random crashes. Now it returns an error with checked builds.
• Enhancement: (Experimental) TMZF_PLOT_TIME_EXPERIMENTAL. Specifying this will generate a plot of times for the given zone.
• Enhancement: (Experimental) You can now register your own debug display function for the run time using tmSetParameter with TMP_DEBUG_PRINTER
• Bugfix: Win32 build was generating a benign warning/error with AppVerifier due to attempting to OpenThread a NULL handle
• Bugfix: TMXGPU: passing a NULL context to tmxgpu would result in crashes
• Bugfix: Linux and OS X 32-bit: dynamic strings would sometimes crash the run time

Visualizer

• Bugfix: TMMF_ICON_* messages weren't showing up in the message window
• Bugfix: Multiple different blob plugins were not handled correctly
• Bugfix: Fixed weird UI issues where zone display would pick up incorrect mouse events
• Bugfix: OS X: Ok/Cancel buttons in message box were reversed
• Bugfix: D3D: Too many plot samples crashed Visualizer
• Change: OS X: Defaults to 1280x800 window size

Server

• Bugfix: Fixed a couple causes of the 'hanshake times out' issue
• Bugfix: Fixed some server instability
• Bugfix: Blobs were not served up properly
• Enhancement: 32-bit Linux version available now

Other

• Bugfix: Updated UE3 integration docs
• Bugfix: Fixed missing documentation for tmsRegisterMessageHandler, tmPPURegisterSPUProgram and TmOpenFlag
• Enhancement: Added embedded server tutorial code

Telemetry 1.1c Released!!

This is a maintenance fix for Telemetry 1.1. As always, update all components in tandem!

Runtime

• Change: TMXERR_GPU_INVALID_CONTEXT is now returned by the tmxGpuInit functions if a NULL context is passed
• Change: Fixed spurious warnings when trying to close a NULL socket (now silently ignored)
• Bugfix: Unused parameter warning in tmxgpu
• Bugfix: Dynamic strings were not working properly on 32-bit OS X run time
• Bugfix: tmSendCallStack did not send proper data when given a callstack previously fetched via tmGetCallStack

Visualizer

• Changes: Zones are no longer arbitrarily split across frame boundaries
• Bugfix: Context switch rendering on 360 regression for 1.1
• Bugfix: Newlines were not always handled correctly in license file leading to "Corrupt Visualizer license" messages
• Bugfix: Minimize-then-maximize to black screen
• Bugfix: Visualizer's spawned editor is now found in path instead of requiring a full path name
• Bugfix: TMMF_ZONE_LABEL was inadvertently replacing labels even on aggregate zones (e.g. LOD zones)
• Bugfix: thin zones were incorrectly colored as LOD zones
• Bugfix: Visualizer plugins were occasionally not found at startup

Server

• Bugfix: Unbalanced leaves with enters were causing intermittent server crashes

Other

• Bugfix: Symbolizer bugs fixed
• Enhancement: Symbolizer now works correctly on OS X
• Enhancement: Additional platform support for symbolizer
• Change: UE3 integration docs updated

Telemetry 1.1a Released!!

Wow, been a long time coming, but this is a massive update that has implemented a ton of features and new platform support!

YOU WILL HAVE TO UPDATE ALL COMPONENTS AND REBUILD YOUR GAMES!! The protocols have changed to support our new features, so make sure you rebuild, relink, update your DLLs and run the new server and Visualizer.

SPU Support

So the big feature people have been waiting for is support for the PS3 SPU on Telemetry, and this required a significant amount of architectural work (since it effectively meant multiple processes), but it's done! It's a standard part of the Telemetry/PS3 distribution now.

OS X Support

That's right, we have OS X support, both 32-bit and 64-bit run-times, along with the Visualizer and command line server! The run-time is part of Telemetry/Mac but the Visualizer and server are available to licensees of any platform.

GPU Support

We now have support for benchmarking GPU calls on XBOX 360, PS3 and Direct3D (DX9,DX10, and DX11). We provide this as a layer on top of Telemetry's timespan features, and we support both synchronous and asynchronous operation.

Visualizer/D3D

The Visualizer now uses Direct3D on Windows instead of OpenGL. The OpenGL version is still available and will continue to be maintained (it's viz64gl.exe in the distribution). Using Direct3D should reduce some of the compatibility problems customers have seen with different video cards and drivers.

Linux Support

We now have support for 32-bit and 64-bit Linux run-times on Intel architectures, along with a command line server for Linux.

Additional changes are as follows:

Run-Time

• Change: tmSetParameter now takes a pointer to parameter instead of the parameter itself


• Change: PS3 library has been renamed to libtelemetryPS3.a


• Change: Linux static library now called libTelemetryX86.link.a or libTelemetryX64.link.a


• Bugfix: calling tmOpen after tmClose didn't work properly


• Bugfix: PS3 network connection issues


• Bugfix: NTELEMETRY was masking connection status constants


• Bugfix: Time wrapping on Linux wasn't handled properly

Visualizer

• Enhancement: Number of zones in a thread now displayed as part of tooltip over trackhead


• Change: Implicitly generated plots for internal Telemetry events (number of allocs, number of locks, etc.) removed


• Change: Timespans generate virtual Telemetry messages


• Change: Profiler rows are reasier to see


• Change: Visualizer input buffer is now larger, helping with extreme data flooding cases


• Change: Timespans now go to microsecond display if < 0.5ms in duration


• Bugfix: race condition that would lead to occasional crash


• Bugfix: Some stall zones were not being rendered properly


• Bugfix: Timespan rendering was broken in some cases


• Bugfix: Annotation handling resulted in loss of annotations if they were numeric


• Bugfix: Zone depth computations were inaccurate in some cases


• Bugfix: Timeline state background colors are stable


• Bugfix: Fragmented input data packets would occasionally cause a hang


• Bugfix: Out of range 64-bit plot values would cause a crash


• Bugfix: High CPU utilization in data processing and server connection threads in some situations

Server

• Enhancemenent: added -keeptmp option for testing


• Enhancement: Added offline processing of previousy stored tmp files


• Bugfix: Minor memory leak


• Bugfix: Server would occasionally resend duplicate lock events in certain flooding conditions


• Change: No longer limited to 64K top level zones per frame


• Change: Server buffers frame data instead of just the most recent frame


• Change: You can now register your own message handler for warning dialogs, etc. with the tms server library

Telemetry 1.0n Released!!

While working on the SPU implementation of Telemetry we realized that the amount of required server work would be substantial. Instead of holding off on a bunch of smaller fixes and changes while we finished that we decided to temporarily pause SPU development and give you guys a really solid interim release before we dive back in.

Unfortunately because of the tweaks this requires another full upgrade -- you must upgrade your run-time, server and Visualizer simultaneously! You'll also want to do a full rebuild since some function signatures and constants may have changed.

The big changes are crash fixes for the server; major UI improvements to the Visualizer (including fixing some bugs with timespan rendering); and the embedded server library is now available in beta form. But there are a ton of other things in here too, so I encourage you to read the change list. Probably 50% or more of these changes were due to direct customer feedback, so please keep us up to date on problems, questions, suggestions or just general comments!

Run-Time

• Change: PROTOCOL CHANGED! Make sure you are using only 1.0n components in your Telemetry chain (server, Visualizer and run-time)
• Change: 360 now sends over profile-0 username as session username
• Bugfix: Internal stall zones that result from flushing the Telemetry buffer are now placed in the correct place in the timeline so that they don't look like displaced child zones. They now look like siblings of the calling zone that forced the flush.
• Change: Reduced string table max size to 16K sdtrings -- please let me know if this causes problems. We're trying to get the minimum context size down.
• Bugfix: Win32: No longer search symbol path for PDBs at startup. This was causing weird hiccups/hangs/stalls for some customers with symbol server paths but without a valid symbol server running.
• Bugfix: NTELEMETRY versions of tmPlotI32 and tmPlotI64 were missing. NTELEMETRY version of tmGetAccumulationStart was also incorrect.
• Enhancement: Command line and parameters are now stored, sent and displayed by the Visualizer as part of session information.

Visualizer

• Enhancement: Cut-and-paste the active tooltip by holding down control as the tooltip pops up. This is a stop gap measure until we get global cut-and-paste for all UI elements. You must enable this first in Options.
• Change: Protocol changed! Make sure you update run-time, server and Visualizer simultaneously!
• Enhancement: MAJOR enhancements to the level of detail rendering of small zones and groups of small zones!! Zones should never LOD away completely; they now look better and are a different color when it's an LOD aggregated zone; you should see a tall 'stalactite' if there is even one tiny child zone deep in a hierarchy; top-level zones now LOD properly all the time; LOD aggregate zones display actual summed child times instead of LOD aggregate displayed time; LOD aggregate zones only display zone name if it's the same zone name (otherwise displays "").
• Enhancement: User messages (tmMessage) are now put in a separate tree branch next to Telemetry messages instead of using a separate checkbox.
• Enhancement: Tooltips now render callstacks vertically.
• Enhancement: Session list now shows numberr of frames in the session
• Change: Session info at bottom of screen now in 12H format instead of 24H format
• Enhancement: Choose microseconds or cycles as display unit for short duration zones.
• Enhancement: Tree views now have a 'half-on' checkbox state to indicate a mix of checked/unchecked children
• Enhancement: Zones that contain message text now display with a letter 'M'
• Enhancement: Generated plots now have an asterisk in their name and can now also be removed.
• Enhancement: Right-click on track head has a "Minimize all but this track" option
• Enhancement: Dynamic tick mark spacing along top of zone view
• Enhancement: Persistent outlining of zones supported, i.e. you can do right-click "Highlight all these zones"
• Change: Zoom is now clamped to 10K clock cycles.
• Bugfix: Minimized tracks that have no data (e.g. lock track if you haven't record any lock events) are no longer displayed
• Bugfix: timeline bars don't disappear no matter how thin they get
• Bugfix: fixed spacing and rendering problems with timespans

Server

• Change: Protocol versions changed!! Make sure you are using 1.0n versions of server, Visualizer and run-time simultaneously!
• Enhancement: Separate server lib is now available (tmsserver*.dll) on Win32. An example program is also supplied.
• Bugfix: Server now puts up message boxes in case of fatal session errors.
• Bugfix: On overflow (too many zones) the server is more aggressive killing the offending session
• Bugfix: More exception handling wrapper to prevent crashes
• Enhancement: Tmp files are only flushed/written if there is incoming data, so a paused (e.g. breakpointed) connection not keep generating a bunch of little files

Telemetry 1.0M released!

This is mostly a maintenance release to get some bug fixes, polish, etc. out there.

This WILL require another full rebuild! Sorry about the inconvenience (again), the APIs have been tweaked to be a little friendlier, specifically the tmEntryEx and tmTryLockEx APIs take a pointer to a matchid instead of a reference.

• tmEnterEx and tmTryLockEx have slightly different signatures
• TMZF_FILTER has been removed since it's no longer necessary
• callstack propagation bug fixed when callstacks were only 1 deep
• Visualizer crash bug fixed if a session ended unexpectedly while the Visualizer was trying to load it
• Track minimization state is now persistent between sessions -- no more constantly minimizing unwanted tracks!
• Currently loaded session is highlighted in the session selector
• Some server exception handling propagation issues fixed
• Memory utilization in the Visualizer reduced considerably for lock events
• Very simple text-only memory reporting available by right clicking on a timeline bar or a memory event. This is just to show some of the data we're acquiring and processing as we move towards memory management feature enhancement.

RAD Telemetry 1.0L Released!

Lots of changes big and small. Hit us up if you want access to the update!

WARNING: You must do a full rebuild with this version since some APIs have changed!

The big changes are that you can now filter zones out on the client based on a time threshold specified in tmEnterEx or tmZoneFiltered. In addition, tmTryLockEx has a time threshold for the same reason.

Unfortunately the overhead for boosting priority in an attempt to solve some customer's priority inversion problems was too high, so we've reverted to not boosting priority by default, but you can still enable it with TMO_BOOST_PRIORITY in tmSetOption.

Linux had two significant bug fixes leading to either server crashes or corrupted plot values, those should be fixed now as well.

Other changes:

• (Linux) renamed static lib to libtelemetry32.a so you can do "-ltelemetry32" link flags
• tmLeaveThread and tmEnterThread have been removed -- the same functionality is available via tmEnterEx and tmLeaveEx
• tmEnterEx and tmLeaveEx have new parameters, so if you use those there's a chance that you'll have to update your code in some places for it to compile
• timestamps are sampled in slightly different places than before so that the overhead of Telemetry isn't misattributed to your code
• (Windows) PDB names weren't being properly sent to the server
• Visualizer: you can now delete sessions from within the Visualizer!
• Visualizer: if there are no memory events then the mem event track will not appear
• Visualizer: timeline backgrounds have significantly more contrast with the new color scheme
• Visualizer: default generated plot window size is now 1s instead of 100ms
• Visualizer: lock events were not always reaching the Visualizer during live captures
• Visualizer: filename wasn't being quoted in the "%F" portion of the editor command line option
• Visualizer: we now specify /edit by default in the editor command line
• Examples: example_basic_threads.cpp updated to handle Linux better
• Server: massive performance improvements while writing frames to disk

As always, your comments, questions, and bug reports are greatly appreciated!

Telemetry 1.0k released! Linux support!

The big news with 1.0k is that we now have beta support for 32-bit Linux (x86) as a run-time platform. If you've wanted to check out Telemetry but didn't because we didn't support Linux, drop us a line and we'll get you hooked up with a trial!

Another major change we did was a partial fix for priority inversion issues on platforms that don't have schedulers that can detect that. Priority inversion is a nasty deadlock that can occur because of inefficient scheduling -- a common case is where a low priority work thread grabs the Telemetry lock via an API call but is then switched away. At that point if multiple higher priority threads take over and then never relinquish control back to the lower priority thread, while also attempting to enter Telemetry, they'll hang. A worst case situation might be a time critical thread that spin waits (i.e. never yields) on others while also trying to enter Telemetry, thus never letting the kernel switch to a lower priority thread.

We have a fix for this that handles many cases, but it's still possible to deadlock on Telemetry if your threading code is misbehaved. Consider this a benefit of Telemetry -- if you're deadlocking Telemetry then that's likely showing a latent bug in your threading implementation if you share resources between different priority threads!

Note that you'll have to recompile your Telemetry enabled applications with this build since some constants have changed.

Some other changes:

• Bugfix: tmEnterEx declaration was wrong, resulting in compile time error


• Bugfix: context switch thread would occasionally crash when shutting down


• Change: maximum zone depth is now 32 in the Visualizer


• Change: changed color scheme a bit in Visualizer


• Bugfix: by-frame % calculations were wrong in some cases in profiler view in Visualizer


• Enhancement: highlighting a plot now highlights its label as well


• Enhancement: you can invert sense of mousewheel vs. control-mousewheel


• Bugfix: fixed some server crash issues


• Enhancement: updated documentation

RAD Telemetry 1.0j shipped!

Telemetry 1.0j is out the door! The big news, of course, is that there is alpha support for PS3 (PPU only) right now. If you're a PS3 developer please check it out and hammer on it, we need some data and use cases from the field.

Here are some of the major changes:

• PS3 support (PPU only)



• direct va_list support in all the vararg APIs. This was heavily requested and now it's in there!



• race conditions/crashes in the run-time when the server went away unexpectedly should be reduced



• added tmEndTryLockEx (just like tmEndTryLock but with explicit file/line)



• new APIs! tmEnterThread and tmLeaveThread which allow you to specify the thread for the zone. This was crucial for 'logical thread' support (e.g. fibers, cooperative threads, etc.)



• new APIs #2! tmBeginTimeSpanAt and tmEndTimeSpanAt allow you to specify the timestamps for timespan events, so you can time an external event and insert it retroactively



• 360/Windows: significant performance boost in the context switch thread ("telemetry:ctxs")



• TMOF_DEFAULT turns off context switches by default on Windows -- the overhead was just too high for users that didn't need it

The Visualizer got some love too:

• Filter session lists by name and computer! This was requested quite a bit for customers had thousands of sessions to sift through.



• Massive memory usage reduction for messages



• Plot and memory events generate virtual messages in the message window (with a different color, and you can even now filter out all user messages if you want)



• A bug with lock rendering was causing massive slow downs in certain special cases, fixed!



• Localhost option has been removed since it was confusing and not very useful



• Text entry fields should be normal now



• Plots no longer render over the slider in the mini-timeline view



• Clicking and dragging on a context switch span should still drag/scroll



• Minimized timespan track would disappear



• Paused frames are no rendered with a different color in the timeline view



• Highlighting a context switch or lock event span highlights the appropriate subsection of a zone in the zone view



• Timestamps are now inserted into the log entries when you use "Copy/Save To File"

And the server had some minor changes:

• Bugfix: callstack processing wasn't processing last entry all the time



• Change: -localhost option removed since it wasn't useful and confused people most of the time



• Crash dump generator added, along with -minidump option to disable full memory crash dumps

Arena/heap memory APIs

Telemetry currently has a simple API to let you mark memory allocations and frees (tmAlloc and tmFree respectively). You can specify a hierarchical path-like structure to categorize the type of allocations, e.g.: tmAlloc( cx, ptr, size, "/meshes/vehicles/supertank" ); A commonly requested feature is the ability to handle different heaps/allocators/arenas such as:

• arenas that are freed in one shot (so you don't have to call tmFree on all the suballocations)


• handle or index based allocators with a base and range, resulting in overlapping 'address spaces'


• small allocators for strings, particles, and small temporaries

And there are many other examples I'm sure, which is why I'm posting here -- what are your needs in terms of expressive memory allocation tracking? Drop me a line (or post here) at telemetry@radgametools.com since I'd like to tackle this right after the PS3/PPU release!

Plot enhancements

One of the most powerful elements of Telemetry is the ability to plot arbitrary game data and see how it correlates to performance. While the plots are great already, we do think that they can and should be better and have been talking to customers about what new features they'd like to see.

Some features up for discussion and possible implementation in the coming weeks and months include:

• discrete plot events. Instead of just assuming that plot events are somehow related, handle each sample as a discrete, independent event, generating a scatter plot instead of a line chart.
• plot limits/budgets. Set plot min/max values that generate flashing text, warning colors, etc. on the Visualizer if those limits are broken. Hovering over the plot would draw horizontal lines showing the budget(s) set. This way you can set resource budgets for your artists and they'll be able to spot them easily.
• Visual enhancements such as better contrast/rendering.

So what are your thoughts? Feel free to post here or drop me an email!

Telemetry/PS3/PPU is up

Telemetry now passes its basic tests on PS3/PPU. It may take a while to achieve feature parity with 360 (e.g. context switches and thread cycle times) but the important thing is that it will at least hopefully be available relatively soon. Now we have to address performance and features.

More message improvements

Minor things, but every little bit helps:

• memory and plot events generate virtual messages in the message window in the generated message group "telemetry/events/XXX". It's a quick and easy way to get a dump of plot and memory events until we get the export feature done.
• virtual messages are colored so they're easy to tell apart from other messages
• virtual messages can be filtered away just like other messages
• user messages (via tmMessage) can be masked off as well with a new checkbox
• the "Warnings" and "Error" checkbox labels are color coded
• "Copy/Save To File" messages now have timestamps prepended

We'll get search/filtering done "soon" once we figure out the ideal UI for it (some people want to search, others want incremental filtering, etc.)

tmEnterThread and tmBeginTimeSpanAt APIs added!

By request from a couple customers we've added two variants of existing APIs. tmEnterThread/tmLeaveThread let you manually specify your thread IDs, thus giving support for logical threads. This is handy if you have a single thread for something like AI but you want to visualize different subtasks/coroutines/fibers individually.

The other API addition is the ability to pass a specific time value to tmBeginTimeSpan and tmEndTimeSpan via the tmBegin/EndTimeSpanAt variants. Now you can measure an event indirectly, such as a GPU fence, and see it on the timeline.

Road map for Telemetry

One of the advantages of RAD Game Tools is that we're highly customer responsive and have a very aggressive release schedule. New features, optimizations, and bug fixes are rolled out at a very fast rate, usually at least once a month and often times a couple times a month.

After GDC we got a lot of good information from new and potential customers, and based on their feedback I wanted to mention what our tentative road map is looking like.

• Summer: PS3 support for PPU. We may have a beta of this as soon as April, but no promises. It's our highest priority major feature.
• Late Summer: basic GPU support on Windows and 360, at the very least allowing you to identify how long a command buffer took to execute and to visualize overlap with your CPU threads.
• Early Fall: PS3 support for SPU. The major issue is that all of our cool features do require space on the SPU, and most PS3 developers don't want to give up any room on their SPUs!
• Fall: deep memory visualization. The current memory stuff is very simple and just tracks allocations and frees, but we want to go crazy and do for memory everything we've done for performance. Fragmentation maps, current usage tree maps, multiheap support, and more!
• Winter: significant architectural changes on the back end with the goal being to remove the entire concept of hard coded frames. Instead we'd like you to be able to define arbitrary 'framesets' so you can refactor the zone view however you'd like, by 'render ticks', 'game ticks', 'physics ticks', etc. Getting this in place opens a huge host of opportunities.
• Winter+: continuous integration, regression testing, and visual diff support

Of course, during all this we'll be implementing lots of small features, polish items, customer requests, bug fixes, and optimizations, and if a new platform gets a lot of requests we'll support it as well. But barring stuff like that this is our current road map if anything changes I'll try to post about it here.

1.0i has shipped!

Telemetry 1.0i is out the door with a host of fixes and enhancements:

• new plots generates automatically showing number of context switches and current telemetry buffer size
• context switch background processing optimized significantly on Windows
• networking back end much faster and trickles data, avoiding network backups under extreme frame rate (500fps+) situations

The Visualizer also got some improvements:

• minimized tracks now stack vertically if they run out of room horizontally
• message window revamped so that it's faster and better looking (can handle millions of messages now)
• significant speed up to lock rendering
• fixed crash bug for certain windowed generated plots
• fixed memory corruption error if there were more than 32K plot points in a single plot
• non-idle, non-system processes in the context switch track are now rendered in light gray so you can tell them apart from 'system idle'