Telemetry 1.1d Released!

This is another maintenance release for Telemetry. Thanks to everyone who submitted bug reports and feature requests!

Runtime

• Bugfix: calling tmInitializeContext on an active context would result in random crashes. Now it returns an error with checked builds.
• Enhancement: (Experimental) TMZF_PLOT_TIME_EXPERIMENTAL. Specifying this will generate a plot of times for the given zone.
• Enhancement: (Experimental) You can now register your own debug display function for the run time using tmSetParameter with TMP_DEBUG_PRINTER
• Bugfix: Win32 build was generating a benign warning/error with AppVerifier due to attempting to OpenThread a NULL handle
• Bugfix: TMXGPU: passing a NULL context to tmxgpu would result in crashes
• Bugfix: Linux and OS X 32-bit: dynamic strings would sometimes crash the run time

Visualizer

• Bugfix: TMMF_ICON_* messages weren't showing up in the message window
• Bugfix: Multiple different blob plugins were not handled correctly
• Bugfix: Fixed weird UI issues where zone display would pick up incorrect mouse events
• Bugfix: OS X: Ok/Cancel buttons in message box were reversed
• Bugfix: D3D: Too many plot samples crashed Visualizer
• Change: OS X: Defaults to 1280x800 window size

Server

• Bugfix: Fixed a couple causes of the 'hanshake times out' issue
• Bugfix: Fixed some server instability
• Bugfix: Blobs were not served up properly
• Enhancement: 32-bit Linux version available now

Other

• Bugfix: Updated UE3 integration docs
• Bugfix: Fixed missing documentation for tmsRegisterMessageHandler, tmPPURegisterSPUProgram and TmOpenFlag
• Enhancement: Added embedded server tutorial code

Telemetry 1.1c Released!!

This is a maintenance fix for Telemetry 1.1. As always, update all components in tandem!

Runtime

• Change: TMXERR_GPU_INVALID_CONTEXT is now returned by the tmxGpuInit functions if a NULL context is passed
• Change: Fixed spurious warnings when trying to close a NULL socket (now silently ignored)
• Bugfix: Unused parameter warning in tmxgpu
• Bugfix: Dynamic strings were not working properly on 32-bit OS X run time
• Bugfix: tmSendCallStack did not send proper data when given a callstack previously fetched via tmGetCallStack

Visualizer

• Changes: Zones are no longer arbitrarily split across frame boundaries
• Bugfix: Context switch rendering on 360 regression for 1.1
• Bugfix: Newlines were not always handled correctly in license file leading to "Corrupt Visualizer license" messages
• Bugfix: Minimize-then-maximize to black screen
• Bugfix: Visualizer's spawned editor is now found in path instead of requiring a full path name
• Bugfix: TMMF_ZONE_LABEL was inadvertently replacing labels even on aggregate zones (e.g. LOD zones)
• Bugfix: thin zones were incorrectly colored as LOD zones
• Bugfix: Visualizer plugins were occasionally not found at startup

Server

• Bugfix: Unbalanced leaves with enters were causing intermittent server crashes

Other

• Bugfix: Symbolizer bugs fixed
• Enhancement: Symbolizer now works correctly on OS X
• Enhancement: Additional platform support for symbolizer
• Change: UE3 integration docs updated

Telemetry 1.1a Released!!

Wow, been a long time coming, but this is a massive update that has implemented a ton of features and new platform support!

YOU WILL HAVE TO UPDATE ALL COMPONENTS AND REBUILD YOUR GAMES!! The protocols have changed to support our new features, so make sure you rebuild, relink, update your DLLs and run the new server and Visualizer.

SPU Support

So the big feature people have been waiting for is support for the PS3 SPU on Telemetry, and this required a significant amount of architectural work (since it effectively meant multiple processes), but it's done! It's a standard part of the Telemetry/PS3 distribution now.

OS X Support

That's right, we have OS X support, both 32-bit and 64-bit run-times, along with the Visualizer and command line server! The run-time is part of Telemetry/Mac but the Visualizer and server are available to licensees of any platform.

GPU Support

We now have support for benchmarking GPU calls on XBOX 360, PS3 and Direct3D (DX9,DX10, and DX11). We provide this as a layer on top of Telemetry's timespan features, and we support both synchronous and asynchronous operation.

Visualizer/D3D

The Visualizer now uses Direct3D on Windows instead of OpenGL. The OpenGL version is still available and will continue to be maintained (it's viz64gl.exe in the distribution). Using Direct3D should reduce some of the compatibility problems customers have seen with different video cards and drivers.

Linux Support

We now have support for 32-bit and 64-bit Linux run-times on Intel architectures, along with a command line server for Linux.

Additional changes are as follows:

Run-Time

• Change: tmSetParameter now takes a pointer to parameter instead of the parameter itself


• Change: PS3 library has been renamed to libtelemetryPS3.a


• Change: Linux static library now called libTelemetryX86.link.a or libTelemetryX64.link.a


• Bugfix: calling tmOpen after tmClose didn't work properly


• Bugfix: PS3 network connection issues


• Bugfix: NTELEMETRY was masking connection status constants


• Bugfix: Time wrapping on Linux wasn't handled properly

Visualizer

• Enhancement: Number of zones in a thread now displayed as part of tooltip over trackhead


• Change: Implicitly generated plots for internal Telemetry events (number of allocs, number of locks, etc.) removed


• Change: Timespans generate virtual Telemetry messages


• Change: Profiler rows are reasier to see


• Change: Visualizer input buffer is now larger, helping with extreme data flooding cases


• Change: Timespans now go to microsecond display if < 0.5ms in duration


• Bugfix: race condition that would lead to occasional crash


• Bugfix: Some stall zones were not being rendered properly


• Bugfix: Timespan rendering was broken in some cases


• Bugfix: Annotation handling resulted in loss of annotations if they were numeric


• Bugfix: Zone depth computations were inaccurate in some cases


• Bugfix: Timeline state background colors are stable


• Bugfix: Fragmented input data packets would occasionally cause a hang


• Bugfix: Out of range 64-bit plot values would cause a crash


• Bugfix: High CPU utilization in data processing and server connection threads in some situations

Server

• Enhancemenent: added -keeptmp option for testing


• Enhancement: Added offline processing of previousy stored tmp files


• Bugfix: Minor memory leak


• Bugfix: Server would occasionally resend duplicate lock events in certain flooding conditions


• Change: No longer limited to 64K top level zones per frame


• Change: Server buffers frame data instead of just the most recent frame


• Change: You can now register your own message handler for warning dialogs, etc. with the tms server library

Telemetry 1.0n Released!!

While working on the SPU implementation of Telemetry we realized that the amount of required server work would be substantial. Instead of holding off on a bunch of smaller fixes and changes while we finished that we decided to temporarily pause SPU development and give you guys a really solid interim release before we dive back in.

Unfortunately because of the tweaks this requires another full upgrade -- you must upgrade your run-time, server and Visualizer simultaneously! You'll also want to do a full rebuild since some function signatures and constants may have changed.

The big changes are crash fixes for the server; major UI improvements to the Visualizer (including fixing some bugs with timespan rendering); and the embedded server library is now available in beta form. But there are a ton of other things in here too, so I encourage you to read the change list. Probably 50% or more of these changes were due to direct customer feedback, so please keep us up to date on problems, questions, suggestions or just general comments!

Run-Time

• Change: PROTOCOL CHANGED! Make sure you are using only 1.0n components in your Telemetry chain (server, Visualizer and run-time)
• Change: 360 now sends over profile-0 username as session username
• Bugfix: Internal stall zones that result from flushing the Telemetry buffer are now placed in the correct place in the timeline so that they don't look like displaced child zones. They now look like siblings of the calling zone that forced the flush.
• Change: Reduced string table max size to 16K sdtrings -- please let me know if this causes problems. We're trying to get the minimum context size down.
• Bugfix: Win32: No longer search symbol path for PDBs at startup. This was causing weird hiccups/hangs/stalls for some customers with symbol server paths but without a valid symbol server running.
• Bugfix: NTELEMETRY versions of tmPlotI32 and tmPlotI64 were missing. NTELEMETRY version of tmGetAccumulationStart was also incorrect.
• Enhancement: Command line and parameters are now stored, sent and displayed by the Visualizer as part of session information.

Visualizer

• Enhancement: Cut-and-paste the active tooltip by holding down control as the tooltip pops up. This is a stop gap measure until we get global cut-and-paste for all UI elements. You must enable this first in Options.
• Change: Protocol changed! Make sure you update run-time, server and Visualizer simultaneously!
• Enhancement: MAJOR enhancements to the level of detail rendering of small zones and groups of small zones!! Zones should never LOD away completely; they now look better and are a different color when it's an LOD aggregated zone; you should see a tall 'stalactite' if there is even one tiny child zone deep in a hierarchy; top-level zones now LOD properly all the time; LOD aggregate zones display actual summed child times instead of LOD aggregate displayed time; LOD aggregate zones only display zone name if it's the same zone name (otherwise displays "").
• Enhancement: User messages (tmMessage) are now put in a separate tree branch next to Telemetry messages instead of using a separate checkbox.
• Enhancement: Tooltips now render callstacks vertically.
• Enhancement: Session list now shows numberr of frames in the session
• Change: Session info at bottom of screen now in 12H format instead of 24H format
• Enhancement: Choose microseconds or cycles as display unit for short duration zones.
• Enhancement: Tree views now have a 'half-on' checkbox state to indicate a mix of checked/unchecked children
• Enhancement: Zones that contain message text now display with a letter 'M'
• Enhancement: Generated plots now have an asterisk in their name and can now also be removed.
• Enhancement: Right-click on track head has a "Minimize all but this track" option
• Enhancement: Dynamic tick mark spacing along top of zone view
• Enhancement: Persistent outlining of zones supported, i.e. you can do right-click "Highlight all these zones"
• Change: Zoom is now clamped to 10K clock cycles.
• Bugfix: Minimized tracks that have no data (e.g. lock track if you haven't record any lock events) are no longer displayed
• Bugfix: timeline bars don't disappear no matter how thin they get
• Bugfix: fixed spacing and rendering problems with timespans

Server

• Change: Protocol versions changed!! Make sure you are using 1.0n versions of server, Visualizer and run-time simultaneously!
• Enhancement: Separate server lib is now available (tmsserver*.dll) on Win32. An example program is also supplied.
• Bugfix: Server now puts up message boxes in case of fatal session errors.
• Bugfix: On overflow (too many zones) the server is more aggressive killing the offending session
• Bugfix: More exception handling wrapper to prevent crashes
• Enhancement: Tmp files are only flushed/written if there is incoming data, so a paused (e.g. breakpointed) connection not keep generating a bunch of little files

Telemetry 1.0M released!

This is mostly a maintenance release to get some bug fixes, polish, etc. out there.

This WILL require another full rebuild! Sorry about the inconvenience (again), the APIs have been tweaked to be a little friendlier, specifically the tmEntryEx and tmTryLockEx APIs take a pointer to a matchid instead of a reference.

• tmEnterEx and tmTryLockEx have slightly different signatures
• TMZF_FILTER has been removed since it's no longer necessary
• callstack propagation bug fixed when callstacks were only 1 deep
• Visualizer crash bug fixed if a session ended unexpectedly while the Visualizer was trying to load it
• Track minimization state is now persistent between sessions -- no more constantly minimizing unwanted tracks!
• Currently loaded session is highlighted in the session selector
• Some server exception handling propagation issues fixed
• Memory utilization in the Visualizer reduced considerably for lock events
• Very simple text-only memory reporting available by right clicking on a timeline bar or a memory event. This is just to show some of the data we're acquiring and processing as we move towards memory management feature enhancement.

RAD Telemetry 1.0L Released!

Lots of changes big and small. Hit us up if you want access to the update!

WARNING: You must do a full rebuild with this version since some APIs have changed!

The big changes are that you can now filter zones out on the client based on a time threshold specified in tmEnterEx or tmZoneFiltered. In addition, tmTryLockEx has a time threshold for the same reason.

Unfortunately the overhead for boosting priority in an attempt to solve some customer's priority inversion problems was too high, so we've reverted to not boosting priority by default, but you can still enable it with TMO_BOOST_PRIORITY in tmSetOption.

Linux had two significant bug fixes leading to either server crashes or corrupted plot values, those should be fixed now as well.

Other changes:

• (Linux) renamed static lib to libtelemetry32.a so you can do "-ltelemetry32" link flags
• tmLeaveThread and tmEnterThread have been removed -- the same functionality is available via tmEnterEx and tmLeaveEx
• tmEnterEx and tmLeaveEx have new parameters, so if you use those there's a chance that you'll have to update your code in some places for it to compile
• timestamps are sampled in slightly different places than before so that the overhead of Telemetry isn't misattributed to your code
• (Windows) PDB names weren't being properly sent to the server
• Visualizer: you can now delete sessions from within the Visualizer!
• Visualizer: if there are no memory events then the mem event track will not appear
• Visualizer: timeline backgrounds have significantly more contrast with the new color scheme
• Visualizer: default generated plot window size is now 1s instead of 100ms
• Visualizer: lock events were not always reaching the Visualizer during live captures
• Visualizer: filename wasn't being quoted in the "%F" portion of the editor command line option
• Visualizer: we now specify /edit by default in the editor command line
• Examples: example_basic_threads.cpp updated to handle Linux better
• Server: massive performance improvements while writing frames to disk

As always, your comments, questions, and bug reports are greatly appreciated!

Telemetry 1.0k released! Linux support!

The big news with 1.0k is that we now have beta support for 32-bit Linux (x86) as a run-time platform. If you've wanted to check out Telemetry but didn't because we didn't support Linux, drop us a line and we'll get you hooked up with a trial!

Another major change we did was a partial fix for priority inversion issues on platforms that don't have schedulers that can detect that. Priority inversion is a nasty deadlock that can occur because of inefficient scheduling -- a common case is where a low priority work thread grabs the Telemetry lock via an API call but is then switched away. At that point if multiple higher priority threads take over and then never relinquish control back to the lower priority thread, while also attempting to enter Telemetry, they'll hang. A worst case situation might be a time critical thread that spin waits (i.e. never yields) on others while also trying to enter Telemetry, thus never letting the kernel switch to a lower priority thread.

We have a fix for this that handles many cases, but it's still possible to deadlock on Telemetry if your threading code is misbehaved. Consider this a benefit of Telemetry -- if you're deadlocking Telemetry then that's likely showing a latent bug in your threading implementation if you share resources between different priority threads!

Note that you'll have to recompile your Telemetry enabled applications with this build since some constants have changed.

Some other changes:

• Bugfix: tmEnterEx declaration was wrong, resulting in compile time error


• Bugfix: context switch thread would occasionally crash when shutting down


• Change: maximum zone depth is now 32 in the Visualizer


• Change: changed color scheme a bit in Visualizer


• Bugfix: by-frame % calculations were wrong in some cases in profiler view in Visualizer


• Enhancement: highlighting a plot now highlights its label as well


• Enhancement: you can invert sense of mousewheel vs. control-mousewheel


• Bugfix: fixed some server crash issues


• Enhancement: updated documentation