Hi, my name is Tong Wynn. I am a developer on the Remote Desktop Virtualization (RDV) team. Our team is responsible for developing Microsoft RemoteFX in the Windows Server 2012 and Windows 8 release .
Included in RemoteFX are many significant improvements in graphics remoting to support a fast and fluid user experience in a remote session. A key change was to redesign the graphics remoting stack from the ground up to adapt to various runtime conditions (such as graphics content types, CPU and network bandwidth availability, and client rendering speed). In this blog, we will discuss in more detail the remoting of adaptive graphics in RemoteFX.
Host-side rendering
In Windows Server 2012 and Windows 8, the full local desktop is composed and rendered on the host in a remote session before the resulting image is encoded and sent by RemoteFX. In other words, RemoteFX is based on host-side rendering. As a result, the full desktop experience of the Windows 8 desktop as well as all application user interfaces (UIs) is consistently sent remotely at the highest quality with 32 bits per pixel (BPP). To balance user experience with encoding and bandwidth cost, the target frame rate in a remote session is 30 frames per second (FPS).
Compared to the approach of remoting part of the graphics as primitives and rendering at the client side, host-side rendering provides a more consistent user experience to a much wider range of clients (for example, it supports thin clients as well as rich clients). It also provides the flexibility to be more adaptive to runtime conditions to provide a better balance between responsiveness, scalability, image quality, and bandwidth consumption.
Windows Server 2012 and Windows 8 desktop composition is optimized to minimize performance impacts of host-side rendering. In addition, RemoteFX uses many techniques for generating what was previously provided by primitives in earlier versions (such as detection of screen region changes and motion, and advanced caching). Other innovations further improve bitmap encoding to be adaptive to runtime conditions and improve the end-to-end user experience. The following sections will discuss what’s new in adaptive graphics for RemoteFX.
Bitmap encoding adapted to content type
One important aspect of adaptive graphics in RemoteFX is that we leverage multiple bitmap codecs that are tuned to be very effective at encoding specific content types.
In previous versions of Remote Desktop Protocol (RDP), after a remote session was established, only one codec (the best codec for the clientserver capabilities) was used to encode all bitmaps.
In RemoteFX, several new codecs are supported, and we detect the content type at runtime to choose the best codec to encode different parts of the frame. RemoteFX differentiates the following types of content at runtime: text, synthetic image, natural image, and video. Each type of content is encoded with a dedicated codec that is tuned to best support the type of content.
The following example illustrates the content type classification of text and image on a web page:
Text is the most common content type in Windows, so supporting a codec that is highly efficient for text is critical. RemoteFX supports a new codec dedicated to encode text content with a high compression ratio and high quality.
The following charts show the compression ratio improvements when using the new text codec, comparing to the Windows 7 codec, for scenarios with mostly text content.
The Y axis in the following chart shows the compression ratio, for example, uncompressed sizecompressed size. A higher value indicates better compression. For example, in the “Knowledge Worker” scenario, when using the RDP7 codec, the overall compression ratio is 400:1, while using the RemoteFX text codec under the same conditions raises the compression ratio to 500:1.
For video content, we leverage the H.264 video encoder supported in Windows 8. For natural images, we use a new encoder that supports progressive encoding, which means it is able to refine the image quality across multiple steps.
This approach allows for balance between network bandwidth consumption, image quality, and frame rate (responsiveness) at runtime. When on WAN, the network bandwidth is the bottleneck; we are able to send text remotely with clarity while reducing the quality of natural images without impacting the user’s ability to view the image, as illustrated in the following photograph.
The following charts illustrate the significant improvement in compression ratio by enabling the content classifier and by using the new text and image codecs together, compared to the Windows 7 codec. The scenarios all have mixed text and image content.
RemoteFX progressive rendering adapted to bandwidth
RemoteFX supports a new codec that can encode a bitmap or regions of a bitmap “progressively,” known as RemoteFX progressive rendering. This means that the image can be encoded and sent over several stages, and the image quality becomes progressively clearer at each stage.
In previous versions of RDP, when network bandwidth was limited, the user session could appear to be “stuck” because a large frame would block further updates for a long time.
Progressive rendering solves this problem by sending the frame in a lower quality when potential network congestion is first detected. If the image changes on the server before it reaches full quality, the low quality image is canceled to allow faster screen updates. The user experience is that the image quality gradually improves across several frames when there is a network bottleneck, with minimal impact on frame rate.
The following images illustrate a desktop background being encoded progressively in 4 steps. In this example, the first step compressed the image by 77 times in size. The next 3 steps incrementally add refinement, and the last step that generates the highest quality image compressed the image by 15 times overall.
(Please click on the image for the full size version.)
Adaptive encoding stack and frame throttling
In addition to bitmap encoding improvements, RemoteFX is designed to adapt graphics encoding stack, encoding frame rate, and image quality to runtime constraints such as CPU and network bandwidth availability, and client rendering speed. The goal is to achieve the best balance among user experience (fast and fluid graphics with high image quality), server scalability and network bandwidth consumption for every RemoteFX session.
Firstly, RemoteFX leverages progressive encoding by adjusting the quality of each frame according to available bandwidth. On LAN, when there is sufficient bandwidth, each frame is encoded to the highest quality with minimal encoding cost. On WAN, when there is potential network congestion, image quality is reduced to use less bandwidth and keep frame rate higher. In other words, the image quality is adaptive to available bandwidth for each frame.
Secondly, RemoteFX supports different graphics encoding stacks that are optimized for server scalability or network bandwidth. By default, RemoteFX dynamically switches between the encoding stacks by monitoring runtime conditions such as frame rate, network bandwidth, and encoding cost. If a problem is detected and there is a better configuration to improve the situation, RemoteFX would switch to the optimal configuration to keep up the frame rate. The graphics stack configuration is also configurable through a Group Policy setting for administrators to optimize for server scalability or bandwidth.
For example, when remoting to a virtual machine with a virtual graphics processing unit (vGPU) enabled , two graphics encoding stacks are supported. In one configuration all encoding is done in the GPU which results in less processing time on the server and higher frame rate but requires higher bandwidth. When network bandwidth is the bottleneck, we dynamically switch to another configuration that involves more components to pre-analyze the image and leverage multiple codecs to achieve a higher compression ratio, which significantly improves frame rate on WAN. Also, when network bandwidth is constrained, progressive encoding is configured to start with a lower quality than on LAN.
Thirdly, after choosing the optimal encoding stack, if there are still constraints that prevent RemoteFX from achieving the target frame rate, the encoding frame rate is adjusted dynamically to ensure that the most up-to-date frame is sent to the client. For example, in Windows 7 doing a directory listing on WAN may cause the remote session to appear “hang”; now the session is responsive under the same conditions with the latest frame displayed.
Last but not least, in addition to the in-session configurations, RemoteFX also accounts for the per user session encoding time in Fair Share CPU Scheduling in the Remote Desktop Session Host service, such that one user session on a server does not consume all encoding time from all other users on the same server.
Caching improvements
Caching is a powerful tool for bandwidth reduction in many scenarios. While caching is not new to RDP, it is optimized in RemoteFX to further improve the user experience.
First, the cache algorithm is improved to find more cache hits with a lower CPU cost. The default cache size increased to 100 MB, which is the “sweet spot” for cache hit vs. cache size from our testing.
Second, asynchronous cache upload is supported which means we can now start a session while the server and client cache are synchronized in parallel. This improves responsiveness on WAN.
Third, we support multiple instances of cache. If two or more connections are going on simultaneously from the same client computer, each gets its own cache, where previously only the first connection got lucky.
Last but not least, we reduced disk access on the client when “persistent bitmaps” is enabled. This improves speed and reduces cost such as battery life.
Configuring RemoteFX Adaptive Graphics
In addition to being adaptive to runtime conditions, RemoteFX also supports two Group Policy settings that give administrators the flexibility to manually choose the best configuration for their scenario. Both policies are under this path: “ComputerConfigurationAdministrativeTemplatesWindowsComponentsRemote Desktop ServicesRemote Desktop Session HostRemote Session Environment.”
The first policy setting is “Configure image quality for RemoteFX Adaptive Graphics.” This policy setting specifies the visual quality for a remote session. Administrators can use this option to balance network bandwidth usage with visual quality delivered.
The options are Medium (default), High, and Lossless. “Medium” quality consumes the lowest amount of bandwidth, “High” quality raises the image quality with a moderate raise in bandwidth consumption, while “Lossless” uses lossless encoding, which preserves full color integrity but requires significant increase in bandwidth.
The second policy setting is “Configure RemoteFX Adaptive Graphics.” This policy setting allows the administrator to choose the encoding configuration to be optimized for server scalability or bandwidth usage.
As discussed in the “Adaptive encoding stack and frame throttling” section, by default RemoteFX chooses the best configuration at runtime, and could dynamically switch between configurations based on network condition.
Summary
In summary, RemoteFX supports adaptive graphics encoding that smartly adapts to a number of parameters including client capabilities, content type, network bandwidth, and CPU load. Codecs include (but are not limited to) codecs that are text-optimized, image-optimized, and video-optimized. Combined with other techniques such as progressive rendering and dynamic encoding stack configuration, RemoteFX delivers a very adaptive and optimal user experience for all content types on all networks.
NOTE: Questions and comments are welcome. However, please DO NOT post a request for troubleshooting by using the comment tool at the end of this post. Instead, post a new thread in the RDS & TS forum .