Ben Waggoner

AVS2ASF – Open Source VC-1 Encoder SDK application

benwaggoner — Wed, 07 Jan 2009 06:31:00 GMT

As you may recall, we released the VC-1 Encoder SDK Professional as as free download. That included a very basic command-line encoder along the lines of Alex Zambelli’s WMCmd.vbs, but with much more limited input support and no preprocessing.

A fellow called Kurtnoise13 jumped on it on the Doom9 Forums discussion about the SDK. He quickly created a new version that supports AVISynth files as source as well. The syntax for a simple PAL 1.2 Mbps encode (from the ReadMe) looks like:

AVS2ASF -i VideoWithAudio.avs -o file.asf -framerate 25 -rate 1200 -profiletype 2 -bframes 2 -maxkeydist 6 -vbv 300000 -peakrate 2000 -pr AudioEncode.prx

Be warned that while the syntax looks a lot like WMCmd.vbs the details can be deceptively different. In particular, units can be different (Kbps instead of bps makes a big difference!). Also, since the VC-1 Encoder SDK is video only, a PRX file defining the audio setting also has to be used; the Format SDK is called to encode audio and mux to WMV.

AS2ASF also supports elementary stream output, so it’s able to generate a .vc1 file for Blu-ray authoring.

For those not familiar with it, AVISynth is an open source frameserver that dynamically processes video and lets it look like any DirectShow source to other tools. This means that intermediate files aren’t needed – an AVISynth “.avs” script can define the source and all the processing to apply to it. And there’s a huge community of AVS plugins to handle new format times and processing modes; almost any kind of file out there can be handled by AVISynth, and almost any kind of processing you’d like to apply is available. For a simple example, the below script

Opens a PNG sequence at 24 fps
Associates a WAV file with it
Crops it from 640x360 to 640x352
Outputs it in YV12 (8-bit 4:2:0)

a=imageSource("D:\Elephant's Dream\PNG 360\%05d.png",1,15691,24)
b=wavsource("D:\Elephant's Dream\ED-CM-St-16bit.wav")
audiodub(a,b)
crop (0,4,0,-4) # crop height 352
ConvertToYV12()

Anyway, I wanted to highlight this technology. The source code is available as a Visual Studio project, and I’d love to see other members of the community continue to enhance it.

A few things that’d be cool:

A simple batch automation GUI
A simple automatic AVS generator, so a .avi or .mov file could be specified as source and the right AVS to wrap it could be build on the fly.

Best practices for Windows Media Encoder in 2009

benwaggoner — Tue, 06 Jan 2009 01:54:00 GMT

I’ve had a bunch of emails lately from people still using Windows Media Encoder for a variety of reasons. A surprising number of people seem to be using ancient versions of actual encoder .dll files, so I figured it was time for one-last roundup post for best practices with WME and the old Format SDK. Hopefully everyone’s planning their migration to a VC-1 Encoder SDK based product, but for those who can’t move yet, please follow these best practices.

Bear in mind that Windows Media Encoder is older than five years now, which is when multimedia products start moving from Mainstream Support to Extended Support in the support lifecycle. We don’t set support policy for unreleased products, so note that there’s been no formal guidence on whether WME will be supported on Windows 7.

Hotfixes

WME is ancient code at this point, and all updates since the original Windows Media 9 Series launch have been done as hotfixes. You need to have these installed for security, stability, and performance.

Vista compatibility hotfix

If you’re on Vista, you’ll absolutely want to install this one.

FIX: You may experience issues when you use Windows Media Encoder 9 Series on a computer that is running Windows Vista

I covered this in my first non HD DVD blog post here on On10.net!

It can be a bit tricky to install sometimes, so follow these instructions from the invaluable

Windows Media and DirectShow tips, gotchas and code bits blog.

This hotfix address three issues:

Issue 1

When you run the Windows Media Encoder command-line script WMCmd.vbs, the script host Cscript.exe may crash.

Issue 2

The icons that appear on the encoder toolbar and in the encoder dialog boxes are displayed by using a low bit depth. Therefore, the icons appear to have a low resolution.

Issue 3

When you configure the encoding profile or start an encoding session, the encoder may crash.

Multithreaded Multiple bitrate encoding hotfix

This one fixes a pretty embarrassing bug – Format SDK 11 wouldn’t use multiple processors when encoding in multiple bitrates (aka “Intelligent Streaming”). Originally blogged here.

FIX: The WMV9 encoder multi-bitrate performance on a multiprocessor system is severely impaired

Critical Security Fix

This is a security fix for a critical vulnerability that could allow remote code execution on a machine running WME. Install it before you launch WME! Note that the vulnerability requires two things that shouldn’t be happening anyway on a production encoder

WME is running logged in as an administrator
The machine is being used to browse the web

Microsoft Security Bulletin MS08-053 – Critical. Vulnerability in Windows Media Encoder 9 Could Allow Remote Code Execution (954156)

Current Format SDK version

WME in the end is mainly a graphical front-end to the Windows Media Format SDK. The actual codec version you’re running is determined by the SDK. Most people get those updates bundled with Windows Media Player, so as long as you have the most recent WMP version for your platform, you’re good to go.

If there’s some reason that you don’t want to update WMP but still want the current codec .dlls, you can also install the Format SDK directly. The SDK version also includes lots of samples and all the normal SDK goodness for building apps for authoring Windows Media content.

For vendors of compression tools, it’s always a good idea to chain the redistributable installer (WMFDist11-WindowsXP-X86-ENU.exe) along with your installer, so you can ensure that users have the current version. That installer can also un manually to install .dll updates without updating .

I’ve seen way too many production encoders in the last few months that have WMP 9 and thus the 9.0 SDK. That’s a good six years old now, and will offer much lower quality and perf than the current versions deliver. Remember this comparison?

Windows XP, Vista, or Server 2008: Format SDK 11

Windows XP shipped with Windows Media Player XP, which was a flavor of WMP 7. However, the almost universally installed Service Pack 2 included WMP 9. WMP 11 is also available for XP, and should absolutely be installed on any XP-based encoder (and really all XP machines in general).

Windows Vista shipped with WMP preinstalled, so no action is required there.

On Windows Server 2008, Windows Media Player, and hence the FSDK, is installed as part of the Desktop Experience feature. You’ll want to enable the DEP on any encoder box (and in general for any 2K8 box you’ll be using as a workstation).

Windows Server 2003: Format SDK 9.5

The most recent version for Server 2003 is the older Format SDK 9.5 (which is still better than 9.0). As service to our customers looking for a high degree of confusion, Format SDK 9.5 was released with Windows Media Player 10.

Note that running on Server 2003 means you won’t have access to the SMPTE-compliant “WVC1” flavor of Windows Media Video 9 Advanced Profile, nor the excellent WMA 10 Pro LBR audio codec, and hence the 32-96 Kbps range for WMA Pro. The codec is also only 2-way threaded instead of 4-way threaded, slower in general, and lacks support for Tarari acceleration and the advanced registry keys.

If you need to encode on a Server OS, you should use a VC-1 Encoder SDK based product or upgrade to Server 2008 (ideally both!), which will give you better video. However, WMA 10 Pro will still not be available on Server 2003.

There have been reports of people using the WMFDist11-WindowsXP-X86-ENU.exe installer from FSDK 11 set to XP compatibility mode to install the FSDK 11 .dll files onto Server 2003. While there aren’t known issues with this, I should point out that this isn’t a supported configuration for Server 2003, and so Microsoft support will not be available for any issues when running with mismatched .dll files.

64-bit Windows: Format SDK 11 (32-bit)

While there is a “Windows Media Encoder 64-bit edition”, don’t use it (I’m not even going to link to it). It predates FSDK 11, and offers lower quality and performance than FSDK 11 does; the performance advantage of running 64-bit native code is a lot smaller than the other improvements in FSDK 11.

So, keep using the 32-bit stuff on 64-bit as well as 32-bit versions of Windows.

Registry keys

Once upon a time, this blog seemed to be mainly about using special registry key options in the Format SDK. Thankfully, with Expression Encoder and the other VC-1 Encoder SDK based products mean we have GUI access to all of these now in modern products.

First off, install Alex Zambelli’s WMV9 PowerToy, which is a simple .NET GUI for setting the codec settings. The tooltips are a better reference than anything I’ll type here; I’ll just give some basic recommendations for different scenarios.

“Always On” choices

This is a good set of defaults with no significant performance downside that almost always helps quality.

Perceptional Option: Adaptive Dead-Zone 1. This maps to the “Adaptive Dead Zone: Conservative” option from Expression Encoder. This lets the codec reduce detail before introducing artifacts, and generally improves quality at lower bitrates.

In-Loop Filter: On. This turns on the In-Loop deblocking filter which softens the edges of block artifacts. This improves the current frame, and also future frames based on it.

Overlap Filter: On. This further smoothes the edges of blocks. It can reduce detail a little at high bitrates, but is almost always helpful at typical web bitrates.

B-Frame Number: 1. Turns on B-frames, and hence enables flash/fade detection when using Lookahead or 2-pass encoding, and also improves compression efficiency.

Lookahead: 16. Tells the codec to buffer ahead 16 frames in 1-pass (CBR or VBR) encoding, letting the codec detect flash frames and fades and switch the frame type based on it. Maps to the “Scene Change Detection” option in Expression Encoder. It will increase end-to-end latency by that many frames in live encoding, but is generally worth it due to quality improvements.

High Quality Live

Assuming you have a fast enough machine to run these settings in at least Complexity 3, they will improve the live experience.

Motion Search Level: Fixed Integer Chroma. This adds basic chroma search to the encoding, which can help the quality of motion graphics significantly. It’s a pretty small boost with more typical camera-shot content, so turn it off if you have perf issues; getting to Complexity 3 is more important.

Motion Search Range: Adaptive. This tells the encoder to switch to a bigger motion search range for frames with high motion, and then go back to a smaller range when motion dies down. This dramatically improves quality with higher motion at bigger frame sizes. The default range is 64 pixels left/right and 32 pixels up/down, so if any objects move more than that between any two P-frames (if you’re using B-frames, that’ll be 2 frames apart), than this feature will help.

High Quality Offline

These settings offer maximum quality for offline encoding, and are slower yet. Use them when you’ve got the time, and run at Complexity 4. Complexity 5 ignores the Motion Search Level and Motion Match Method settings, making it lower quality. The only time you need to use Complexity 5 is if you can’t set registry keys (in which case it’s quite a bit better) or need to use WME to scale your video, as scaling quality is much better in Complexity 5. However, you’re better off preprocessing in another tool if at all possible.

Motion Search Level: Fixed True Chroma. This is a full-precision motion search for chroma. It never hurts, and can help quality a lot with motion graphics and animation.

Motion Match Method: Adaptive. This switches between the Sum of Absolute Differences (SAD) and the Hadamard method to compare motion between frames as appropriate for each macroblock. Full Hadamard can be higher quality for some very complex content, but the Adaptive mode is faster and better most of the time.

Use hardware preprocessing

WME’s preprocessing was designed for good live encoding performance on circa 2002 machines, which means it’s tuned far more for speed than quality. In particular, deinterlacing and scaling aren’t very good (to say the least). So if you’re using WME with a hardware capture card like an Osprey, you’ll get better results doing all the preprocessing on the card, and just passing the final scaled, cropped, deinterlaced YV12 bitmaps off to WME. This also saves some additional CPU cycles for the codec. For file-to-file encoding, you also want to do any deinterlacing and scaling before you import the file into WME. Expression Encoder has great scaling and good deinterlacing, so that’s another reason to use it over WME.

Optimum hardware

If WMP 9 is all you have on your encoder because that’s all there was when you configured it, you’re going to be running on some very old and very slow hardware! Bear in mind that Moore’s Law predicts you can get twice the computer for your dollar every 18 months or so. So today’s best machines will have about 16x the encoding horsepower of what you could have had when WME was released!

Encoder Complexity

When tuning hardware, the goal is to enable higher encoder complexity values to be used. Your target is at least complexity of 3; that’s most of the way towards optimal quality; lower values sacrifice a quite a bit of quality for improved speed, while higher values sacrifice a lot of speed for only a little additional quality improvement. Complexity 4 is optimum when time/performance isn’t a concern. As mentioned above, Complexity 5 in FSDK 11 invalidates some registry keys (not an issue with VC-1 Encoder SDK).

For live encoding, the best way to test is to run the encode using real-world sources and make sure you’re not dropping frames at your target frame size. You’ll want to set Video Smoothness to 0 for this test to make sure frames aren’t being dropped for quality reasons instead of performance reasons.

CPU

Since the encoder is 4-way threaded, you want to be on at least a 4-core machine. These are dirt cheap these days, with affordable 4-core laptops coming soon. Generally speaking, a 4-core is a lot faster than a 2-core at a slightly faster clock speed.

Clock speed is a big factor as well, but it only matters in context to the generation of processors. For example, the new Intel i7 “Nehalem” get a whole lot more work done at 3 GHz than the original P4 “NetBurst” CPUs did at 3 GHz.

Memory

Memory isn’t a big factor in encoding, as long as you have enough. You never want an encoder to start swapping, as performance will fall through the floor. 2 GB should be plenty for a dedicated encoder, even doing HD encoding.

Tarari Encoder Accelerator

If you’re encoding to higher resolutions and just can’t hit the performance you need at a sufficient quality, check out the Tarari Encoder Accelerator. This offloads a number of the computationally intensive parts of compression to a PCI-X board. While it can’t produce the absolutely best quality a tweaked Complexity 4 + registry key software encode can, it can certainly deliver much higher quality for live HD than pure software, particularly on older computers.

The Tarari board is also fully supported by VC-1 Encoder SDK products, so your investment isn’t tied to a FSDK workflow.

Move to a VC-1 Encoder SDK based product!

So, that’s how to get best results out of legacy Format SDK encoders, particularly WME. But really, it’s time to start planning a migration to a VC-1 Encoder SDK based encoder. If there’s something you need that they simply don’t provide, please let me know. Future improvements in VC-1 and Windows Media encoding are going to focus on the SDK, so the gap between Windows Media Encoder and the best encoders is going to be growing bigger over time.

Extra – rollup of Windows Media Encoder related posts

And just so they don’t disappear off the bottom of the blog, here are my blog posts about WME and the FSDK 11 codecs that readers may still find useful:

Low Latency webcasting with Windows Media and Silverlight

Compression for Silverlight Webcast posted

New WMcmd.vbs with improved multithreading and new presets

http://on10.net/blogs/benwagg/New-PowerToy-and-registry-key-documentation/

Hands-on with the Temporary Insanity trailer

http://on10.net/blogs/benwagg/Encoding-for-the-Zune/

http://on10.net/blogs/benwagg/Elephants-Dream-720p--2-Mbps/

http://on10.net/blogs/benwagg/My-Live-Webcasting-with-Windows-Media-presentation-from-Streaming-Media-East/

http://on10.net/blogs/benwagg/My-Mix-07-session-about-WMV-Encoding-and-Expression-Media-Encoder/

Moonlight 1.0 beta 1 is out

benwaggoner — Tue, 16 Dec 2008 07:35:00 GMT

Moonlight is a GPL'ed open source implementation of Silverlight, created by the Mono team at Novell. While it's a Novell project, we're helping them out, and I'm excited at the progress that they've made, and that we'll have a free software playback option for Silverlight.

And thus, the beta 1 of Moonlight 1.0 is now available. Moonlight 1.0 maps to Silverlight 1.0, so you've got good XAML support. But there's no .NET bytecode support, so logic takes places inside the browser's JavaScript engine. Beta 1 also includes support for the "Microsoft Media Pack" described here. Moonlight will automatically offer to download it the first time you hit a media file. Microsoft covers the patent licensing for the included codecs.

Here's a list of supported sites. I imagine most of my demo projects up on Silverlight Streaming should work, as they're all Silverlight 1.0 compatible at this point. Someone who gets this installed, let me know (my personal lameness documented below).

Ars Technica also has a good article about beta 1. It contains plenty of screen shots and such, which I have once again failed to provide due to my lamentable lack of Linux-fu.

Anyway, huzzah to the Moonlight team. I can't wait for the final release of Moonlight 1.0 and the first beta of Moonlight 2.0.

(Yes I did try to install Ubuntu, but as with every attempt I've made to run Linux in the last five years, I quickly got stymied by video driver issues. While I got basic display working, I was using my backup Barcelona box, which has my beloved but ancient LaCie ElectronBlu 22: CRT monitor and a cheapo 1280x1024 LCD. Out of the box, it wouldn't let me run the main monitor at more than the small displays resolution or (more painfully) refresh rate. I tried to fix it until I got to the "type sudo..." phase. I bumped into Miguel de Icaza himself at an event last week (we had a fun chat about GPU compositing), who tells me that the new OpenSUSE 11.1 has a nice GUI configuration utility that should fix the issue.)

Custom Slider for Silverlight 2 Media Players available for download

benwaggoner — Wed, 26 Nov 2008 01:03:00 GMT

Sorry it's taken so long, but we've now got the Custom Slider for Silverlight 2 Media Players up on MSDN.

Unlike the snippits I included earlier, this is a full project that can be modified. It’s being made available under the Microsoft Public License.

Code Gallery Page
Project Download

Enjoy!

Netflix’s Neil Hunt shares encoding workflow info

benwaggoner — Thu, 20 Nov 2008 01:50:00 GMT

Now, this is a blog post! Neil Hunt, Chief Product Officer for Netflix, has just put up a great blog post talking about their encoding workflow for their video streaming services.

It’s full of awesomeness, but I wanted to excerpt the section describing their 1st gen, 2nd gen, and HD encoding settings and workflow.

First Generation Encoding

Our first set of encodes are based on WMV3 and WMA in ASF with WMDRM10 (Janus). We chose these standards because the Janus components have been widely adopted by our CE partners such as Roku, LG Electronics, Samsung, TiVo, and of course Microsoft Xbox.
We encode most content at 500, 1000, 1600, and 2200kbps VBR, but some titles whose source quality merits it have also been encoded at 3400kbps. The highest bitrate encodes are fit into 720x480 non-square pixels (the usual 1.2 PAR for widescreen content, 0.9 PAR for 4:3), but optimum encoding at lower bitrates is achieved with fewer pixels. Encoded films are normally at 24fps to match the source, while shot-to-video and mixed material is de-interlaced to 30fps (or 25fps for PAL content).

Netflix has been using anamorphic video all along, which I think is an underused feature of Windows Media and other formats. When you’re limited to 720x480 pixels, you want to encode all the pixels you’ve got, without having to synthesize any extra.

Second Generation Encoding

The new Silverlight player (that some users are helping us test as I write) uses VC1 Advanced Profile encoding with PlayReady DRM. A key property is that each GOP header includes frame size and resolution, which allows us to assemble a stream on the fly from different bitrate encodes as your broadband bandwidth fluctuates. (Another key feature is more coverage, including Intel Macs and Firefox users.) We expect to switch completely to the new player later this year.
The VC1 encoders are more efficient than the WMV3 encoders, so we are currently encoding VC1AP at slightly lower birates: 375, 500, 1000, and 1500kbps, all square pixel. At some point we are likely to add a couple more resolutions of non-square pixel encodes capturing the original pixel-aspect-ratio of the source.
We are also re-wrapping the VC1AP encodes in WMDRM10 for CE devices, which will gradually switch to the more efficient encodes in future firmware upgrades.

This is a great example of the improved efficiency of the VC-1 Encoder SDK and tools based on it. Not all Windows Media/VC-1 encoding is equal; the latest tools can offer a very meaningful reduction in bitrate required for a give quality level, improving user experiences and the cost of delivering the content.

The new encodes are backwards compatible to older hardware and software encoders, so even older devices can take advantage of the improvements.

High Definition Encodes

Today we have rights to deliver about 400 streams in HD (720p). More titles will be added over time. We experimented with first-generation WMV3 encodes at 4000kbps and 5500kbps, but settled on second-generation HD encodes with VC1AP at 2600kbps and 3800kbps, which extends their accessibility down to lower home broadband connections. As with SD, encodes of film material are at 24fps, and encodes of shot-to-video material are at 30fps (or 25fps for PAL), rather than the 60fps that would come from a Blu-ray disc - we judged the 60fps content as too expensive of bandwidth for now. In general, these encodes are definitively better than SD, but won't challenge well-executed Blu-ray encodes - that would require a bitrate out of reach for most domestic broadband today. We believe Moore's law will drive home broadband higher and higher enabling full 1080p60 encodes in a few years.

There’s codecs for you – a good 720p experience at 3800 Kbps, which is probably a little below the average 480p bitrate used for MPEG-2 on the DVDs that fly around the nation in those Netflix envelopes.

I’m not that worried about 1080p60 encoding myself. The vast majority of fictional content is shot 24p, including 99% at least of Blu-ray discs. While Blu-ray players may output 1080p60, the encodes, and hence bitrate requirements, are still 1080p24. I find that typical film content in VC-1 wants around 6-8 Mbps for a 1080p24 experience enough better than 720p to be worth the trouble.

I just got my Xbox set up to handle the Netflix streams – now I’ve got to go check out some of those 400 HD titles! Hmm. Pan’s Labyrinth and Heroes.

Anyway, it’s great stuff. I love it when partners can share this kind of detail about what they’re doing.

Silverlight 2 launch party in Portland tonight

benwaggoner — Wed, 12 Nov 2008 00:06:00 GMT

PDXUX.Net is hosting a Silverlight 2 launch party in Portland tonight, at North.

Tim Heuer will be the featured presenter, and I’ll come in and do fifteen minutes or so of media demos. There will be ample time for mingling and Q&A as well.

Sorry it’s such a last minute announce – we only just worked out the plan.

Building high-performance Silverlight Media Players

benwaggoner — Mon, 03 Nov 2008 23:33:00 GMT

Overview

So, I get involved in a ton of conversations with various internal and external customers about building Silverlight media players. The great thing about Silverlight is that it's deep, and provides lots of ways to build complex applications that include media playback. The flip side of that is that, like all software engineering, there's plenty of sub-optimal ways to do thing that can have a negative impact on media playback performance.

While it's tempting to assume everyone's got a hopping dual-core machine these days, that's not the case. When we reviewed the demographics for the NBC Olympics player we were surprised at what a big chunk of the home users had older, slower, < 2 GHz single-core PCs. We were also somewhat surprised by how fast average broadband has gotten. It's not unusual for consumer to now be bound by CPU power more than bandwidth power as far as the highest bitrate they can play.

That said, don't go overboard – plenty of these techniques are just best practices, but some can limit the complexity of the player you can build. Unless you’re getting dropped frames on the target platforms, easier playback scenarios can support all kinds of effects. Just make sure that the payoff in improved user experience from using the more advanced techniques is worth the perf hit.

Fast Path

First up, let's talk about getting players on the Fast Path - this is when Silverlight doesn't have to do any scaling or compositing of the video rectangle, saving a good chunk of CPU power as well as memory bandwidth. Again, these are really about HD content – SD and below should have plenty of perf even on older machines.

No scaling

The video's MediaElement should be exactly the size it was encoded at. The simplest way to do that is to just remove the Height and Width elements from the MediaElement. The perf differential is really just scaling versus no scaling; there's no significant advantage to using scaling tricks like exact 2x or 3x scaling, or just scaling on one axis. Just leaving Height= and Width= blank in the MediaElement is probably the easiest way to turn off scaling.

Note that this applies to non-square pixel encoded video. For example, 720x480 encoded as 4:3 won't ever use the fast path; you'd have to encode as square pixel 640x480 instead.

And this applies to scaling down as much as scaling up. Playing 640x480 video at 320x240 will actually take more CPU than just leaving it at 640x480.

If you’re building a video browser that plays multiple video streams at once, it can be worth it to provide low resolution thumbnails at the display size; that’ll allow a lot more clips to be played at once.

Pixel-aligned

The pixels also need to be exactly aligned with the grid, so no decimal coordinates. Pixel Snapping will do this for you automatically in Silverlight 2.

MediaElement, not VideoBrush

VideoBrush can enable some great effects like mirroring, but isn't compatible with Fast Path. Unless you’re doing something that requires VideoBrush, stick with MediaElement.

General XAML tips

Frame Rate

The default frame rate of a Silverlight application is 60 fps, while most media encoded for Silverlight is 30 fps or less. Setting the fps of the application to that of the media. That’ll provide better performance, and and make the video and GUI elements seem more in sync.

Don't use Windowless

Windowless mode has a slight perf hit on Windows (it doesn't have a significant impact on Mac). Windowless mode is mainly used to mix Silverlight with HTML or other web elements in the same part of the screen.

Minimize overlays

While the fast path can operate with overlays, they do take additional CPU to process, so minimize the use and size of overlays to what’s useful. Even an invisible object that overlaps the video, like a play control set to transparent, still gets composited. Instead, when the control is going to go invisible, have it move entirely outside of the video rectangle. As long as it's not overlapping the media, no problem.

Keep MediaElement opaque

Speaking of keeping rendering of transparent elements to a minimum, you definitely want the video itself to be opaque, particularly at bigger frame sizes. Even a hint of transparency will require additional processing of every pixel of every frame.

Media Encoding

Constrain video peak bitrate

The main factors in CPU load for video decoding are how many pixels/second are being displayed (height * width * frames per second), and what the peak data rate of the video is.

With a CBR (Constant Bitrate) encode, the peak and average bitrate are identical. The only variability is the buffer duration. Using a very long duration buffer (like 20 seconds) can make for a data rate spike within the buffer that can make for challenging encoding.
With a VBR (Variable Bitrate) encode, the peak is higher than the average. There’s no rule about what the difference is. Typically the peak is at least 1.5x the average, but it can go a lot higher.
A Buffer Window 1-2x your keyframe interval is a good starting point. You don’t want a buffer smaller than the keyframe rate, as the keyframes can wind up be starved for bits, resulting in the “blur-in” effect when the quality drops after a scene change.

If you want to calibrate what VBR peak bitrate you can use on a particular system, it works to test with CBR. Just find out the bitrate + buffer duration you can use with CBR, and use that as the peak bitrate and buffer duration with your VBR encodes.

For high bitrate content where the perf ceiling of the peak buffer is a more important limitation than average bitrate, go ahead and use CBR encoding. That generally provides better results than VBR when the peak would be much less than 1.5x the average.

Encode audio at 44.1 KHz mono or stereo

Silverlight’s internal audio pipeline runs at 44.1 KHz, so even if the audio comes in at a higher rate, you should resample to 44.1 KHz on encode. If you have a lower sample rate source, it’s fine to leave it at that.

And while WMA 10 Professional supports 5.1 and 7.1 audio, Silverlight 2 always mixes down to stereo. So if you’re targeting Silverlight only, convert multichannel sources to stereo before encoding. This also enables the much more efficient WMA 10 Pro codec.

Crop letterboxing

This shouldn’t be news, but don’t encode non-image parts of the video frame, like letterboxing. A 640x480 frame with standard 1.85:1 letterboxing can be cropped and encoded at 640x352 without losing any visual information, but making encoding and decoding faster (36% fewer pixels need to be processed). Silverlight is more than capable of drawing the black rectangles for you client-side if you must have them.

Use Inverse Telecine with 3:2 pulldown

If the source content was sourced from 24p film, but transferred to 29.97i video with 3:2 pulldown, the video file will see a repeating pattern of three progressive and two interlaced frames. Instead of deinterlacing that video (with the inevitable artifacts) and encoding at 29.97, inverse telecine can restore the original 24p, eliminating deinterlacing artifacts, the framerate judder from the 24 to 30 remapping, and providing more bits per frame. And, of course, it’s 25% easier to decode and display 24 frames a second than 30. Make sure to turn the Silverlight application’s frame rate down to 24 fps as well.

Encode as square pixel for fast path

If you’re shooting for the fast path, note that the scaling required for playing back anamorphic video will turn it off. If you have 16:9 720x480 source, you want to play back in a 848x480 window (16:9 480p) , you’ll need to encode at 848x480 if you want to get the fast path.

Depending on the design of your player and the performance of the system, it’ll vary as to whether you’re better off decoding fewer pixels (720x480 instead of 848x480), but losing the fast path. Testing both ways on your target platforms is, as always, the best thing to do.

Encode anamorphic sources as anamorphic for non-fast path

If you’re not going for the fast path, you might as well encode content authored as anamorphic, as anamorphic. For example, the DVCPROHD codec is internally 960x720 in 720p mode. That would be compressed as 1280x720 for 720p playback in square pixels. But since the source is only 960 wide, encoding at 960x720 (set to a 16:9 aspect ratio) will be 33% more efficient in terms of bitrate and decode performance. Depending on the platform, that can be well-worth the sacrifice of Fast Path.

Silverlight 2 specific

Just use Silverlight 2

Silverlight 2 has a bunch of perf improvements over Silverlight 1.0, the most notable being a faster VC-1 decoder and better scaling performance (with better quality to boot). The latter means that not using the Fast Path has less of an impact in Silverlight 2 than before.

The good news is that Silverlight automatically updates, so you don’t need to do anything specific to force this. It can be worth it to retest Silverlight 1.0 and Silverlight 2 Beta applications in Silverlight 2 to see if suggested system requirements can be lowered.

Use Andre & Akshay’s Custom Slider

Our Silverlight teammates Andre Michaud and Akshay Johar have build a custom slider that offers better performance for video playback, particularly with streaming content. This one doesn’t continuously generate valueChanged events, which would then turn into new seeks in the media file. Instead, it waits to issue them until either mouse up on the slider thumb or slider tracker, or just the last seek of a bunch of them if they come at once. So, the user can wiggle the mouse willly-nilly, but it won’t turn into a seek until the movement slows down, or they let go.

The code was in my previous blog post.

Use Expression Encoder 2 Service Pack 1 Templates

The new Expression Encoder 2 SP1 (which I really need to blog about, but then I should blog about Adaptive Streaming first, which is going to be another long post…) adds new Silverlight 2 templates. These implement the described best practices, including using the fast path if the Job Output’s Stretch Mode=None, and a less manic slider.

Test early, test often, test on your target platforms

This was one of my most-used articles of wisdom imparted to me by the legendary Charles Wiltgen of Kinoma. In any project, you want to define your minimum and recommended system specs. You need to have machines of those specs available to test to make sure they actually work. It’s obvious, but often missed. And for complex players with a lot of custom XAML, a Core 2 Extreme can get away with suboptimal design that’ll turn an older P4 into a filmstrip emulator (and you don’t even get that “beep”).

NetFlix Silverlight player now in open Beta

benwaggoner — Mon, 03 Nov 2008 19:18:00 GMT

Just a quick note - NetFlix is offering an open beta for their new Silverlight-based player.

It's an opt-in, and once you opt-in, you need to use Silverlight on all your machines you use for "Watch Instantly."

Friday, October 31, 2008

Opt-in for the new Netflix movie player

Hi, Brent here with an update on our new player for the Mac and PC. We're past the first phase of our beta already, so I'd like to give you a chance to join in. But before doing so you need to consider the following:

There may be bugs. We are logging all errors, but if you run into problems you can help out by posting details here in the blog comments.

Not all movies are available to watch with Silverlight. You may notice errors or lower than normal quality when watching certain titles.

Our new player works on PCs and Intel-based Macs.

Windows users should be aware that if you opt in, you will need to use Silverlight on all the machines you use to watch instantly.

Once you get up and running check out some of our newest content. We recently added a lot more movies, as well as recent episodes of many TV shows including Heroes, CSI and NCIS. You’ll find all these movies & TV episodes under the “Watch Instantly” tab of the website.

So here’s the opt-in link:
www.netflix.com/silverlightoptin

Enjoy!

Posted by Brent at 1:45 PM

New Custom Slider for Silverlight 2 media players

benwaggoner — Tue, 28 Oct 2008 22:13:00 GMT

My Silverlight teammates Akshay Johar and Andre Michaud have built a new custom slider for use in Silverlight 2 media players.

When you scrub the default slider in a MediaElement today, it generates a whole slew of valueChanged events, resulting in a huge number of seeks per second. This can confuse media playback, particularly when streaming from Windows Media Services, which then gets flooded with many requests a second.

Essentially it detects a “crazy mode” scrub, and quietly ignores it while the “crazy mode” is in place. Once out of this mode, it happily sends and receives a successful seek.

So, the player only fires ValueChanged on mouse up (on the slider thumb or the slider tracker), or the final seek request of a cluster that comes in at once.

It’s also uses Tim Heuer’s absolute value slider.

Now, let me try this new Windows Live Writer plug-in for Insert Code Snippet for the C# code.

(EDIT) A sample project is also now available for download.

 1: using System;

 2: using System.Net;

 3: using System.Windows;

 4: using System.Windows.Controls;

 5: using System.Windows.Documents;

 6: using System.Windows.Ink;

 7: using System.Windows.Input;

 8: using System.Windows.Media;

 9: using System.Windows.Media.Animation;

 10: using System.Windows.Shapes;

 11: using System.Windows.Controls.Primitives;

 12: using System.Windows.Threading;

13:

 14: namespace VirtualizedSlider

 15: {

 16:     public class CustomSlider : Slider

 17:     {

 18:         public Thumb horizontalThumb;

 19:         private FrameworkElement horizontalLeftTrack;

 20:         private FrameworkElement horizontalRightTrack;

 21:         private double oldValue = 0, newValue = 0, prevNewValue = 0;

 22:         public event RoutedPropertyChangedEventHandler<double> MyValueChanged;

 23:         public event RoutedPropertyChangedEventHandler<double> MyValueChangedInDrag;

24:

 25:         private DispatcherTimer dragtimer = new DispatcherTimer();

 26:         private double dragTimeElapsed = 0;

 27:         private const short DragWaitThreshold = 200, DragWaitInterval = 100;

 28:         public Rectangle progressRect = null;

 29:         private bool dragSeekJustFired = false;

30:

 31:         public CustomSlider()

 32:         {

 33:             this.ValueChanged += new RoutedPropertyChangedEventHandler<double>(CustomSlider_ValueChanged);

 34:             dragtimer.Interval = new TimeSpan(0, 0, 0, 0, DragWaitInterval);

 35:             dragtimer.Tick += new EventHandler(dragtimer_Tick);

36:

 37:         }

38:

 39:         void dragtimer_Tick(object sender, EventArgs e)

 40:         {

 41:             dragTimeElapsed += DragWaitInterval;

 42:             if (dragTimeElapsed >= DragWaitThreshold)

 43:             {

 44:                 RoutedPropertyChangedEventHandler<double> handler = MyValueChangedInDrag;

 45:                 if ((handler != null) && (newValue != prevNewValue))

 46:                 {

 47:                     handler(this, new RoutedPropertyChangedEventArgs<double>(oldValue, newValue));

 48:                     dragSeekJustFired = true;

 49:                     prevNewValue = newValue;

 50:                 }

 51:                 dragTimeElapsed = 0;

 52:             }

 53:         }

54:

 55:         void CustomSlider_ValueChanged(object sender, RoutedPropertyChangedEventArgs<double> e)

 56:         {

 57:             oldValue = e.OldValue;

 58:             newValue = e.NewValue;

 59:             if (horizontalThumb.IsDragging)

 60:             {

 61:                 dragTimeElapsed = 0;

 62:                 dragtimer.Stop();

 63:                 dragtimer.Start();

 64:                 dragSeekJustFired = false;

 65:             }

 66:         }

67:

 68:         public override void OnApplyTemplate()

 69:         {

 70:             base.OnApplyTemplate();

71:

 72:             horizontalThumb = GetTemplateChild("HorizontalThumb") as Thumb;

73:

 74:             horizontalLeftTrack = GetTemplateChild("LeftTrack") as FrameworkElement;

 75:             horizontalRightTrack = GetTemplateChild("RightTrack") as FrameworkElement;

 76:             progressRect = GetTemplateChild("progressRect") as Rectangle;

77:

 78:             if (horizontalLeftTrack != null) horizontalLeftTrack.MouseLeftButtonDown += new MouseButtonEventHandler(OnMoveThumbToMouse);

 79:             if (horizontalRightTrack != null) horizontalRightTrack.MouseLeftButtonDown += new MouseButtonEventHandler(OnMoveThumbToMouse);

80:

 81:             horizontalThumb.DragCompleted += new DragCompletedEventHandler(DragCompleted);

 82:             progressRect.Width = this.Width;

 83:         }

84:

 85:         private void OnMoveThumbToMouse(object sender, MouseButtonEventArgs e)

 86:         {

 87:             e.Handled = true;

 88:             Point p = e.GetPosition(this);

89:

 90:             if (this.Orientation == Orientation.Horizontal)

 91:             {

 92:                 Value = (p.X - (horizontalThumb.ActualWidth / 2)) / (ActualWidth - horizontalThumb.ActualWidth) * Maximum;

 93:             }

 94:             RoutedPropertyChangedEventHandler<double> handler = MyValueChanged;

 95:             if (handler != null)

 96:             {

 97:                 handler(this, new RoutedPropertyChangedEventArgs<double>(oldValue, Value));

 98:             }

 99:         }

 100:

 101:         private void DragCompleted(object sender, DragCompletedEventArgs e)

 102:         {

 103:             dragtimer.Stop();

 104:             dragTimeElapsed = 0;

 105:             RoutedPropertyChangedEventHandler<double> handler = MyValueChanged;

 106:             if ((handler != null) && (!dragSeekJustFired))

 107:             {

 108:                 handler(this, new RoutedPropertyChangedEventArgs<double>(oldValue, this.Value));

 109:             }

 110:         }

 111:     }

 112: }

And here’s the XAML from the progressSliderStyle, demonstrating how the custom slider knows about mouse clicks on the track (the “LeftTrack” and “RightTrack” rectangles inserted in the template handle this):

 1: <Setter Property="Template">

 2:                  <Setter.Value>

 3:                      <ControlTemplate TargetType="Slider">

 4:                          <Grid x:Name="Root">

 5:                              <Grid.Resources>

 6:                                  <ControlTemplate x:Key="RepeatButtonTemplate">

 7:                                      <Grid x:Name="Root" Opacity="0" Background="Transparent"/>

 8:                                  </ControlTemplate>

 9:                              </Grid.Resources>

 10:                              <vsm:VisualStateManager.VisualStateGroups>

 11:                                  <vsm:VisualStateGroup x:Name="CommonStates">

 12:                                      <vsm:VisualState x:Name="Normal"/>

 13:                                      <vsm:VisualState x:Name="MouseOver"/>

 14:                                      <vsm:VisualState x:Name="Disabled">

 15:                                          <Storyboard>

 16:                                              <DoubleAnimation Storyboard.TargetName="Root" Storyboard.TargetProperty="(UIElement.Opacity)" To="0.5"/>

 17:                                          </Storyboard>

 18:                                      </vsm:VisualState>

 19:                                  </vsm:VisualStateGroup>

 20:                              </vsm:VisualStateManager.VisualStateGroups>

21:

 22:                              <!-- Horizontal Template -->

 23:                              <Grid x:Name="HorizontalTemplate" Background="{TemplateBinding Background}">

 24:                                  <Grid.ColumnDefinitions>

 25:                                      <ColumnDefinition Width="Auto"/>

 26:                                      <ColumnDefinition Width="Auto"/>

 27:                                      <ColumnDefinition Width="*"/>

 28:                                  </Grid.ColumnDefinitions>

29:

 30:                                  <!-- Track Layer -->

 31:                                  <Rectangle x:Name="progressRect" Stroke="#FFA3AEB9" StrokeThickness="{TemplateBinding BorderThickness}" Fill="Red" Grid.Column="0" Grid.ColumnSpan="3" Height="3" Width="5" RadiusX="1" RadiusY="1" Margin="5,0,5,0" Canvas.ZIndex="1" Visibility="Collapsed" HorizontalAlignment="Left" />

 32:                                  <Rectangle Stroke="#FFA3AEB9" StrokeThickness="{TemplateBinding BorderThickness}" Fill="#FFE6EFF7" Grid.Column="0" Grid.ColumnSpan="3" Height="3" RadiusX="1" RadiusY="1" Margin="5,0,5,0"  Canvas.ZIndex="0" />

33:

 34:                                  <!-- Repeat Buttons + Thumb -->

 35:                                  <RepeatButton x:Name="HorizontalTrackLargeChangeDecreaseRepeatButton" IsTabStop="False" Template="{StaticResource RepeatButtonTemplate}" Grid.Column="0" Canvas.ZIndex="1"/>

 36:                                  <Rectangle x:Name="LeftTrack" Grid.Column="0" Fill="#00FFFFFF" Cursor="Hand" Canvas.ZIndex="1"/>

 37:                                  <Thumb Height="18" x:Name="HorizontalThumb" Width="11" Grid.Column="1" IsTabStop="True" Canvas.ZIndex="1"/>

 38:                                  <RepeatButton x:Name="HorizontalTrackLargeChangeIncreaseRepeatButton" IsTabStop="False" Template="{StaticResource RepeatButtonTemplate}" Grid.Column="2" Canvas.ZIndex="1"/>

 39:                                  <Rectangle x:Name="RightTrack" Grid.Column="2" Fill="#00FFFFFF" Cursor="Hand" Canvas.ZIndex="1"/>

 40:                              </Grid>

 41:                          </Grid>

 42:                      </ControlTemplate>

 43:                  </Setter.Value>

 44:              </Setter>

Silverlight 2 powering Netflix on Mac

benwaggoner — Tue, 28 Oct 2008 00:38:00 GMT

It's been discussed for a while, but the time is now - Netflix has just announced Silverlight 2 will power their streaming video service, which had been Windows-only until now. Silverlight 2 adds support for PlayReady DRM, so we now have the same DRM support on Mac and Windows. The service will be available on Xbox 360 as well.

You have to be a Netflix subscriber of course, but the content is commercial-free and much higher quality than the various free video services. They encode up to a full 720x480, in proper 4:3 or 16:9, at a range of bitrates. The higher rates definitely outperform standard def cable and satellite in video quality.

Lots of good stuff in the press release, but I had a couple of things I wanted to call out (bolding mine).

Netflix, Inc. (NASDAQ: NFLX), the world's largest online movie rental service, today announced it has begun the deployment of Microsoft Silverlight to enhance the instant watching component of the Netflix service and to allow subscribers for the first time to watch movies and TV episodes instantly on their Intel-based Apple Macintosh computers. The deployment, which will initially touch a small percentage of new Netflix subscribers, is the first step in an anticipated roll-out of the new platform to all Netflix subscribers by the end of the year.

Note that this won’t be immediately available to all customers.

Silverlight is designed for delivery of cross-platform, cross-browser media experiences inside a Web browser. It is expected that Netflix members who watch movies and TV episodes instantly on their computers will enjoy a faster, easier connection and a more robust viewing experience with Silverlight, due to the quality built directly into the player. Among the viewing enhancements with the new player is a breakthrough in timeline navigation that vastly improves the use of fast-forwarding and rewinding. The new Netflix player takes advantage of PlayReady DRM, which is built into Silverlight, for the playback of protected content on both Windows-based PCs and on Macs. That had not been possible with previous generation technologies.

This is one of the first big rollouts of PlayReady with Silverlight in the USA. When you see the new navigation, it’s a real testament to the difference between a media player plugin like the WMP OCX, and a rich application runtime that plays media well like Silverlight.

The rich user experience support and DRM, what else does Silverlight bring to the equation? One big thing is that PlayReady DRM supports http delivery of pre-encrypted content, unlike other platforms that require a proprietary streaming server and protocol be used to deliver protected content, applying DRM on the fly. Pre-encrypted content is both more scalable (since the encryption processing is only done once, not per user), and more secure (since an in-the-clear media file doesn’t need to be transmitted and stored on the server).

There’s been two big technical challenges in web video for the last dozen years or so.

How do we deliver the best experience to each user, in a world of variable bandwidth and hardware power
How do we deliver that experience as cost effectively as possible, enabling new business models in video distribution.

We’re at the point now where a user with a fast connection and a decent computer should have a darn good video and audio experience. Netflix is a testament to what’s possible as the cost to deliver a MB of video gets low enough, from a business perspective. While finally delivering a user experience better than VideoCD.

Final Olympics numbers

benwaggoner — Mon, 20 Oct 2008 01:49:00 GMT

People kept asking me about this on my just completed tour of Asia, so here’s a blog post that’s got the final numbers for the NBCOlympics.com video delivery with Silverlight.

The data is from the Silverlight 2 Customer Evidence and Quote Sheet.

During the 17 days of the games, the site had

51.9 million unique visitors (106% more than Athens and Torino combined)
1.3 billion page views
75.5 million video streams initiated (601% more than Athens and Torino combined)
9.9 million hours of video watched (equivalent of 1,126 years of video).
More than 27 minutes per user average viewing time (compared to 3 minutes for sites using other streaming video technologies)

And these great quotes from Perkins Miller, Senior Vice President, Digital Media, NBC Sports and Olympics:

“NBC had plans to create the most ambitious online video event in history — more than 2,200 hours of Olympic coverage in 25 sports in less than 17 days – with 3,500 hours on demand. NBC needed to ensure the experience was versatile, stable, and — above all — engaging and entertaining.”

NBC Sports chose Silverlight 2 because “with such a vast number of events and hours of live and on-demand video, we needed a scalable, flexible platform that would deliver an immersive and engaging experience.”

Beyond the NBC in the USA, Silverlight was used for Olympics coverage in other countries, including:

France (FranceTV)
Netherlands (NOS)
Russia (Sportbox.ru)
Italy (RAI)

Silverlight was a great coming-of-age story for Silverlight as we prepared to launch Silverlight 2. The“does Silverlight scale” and “will customers install it” questions have largely gone away in the last couple of months.

Silverlight 2 Released!

benwaggoner — Tue, 14 Oct 2008 16:24:00 GMT

It was a long time coming, but today was the day. Silverlight 2 was released to the web today!

I'm in Shenzen, where it's after midnight, so I'll just list a few links.

First, Scott Guthrie's blog has, as always, the best summary of what's new and interesting in Silverlight 2.
Silverlight.net has been updated, particularly the Get Started section which has links to all the downloads you need.
Which in particular includes Blend 2 SP1
We're also working with Soyatec to build Eclipse support for Silverlight.
I covered the new media features in this post, with examples here.
We even got covered in a New York Times article.

Anyway, enjoy!

Demo of Silverlight 2 scaling quality improvements

benwaggoner — Thu, 02 Oct 2008 21:03:00 GMT

My colleague Donald Karlov did a A/B compare for me showing the quality improvement for scaling between Silverlight 2 Beta 2 and the new Silverlight 2 RC0. Check this detail of a shot from an encode I did of Big Buck Bunny, scaled to 2.5x (640x360 to 1600x900 – a pretty common scaling ratio for full-screen playback on a smaller LCD or a laptop).

Silverlight 1.0 through Silverlight 2 Beta 2

Silverlight 2 RC0 and later

Note the improvements in detail, and the elimination of some stray pixels at the edge of the butterfly wing. Most shots won’t be as dramatically improved, but all will be helped at least somewhat.

And even with the better quality, playback CPU load dropped by 10-15%, due to better perf in the scaler and decoder.

Silverlight 2 RC0 is out

benwaggoner — Wed, 01 Oct 2008 15:01:00 GMT

The first public release canditate (RCo) for Silverlight 2 is now available for download by developers. While anyone can install it, the main goal is for developers to make sure they aren't hit by any breaking changes for Silverlight 2 RTM so they can fix any issues. No one will be auto-updated to this release. Users will be auto-updated to Silverlight 2's final release when it's posted.

All Silverlight 1.0 projects should work fine, but there may be some Silverlight 2 Beta projects that require updates.

For the media folks, there's a couple of new things I want to mention.

There's a new scaling algorithm that's a lot faster and much higher quality - a nice compromise when you can get it!
Additional optimizations have been done for the VC-1 video decoder, so playback will be faster. The gains are biggest for content using B-frames.

Since the default Expression Encoder templates have "always on" scaling and its presets use B-frames by default, the above will provide a nice performance boost for existing content. The above are "always on" features - you don't need to update anything to take advantage of them. So expect smoother frame rates on lower-end machines.

Further information:

EDIT: Added ScottGu link

First MediaStreamSource example is up

benwaggoner — Wed, 01 Oct 2008 14:04:00 GMT

My colleague Larry Olson has posted the first public demo project for MediaStreamSource. This new Silverlight 2 feature enables file parsers and network protocols to be implemented in managed code inside of Silverlight, thus enabling support beyond the built-in native formats. Since MSS passes off the compressed media samples to the native decoders, it should offer nearly the same performance as a native code implementation (parsers aren't that expensive). But since it runs inside of the Silverlight managed code sandbox, support for new formats and protocols can be just added into player's .XAP package file, and will be completely transparent to the end user.

MediaStreamSource is one of the biggest enabling features in Silverlight 2. At IBC, people came up with all kinds of crazy, awesome, and even useful ideas of what to do with it beyond anything I'd ever imagined. Really, the only big limitation is that you can't use UDP packets, as they're not supported by the Silverlight networking stack.

Some people have had trouble finding the MSDN docs about MSS, so I'll list them here as well:

MediaStreamSource Class
MediaStreamSource Members

Expression Encoder Service Pack 1 preview

benwaggoner — Wed, 24 Sep 2008 03:19:00 GMT

James Clarke has a blog post up describing some of the new features in the forthcoming Expression Encoder 2 Service Pack 1.

It's got some great new stuff, and we're demoing it here at Streaming Media West this week.

Read the whole post - there's lots of good stuff in there. A couple of my favories are

A new Silverlight 2 base player using .NET. Among other things, this will enable players that display the video at 100% scale by default, improving quality and performance.
New A/B compare modes. Awesome stuff for high-touch encoding and codec tweaking
Some even more VC-1 advanced options for High Codec Nerditry.
The first public release of Microsoft H.264 compression technology.

Anyway, a lot of good stuff for a SP1. I'm looking forward to it.

Ben's IBC Schedule

benwaggoner — Wed, 10 Sep 2008 16:54:17 GMT

So, if you're coming to IBC, come swing by the Topaz Lounge and say hello. I'll be demoing on the below schedule, alternating between the Silverlight Formats pod and floating between pods answering questions. And yes, we'll be showing H.264 playback in Silverlight at the Formats pod.

The Topaz Lounge is a little out of the way, but that means it's not nearly as deafening as the show floor. Last year we had a complementary coffee/soda bar set up.

Friday, September 12th

11:45am - 3:00pm: Silverlight Formats

Saturday, September 13th

9:15 am - 1:30pm: Floating

Sunday, September 14th

9:15 am - 1:30pm: Silverlight Formats

Monday, September 15th

1:30pm - 6:00pm: Silverlight Formats

Tuesday, September 16th

9:15 am - 12:30pm: Floating

H.264 and AAC support coming in Silverlight

benwaggoner — Tue, 09 Sep 2008 19:09:00 GMT

So, our big IBC press announcement went out this morning. Lots of blog-worthy stuff in there, but as a compression nerd, it's the codec stuff I'm going to talk about first.

The big news is that, yes, we're going to add support for MPEG-4 to Silverlight, in the version coming after the fast-approaching fall release of Silverlight 2. Specifically, this will be H.264 for video and AAC for audio.

Why?

So, why MPEG-4 support in Silverlight? It's pretty straightforward, really. We have customers with libraries of H.264 content they wanted to publish to Silverlight, but didn't want to reencode to VC-1. Silverlight's strengths go far beyond media playback, and customers wanted the choice to deploy a wide variety of existing content within Silverlight. Silverlight aspires to provide as much choice as feasible as to how Silverlight can be authored and delivered. H.264 support is something we'd considered for past versions, but there were higher priority features we needed to deliver first. Silverlight 2 provides us a very rich base for delivering web apps, so we can start spreading our wings a bit.

What?

We've got more technical details we'll be sharing at IBC and later, but today, I'll just quote the details from Scott Guthrie's interview:

PressPass: Will you be showing or announcing anything new at the IBC conference this week?

Guthrie: Yes. At IBC 2008 we will be demonstrating a technology preview of H.264 video and Advanced Audio Coding (AAC) audio playback support in Silverlight, and H.264 authoring using Microsoft Expression Encoder and Windows Server 2008 for delivery. Until now, Silverlight has supported the SMPTE VC-1 and Windows Media formats, as well as MP3 for audio, enabling customers to take advantage of broad support across the Windows Media ecosystem, including third-party tools, service providers and content delivery networks.

We’ve always wanted Silverlight to support a variety of formats, so today we’re announcing that H.264 and AAC support will be available in a future version of Silverlight, which will offer content owners greater flexibility and choice to deliver video and audio.

PressPass: Historically, people have associated Microsoft with VC-1. Does this signal a change in direction?

Guthrie: No. Although we have been working with VC-1 for some time, it’s not widely recognized that Microsoft has also been an active participant in the standardization of H.264/MPEG AVC for many years, and we’ve included H.264 support in several Microsoft products. Microsoft’s Gary Sullivan was the chairman of the Joint Video Team (JVT), which developed the H.264 standard, and he recently accepted an Emmy Award on behalf of the JVT.

PressPass: Does this mean that Silverlight is moving away from Windows Media?

Guthrie: Not at all. This is about offering our customers more choice. Media producers and distributors around the world have enjoyed the high quality, flexibility and affordability of Windows Media formats for over a decade. As a testament to its pervasiveness, Windows Media can be found in almost every conceivable media scenario from desktop home video to feature films and TV broadcasts.

Again, don't read this as as a big change around our strategy for media formats. We've long-supported WMV and MPEG-4 side-by-side in products like Xbox and Zune. As I posted last week, Microsoft has been contributing to H.264 since its inception.

This doesn't represent any change in our support for Windows Media. Windows Media continues to work well for today's Silverlight customers. I expect (not a goal, just a prediction) that the majority of Silverlight content will remain in WMV well after we release MPEG-4 support.

In particular, VC-1 will likely remain the codec of choice for HD for some time. Comparing VC-1 Advanced Profile to H.264 High Profile with all the bells and whistles turned on, VC-1 only needs about half as many MIPS per pixel for playback. This won't matter as much for lower resolution content, or podcasting stuff that's in the simpler Baseline profile, but makes for a big reduction in system requirements for 720p and higher.

Where/When can I learn more?

If you're at IBC, swing by! If you want a head start on using MPEG-4 in Silverlight, feel free contact me directly. And we'll have plenty of more technical info to share down the road a bit.

Technical Emmy for H.264/MPEG-4 AVC

benwaggoner — Thu, 28 Aug 2008 19:05:00 GMT

I have many ways of reminding my wife I'm an hopeless video nerd. Among them is that I only pay attention to the Emmy Awards she's watching when the recap of the technical awards comes on.

So, I was really excited that our own Dr. Gary Sullivan, chairman of the Joint Video Team, was on hand last week to receive a 2008 Primetime Emmy^® Engineering Award for developing H.264/MPEG-4 AVC High Profile. In the words of the Academy:

Joint Video Team Standards Committee (JVT) for the development of the High Profile for H.264 / MPEG-4 AVC. The High Profile enables high definition images in the H.264 video coding system used today to deliver HD video over satellite and cable TV as well as Blu-ray Disc. The JVT is a standardization team comprised of members from the International Standardization Organization (ISO), the International Electrotechnical Commission (IEC), and the International Telecommunication Union (ITU).

The award recognized Broadcom, NTT DoCoMo, Dolby Laboratories, Fraunhofer Heinrich Hertz Institute, Microsoft, Motorola, Panasonic, Sony, and Thomson as contributing companies, as well as standards organizations ISO, IEC, and ITU.

Gary has long represented Microsoft as as chair and co-chair of the Joint Video Team that coordinated the H.264/MPEG-4 AVC effort between ITU-T VCEG and ISO/IEC MPEG, amongst his many other efforts like DXVA; his page on Wikipedia has more details.

I suppose some people might be surprised at Microsoft's inclusion here, but they shouldn't be. While Microsoft has a long heritage of making our own ground-up codecs like VC-1, we've also long been involved in standards-based codecs, including the original MPEG-4 Part 2 as well as MPEG-4 Part 10/H.264. And we've got a number of products that incorporate H.264 today, including Mediaroom, the Zune 4/8 and 80, and Xbox 360. H.264 are and VC-1 are both great codecs, with somewhat different sweet spots, and we use either or both as appropriate in our products, platforms, and services.

The picture above has, left to right: Malcolm Johnson (director of the ITU standardization bureau), Ajay Luthra (JVT vice-chair from Motorola), Gary Sullivan (JVT chair from Microsoft), Thomas Wiegand (JVT vice-chair from Fraunhofer HHI), and Scott Jameson (chair of ISO/IEC JTC 1)

ANSI

EE Times

Dolby

Minoru "Mick" Etoh's blog (who represented DoCoMo at the ceremony)