Tuesday, September 27, 2011

Silverlight 5 P-Invoke: Using Kinect SDK from Silverlight

Away from all the discussions about whether Silverlight is dead or not, The Silverlight 5 RC contains the previously announced P-Invoke feature which enables you to call Win32 style APIs from a trusted Silverlight application.

There are some attempts to use Kinect from Silverlight (here) but this was before MS released the official SDK. we will use Silverlight 5 P-Invoke feature to call the Kinect SDK APIs.

P-Invoke in Silverlight works just like P-Invoke on the desktop. you use the DllImport attribute to import the APIs and you declare in your code the dependent types (Enums, structs, etc.)

  1. [DllImport("MSRKINECTNUI.DLL")]
  2.       private static extern HRESULT NuiInitialize(uint dwFlags);

I tried to keep the classes and methods in library identical to the ones in the official SDK. I implemented only a couple of the available APIs (with the help of the Coding4Fun Kinect Toolkit and Reflector).

To test the library we will create a simple application:

- Create a new Silverlight 5 application

- Add a reference to the KinectSilverlightLibrary.

- Change the properties of the project: enable running out of browser.

Enable running out of browser

- Change the Out-of-Browser settings to require elevated trust

Require elevated trust

- In the App.xaml.cs file, define a new property of type Runtime, this property represents the Kinect runtime, in the Application_Startup event handler initialize the runtime and in the Application_Exit event handler shut down the runtime

  1. Runtime kinectRuntime = new Runtime();
  2. public Runtime KinectRuntime
  3. {
  4.     get
  5.     { return kinectRuntime; }
  6. }
  7. private void Application_Startup(object sender, StartupEventArgs e)
  8. {
  9.     this.RootVisual = new MainPage();
  10.     kinectRuntime.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseDepth );
  11. }
  12. private void Application_Exit(object sender, EventArgs e)
  13. {
  14.     kinectRuntime.Uninitialize();
  15. }

- In the MainPage.xaml file add two image controls that we will use to display the video and depth streams coming from Kinect

  1. <Grid x:Name="LayoutRoot" Background="White" Loaded="LayoutRoot_Loaded">
  2.         <Grid.ColumnDefinitions>
  3.             <ColumnDefinition/>
  4.             <ColumnDefinition/>
  5.         </Grid.ColumnDefinitions>
  6.         <Image Name="image1" Stretch="Fill" Grid.Column="0" />
  7.         <Image Name="image2" Stretch="Fill" Grid.Column="1" />
  8.     </Grid>

- In the MainPage.xaml.cs add the following lines of code

  1. private void LayoutRoot_Loaded(object sender, RoutedEventArgs e)
  2.         {
  3.            Runtime runtime = (App.Current as App).KinectRuntime;
  4.            runtime.VideoStream.OpenStream(ImageType.Color, ImageResolution.Resolution640x480);
  5.           runtime.DepthStream.OpenStream(ImageType.Depth, ImageResolution.Resolution320x240);
  6.          runtime.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(runtime_VideoFrameReady);
  7.            runtime.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(runtime_DepthFrameReady);
  8.         }
  9.     
  10.         void runtime_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
  11.         {
  12.             Dispatcher.BeginInvoke(() => {
  13.                 image1.Source = e.ImageFrame.ToBitmapSource();
  14.             });
  15.         }
  16.         void runtime_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)
  17.         {
  18.             Dispatcher.BeginInvoke(() =>
  19.             {
  20.                 image2.Source = e.ImageFrame.ToBitmapSource();
  21.             });
  22.  
  23.         }

When the page is loaded we open the video and depth streams and subscribe to the VideoFrameReady and DepthFrameReady events, in the event handler we retrieve the ImageFrame and convert it to a bitmap (I used the WritableBitmapEx library)

Here’s the application running

Kinect Video and Depth Streams 

You are free to continue adding the rest of the APIs to this library.

You can download the source code from here:

6 comments:

Anonymous said...

Hey....that is a really cool. Do you have any examples of how to read/write to the COM/Serial ports from SL5 using PInvoke?

Anonymous said...

Ciao.. thank you for sharing, this is really cool. Take a look at MIRIA SDK http://miria.codeplex.com . It enables using of Kinect, Wii mote and other devices already starting from Silverlight v2. =)

Programmer Connor said...

Thank you for sharing SL library and source code. I'm downloading and running source code. But there have some problem.
show you picture:
http://4.bp.blogspot.com/-OoQt5sW8bcQ/TwtdwgOI0GI/AAAAAAAAAPk/9cKDNIaoh8E/s1600/1-10-2012%2B5-32-41%2BAM.jpg

I'm using Windows Seven x64 OS. I think SL Library not support x64 OS?

Gwenaƫl said...

Looks very cool. Unfortunately I have exactly the same exception thrown as Programmer Connor has.
Could it be because of use of beta 2 or x64?
Thanks for answering

WINDOWS PHONE 7 DEVELOPMENT said...

Thanks for sharing your info. I really appreciate your efforts and I will be waiting for your further write ups thanks once again.
SEO tools

Anonymous said...

Hi, I tried to work it out but I failed, do you have an updated version? For kinect for windows v1?