Uploading Large Files To Self Hosted WCF Rest Service

There are several helpful posts regarding uploading and downloading large files from REST and SOAP based WCF services. I wanted to create my own example of uploading large files to a simple WCF 4.0 REST based service. By large, I mean files 2GiB or greater in size.

To do this for WCF, streaming has to be used. WCF, REST, and Streaming are 3 main search teams that helped me Google this in my research of this topic. Here is one helpful post I came across:

http://social.msdn.microsoft.com/Forums/en/wcf/thread/fb9efac5-8b57-417e-9f71-35d48d421eb4.

I got to thinking, if the file is large, processing the request server side will take a long time, tying up a WCF worker pool thread. Processing the request asynchronously on the server will prevent worker pool threads from being tied up for a long time. A useful example of this is given at:

http://msdn.microsoft.com/en-us/library/ms731177.aspx

For this example, a WCF service library was created for handling the uploads and was hosted in a console application. In a later post, we’ll try this in IIS 7.5 hosting this same library. To setup streaming for a WCF 4.0 REST service, we need to set some settings and this was done in the config file.

[source language=”xml” collapse=”1″]
<?xml version="1.0"?>
<configuration>
<startup><supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/></startup>
<system.serviceModel>
<bindings>
<webHttpBinding>
<binding name="streamWebHttpBinding" maxReceivedMessageSize="1000000000000" receiveTimeout="01:00:00" sendTimeout="01:00:00" transferMode="Streamed" />
</webHttpBinding>
</bindings>
<behaviors>
<serviceBehaviors>
<behavior name="LargeUploadBehavior">
<serviceDebug includeExceptionDetailInFaults="true"/>
</behavior>
</serviceBehaviors>
<endpointBehaviors>
<behavior name="RestBehavior">
<webHttp />
</behavior>
</endpointBehaviors>
</behaviors>
<services>
<service name="LargeUpload.UploadData" behaviorConfiguration="LargeUploadBehavior" >
<endpoint behaviorConfiguration="RestBehavior" address="http://localhost:8085/lupload" contract="LargeUpload.IUpload" binding="webHttpBinding" bindingConfiguration="streamWebHttpBinding" />
</service>
</services>
</system.serviceModel>
</configuration>
[/source]

The transferMode attribute is set to Streamed for webHttpBinding in bindings. The send and receive timeouts need to be lengthened and the maxReceivedMessageSize also needs to be set to a large enough value. The client typically will use an HttpWebRequest to upload the file and for streaming will need to set AllowWriteStreamBuffering = false and SendChunked = true.

The service contract for the service is shown below:

[source language=”csharp” collapsed=”1″]
/// <summary>
/// Interface for sample upload WCF service.
/// </summary>
[ServiceContract]
public interface IUpload
{
/// <summary>
/// Returns a simple string to a client to make sure the service is working.
/// </summary>
[OperationContract]
[WebGet(UriTemplate="")]
Stream Default();

/// <summary>
/// Upload, runs synchronously service side.
/// </summary>
/// <param name="token">An application arbitrary piece of data. Can be used for request obfuscation.</param>
/// <param name="data">The data being uploaded.</param>
[OperationContract]
[WebInvoke(UriTemplate="upload/{token}")]
void Upload(string token, Stream data);

/// <summary>
/// An asynchronous service side upload operation.
/// </summary>
/// <param name="token">An application arbitrary piece of data. Can be used for request obfuscation.</param>
/// <param name="data">The data being uploaded.</param>
/// <param name="callback">Callback for async pattern, client does not pass this.</param>
/// <param name="asyncState">User state for async pattern, client does not pass this.</param>
/// <remarks>
/// The <paramref name="token"/> parameter is the only parameter passed in the URL by the client. The <paramref name="data"/>
/// parameter is the request body, the file being uploaded.
/// </remarks>
/// <returns></returns>
[OperationContract(AsyncPattern = true)]
[WebInvoke(UriTemplate="asyncupload/{token}")]
IAsyncResult BeginAsyncUpload(string token, Stream data, AsyncCallback callback, object asyncState);

/// <summary>
/// Ends the asynchonous operation initiated by the call to <see cref="BeginAsyncUpload"/>.
/// </summary>
/// <remarks>
/// This is called by the WCF framework service side. NOTE: There is on <see cref="OperationContractAttribute"/> decorating
/// this method.
/// </remarks>
/// <param name="ar"></param>
void EndAsyncUpload(IAsyncResult ar);

}
[/source]

Following is the service implentation:

[source language=”csharp” collapsed=”1″]
/// <summary>
/// Implementation of <see cref="IUpload.Upload"/> for
/// </summary>
[ServiceBehavior]
public class UploadData : IUpload
{

/// <summary>
/// <see cref="IUpload.Default"/>. All this does is provide a test for whether or not the
/// service is working, for debugging only.
/// </summary>
public Stream Default()
{
string retStream = "success";
MemoryStream ms = new MemoryStream(Encoding.UTF8.GetBytes(retStream));

WebOperationContext ctx = WebOperationContext.Current;
ctx.OutgoingResponse.ContentType = "text/plain";
ctx.OutgoingResponse.ContentLength = ms.Length;

return ms;
}

#region IUpload

/// <summary>
/// <see cref="IUpload.Upload"/>
/// </summary>
/// <param name="token">This parameter is ignored.</param>
/// <param name="data">Data being uploaded.</param>
public void Upload(string token, System.IO.Stream data)
{
_streamToFile(data);
}

/// <summary>
/// <see cref="IUpload.Upload"/>
/// </summary>
/// <param name="token">This parameter is ignored.</param>
/// <param name="data">Data being uploaded.</param>
/// <param name="callback">Async callback.</param>
/// <param name="asyncState">Async user state.</param>
public IAsyncResult BeginAsyncUpload(string token, Stream data, AsyncCallback callback, object asyncState)
{
return new CompletedAsyncResult<Stream>(data);
}

/// <summary>
/// <see cref="IUpload.EndAsyncUpload"/>
/// </summary>
public void EndAsyncUpload(IAsyncResult ar)
{
Stream data = ((CompletedAsyncResult<Stream>)ar).Data;
_streamToFile(data);
}

#endregion

/// <summary>
/// Writes the uploaded stream to a file.
/// </summary>
/// <remarks>
/// This function is just to prove a test. This simple saves the uploaded data into a file named &quot;upload.dat&quot; in a subdirectory
/// whose name is created by a generated guid.
/// </remarks>
private static void _streamToFile(Stream data)
{
// create name of subdirectory
string subDir = Guid.NewGuid().ToString("N");

// get full path to and create the directory to save file in
string uploadDir = Path.Combine(Path.GetDirectoryName(typeof(UploadData).Assembly.Location), subDir);
Directory.CreateDirectory(uploadDir);

// 64 KiB buffer
byte[] buff = new byte[0x10000];

// save the file in chunks
using (FileStream fs = new FileStream(Path.Combine(uploadDir, "upload.dat"), FileMode.Create))
{
int bytesRead = data.Read(buff, 0, buff.Length);
while (bytesRead > 0)
{
fs.Write(buff, 0, bytesRead);
bytesRead = data.Read(buff, 0, buff.Length);
}
}
}

}
[/source]

Using this test setup, I was able to upload files larger than 1 GiB. The solution has been zipped up and included as an attachment.

Here is the Visual Studio 2010 solution if you wish to download it:

REST Large Upload Demo

This entry was posted in ASP .NET, C#, WCF and tagged , , . Bookmark the permalink.

2 Responses to Uploading Large Files To Self Hosted WCF Rest Service

  1. Doug says:

    Phillipp,

    I believe this post relates to the hosting under IIS issue. I just found this today. http://weblogs.asp.net/jclarknet/archive/2008/02/14/wcf-streaming-issue-under-iis.aspx

  2. Philipp says:

    Have you ever postet the solution with the IIS host? Cause i have a lot of problems when i want to host a restful-wcf-svc-file in iis. In my opinion there is a problem with the length of existing buffers.

Leave a Reply

Your email address will not be published. Required fields are marked *