Uploading Large Files To Self Hosted WCF Rest Service

There are several helpful posts regarding uploading and downloading large files from REST and SOAP based WCF services. I wanted to create my own example of uploading large files to a simple WCF 4.0 REST based service. By large, I mean files 2GiB or greater in size.

To do this for WCF, streaming has to be used. WCF, REST, and Streaming are 3 main search teams that helped me Google this in my research of this topic. Here is one helpful post I came across:

http://social.msdn.microsoft.com/Forums/en/wcf/thread/fb9efac5-8b57-417e-9f71-35d48d421eb4.

I got to thinking, if the file is large, processing the request server side will take a long time, tying up a WCF worker pool thread. Processing the request asynchronously on the server will prevent worker pool threads from being tied up for a long time. A useful example of this is given at:

http://msdn.microsoft.com/en-us/library/ms731177.aspx

For this example, a WCF service library was created for handling the uploads and was hosted in a console application. In a later post, we’ll try this in IIS 7.5 hosting this same library. To setup streaming for a WCF 4.0 REST service, we need to set some settings and this was done in the config file.

<?xml version="1.0"?>
<configuration>
  <startup><supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/></startup>
  <system.serviceModel>
    <bindings>
      <webHttpBinding>
        <binding name="streamWebHttpBinding" maxReceivedMessageSize="1000000000000" receiveTimeout="01:00:00" sendTimeout="01:00:00" transferMode="Streamed" />
      </webHttpBinding>
    </bindings>
    <behaviors>
      <serviceBehaviors>
        <behavior name="LargeUploadBehavior">
          <serviceDebug includeExceptionDetailInFaults="true"/>
        </behavior>
      </serviceBehaviors>
      <endpointBehaviors>
        <behavior name="RestBehavior">
          <webHttp />
        </behavior>
      </endpointBehaviors>
    </behaviors>
    <services>
      <service name="LargeUpload.UploadData" behaviorConfiguration="LargeUploadBehavior"  >        
        <endpoint behaviorConfiguration="RestBehavior" address="http://localhost:8085/lupload" contract="LargeUpload.IUpload" binding="webHttpBinding" bindingConfiguration="streamWebHttpBinding" />
      </service>
    </services>
  </system.serviceModel>
</configuration>

The transferMode attribute is set to Streamed for webHttpBinding in bindings. The send and receive timeouts need to be lengthened and the maxReceivedMessageSize also needs to be set to a large enough value. The client typically will use an HttpWebRequest to upload the file and for streaming will need to set AllowWriteStreamBuffering = false and SendChunked = true.

The service contract for the service is shown below:

    /// <summary>
    /// Interface for sample upload WCF service.
    /// </summary>
    [ServiceContract]
    public interface IUpload
    {
        /// <summary>
        /// Returns a simple string to a client to make sure the service is working.
        /// </summary>
        [OperationContract]
        [WebGet(UriTemplate="")]
        Stream Default();

        /// <summary>
        /// Upload, runs synchronously service side.
        /// </summary>
        /// <param name="token">An application arbitrary piece of data.  Can be used for request obfuscation.</param>
        /// <param name="data">The data being uploaded.</param>
        [OperationContract]
        [WebInvoke(UriTemplate="upload/{token}")]
        void Upload(string token, Stream data);

        /// <summary>
        /// An asynchronous service side upload operation.
        /// </summary>
        /// <param name="token">An application arbitrary piece of data.  Can be used for request obfuscation.</param>
        /// <param name="data">The data being uploaded.</param>
        /// <param name="callback">Callback for async pattern, client does not pass this.</param>
        /// <param name="asyncState">User state for async pattern, client does not pass this.</param>
        /// <remarks>
        /// The <paramref name="token"/> parameter is the only parameter passed in the URL by the client.  The <paramref name="data"/>
        /// parameter is the request body, the file being uploaded.
        /// </remarks>
        /// <returns></returns>
        [OperationContract(AsyncPattern = true)]
        [WebInvoke(UriTemplate="asyncupload/{token}")]
        IAsyncResult BeginAsyncUpload(string token, Stream data, AsyncCallback callback, object asyncState);


        /// <summary>
        /// Ends the asynchonous operation initiated by the call to <see cref="BeginAsyncUpload"/>.
        /// </summary>
        /// <remarks>
        /// This is called by the WCF framework service side.  NOTE:  There is on <see cref="OperationContractAttribute"/> decorating
        /// this method.
        /// </remarks>
        /// <param name="ar"></param>
        void EndAsyncUpload(IAsyncResult ar);

    }

Following is the service implentation:

    /// <summary>
    /// Implementation of <see cref="IUpload.Upload"/> for 
    /// </summary>
    [ServiceBehavior]
    public class UploadData : IUpload
    {
        
        /// <summary>
        /// <see cref="IUpload.Default"/>.  All this does is provide a test for whether or not the
        /// service is working, for debugging only.
        /// </summary>
        public Stream Default()
        {
            string retStream = "success";
            MemoryStream ms = new MemoryStream(Encoding.UTF8.GetBytes(retStream));

            WebOperationContext ctx = WebOperationContext.Current;
            ctx.OutgoingResponse.ContentType = "text/plain";
            ctx.OutgoingResponse.ContentLength = ms.Length;

            return ms;
        }

        #region IUpload

        /// <summary>
        /// <see cref="IUpload.Upload"/>
        /// </summary>
        /// <param name="token">This parameter is ignored.</param>
        /// <param name="data">Data being uploaded.</param>
        public void Upload(string token, System.IO.Stream data)
        {
            _streamToFile(data);
        }

        /// <summary>
        /// <see cref="IUpload.Upload"/>
        /// </summary>
        /// <param name="token">This parameter is ignored.</param>
        /// <param name="data">Data being uploaded.</param>
        /// <param name="callback">Async callback.</param>
        /// <param name="asyncState">Async user state.</param>
        public IAsyncResult BeginAsyncUpload(string token, Stream data, AsyncCallback callback, object asyncState)
        {
            return new CompletedAsyncResult<Stream>(data);
        }

        /// <summary>
        /// <see cref="IUpload.EndAsyncUpload"/>
        /// </summary>
        public void EndAsyncUpload(IAsyncResult ar)
        {
            Stream data = ((CompletedAsyncResult<Stream>)ar).Data;
            _streamToFile(data);
        }

        #endregion

        /// <summary>
        /// Writes the uploaded stream to a file.
        /// </summary>
        /// <remarks>
        /// This function is just to prove a test.  This simple saves the uploaded data into a file named &quot;upload.dat&quot; in a subdirectory
        /// whose name is created by a generated guid.
        /// </remarks>
        private static void _streamToFile(Stream data)
        {
            // create name of subdirectory
            string subDir = Guid.NewGuid().ToString("N");

            // get full path to and create the directory to save file in
            string uploadDir = Path.Combine(Path.GetDirectoryName(typeof(UploadData).Assembly.Location), subDir);
            Directory.CreateDirectory(uploadDir);

            // 64 KiB buffer
            byte[] buff = new byte[0x10000];

            // save the file in chunks
            using (FileStream fs = new FileStream(Path.Combine(uploadDir, "upload.dat"), FileMode.Create))
            {
                int bytesRead = data.Read(buff, 0, buff.Length);
                while (bytesRead > 0)
                {
                    fs.Write(buff, 0, bytesRead);
                    bytesRead = data.Read(buff, 0, buff.Length);
                }
            }
        }


    }

Using this test setup, I was able to upload files larger than 1 GiB. The solution has been zipped up and included as an attachment.

Here is the Visual Studio 2010 solution if you wish to download it:

REST Large Upload Demo

Download PDF
This entry was posted in ASP .NET, C#, WCF and tagged , , . Bookmark the permalink.

2 Responses to Uploading Large Files To Self Hosted WCF Rest Service

  1. Doug says:

    Phillipp,

    I believe this post relates to the hosting under IIS issue. I just found this today. http://weblogs.asp.net/jclarknet/archive/2008/02/14/wcf-streaming-issue-under-iis.aspx

  2. Philipp says:

    Have you ever postet the solution with the IIS host? Cause i have a lot of problems when i want to host a restful-wcf-svc-file in iis. In my opinion there is a problem with the length of existing buffers.

Leave a Reply

Your email address will not be published. Required fields are marked *