Avatars must be image file types.

I am attempting to update a user's avatar with a script that generates an image with user's initials.  When I try execute the following code, I get a validation error that "Avatars must be image file types"

The URL loads in the browser and renders the avatar correctly.

               var response = client.DownloadData(endpoint);
                var res = Apis.Get<IUserAvatars>().Update(e.Id.Value, new UserAvatarsUpdateOptions { FileData = response, FileName = e.Id.Value.ToString() + "avatar.jpg" });

The endpoint is a URL on our domain that renders a jpg

My code snippet that generates the avatar is as follows:

                context.Response.ContentType = "image/jpg";
                MemoryStream temp = new MemoryStream();
                bmp.Save(temp, ImageFormat.Jpeg);
                byte[] buffer = temp.GetBuffer();
                context.Response.OutputStream.Write(buffer, 0, buffer.Length);
            }