C#/ASP.NET HTTP POST to page with generated content

Hello all,

Let's say we have a webpage, www.site.com. It has a basic form and input boxes for a username and password. It uses a POST to send the username and password.

If you wanted to login to the site externally, programmatically, from a different site etc... You could use WebClient() OR WebRequest() to do so and get a respone.

Pretty simple to do.

Now, taking that same situation, except everytime you visit the page a unique md5 hash is generated and is ALSO sent with your username/pass when you click 'submit' or whatever and it POSTs. So:

www.site.com/login.php?md5='md5hashgoeshere'&user=John&pass=Doe


So first you have to find the md5 hash because when you go to the site, it is generated before you login (it's more for a session ID sorta thing).

So let's say you have a small script that crawls through the page and finds the hash, In order to do this you have to download the data from your webpage.

Now logically it would make sense to of course use the same url you used to find the md5 (www.site.com). BUT, when you use it say for a webclient.uploadData(POST in here) it doesn't use that same *instance* of the site to find the hash. Same URL, but new instance of the website so a new hash.

Is there a way to use the same instance of the website to find the hash and send a POST method to the same instance of the website that you used to find the hash? Otherwise the hash will be generated again and you will fail to login.

Long post, sorry! Thanks!

 

Anyone have an idea as to what I am asking and/or talking about ?

 

no clue. maybe you should explain a little about why you need to do this? Just to put some context on it.

 

Why? No specific reason, more to learn about POST and GET methods and trying to retrieve information from a website externally, like from a different website.

Instead of logging into www.site.com directly, why not log into the site from www.site355.com

 

Okay let me try explaining it this way

I start out by using webrequest:

WebRequest req;
req = WebRequest.create(url)

Code (markup):

I then create a stream of the website and read it (like looking at the source via a text editor)

Stream webStream;
webStream = req.GetResponse().GetResponseStream();
StreamReader webReader = new StreamReader(webStream);
string fContents = "";
fContents = webReader.ReadToEnd();

Code (markup):

So now, I read the stream and then I do stuff with the data and now i want to send a POST to the website through the same stream as to not lose the instance of the page i am in right now.

string data = "something";
byte[] bytes = Encoding.ASCII.GetBytes(data);
req.ContentLength = bytes.Length;
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
Stream dataStream = req.GetRequestStream();
dataStream.Write(bytes, 0, bytes.Length);
dataStream.Close();
WebResponse response = req.GetResponse();
Stream rData = response.GetResponseStream();

Code (markup):

So to recap:

//Create the instance of the website
WebRequest req;
req = WebRequest.create(url)

//Website is put into a stream that is then read
Stream webStream;
webStream = req.GetResponse().GetResponseStream();
StreamReader webReader = new StreamReader(webStream);
string fContents = "";
fContents = webReader.ReadToEnd();

// do something with the data and put it in a string variable called "data"
string data = "something";

//Send the data back to the website via a POST method
byte[] bytes = Encoding.ASCII.GetBytes(data);
req.ContentLength = bytes.Length;
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
Stream dataStream = req.GetRequestStream();
dataStream.Write(bytes, 0, bytes.Length);
dataStream.Close();

//Get a final response from the website
WebResponse response = req.GetResponse();
Stream rData = response.GetResponseStream();

Code (markup):

Hopefully this now makes a lot more sense.

My first problem is that, I don't know if this is even going to happen correctly because I can't test it. I get an error:

Exception Details: System.InvalidOperationException: This property cannot be set after writing has started.

req.ContentLength = bytes.Length; //this is the line it errors on

Code (markup):

Anyone have an idea what this means? Many thanks in advance!

-Ferbal

 

Ok I *Think* I know what you are talking about.

A lot of sites do this, and the way around it is to scrape the screen, get the hash - and store the cookies. When you make the next POST, supply the cookie value - that way you will use the same session.

EG (ASP 2.0, but this works just as well using ASP 1.1)


Private Function sGetWeb(ByVal sURL As String, ByVal sURLParams As String, ByRef oCookies As CookieContainer)
Dim oCertOverride As New CertificateOverride

Dim Writer As StreamWriter = Nothing

'-- over ride the bad certificate error
ServicePointManager.ServerCertificateValidationCallback = AddressOf oCertOverride.RemoteCertificateValidationCallback

'-- open the channel to web site
Dim oReq As HttpWebRequest = System.Net.HttpWebRequest.Create(sURL)
oReq.ContentType = "application/x-www-form-urlencoded"
oReq.ContentLength = Len(sURLParams)
oReq.CookieContainer = oCookies

oReq.Method = "POST"
'-- set the credentials for HTTPS
Dim oCred As New System.Net.NetworkCredential("", "")
oReq.Credentials = oCred

Writer = New StreamWriter(oReq.GetRequestStream())
Writer.Write(sURLParams)
Writer.Close()
Writer = Nothing


'-- get a response from the site
Dim oResp As WebResponse = oReq.GetResponse()
'-- attach the stream to a reader
Dim oSRead As New StreamReader(oResp.GetResponseStream)

'-- get the content
Dim cContent As String = oSRead.ReadToEnd

sGetWeb = cContent
End Function

This function uses POST, but you can also change the method to GET. It also supplys credentials for HTTPS sites, and you might not need that.

To call it, use:

Dim oCookies As New CookieContainer
Dim sURL As String
Dim sURLParams As String
Dim sOutText As String
Dim sHashCode as string

sURL = "https://SiteToLoginTo.com/Login.jsp"
sURLParams = ""

sOutText = sGetWeb(sURL, sURLParams, oCookies)
sHashCode = sExtractHashCode(sOutText)
sOutText = sGetWeb("https://SiteToLoginTo.com/LoginSubmit.jsp", "Hash=" & sHashCode, oCookies)


Of course, as well as submitting the hash code, you have to submit the username & password & any other form vars.

Hopefully this helps.

 

FYI: All params are brought in from web.config.
(Notice the ConfigurationSettings.AppSettings["Key"]; calls)

The values in relation to each key look something like this:
loginId=foo&password=bar

public void setTvCrawlDisplay(String tvText, DateTime tvExpiration)
{
   //JFB - Calculate minutes for warning display.    
   DateTime currentDateTime = DateTime.Now;

   //JFB - Difference in days, hours, and minutes.    
   TimeSpan tvTimeSpan = tvExpiration - currentDateTime;
      
   //JFB - Difference in minutes.    
   int differenceInMinutes = (int) tvTimeSpan.TotalMinutes;   
   
   String url = ConfigurationSettings.AppSettings["URL"];
   String strPost = ConfigurationSettings.AppSettings["LoginPassword"];
   StreamWriter myWriter = null;
   CookieContainer myContainer = new CookieContainer();
   
   //Request #1 (the login)    
   HttpWebRequest objRequest = (HttpWebRequest)WebRequest.Create(url);
   objRequest.Method = "POST";
   objRequest.ContentLength = strPost.Length;
   objRequest.ContentType = "application/x-www-form-urlencoded";         
   objRequest.CookieContainer = new CookieContainer();

   try
   {
      myWriter = new StreamWriter(objRequest.GetRequestStream());
      myWriter.Write(strPost);
   }
   catch (Exception e) 
   {
      Console.WriteLine(e.Message);
   }
   finally 
   {
      myWriter.Close();
   }
       
   HttpWebResponse objResponse = (HttpWebResponse)objRequest.GetResponse();
   //retain the cookies    
   foreach (Cookie cook in objResponse.Cookies)
   {
      myContainer.Add(cook); 
   }
   
   //Check out the html.    
   using (StreamReader sr = 
          new StreamReader(objResponse.GetResponseStream()) )
   {
      String test = sr.ReadToEnd();

      // Close and clean up the StreamReader       
      sr.Close();
   }
   
   //Request #2 (select the proper submenu)    
   objRequest = (HttpWebRequest)WebRequest.Create(url);
   strPost = strPost = ConfigurationSettings.AppSettings["EditMenu"];
   objRequest = (HttpWebRequest)WebRequest.Create(url);
   objRequest.Method = "POST";
   objRequest.ContentLength = strPost.Length;
   objRequest.ContentType = "application/x-www-form-urlencoded";         
   objRequest.CookieContainer = myContainer;

   try
   {
      myWriter = new StreamWriter(objRequest.GetRequestStream());
      myWriter.Write(strPost);
   }
   catch (Exception e) 
   {
      Console.WriteLine(e.Message);
   }
   finally 
   {
      myWriter.Close();
   }
       
   objResponse = (HttpWebResponse)objRequest.GetResponse();
   
   //Check out the html.
   
   using (StreamReader sr = 
          new StreamReader(objResponse.GetResponseStream()) )
   {
      String test = sr.ReadToEnd();

      // Close and clean up the StreamReader       
      sr.Close();
   }
}

Code (markup):

try this fore more info

i also try to login and yahoo address book download but still not done if any one know please relpay me