1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

C#/ASP.NET HTTP POST to page with generated content

Discussion in 'C#' started by Ferbal, Aug 2, 2006.

  1. #1
    Hello all,

    Let's say we have a webpage, www.site.com. It has a basic form and input boxes for a username and password. It uses a POST to send the username and password.

    If you wanted to login to the site externally, programmatically, from a different site etc... You could use WebClient() OR WebRequest() to do so and get a respone.

    Pretty simple to do.

    Now, taking that same situation, except everytime you visit the page a unique md5 hash is generated and is ALSO sent with your username/pass when you click 'submit' or whatever and it POSTs. So:

    www.site.com/login.php?md5='md5hashgoeshere'&user=John&pass=Doe


    So first you have to find the md5 hash because when you go to the site, it is generated before you login (it's more for a session ID sorta thing).

    So let's say you have a small script that crawls through the page and finds the hash, In order to do this you have to download the data from your webpage.

    Now logically it would make sense to of course use the same url you used to find the md5 (www.site.com). BUT, when you use it say for a webclient.uploadData(POST in here) it doesn't use that same *instance* of the site to find the hash. Same URL, but new instance of the website so a new hash.
    SEMrush
    Is there a way to use the same instance of the website to find the hash and send a POST method to the same instance of the website that you used to find the hash? Otherwise the hash will be generated again and you will fail to login.

    Long post, sorry! Thanks!
     
    Ferbal, Aug 2, 2006 IP
    SEMrush
  2. Ferbal

    Ferbal Peon

    Messages:
    9
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Anyone have an idea as to what I am asking and/or talking about :)?
     
    Ferbal, Aug 9, 2006 IP
  3. benjymouse

    benjymouse Peon

    Messages:
    39
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #3
    no clue. maybe you should explain a little about why you need to do this? Just to put some context on it.
     
    benjymouse, Aug 10, 2006 IP
  4. Ferbal

    Ferbal Peon

    Messages:
    9
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Why? No specific reason, more to learn about POST and GET methods and trying to retrieve information from a website externally, like from a different website.

    Instead of logging into www.site.com directly, why not log into the site from www.site355.com
     
    Ferbal, Aug 13, 2006 IP
  5. Ferbal

    Ferbal Peon

    Messages:
    9
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Okay let me try explaining it this way :)

    I start out by using webrequest:

    
    WebRequest req;
    req = WebRequest.create(url)
    
    Code (markup):
    I then create a stream of the website and read it (like looking at the source via a text editor)

    
    Stream webStream;
    webStream = req.GetResponse().GetResponseStream();
    StreamReader webReader = new StreamReader(webStream);
    string fContents = "";
    fContents = webReader.ReadToEnd();
    
    Code (markup):
    So now, I read the stream and then I do stuff with the data and now i want to send a POST to the website through the same stream as to not lose the instance of the page i am in right now.

    
    string data = "something";
    byte[] bytes = Encoding.ASCII.GetBytes(data);
    req.ContentLength = bytes.Length;
    req.Method = "POST";
    req.ContentType = "application/x-www-form-urlencoded";
    Stream dataStream = req.GetRequestStream();
    dataStream.Write(bytes, 0, bytes.Length);
    dataStream.Close();
    WebResponse response = req.GetResponse();
    Stream rData = response.GetResponseStream();
    
    Code (markup):
    So to recap:

    
    //Create the instance of the website
    WebRequest req;
    req = WebRequest.create(url)
    
    //Website is put into a stream that is then read
    Stream webStream;
    webStream = req.GetResponse().GetResponseStream();
    StreamReader webReader = new StreamReader(webStream);
    string fContents = "";
    fContents = webReader.ReadToEnd();
    
    // do something with the data and put it in a string variable called "data"
    string data = "something";
    
    //Send the data back to the website via a POST method
    byte[] bytes = Encoding.ASCII.GetBytes(data);
    req.ContentLength = bytes.Length;
    req.Method = "POST";
    req.ContentType = "application/x-www-form-urlencoded";
    Stream dataStream = req.GetRequestStream();
    dataStream.Write(bytes, 0, bytes.Length);
    dataStream.Close();
    
    //Get a final response from the website
    WebResponse response = req.GetResponse();
    Stream rData = response.GetResponseStream();
    
    Code (markup):
    Hopefully this now makes a lot more sense.

    My first problem is that, I don't know if this is even going to happen correctly because I can't test it. I get an error:

    
    Exception Details: System.InvalidOperationException: This property cannot be set after writing has started.
    
    req.ContentLength = bytes.Length; //this is the line it errors on
    
    Code (markup):
    Anyone have an idea what this means? Many thanks in advance!

    -Ferbal
     
    Ferbal, Aug 18, 2006 IP
  6. BurgerKing

    BurgerKing Active Member

    Messages:
    397
    Likes Received:
    23
    Best Answers:
    0
    Trophy Points:
    58
    #6
    Ok I *Think* I know what you are talking about.

    A lot of sites do this, and the way around it is to scrape the screen, get the hash - and store the cookies. When you make the next POST, supply the cookie value - that way you will use the same session.

    EG (ASP 2.0, but this works just as well using ASP 1.1)


    Private Function sGetWeb(ByVal sURL As String, ByVal sURLParams As String, ByRef oCookies As CookieContainer)
    Dim oCertOverride As New CertificateOverride

    Dim Writer As StreamWriter = Nothing

    '-- over ride the bad certificate error
    ServicePointManager.ServerCertificateValidationCallback = AddressOf oCertOverride.RemoteCertificateValidationCallback

    '-- open the channel to web site
    Dim oReq As HttpWebRequest = System.Net.HttpWebRequest.Create(sURL)
    oReq.ContentType = "application/x-www-form-urlencoded"
    oReq.ContentLength = Len(sURLParams)
    oReq.CookieContainer = oCookies

    oReq.Method = "POST"
    '-- set the credentials for HTTPS
    Dim oCred As New System.Net.NetworkCredential("", "")
    oReq.Credentials = oCred

    Writer = New StreamWriter(oReq.GetRequestStream())
    Writer.Write(sURLParams)
    Writer.Close()
    Writer = Nothing


    '-- get a response from the site
    Dim oResp As WebResponse = oReq.GetResponse()
    '-- attach the stream to a reader
    Dim oSRead As New StreamReader(oResp.GetResponseStream)

    '-- get the content
    Dim cContent As String = oSRead.ReadToEnd

    sGetWeb = cContent
    End Function

    This function uses POST, but you can also change the method to GET. It also supplys credentials for HTTPS sites, and you might not need that.

    To call it, use:

    Dim oCookies As New CookieContainer
    Dim sURL As String
    Dim sURLParams As String
    Dim sOutText As String
    Dim sHashCode as string

    sURL = "https://SiteToLoginTo.com/Login.jsp"
    sURLParams = ""

    sOutText = sGetWeb(sURL, sURLParams, oCookies)
    sHashCode = sExtractHashCode(sOutText)
    sOutText = sGetWeb("https://SiteToLoginTo.com/LoginSubmit.jsp", "Hash=" & sHashCode, oCookies)


    Of course, as well as submitting the hash code, you have to submit the username & password & any other form vars.

    Hopefully this helps.
     
    BurgerKing, Aug 27, 2006 IP
  7. shaileshk

    shaileshk Well-Known Member

    Messages:
    455
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    108
    #7
    FYI: All params are brought in from web.config.
    (Notice the ConfigurationSettings.AppSettings["Key"]; calls)

    The values in relation to each key look something like this:
    loginId=foo&password=bar

    
    public void setTvCrawlDisplay(String tvText, DateTime tvExpiration)
    {
       //JFB - Calculate minutes for warning display.    
       DateTime currentDateTime = DateTime.Now;
    
       //JFB - Difference in days, hours, and minutes.    
       TimeSpan tvTimeSpan = tvExpiration - currentDateTime;
          
       //JFB - Difference in minutes.    
       int differenceInMinutes = (int) tvTimeSpan.TotalMinutes;   
       
       String url = ConfigurationSettings.AppSettings["URL"];
       String strPost = ConfigurationSettings.AppSettings["LoginPassword"];
       StreamWriter myWriter = null;
       CookieContainer myContainer = new CookieContainer();
       
       //Request #1 (the login)    
       HttpWebRequest objRequest = (HttpWebRequest)WebRequest.Create(url);
       objRequest.Method = "POST";
       objRequest.ContentLength = strPost.Length;
       objRequest.ContentType = "application/x-www-form-urlencoded";         
       objRequest.CookieContainer = new CookieContainer();
    
       try
       {
          myWriter = new StreamWriter(objRequest.GetRequestStream());
          myWriter.Write(strPost);
       }
       catch (Exception e) 
       {
          Console.WriteLine(e.Message);
       }
       finally 
       {
          myWriter.Close();
       }
           
       HttpWebResponse objResponse = (HttpWebResponse)objRequest.GetResponse();
       //retain the cookies    
       foreach (Cookie cook in objResponse.Cookies)
       {
          myContainer.Add(cook); 
       }
       
       //Check out the html.    
       using (StreamReader sr = 
              new StreamReader(objResponse.GetResponseStream()) )
       {
          String test = sr.ReadToEnd();
    
          // Close and clean up the StreamReader       
          sr.Close();
       }
       
       //Request #2 (select the proper submenu)    
       objRequest = (HttpWebRequest)WebRequest.Create(url);
       strPost = strPost = ConfigurationSettings.AppSettings["EditMenu"];
       objRequest = (HttpWebRequest)WebRequest.Create(url);
       objRequest.Method = "POST";
       objRequest.ContentLength = strPost.Length;
       objRequest.ContentType = "application/x-www-form-urlencoded";         
       objRequest.CookieContainer = myContainer;
    
       try
       {
          myWriter = new StreamWriter(objRequest.GetRequestStream());
          myWriter.Write(strPost);
       }
       catch (Exception e) 
       {
          Console.WriteLine(e.Message);
       }
       finally 
       {
          myWriter.Close();
       }
           
       objResponse = (HttpWebResponse)objRequest.GetResponse();
       
       //Check out the html.
       
       using (StreamReader sr = 
              new StreamReader(objResponse.GetResponseStream()) )
       {
          String test = sr.ReadToEnd();
    
          // Close and clean up the StreamReader       
          sr.Close();
       }
    }
    
    
    Code (markup):
    try this fore more info

    i also try to login and yahoo address book download but still not done if any one know please relpay me
     
    shaileshk, May 23, 2007 IP