# Doper programmers: little help please?
**URL:** https://boards.straightdope.com/t/doper-programmers-little-help-please/254842
**Category:** In My Humble Opinion
**Created:** [July 13, 2004, 3:47am UTC](https://boards.straightdope.com/t/doper-programmers-little-help-please/254842 "2004-07-13T03:47:40Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@Skywatcher](https://boards.straightdope.com/u/Skywatcher)
#### Post date: [July 13, 2004, 3:47am UTC](https://boards.straightdope.com/t/doper-programmers-little-help-please/254842/1 "2004-07-13T03:47:40Z")
Saw this on another message board I frequent:
> [@](#):
>
> use a reader on a web page on a site and transfer the data to his universe file, if you are not a programmer you’ve no idea how easy this actually is.
By “use a reader on a web page and transfer the data”, he means an automated process to gather baseball statistics from sites like ESPN and [MLB.com](http://MLB.com) and transfering them to a data file. Is this guy just talking out his ass or what?
---
### Author:  [@ccwaterback](https://boards.straightdope.com/u/ccwaterback)
#### Post date: [July 13, 2004, 4:13am UTC](https://boards.straightdope.com/t/doper-programmers-little-help-please/254842/2 "2004-07-13T04:13:31Z")
Not at all. I have a script (written by a generous SDMB fellow) that “reads” a website every night at 7PM and saves it to my PC. To get baseball stats all you would need to so is program more “page parsing” logic.
* * *
Dim oXMLHTTP, sHTML
Dim oFS, oTS, sFileName, sFilePath
Dim sNewHTML
Set oXMLHTTP = CreateObject(“MSXML2.XMLHTTP.3.0”)
oXMLHTTP.open “GET”,“[http://finance.yahoo.com/a3?o=l:0&d=t](http://finance.yahoo.com/a3?o=l:0&d=t)”,false
oXMLHTTP.send
sHTML = oXMLHTTP.responseText
Set oXMLHTTP = Nothing
sNewHTML = Replace(CStr(sHTML), “/q”, “[http://finance.yahoo.com/q](http://finance.yahoo.com/q)”)
sFileName = “up” & CStr(Year(Date)) & “-” & CStr(Month(Date)) & “-” & CStr(Day(Date)) & “.htm”
sFilePath = “C:\Documents and Settings\Administrator\Desktop\StkStuff”
Set oFS = CreateObject(“Scripting.FileSystemObject”)
Set oTS = oFS.CreateTextFile(sFilePath & sFileName,True,False)
oTS.Write CStr(sNewHTML)
oTS.Close
Set oTS = Nothing
Set oFS = Nothing
’ Run manually the first time to get Norton’s blessing.
---
### Author:  [@friedo](https://boards.straightdope.com/u/friedo)
#### Post date: [July 13, 2004, 5:42am UTC](https://boards.straightdope.com/t/doper-programmers-little-help-please/254842/3 "2004-07-13T05:42:39Z")
I’ve written many such “screen-scraping” programs in Perl. Pretty trivial stuff, as long as they don’t radically change the layout of the page. (But if they do, all you have to do is change your parsing logic to match.)
---
### Author:  [@rjung](https://boards.straightdope.com/u/rjung)
#### Post date: [July 13, 2004, 7:20am UTC](https://boards.straightdope.com/t/doper-programmers-little-help-please/254842/4 "2004-07-13T07:20:02Z")
Note that “scraping” is falling out of vogue, though, because it requires maintaining the reader to follow changes in the format of the parent page. Any site that’s going to present updated information on a regular basis really should have an RSS feed instead, which can be parsed easily.
---
### Author:  [@friedo](https://boards.straightdope.com/u/friedo)
#### Post date: [July 13, 2004, 8:20am UTC](https://boards.straightdope.com/t/doper-programmers-little-help-please/254842/5 "2004-07-13T08:20:55Z")
Generally scraping is only used as a last resort; sites which _want_ to distribute their information usually provide an XML feed of some sort (usually RSS). But if you need to extract information from a site that doesn’t want to bother, then scraping is the way to go.
---
### Author:  [@Skywatcher](https://boards.straightdope.com/u/Skywatcher)
#### Post date: [July 13, 2004, 1:06pm UTC](https://boards.straightdope.com/t/doper-programmers-little-help-please/254842/6 "2004-07-13T13:06:40Z")
Thanks, I knew I could count on Dopers to fill me in.
---
### Author:  [@Trigonal\_Planar](https://boards.straightdope.com/u/Trigonal_Planar)
#### Post date: [July 13, 2004, 1:50pm UTC](https://boards.straightdope.com/t/doper-programmers-little-help-please/254842/7 "2004-07-13T13:50:04Z")
I did this for an online game I play. There’s an economic portion to it and I wrote a script that would regularly collect the prices, supply and demand on the different items; this way I could plot graphs of said item over time in an effort to gain an edge on the competition. 🙂