# Text strings ON a web page but NOT in its source code. Getting the former **URL:** \ **Category:** Factual Questions\ **Created:** [May 29, 2023, 6:11pm UTC](https://boards.straightdope.com/t/text-strings-on-a-web-page-but-not-in-its-source-code-getting-the-former/984746 "2023-05-29T18:11:08Z")\ **Posts on this page:** 11\ **Page:** 2
**Author:** ![AHunter3](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/ahunter3/32/368_2.png) [@AHunter3](https://boards.straightdope.com/u/AHunter3)\ **Post date:** [May 30, 2023, 1:19am UTC](https://boards.straightdope.com/t/text-strings-on-a-web-page-but-not-in-its-source-code-getting-the-former/984746/21 "2023-05-30T01:19:20Z")
I could run Max’s WindowsOS shell command in a virtual machine running Windows 10 easily enough, although a MacOS command line equivalent would be more convenient for me as a Mac-centric person… still, one cannot have everything! 🙂 ---
**Author:** ![Max\_S](https://avatars.discourse-cdn.com/v4/letter/m/46a35a/32.png) [@Max\_S](https://boards.straightdope.com/u/Max_S)\ **Post date:** [May 30, 2023, 1:32am UTC](https://boards.straightdope.com/t/text-strings-on-a-web-page-but-not-in-its-source-code-getting-the-former/984746/22 "2023-05-30T01:32:42Z")
I don’t have a device to test this on, but you can try this (note the backslashes before the ampersands): ```auto open -a "Google Chrome" --args --enable-logging --headless=new --dump-dom --virtual-time-budget=10000 https://a030-goat.nyc.gov/goat/Function1B?borough=1\&street=W%20180th%20Street\&address=500 > dom.txt ``` ~Max ---
**Author:** ![AHunter3](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/ahunter3/32/368_2.png) [@AHunter3](https://boards.straightdope.com/u/AHunter3)\ **Post date:** [May 30, 2023, 2:12am UTC](https://boards.straightdope.com/t/text-strings-on-a-web-page-but-not-in-its-source-code-getting-the-former/984746/23 "2023-05-30T02:12:43Z")
Thanks **Max!** It runs, it deposits dom.txt at the root of my user folder, but dom.txt is an empty text file. ---
**Author:** ![AHunter3](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/ahunter3/32/368_2.png) [@AHunter3](https://boards.straightdope.com/u/AHunter3)\ **Post date:** [May 30, 2023, 6:09pm UTC](https://boards.straightdope.com/t/text-strings-on-a-web-page-but-not-in-its-source-code-getting-the-former/984746/24 "2023-05-30T18:09:46Z")
Max\_S’s Windows command worked when I tried it on a Windows 10 virtual machine, so if I encounter this kind of situation in the future I could write a FileMaker script that executes an instruction to cmd, calculating the string akin to Max\_S’s example and then imports from dom.txt and parses the result. Still would like a MacOS equivalent… I’ve been reading up on curl and it seems like if I understood the syntax better, it should be able to spit out a text file that contains the text that would be in a web browser’s window for the specified URL. ---
**Author:** ![N9IWP](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/n9iwp/32/3154_2.png) [@N9IWP](https://boards.straightdope.com/u/N9IWP)\ **Post date:** [May 30, 2023, 10:39pm UTC](https://boards.straightdope.com/t/text-strings-on-a-web-page-but-not-in-its-source-code-getting-the-former/984746/25 "2023-05-30T22:39:07Z")
based on > **[How to Use Headless Chrome on Apple OSX](https://medium.com/@anotherplanet/how-to-use-headless-chrome-on-apple-osx-d9f7f3a54983)** > > StepByStep how to get headless Chrome up and running on OSX I’d say `alias chrome="/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome` then `chrome --enable-logging --headless` (rest of line) But that is a guess Brian ---
**Author:** ![Dr.Strangelove](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/dr.strangelove/32/6613_2.png) [@Dr.Strangelove](https://boards.straightdope.com/u/Dr.Strangelove)\ **Post date:** [May 30, 2023, 10:49pm UTC](https://boards.straightdope.com/t/text-strings-on-a-web-page-but-not-in-its-source-code-getting-the-former/984746/26 "2023-05-30T22:49:38Z")
> [@AHunter3](#): > > it should be able to spit out a text file that contains the text that would be in a web browser’s window for the specified URL. Yes, but as others have said, _that is not the information you want_. The information simply isn’t there until the dynamic elements (JavaScript, in particular) have done their thing. **Max\_S** solution is a neat one, as it dumps the document after it has gone through this step. Only a proper browser is capable of this. Curl does not contain a JavaScript engine or any other HTML processing and so is incapable of it. ---
**Author:** ![AHunter3](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/ahunter3/32/368_2.png) [@AHunter3](https://boards.straightdope.com/u/AHunter3)\ **Post date:** [May 31, 2023, 12:52am UTC](https://boards.straightdope.com/t/text-strings-on-a-web-page-but-not-in-its-source-code-getting-the-former/984746/27 "2023-05-31T00:52:50Z")
Aah, thanks for the clarification, Dr.Strangelove! I will continue to play with headless-chrome command-line variants. I’ve got the Windows version working. > [@](#): > > I’d say > `alias chrome="/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome` > then > `chrome --enable-logging --headless` (rest of line) So… `alias chrome="/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome` then ```auto chrome --enable-logging --headless=new --dump-dom --virtual-time-budget=10000 https://a030-goat.nyc.gov/goat/Function1B?borough=1\&street=W%20180th%20Street\&address=500 > dom.txt ``` ?? or then ```auto chrome --enable-logging --headless == new --dump-dom --virtual-time-budget=10000 https://a030-goat.nyc.gov/goat/Function1B?borough=1\&street=W%20180th%20Street\&address=500 > dom.txt ``` ?? ---
**Author:** ![Sam\_Stone](https://avatars.discourse-cdn.com/v4/letter/s/ecccb3/32.png) [@Sam\_Stone](https://boards.straightdope.com/u/Sam_Stone)\ **Post date:** [May 31, 2023, 1:14am UTC](https://boards.straightdope.com/t/text-strings-on-a-web-page-but-not-in-its-source-code-getting-the-former/984746/28 "2023-05-31T01:14:06Z")
Have you considered asking ChatGPT for help? It will even write code for you, but it’s really good at explaining how to do coding tasks, or looking over your code and telling you what’s wrong with it. It will definitely write web page scrapers for you if you can provide enough detail. My son had ChatGPT build up a REACT web site today without having to do any coding. ---
**Author:** ![Dr.Strangelove](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/dr.strangelove/32/6613_2.png) [@Dr.Strangelove](https://boards.straightdope.com/u/Dr.Strangelove)\ **Post date:** [May 31, 2023, 1:26am UTC](https://boards.straightdope.com/t/text-strings-on-a-web-page-but-not-in-its-source-code-getting-the-former/984746/29 "2023-05-31T01:26:04Z")
> [@AHunter3](#): > > Aah, thanks for the clarification, Dr.Strangelove! Sure thing. At a _very_ high level, the steps the browser performs are: 1. Fetches the text representation of a page from the server. 2. Converts the text representation to the “DOM”, or document object model, which allows more convenient internal processing. 3. Runs JavaScript elements, which may update the DOM, and including information pulled from additional fetches from the server. 4. Renders the DOM into an image for display Curl _only_ performs step 1, and so is useless here. Chrome and other browsers perform all steps, but generally just produce an image. **Max\_S** ’s solution replaces step 4 with: 4b) Convert the DOM back to a text representation and save to disk That text representation usually does not even exist, since there is no need for the browser to have it. But it is useful for debugging and so Chrome supports conversion. ---
**Author:** ![BigT](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/bigt/32/12044_2.png) [@BigT](https://boards.straightdope.com/u/BigT)\ **Post date:** [May 31, 2023, 1:37am UTC](https://boards.straightdope.com/t/text-strings-on-a-web-page-but-not-in-its-source-code-getting-the-former/984746/30 "2023-05-31T01:37:54Z")
If what Max says doesn’t work, I’d try it without the backslashes for the alias. I am unsure if both quotes **and** backslashes are necessary. ---
**Author:** ![echoreply](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/echoreply/32/3641_2.png) [@echoreply](https://boards.straightdope.com/u/echoreply)\ **Post date:** [May 31, 2023, 2:30am UTC](https://boards.straightdope.com/t/text-strings-on-a-web-page-but-not-in-its-source-code-getting-the-former/984746/31 "2023-05-31T02:30:02Z")
> [@AHunter3](#): > > alias chrome="/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome > [@BigT](#): > > try it without the backslashes for the alias. I am unsure if both quotes **and** backslashes are necessary Quotes or backslashes, but not both (unless you want the actual character `\`, but you don’t). All `alias` is doing is telling the shell (probably `zsh` on a Mac) that when it sees the command `chrome` it should really run the program at ```auto /Applications/Google Chrome.app/Contents/MacOS/Google Chrome ``` The problem is that the shell sees a space as a separator, so you need either quotes so it sees the entire string as a single unit, or the backslash to escape the space, so it is not seen as a separator. So `alias` is just a timesaver; you can always type the entire `/Applications/...` path to the command to run it. (Caveat on all of this: I have far more experience with `bash` than `zsh`, so if `zsh` is so messed up as to require quotes and an escape to handle a space, then give up and switch shells.) If I were doing this, I would probably have a shell script something like ```bash !#/bin/bash CHROMEEXEC="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" CHROMEARGS="--args --enable-logging --headless=new --dump-dom --virtual-time-budget=10000" URL="https://a030-goat.nyc.gov/goat/Function1B?" ADDRESS="borough=1&street=W%20180th%20Street&address=500" $CHROMEEXEC $CHROMEARGS ${URL}${ADDRESS} ``` Then once it was working for manually entered addresses in the URL and ADDRESS lines, make the script a bit more complicated to construct formatted addresses into the URL. Of course, switch to whatever other URL and appropriate input munging for the next problem. (I couldn’t find my code in a brief search, but last time I needed to convert addresses to a lat/long I used Google’s map API from R, and I think it was trivial, because there were adequate coding examples online. I think it took longer to get signed up for Google map API access than it did to actually write and run the code. The first few thousand lookups are free, but there is a cost after that, so fine for a limited scope jobs.) [Previous page](https://boards.straightdope.com/t/text-strings-on-a-web-page-but-not-in-its-source-code-getting-the-former/984746.md?page=1)