# Unidentified non-display characters **URL:** \ **Category:** Factual Questions\ **Created:** [November 9, 2010, 4:24pm UTC](https://boards.straightdope.com/t/unidentified-non-display-characters/560051 "2010-11-09T16:24:08Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Johnny\_L.A](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/johnny_l.a/32/1084_2.png) [@Johnny\_L.A](https://boards.straightdope.com/u/Johnny_L.A)\ **Post date:** [November 9, 2010, 4:24pm UTC](https://boards.straightdope.com/t/unidentified-non-display-characters/560051/1 "2010-11-09T16:24:08Z")
A client has a new ERP system, so their data has changed. In one field there are pairs of non-display characters. One is a CRLF, which is alt+010. These are easily changed to pipes. The other character, which precedes the alt+010 character, is a mystery. I imported the file into Access, and these mystery characters caused a new record each time they were encountered. (i.e., the record was split.) I can import the resulting file into Excel and move things around manually so that I have single records for each account, in which each field has its own column. But that’s a pain. It would be much easier if I could change these mystery characters into pipes, and use the pipes in Text-to-columns to separate the fields. Does anyone have any idea what the alt code for these mystery NDCs might be? Thanks. ---
**Author:** ![Terminus\_Est](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/terminus_est/32/3087_2.png) [@Terminus\_Est](https://boards.straightdope.com/u/Terminus_Est)\ **Post date:** [November 9, 2010, 4:54pm UTC](https://boards.straightdope.com/t/unidentified-non-display-characters/560051/2 "2010-11-09T16:54:47Z")
You say that this mystery character _always_ precedes alt+010? It’s probably a CR. Using your “alt” terminology: CR = alt+013 LF = alt+010 CRLF = alt+013, alt+010 CR and LF are different characters. ---
**Author:** ![Johnny\_L.A](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/johnny_l.a/32/1084_2.png) [@Johnny\_L.A](https://boards.straightdope.com/u/Johnny_L.A)\ **Post date:** [November 9, 2010, 5:54pm UTC](https://boards.straightdope.com/t/unidentified-non-display-characters/560051/3 "2010-11-09T17:54:16Z")
I’m working from home today, so I’m on a Mac. You can’t make alt characters on a Mac, and they’ve cracked down on Internet usage at work so I can’t come to SDMB when I’m in the office. But I asked a coworker to try the alt+013. I assume she did it correctly, as occasionally I’ll ask her to change the alt+010s when I’m offsite. I checked the file, and the little boxes (NDCs) are still there. So they’re apparently not 013. Maybe an EOR/EOL? Yes, the NDC always precedes the 010. ---
**Author:** ![Terminus\_Est](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/terminus_est/32/3087_2.png) [@Terminus\_Est](https://boards.straightdope.com/u/Terminus_Est)\ **Post date:** [November 9, 2010, 6:06pm UTC](https://boards.straightdope.com/t/unidentified-non-display-characters/560051/4 "2010-11-09T18:06:39Z")
If you’re on a Mac with OS X, then you have the full power of the Unix command line. Perform a hexdump on the file and figure it out from there. od -x filename will get you output in hexadecimal. Here’s an ASCII table if you don’t have one handy: [http://www.asciitable.com/](http://www.asciitable.com/) ---
**Author:** ![Johnny\_L.A](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/johnny_l.a/32/1084_2.png) [@Johnny\_L.A](https://boards.straightdope.com/u/Johnny_L.A)\ **Post date:** [November 9, 2010, 6:18pm UTC](https://boards.straightdope.com/t/unidentified-non-display-characters/560051/5 "2010-11-09T18:18:31Z")
I don’t know how to do a hex dump. The file resides on my PC at the office. I connect using RDC. So I don’t actually have the file on my Mac. ---
**Author:** ![Deflagration](https://avatars.discourse-cdn.com/v4/letter/d/3ab097/32.png) [@Deflagration](https://boards.straightdope.com/u/Deflagration)\ **Post date:** [November 9, 2010, 6:37pm UTC](https://boards.straightdope.com/t/unidentified-non-display-characters/560051/6 "2010-11-09T18:37:28Z")
I too will vote for the mystery character being a Carriage Return (CR). If they’re coming in pairs and the second one is a Line Feed (LF), then it’s your most likely culprit. ---
**Author:** ![Arnold\_Winkelried](https://avatars.discourse-cdn.com/v4/letter/a/3d9bf3/32.png) [@Arnold\_Winkelried](https://boards.straightdope.com/u/Arnold_Winkelried)\ **Post date:** [November 9, 2010, 6:52pm UTC](https://boards.straightdope.com/t/unidentified-non-display-characters/560051/7 "2010-11-09T18:52:29Z")
> [@Johnny\_L.A](#): > > I don’t know how to do a hex dump. > > The file resides on my PC at the office. Download and install a freebie text editor that will show you a hex dump of the file. I use [PSPad](http://www.pspad.com) at work. ---
**Author:** ![Noone\_Special](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/noone_special/32/2863_2.png) [@Noone\_Special](https://boards.straightdope.com/u/Noone_Special)\ **Post date:** [November 9, 2010, 6:52pm UTC](https://boards.straightdope.com/t/unidentified-non-display-characters/560051/8 "2010-11-09T18:52:44Z")
Is the client’s new system Windows-based? If so, I’m going to add to the pile-on saying that the first character almost _has_ to be a CR. You probably don’t see it on your Mac, because all Unix systems, as well as Mac-OS systems (after version 9), only use a LF character as a Line Separator. Windows uses CR+LF. ---
**Author:** ![Johnny\_L.A](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/johnny_l.a/32/1084_2.png) [@Johnny\_L.A](https://boards.straightdope.com/u/Johnny_L.A)\ **Post date:** [November 9, 2010, 7:02pm UTC](https://boards.straightdope.com/t/unidentified-non-display-characters/560051/9 "2010-11-09T19:02:49Z")
> [@Arnold\_Winkelried](#): > > Download and install a freebie text editor that will show you a hex dump of the file. I use [PSPad](http://www.pspad.com) at work. They’ve become very strict about downloading. Any downloads must be done by out outside IT guy, and he’s expensive. The person who gives the OK for computer work doesn’t know much about computers or data files, so it would be difficult to get approval. Is there another alt code for CR? alt-013 doesn’t work. If someone wants to take a look at it, I’ve made a sample file that I can send if someone wants to PM me. ---
**Author:** ![Derleth](https://avatars.discourse-cdn.com/v4/letter/d/b9e5f3/32.png) [@Derleth](https://boards.straightdope.com/u/Derleth)\ **Post date:** [November 9, 2010, 7:08pm UTC](https://boards.straightdope.com/t/unidentified-non-display-characters/560051/10 "2010-11-09T19:08:34Z")
[Here’s a Windows command line hexdump program.](http://www.richpasco.org/utilities/hexdump.html) [Here’s an Internet hexdump program.](http://www.fileformat.info/tool/hexdump.htm) If you feel you can upload your file to some website. ---
**Author:** ![Noone\_Special](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/noone_special/32/2863_2.png) [@Noone\_Special](https://boards.straightdope.com/u/Noone_Special)\ **Post date:** [November 9, 2010, 7:08pm UTC](https://boards.straightdope.com/t/unidentified-non-display-characters/560051/11 "2010-11-09T19:08:37Z")
Give. E-mail address should be in my profile. If you can’t see it either PM me or post here. ETA: Just checked, not it isn’t in my profile. PM being composed as you read this. ---
**Author:** ![Johnny\_L.A](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/johnny_l.a/32/1084_2.png) [@Johnny\_L.A](https://boards.straightdope.com/u/Johnny_L.A)\ **Post date:** [November 9, 2010, 7:17pm UTC](https://boards.straightdope.com/t/unidentified-non-display-characters/560051/12 "2010-11-09T19:17:53Z")
Email sent. Thanks! ---
**Author:** ![Johnny\_L.A](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/johnny_l.a/32/1084_2.png) [@Johnny\_L.A](https://boards.straightdope.com/u/Johnny_L.A)\ **Post date:** [November 9, 2010, 8:41pm UTC](https://boards.straightdope.com/t/unidentified-non-display-characters/560051/13 "2010-11-09T20:41:50Z")
Thank you, **Noone Special** for looking at the file. **Noone Special** says the NDC is a CR alt-013, but Find & Replace (in Excel) isn’t finding and replacing. ---
**Author:** ![Noone\_Special](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/noone_special/32/2863_2.png) [@Noone\_Special](https://boards.straightdope.com/u/Noone_Special)\ **Post date:** [November 9, 2010, 8:56pm UTC](https://boards.straightdope.com/t/unidentified-non-display-characters/560051/14 "2010-11-09T20:56:25Z")
To elaborate – AFAICT, it’s the “find” part that’s failing, due to the fact that, at least for me, any attempt to enter Alt-013 in a user interface (e.g., “Find” box in an editor) results in an actual CRLF being entered (i.e., it’s as if I press the Enter key) Anyone know a windows shell equivalent to sed or awk that will allow running the file through a filter that will catch and delete/replace instances of Non-printable characters like this? Because this is basically what he needs (and I’m not good enough at Windows Shell programming / Excel VBA(?) to pull it off!) ---
**Author:** ![Derleth](https://avatars.discourse-cdn.com/v4/letter/d/b9e5f3/32.png) [@Derleth](https://boards.straightdope.com/u/Derleth)\ **Post date:** [November 9, 2010, 9:03pm UTC](https://boards.straightdope.com/t/unidentified-non-display-characters/560051/15 "2010-11-09T21:03:33Z")
[For this kind of work, I’d just install Cygwin on the Windows machine and use the standard \*nix tools.](http://www.cygwin.com/) Cygwin is a port of the \*nix environment (command line and graphical) to Windows, so you get everything needed to do this in a reasonable fashion. ---
**Author:** ![Noone\_Special](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/noone_special/32/2863_2.png) [@Noone\_Special](https://boards.straightdope.com/u/Noone_Special)\ **Post date:** [November 9, 2010, 9:08pm UTC](https://boards.straightdope.com/t/unidentified-non-display-characters/560051/16 "2010-11-09T21:08:10Z")
> [@Derleth](#): > > [For this kind of work, I’d just install Cygwin on the Windows machine and use the standard \*nix tools.](http://www.cygwin.com/) Cygwin is a port of the \*nix environment (command line and graphical) to Windows, so you get everything needed to do this in a reasonable fashion. Agreed (and in fact that’s what I used to determine that it really _was_ a CR) – however Johnny has indicated (see post #9) that he cannot install _any_ new software on the work PC, therefore my shout-out to anyone who may know how to work around this limitation by using Windows Shell built-ins. ---
**Author:** ![Terminus\_Est](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/terminus_est/32/3087_2.png) [@Terminus\_Est](https://boards.straightdope.com/u/Terminus_Est)\ **Post date:** [November 9, 2010, 9:08pm UTC](https://boards.straightdope.com/t/unidentified-non-display-characters/560051/17 "2010-11-09T21:08:18Z")
Stock Windows doesn’t really have much in the way of command-line tools. It may be better to ask why CR/LF is in the original file to begin with. If they’re there to indicate an actual line break, then the final display app should be programmed to behave appropriately. ---
**Author:** ![Noone\_Special](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/noone_special/32/2863_2.png) [@Noone\_Special](https://boards.straightdope.com/u/Noone_Special)\ **Post date:** [November 9, 2010, 9:11pm UTC](https://boards.straightdope.com/t/unidentified-non-display-characters/560051/18 "2010-11-09T21:11:21Z")
> [@Terminus\_Est](#): > > Stock Windows doesn’t really have much in the way of command-line tools. It may be better to ask why CR/LF is in the original file to begin with. If they’re there to indicate an actual line break, then the final display app should be programmed to behave appropriately. I’ve seen Johnny’s sample data. Without going into too much detail, yes it probably was a Line Break originally, but the several “lines” from the original field need to be merged for Data Mining purposes. Agreed that the best place to nuke the Line Break would have been during data acquisition; but that’s probably spilled milk at this point… ---
**Author:** ![Terminus\_Est](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/terminus_est/32/3087_2.png) [@Terminus\_Est](https://boards.straightdope.com/u/Terminus_Est)\ **Post date:** [November 9, 2010, 9:24pm UTC](https://boards.straightdope.com/t/unidentified-non-display-characters/560051/19 "2010-11-09T21:24:24Z")
The data mining app should be able to ignore CR/LF or, better, treat it as whitespace. I’m sure whatever tool is being used to process these records (Oracle?, DB2?, generic SQL?) could also convert it itself. ---
**Author:** ![Johnny\_L.A](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/johnny_l.a/32/1084_2.png) [@Johnny\_L.A](https://boards.straightdope.com/u/Johnny_L.A)\ **Post date:** [November 9, 2010, 10:00pm UTC](https://boards.straightdope.com/t/unidentified-non-display-characters/560051/20 "2010-11-09T22:00:41Z")
I’ve asked the client a couple of times if it’s something they can fix on their end. I’m dealing with the credit manager and not an IT person, and she’s come back to say she can’t fix it. I don’t know what system they’re using. [Next page](https://boards.straightdope.com/t/unidentified-non-display-characters/560051.md?page=2)