# In \*general\*, how do encryption algorithms prevent 0's mid-string? **URL:** \ **Category:** Factual Questions\ **Created:** [February 26, 2015, 5:48pm UTC](https://boards.straightdope.com/t/in-general-how-do-encryption-algorithms-prevent-0s-mid-string/713683 "2015-02-26T17:48:04Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![Saint\_Cad](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/saint_cad/32/18907_2.png) [@Saint\_Cad](https://boards.straightdope.com/u/Saint_Cad)\ **Post date:** [February 26, 2015, 7:51pm UTC](https://boards.straightdope.com/t/in-general-how-do-encryption-algorithms-prevent-0s-mid-string/713683/21 "2015-02-26T19:51:54Z")
Here’s an idea, have the acceptable range of the encryption be 0 to 254 then add 1 to each encrypted byte. ---
**Author:** ![Punoqllads](https://avatars.discourse-cdn.com/v4/letter/p/d2c977/32.png) [@Punoqllads](https://boards.straightdope.com/u/Punoqllads)\ **Post date:** [February 26, 2015, 7:54pm UTC](https://boards.straightdope.com/t/in-general-how-do-encryption-algorithms-prevent-0s-mid-string/713683/22 "2015-02-26T19:54:58Z")
> [@bup](#): > > How does that ensure the encrypted message has no zeros mid-array? Because long sequences of 0s can be by definition compressed. ---
**Author:** ![leahcim](https://avatars.discourse-cdn.com/v4/letter/l/b4bc9f/32.png) [@leahcim](https://boards.straightdope.com/u/leahcim)\ **Post date:** [February 26, 2015, 8:08pm UTC](https://boards.straightdope.com/t/in-general-how-do-encryption-algorithms-prevent-0s-mid-string/713683/23 "2015-02-26T20:08:44Z")
> [@Punoqllads](#): > > Because long sequences of 0s can be by definition compressed. That doesn’t mean the compressed result will have no zeros in it. ---
**Author:** ![septimus](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/septimus/32/410_2.png) [@septimus](https://boards.straightdope.com/u/septimus)\ **Post date:** [February 26, 2015, 8:12pm UTC](https://boards.straightdope.com/t/in-general-how-do-encryption-algorithms-prevent-0s-mid-string/713683/24 "2015-02-26T20:12:04Z")
> [@bup](#): > > But, I am interested in the general solution. String to string algorithms _exist._ How do they do it? It is a common need, e.g. in file-long encodings like Jpeg. Those boasting of systems which avoid any problem may not have worked with many real-world high-performance encodings. A simple solution – and IIRC something along these lines is used for Jpeg, though it’s solving a slightly different problem :– Replace FF or 00 with FF FF or FF 01 respectively. (Codes FF 02 through FF FE are freed up for use as markers.) 1/128 of the time, 16 bits instead of 8 bits are used to represent a byte, for an average loss of 1/16 bit per byte. With this approach, your file will grow by 0.78% on average (if 00 and FF each occur with 1/256 probability). If this is too much waste, other, more complicated, approaches are possible. (Note that the “waste” _does_ build in a crisp possibly useful way to insert markers.) ---
**Author:** ![Learjeff](https://avatars.discourse-cdn.com/v4/letter/l/94ad74/32.png) [@Learjeff](https://boards.straightdope.com/u/Learjeff)\ **Post date:** [February 26, 2015, 8:22pm UTC](https://boards.straightdope.com/t/in-general-how-do-encryption-algorithms-prevent-0s-mid-string/713683/25 "2015-02-26T20:22:44Z")
> [@ftg](#): > > The best thing to do of course is _not call it a string_! It’s an array of unsigned bytes with a length count. End of hassle. Exactly. It’s no longer a string, it’s an octet sequence, with a specific length. ---
**Author:** ![Mangetout](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/mangetout/32/19_2.png) [@Mangetout](https://boards.straightdope.com/u/Mangetout)\ **Post date:** [February 26, 2015, 8:24pm UTC](https://boards.straightdope.com/t/in-general-how-do-encryption-algorithms-prevent-0s-mid-string/713683/26 "2015-02-26T20:24:31Z")
> [@Saint\_Cad](#): > > Here’s an idea, have the acceptable range of the encryption be 0 to 254 then add 1 to each encrypted byte. Agreed. The algorithm can be ‘unbiased’ within any range of possible values In the same sort of way that a six sided die is no more or less ‘random’ than a 12 sided die. ---
**Author:** ![Learjeff](https://avatars.discourse-cdn.com/v4/letter/l/94ad74/32.png) [@Learjeff](https://boards.straightdope.com/u/Learjeff)\ **Post date:** [February 26, 2015, 8:24pm UTC](https://boards.straightdope.com/t/in-general-how-do-encryption-algorithms-prevent-0s-mid-string/713683/27 "2015-02-26T20:24:57Z")
> [@bup](#): > > Well, the mechanism I have for sending the message to the other party is a string. I can’t get around it. That’s exactly what base64 is for. Unfortunately it takes more space, but it’s pretty trivial and fast to convert. ---
**Author:** ![septimus](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/septimus/32/410_2.png) [@septimus](https://boards.straightdope.com/u/septimus)\ **Post date:** [February 26, 2015, 8:30pm UTC](https://boards.straightdope.com/t/in-general-how-do-encryption-algorithms-prevent-0s-mid-string/713683/28 "2015-02-26T20:30:05Z")
> [@ftg](#): > > Of course you can have a zero (null byte) in a string. No one is forced to write code that assumes that a null byte terminates a string. Well, nobody’s “forced” into a lot of things. It becomes a matter of relative convenience. For some specific problem environments, something like OP’s proposal may often be best. I’ve done something like this at least a dozen times (though I think I’ve also written more code than most Dopers). ---
**Author:** ![septimus](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/septimus/32/410_2.png) [@septimus](https://boards.straightdope.com/u/septimus)\ **Post date:** [February 26, 2015, 8:35pm UTC](https://boards.straightdope.com/t/in-general-how-do-encryption-algorithms-prevent-0s-mid-string/713683/29 "2015-02-26T20:35:34Z")
> [@Saint\_Cad](#): > > Here’s an idea, have the acceptable range of the encryption be 0 to 254 then add 1 to each encrypted byte. Yes, but the simplicity of modular-256 arithmetic over mod-255 will often be too good to pass up. ---
**Author:** ![Learjeff](https://avatars.discourse-cdn.com/v4/letter/l/94ad74/32.png) [@Learjeff](https://boards.straightdope.com/u/Learjeff)\ **Post date:** [February 26, 2015, 8:35pm UTC](https://boards.straightdope.com/t/in-general-how-do-encryption-algorithms-prevent-0s-mid-string/713683/30 "2015-02-26T20:35:47Z")
> [@bup](#): > > Am I missing something? No, AdamF is. Another advantage of base64 is that, since data is passed as strings, you may not know what kind of processing might be applied to “text strings”. If you stick to base64, every byte will contain a well-defined printable ASCII character, with the least likelihood of getting mangled. With base64, your text gets 33% bigger (it takes 4 bytes to carry 3 bytes of data.) That’s the biggest disadvantage. There’s no way to predict how many zeros you’ll get, so with a “zero-replacement” protocol, you have no idea how long your messages might be. If your protocol is efficient enough that the max buffer size can still handle your max cyphertext plus your zero-hiding trick, then sure, go for that if you prefer. But base64 will be recognized and understood by others: less to explain. ---
**Author:** ![gazpacho](https://avatars.discourse-cdn.com/v4/letter/g/6f9a4e/32.png) [@gazpacho](https://boards.straightdope.com/u/gazpacho)\ **Post date:** [February 26, 2015, 8:45pm UTC](https://boards.straightdope.com/t/in-general-how-do-encryption-algorithms-prevent-0s-mid-string/713683/31 "2015-02-26T20:45:22Z")
> [@Learjeff](#): > > That’s exactly what base64 is for. Unfortunately it takes more space, but it’s pretty trivial and fast to convert. PGP uses this or something like this for encrypting emails to ensure that the encrypted email is nice ASCII text with nothing weird in that that would get corrupted by some program on the way. ---
**Author:** ![davidm](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/davidm/32/225_2.png) [@davidm](https://boards.straightdope.com/u/davidm)\ **Post date:** [February 26, 2015, 9:01pm UTC](https://boards.straightdope.com/t/in-general-how-do-encryption-algorithms-prevent-0s-mid-string/713683/32 "2015-02-26T21:01:34Z")
What’s the objection to just using Base64? Is there some need to minimize the length or something? ---
**Author:** ![septimus](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/septimus/32/410_2.png) [@septimus](https://boards.straightdope.com/u/septimus)\ **Post date:** [February 26, 2015, 9:07pm UTC](https://boards.straightdope.com/t/in-general-how-do-encryption-algorithms-prevent-0s-mid-string/713683/33 "2015-02-26T21:07:37Z")
> [@davidm](#): > > What’s the objection to just using Base64? Is there some need to minimize the length or something? Base64 is a fine solution. But sometimes reducing the average length will have merit, e.g. compression schemes, or large tables. ---
**Author:** ![ftg](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/ftg/32/2801_2.png) [@ftg](https://boards.straightdope.com/u/ftg)\ **Post date:** [February 26, 2015, 9:20pm UTC](https://boards.straightdope.com/t/in-general-how-do-encryption-algorithms-prevent-0s-mid-string/713683/34 "2015-02-26T21:20:28Z")
Pffft, these kids today with their fancy base64. In my day we used UUEncode and were glad to have it. Reminder: the issue isn’t a string of 0s, it’s having _a_ zero in a string. I also don’t see why the length variable should be constrained to a byte. Note: the OP’s question really has nothing to do with encryption. It’s just about dealing with nulls in strings. How they got there doesn’t really matter. You can get really creative if you want to. Suppose the string has 53 null bytes. First say: “I’m sending you 54 strings.” Then send the string. At the other end, keep reading until you’ve got 54 strings, concatenating them together _including_ the null byte. (And remembering that 2+ nulls in a row results in empty strings.) And if you are using something like “getstring” on the other end, my God have mercy on your code. ---
**Author:** ![bup](https://avatars.discourse-cdn.com/v4/letter/b/6bbea6/32.png) [@bup](https://boards.straightdope.com/u/bup)\ **Post date:** [February 26, 2015, 9:38pm UTC](https://boards.straightdope.com/t/in-general-how-do-encryption-algorithms-prevent-0s-mid-string/713683/35 "2015-02-26T21:38:40Z")
> [@Learjeff](#): > > That’s exactly what base64 is for. Unfortunately it takes more space, but it’s pretty trivial and fast to convert. Ah - I get it! (I think - at least, I can now do what I want, and understand how string to string works). By taking the 4 byte integers that XTEA gives me, and dividing them out into _5_ bytes (each one holding a number between 0 and 64), I now have a couple of extra bits. I can map the values 0-64 to any characters I want, thus I can make it a string with no zeroes. And I can make it encoding-independent by doing it with math, instead of depending on what bits are where. Take the mod 64, then divide by 64, then take the mode 64 again… I get a 5 digit base-64 number. Thanks for all the input, everyone! Anyone I didn’t understand - I’m sorry. I think I became a little bit better of a code jockey today! ---
**Author:** ![Chronos](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/chronos/32/134_2.png) [@Chronos](https://boards.straightdope.com/u/Chronos)\ **Post date:** [February 27, 2015, 1:16am UTC](https://boards.straightdope.com/t/in-general-how-do-encryption-algorithms-prevent-0s-mid-string/713683/36 "2015-02-27T01:16:43Z")
The other big advantage to using base64 is that it’s a standard technique, so there are standard tools for it. You probably could write your own implementation of it… but you don’t need to, because you can probably find a library for your language of choice that’s already implemented it. ---
**Author:** ![Senegoid](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/senegoid/32/6606_2.png) [@Senegoid](https://boards.straightdope.com/u/Senegoid)\ **Post date:** [February 27, 2015, 2:01am UTC](https://boards.straightdope.com/t/in-general-how-do-encryption-algorithms-prevent-0s-mid-string/713683/37 "2015-02-27T02:01:11Z")
Olde-tyme C programmers were routinely derided for their reluctance to use C Standard Library functions, preferring to re-invent all their own wheels every time. This discussion clarifies exactly why. How do C programmers hunt elephants? > [@](#): > > **C** programmers refuse to buy rifles off the shelf, preferring to take steel rods and a mobile machine shop to Africa intending to build the perfect rifle for the job from scratch while everyone else bags the elephants. [How Programmers Hunt Elephants](http://baetzler.de/humor/programmers_hunt_elephants.html) ---
**Author:** ![AdamF](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/adamf/32/6246_2.png) [@AdamF](https://boards.straightdope.com/u/AdamF)\ **Post date:** [February 27, 2015, 2:44am UTC](https://boards.straightdope.com/t/in-general-how-do-encryption-algorithms-prevent-0s-mid-string/713683/38 "2015-02-27T02:44:43Z")
> [@bup](#): > > Am I missing something? You are correct, my mistake. ---
**Author:** ![septimus](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/septimus/32/410_2.png) [@septimus](https://boards.straightdope.com/u/septimus)\ **Post date:** [February 27, 2015, 8:46am UTC](https://boards.straightdope.com/t/in-general-how-do-encryption-algorithms-prevent-0s-mid-string/713683/39 "2015-02-27T08:46:05Z")
> [@Senegoid](#): > > Olde-tyme C programmers were routinely derided for their reluctance to use C Standard Library functions, preferring to re-invent all their own wheels every time. > > This discussion clarifies exactly why. Cpomments like this confuse me. How many programmer man-minutes do you think are needed to build base64 routines from scratch? Or the simple escape mechanism I described? ---
**Author:** ![si\_blakely](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@si\_blakely](https://boards.straightdope.com/u/si_blakely)\ **Post date:** [February 27, 2015, 10:32am UTC](https://boards.straightdope.com/t/in-general-how-do-encryption-algorithms-prevent-0s-mid-string/713683/40 "2015-02-27T10:32:54Z")
> [@septimus](#): > > Cpomments like this confuse me. How many programmer man-minutes do you think are needed to build base64 routines from scratch? Oh, it might only take a few minutes to implement a straightforward base64 routine that works with some nice clean input. Dealing with the edge cases, error conditions, input that does not conform to expected parameters - that takes the time. And if you don’t take the time when you write it originally, you will take huge amounts of time trying to figure out the unexpected behaviour of your wider system. This is why using a standard library routine that has all the edge cases/error conditions/every possible input covered does actually save time and make the code more reliable. And, as always, I quote one of my Computing lecturers on implementation estimation - “take your original guess, double it, and shift to the next order of magnitude” Experience has shown me that he was pretty accurate. [Previous page](https://boards.straightdope.com/t/in-general-how-do-encryption-algorithms-prevent-0s-mid-string/713683.md?page=1) [Next page](https://boards.straightdope.com/t/in-general-how-do-encryption-algorithms-prevent-0s-mid-string/713683.md?page=3)