# Is anyone clever enough to reinvent Ben's Method for improving disk performance? **URL:** \ **Category:** Miscellaneous and Personal Stuff I Must Share\ **Created:** [August 8, 2018, 11:25am UTC](https://boards.straightdope.com/t/is-anyone-clever-enough-to-reinvent-bens-method-for-improving-disk-performance/819265 "2018-08-08T11:25:48Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![BeepKillBeep](https://avatars.discourse-cdn.com/v4/letter/b/9e8a1a/32.png) [@BeepKillBeep](https://boards.straightdope.com/u/BeepKillBeep)\ **Post date:** [August 10, 2018, 1:31pm UTC](https://boards.straightdope.com/t/is-anyone-clever-enough-to-reinvent-bens-method-for-improving-disk-performance/819265/21 "2018-08-10T13:31:53Z")
> [@septimus](#): > > Ben’s method was proposed for use with a Unix operating system. Some versions of the Unix file system issue their own anticipatory reads of sequential files, unbeknownst to and independent of, any “smarts” the disk controller may be attempting. Anticipatory reads at the disk controller level may supplement OR replace those kernel “pre-fetches.” Ben’s method is implemented at the disk controller level and may interact (sometimes unfavorably?) with kernel-initiated anticipatory reads. > > “Quantum”? The applications issue read requests and then sleep until fulfillment. (Writes can be asynchronous or delayed, but not reads.) I have no idea about this thread because it isn’t something I am familiar with but why are you responding to yourself? ---
**Author:** ![RadioWave](https://avatars.discourse-cdn.com/v4/letter/r/91b2a8/32.png) [@RadioWave](https://boards.straightdope.com/u/RadioWave)\ **Post date:** [August 10, 2018, 2:53pm UTC](https://boards.straightdope.com/t/is-anyone-clever-enough-to-reinvent-bens-method-for-improving-disk-performance/819265/22 "2018-08-10T14:53:25Z")
> [@septimus](#): > > AFAIK Ben’s Method has never been implemented nor published. Comments in the thread only confirm this. Most of us who do this for a living do so under pretty strict NDAs (Non Disclosure Agreements) so we can’t really talk about what we have or haven’t implemented. Secondly, if it hasn’t been published, how did you hear about it? Are you “Ben”? > [@septimus](#): > > Anticipatory reads at the disk controller level may supplement OR replace those kernel “pre-fetches.” I should think any disk controller which replaces I/O requests from the host would be short lived. Again, the problem is that there are so many ways to optimize for particular workloads that it’s difficult to imagine which one you’re thinking of. I would almost guarantee that at this point in time, whatever it is, someone has thought of it and discarded it because it only works for certain patterns. Some general ideas: Does it just start reading from wherever the head lands on the track and then fills in the front of the buffer when it gets there? (very common) Does it place consecutive blocks on different tracks so that the head moves towards the middle of the platter (thinking it would reduce the average next seek time) Are the tracks themselves “curved” so that the head moves back towards the center of the disk? Are you going to eventually tell us what Ben’s method is? The last thread kind of came off as “I know this really genius thing but I’m not going to tell you what it is.” It would be frustrating if there’s no reveal at the end. ---
**Author:** ![Quartz](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/quartz/32/267_2.png) [@Quartz](https://boards.straightdope.com/u/Quartz)\ **Post date:** [August 10, 2018, 3:31pm UTC](https://boards.straightdope.com/t/is-anyone-clever-enough-to-reinvent-bens-method-for-improving-disk-performance/819265/23 "2018-08-10T15:31:04Z")
> [@septimus](#): > > AFAIK Ben’s Method has never been implemented nor published. Comments in the thread only confirm this. I expect that when you make the reveal, many of us will go, “Oh, that old thing?” ---
**Author:** ![RadioWave](https://avatars.discourse-cdn.com/v4/letter/r/91b2a8/32.png) [@RadioWave](https://boards.straightdope.com/u/RadioWave)\ **Post date:** [August 10, 2018, 4:43pm UTC](https://boards.straightdope.com/t/is-anyone-clever-enough-to-reinvent-bens-method-for-improving-disk-performance/819265/24 "2018-08-10T16:43:54Z")
> [@Quartz](#): > > I expect that when you make the reveal, many of us will go, “Oh, that old thing?” I’m sure that’s what you’d like us all to believe but I’m not falling for that old trick. I’m going to implement Ben’s method immediately before you can, dominate the world disk market, make millions of dollars and retire somewhere in the Turks and Caicos islands. He did say it wasn’t patented, didn’t he? ---
**Author:** ![Pantastic](https://avatars.discourse-cdn.com/v4/letter/p/3bc359/32.png) [@Pantastic](https://boards.straightdope.com/u/Pantastic)\ **Post date:** [August 10, 2018, 4:43pm UTC](https://boards.straightdope.com/t/is-anyone-clever-enough-to-reinvent-bens-method-for-improving-disk-performance/819265/25 "2018-08-10T16:43:54Z")
> [@BeepKillBeep](#): > > I have no idea about this thread because it isn’t something I am familiar with but why are you responding to yourself? It makes about as much sense as giving credit for a ‘method’ to a person who never published or implemented it. Realistically, if it’s useful someone has probably already tried it, if it’s not useful then what’s the point? No, I don’t think that everything has already been invented, but I do think that the set of ‘disk optimization methods that might be guessed and discussed on non-technical message boards’ has been pretty thoroughly plundered already. ---
**Author:** ![septimus](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/septimus/32/410_2.png) [@septimus](https://boards.straightdope.com/u/septimus)\ **Post date:** [August 10, 2018, 4:44pm UTC](https://boards.straightdope.com/t/is-anyone-clever-enough-to-reinvent-bens-method-for-improving-disk-performance/819265/26 "2018-08-10T16:44:01Z")
> [@RadioWave](#): > > I should think any disk controller which replaces I/O requests from the host would be short lived. So, instead of attempting to understanding what I wrote, you assume I’m an idiot. Got it. Was it too far-fetched to imagine there might be some explicit or implicit cooperation between disk controller designed and kernel programmer? The blocks(\*) of a sequential file are typically laid out so that they pass under the read head in the order 1, 2, 3, 4, 5, 6, 7, 8, 9, … An alternative, used when inter-sector gaps are too brief for necessary processing is something like 1, 11, 2, 12, 3, 13, 4, 14, 5, 15, 6, … Ben’s method uses an ordering different from either of these. (\* - Irrelevant for our purpose here is whether a “block” is a sector or some other-sized chunk.) ---
**Author:** ![RadioWave](https://avatars.discourse-cdn.com/v4/letter/r/91b2a8/32.png) [@RadioWave](https://boards.straightdope.com/u/RadioWave)\ **Post date:** [August 10, 2018, 4:57pm UTC](https://boards.straightdope.com/t/is-anyone-clever-enough-to-reinvent-bens-method-for-improving-disk-performance/819265/27 "2018-08-10T16:57:56Z")
> [@septimus](#): > > So, instead of attempting to understanding what I wrote, you assume I’m an idiot. Got it. > Was it too far-fetched to imagine there might be some explicit or implicit cooperation between disk controller designed and kernel programmer? No need to take offense, I was simply replying to what you said. It’s been years since I’ve worked at the low level (SCSI CDB, ATA, etc) but I don’t remember any hints that an operation could be replaced by something else. I suppose in a captive market like IBM SLEDs or something they could have done something like that. ---
**Author:** ![ftg](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/ftg/32/2801_2.png) [@ftg](https://boards.straightdope.com/u/ftg)\ **Post date:** [August 10, 2018, 6:09pm UTC](https://boards.straightdope.com/t/is-anyone-clever-enough-to-reinvent-bens-method-for-improving-disk-performance/819265/28 "2018-08-10T18:09:59Z")
First of all, the OP’s “method” is so vaguely worded it cannot really be compared to any actual method in use. Second of all, prefetching from a disk has been well known since there were disks. So way before “the late1980s”. Lots of ways of doing it including in software and disk firmware. It is not remotely novel. It’s like claiming to have invented the idea of putting tasks to be done on a queue. Only less specific. (Although I came across a patent for that given in the late 90s.) ---
**Author:** ![Quartz](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/quartz/32/267_2.png) [@Quartz](https://boards.straightdope.com/u/Quartz)\ **Post date:** [August 10, 2018, 7:04pm UTC](https://boards.straightdope.com/t/is-anyone-clever-enough-to-reinvent-bens-method-for-improving-disk-performance/819265/29 "2018-08-10T19:04:37Z")
> [@septimus](#): > > Ben’s method uses an ordering different from either of these. So it’s just another method of organising interleaving. Big whoop. We looked at that in our first or second year at university. IIRC - it’s been 30 years - we looked at various intervals, varying sector counts (inner tracks having fewer sectors), and much, much more. ---
**Author:** ![iamthewalrus\_3](https://avatars.discourse-cdn.com/v4/letter/i/258eb7/32.png) [@iamthewalrus\_3](https://boards.straightdope.com/u/iamthewalrus_3)\ **Post date:** [August 10, 2018, 8:58pm UTC](https://boards.straightdope.com/t/is-anyone-clever-enough-to-reinvent-bens-method-for-improving-disk-performance/819265/30 "2018-08-10T20:58:05Z")
My guess is that there’s something clever here that may or may not really be novel to people in the field (I am not in the field, so it’s probably going to seem clever to me), but that the problem is far too underspecified for people to guess at what you’re getting at, **septimus**. ---
**Author:** ![beowulff](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/beowulff/32/542_2.png) [@beowulff](https://boards.straightdope.com/u/beowulff)\ **Post date:** [August 10, 2018, 9:11pm UTC](https://boards.straightdope.com/t/is-anyone-clever-enough-to-reinvent-bens-method-for-improving-disk-performance/819265/31 "2018-08-10T21:11:16Z")
Are the blocks interleaved in some fashion that optimizes cache filling and emptying? For example, if the cache is 64K, and the blocks are 8K, and it takes (say) 4 blocks worth of time to empty the cache, then arrange them like: 1,2,3,4,5,6,7,8,13,14,15,16,17,18,19,20. ---
**Author:** ![septimus](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/septimus/32/410_2.png) [@septimus](https://boards.straightdope.com/u/septimus)\ **Post date:** [August 10, 2018, 10:36pm UTC](https://boards.straightdope.com/t/is-anyone-clever-enough-to-reinvent-bens-method-for-improving-disk-performance/819265/32 "2018-08-10T22:36:00Z")
It’s exciting to hear there are other disk controller experts here. But I’m sorry none of you re-invented Ben’s Method with a minimum of hints. My #26 was so explicit: > [@septimus](#): > > The blocks(\*) of a sequential file are typically laid out so that they pass under the read head in the order > 1, 2, 3, 4, 5, 6, 7, 8, 9, … > > Ben’s method uses an ordering different from [this]. … that a short string of integers will now constitute “solution”. The “brass ring” is off the table by now, but still — five brownie points to whoever figures it out first! Replies might have been less condescending if my credentials were on display. To detail the variety of work I’ve done with disk systems might seem an exaggeration, and disks were seldom my main focus. I wrote a large immodest paragraph that would impress anyone who believed it, but erased it — we don’t need a “pissing contest.” I did however already mention one resume bullet that is easy to prove: > [@septimus](#): > > To lay some context, I was myself an inventor, with dozens of U.S. patents (not even counting continuations-in-part). Though none of my patents were in the fields of disk storage or file system design, I had considerable experience therewith. Among the wide variety of tasks I did as a computer system consultant was reviewing others’ inventions. I think I had a fairly good eye for novelty and “niftiness.” Whatever the limitations of Ben’s Method, it was novel and nifty. (Niftier than the majority of my own 30+ patents.) As for non-obviousness … well, despite all the prattle here, nobody’s come close to writing down the single sentence, or even simple list of integers, that would describe Ben’s Method. > [@RadioWave](#): > > It would be frustrating if there’s no reveal at the end. I won’t leave it unsolved for more than another day or two, but want to give people a final chance for the pleasure of their own ‘Aha!’ I’ll PM the solution to anyone unable to wait for two days and _ **promising not to spoil it for others** _. Please PM back to me with feedback showing how the method was already well-known. :rolleyes: > [@BeepKillBeep](#): > > I have no idea about this thread because it isn’t something I am familiar with but why are you responding to yourself? Uh … the 5-minute Edit window had expired? Yours is the most peculiar post in the thread so far … and there have been some peculiar posts! 🙂 > [@beowulff](#): > > Are the blocks interleaved in some fashion that optimizes cache filling and emptying? > For example, if the cache is 64K, and the blocks are 8K, and it takes (say) 4 blocks worth of time to empty the cache, then arrange them like: 1,2,3,4,5,6,7,8,13,14,15,16,17,18,19,20. At last! An actual “contest” submission! And an idea with possible merit. But No, that isn’t Ben’s Method. However, as you have done here, Ben’s Method can _also_ be presented with a simple ordered list of integers. ---
**Author:** ![Quartz](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/quartz/32/267_2.png) [@Quartz](https://boards.straightdope.com/u/Quartz)\ **Post date:** [August 10, 2018, 10:43pm UTC](https://boards.straightdope.com/t/is-anyone-clever-enough-to-reinvent-bens-method-for-improving-disk-performance/819265/33 "2018-08-10T22:43:11Z")
That’s one basis of clustering. Another being a minimum addressability issue. For instance, you might have an interleave of 1+2+3+4, 9+10+11+12, 5+6+7+8, and 13+14+16+16. An advantage of clustering is that it can be logical, not physical. The drive controller can map the logical clusters to physical sectors in what the manufacturer thinks is the optimum manner. It also allows the drive to invisibly swap out bad sectors. I think this came to the consumer market with IDE drives, but it would not surprise me if there were MFM and RLL precursors. ---
**Author:** ![BeepKillBeep](https://avatars.discourse-cdn.com/v4/letter/b/9e8a1a/32.png) [@BeepKillBeep](https://boards.straightdope.com/u/BeepKillBeep)\ **Post date:** [August 11, 2018, 1:01pm UTC](https://boards.straightdope.com/t/is-anyone-clever-enough-to-reinvent-bens-method-for-improving-disk-performance/819265/34 "2018-08-11T13:01:45Z")
> [@septimus](#): > > Uh … the 5-minute Edit window had expired? Yours is the most peculiar post in the thread so far … and there have been some peculiar posts! 🙂 It sure didn’t come across that way. It really came across as you replying to yourself. I mean go have a look at it and tell me that doesn’t look like you’re replying to yourself. It is no big deal, I thought it was kind of funny. I’m glad I won the most peculiar post award though. Does it come with a t-shirt? ---
**Author:** ![pulykamell](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/pulykamell/32/3166_2.png) [@pulykamell](https://boards.straightdope.com/u/pulykamell)\ **Post date:** [August 11, 2018, 1:31pm UTC](https://boards.straightdope.com/t/is-anyone-clever-enough-to-reinvent-bens-method-for-improving-disk-performance/819265/35 "2018-08-11T13:31:39Z")
> [@BeepKillBeep](#): > > It sure didn’t come across that way. It really came across as you replying to yourself. I mean go have a look at it and tell me that doesn’t look like you’re replying to yourself. It is no big deal, I thought it was kind of funny. I’m glad I won the most peculiar post award though. Does it come with a t-shirt? FWIW, you’re not the only one that was confused, especially the part of the post where he does quote and reply to himself and questions the use of the word “quantum.” I couldn’t figure out what was going on. ---
**Author:** ![septimus](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/septimus/32/410_2.png) [@septimus](https://boards.straightdope.com/u/septimus)\ **Post date:** [August 11, 2018, 2:27pm UTC](https://boards.straightdope.com/t/is-anyone-clever-enough-to-reinvent-bens-method-for-improving-disk-performance/819265/36 "2018-08-11T14:27:24Z")
I am confused by the confusion of y’all! 🙂 > [@](#): > > **clar·i·fi·ca·tion** > ˌklerəfəˈkāSH(ə)n > > noun > > the action of making a statement or situation less confused and more comprehensible. > > Example: “please advise us if you require further clarification” **septimus** used the word “quantum” in an unfamiliar way that might confuse. Someone ( **septimus**!) noticed the potential confusion and provided a **clar·i·fi·ca·tion**. Does this … uh … _clarify_? 😛 Or just confuse further? :eek: ---
**Author:** ![pulykamell](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/pulykamell/32/3166_2.png) [@pulykamell](https://boards.straightdope.com/u/pulykamell)\ **Post date:** [August 11, 2018, 2:44pm UTC](https://boards.straightdope.com/t/is-anyone-clever-enough-to-reinvent-bens-method-for-improving-disk-performance/819265/37 "2018-08-11T14:44:09Z")
Well, duh, I understand now after the recent comments, but I was confused by it, too, at the time of reading. Sheesh. ---
**Author:** ![BeepKillBeep](https://avatars.discourse-cdn.com/v4/letter/b/9e8a1a/32.png) [@BeepKillBeep](https://boards.straightdope.com/u/BeepKillBeep)\ **Post date:** [August 11, 2018, 2:57pm UTC](https://boards.straightdope.com/t/is-anyone-clever-enough-to-reinvent-bens-method-for-improving-disk-performance/819265/38 "2018-08-11T14:57:24Z")
> [@septimus](#): > > I am confused by the confusion of y’all! 🙂 > **septimus** used the word “quantum” in an unfamiliar way that might confuse. > Someone ( **septimus**!) noticed the potential confusion and provided a **clar·i·fi·ca·tion**. > > Does this … uh … _clarify_? 😛 Or just confuse further? :eek: Sadly, my fascination with the whole thing (the reply) has kind of passed. I’m looking forward to the big reveal of Ben’s method just for curiosity, this isn’t my area of expertise at all. ---
**Author:** ![RaftPeople](https://avatars.discourse-cdn.com/v4/letter/r/6f9a4e/32.png) [@RaftPeople](https://boards.straightdope.com/u/RaftPeople)\ **Post date:** [August 11, 2018, 5:47pm UTC](https://boards.straightdope.com/t/is-anyone-clever-enough-to-reinvent-bens-method-for-improving-disk-performance/819265/39 "2018-08-11T17:47:14Z")
> [@RadioWave](#): > > Again, the problem is that there are so many ways to optimize for particular workloads that it’s difficult to imagine which one you’re thinking of. I would almost guarantee that at this point in time, whatever it is, someone has thought of it and discarded it because it only works for certain patterns. I was thinking the same thing. For most problems in software/hardware, it’s easy to consider a boatload of permutations and combinations of optimizing for various specific variables or use cases, but for any one of those variables/use cases that is highly optimized, there’s another one that just went the opposite direction. It’s pretty rare to come up with something truly novel/clever that is generally applicable, which is why I don’t think software should be patentable (but that’s a different discussion). Regarding the OP: We know it’s some form of interleaving and that it’s helps when the situation is multiple long sequential reads, so the thing that interests me is trying to predict which use cases are hurt by this method. First thought would be many small random reads, but it’s possible the interleaving method wouldn’t be any worse for that. ---
**Author:** ![septimus](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/septimus/32/410_2.png) [@septimus](https://boards.straightdope.com/u/septimus)\ **Post date:** [August 11, 2018, 7:33pm UTC](https://boards.straightdope.com/t/is-anyone-clever-enough-to-reinvent-bens-method-for-improving-disk-performance/819265/40 "2018-08-11T19:33:27Z")
Let’s first recapitulate. Recall that the disk is kept busy with a queue of requests; we don’t have the luxury of doing arbitrary _speculative_ reads; there may be many _actually-requested_ reads pending. We seek to a track containing part of a sequential file, having been directed to read, for example, blocks 7-8-9 of that file. Since we have not calibrated the disk, the seek will complete at an effectively _random_ point on that track. If we land in the middle of the requested target – say block 8 – it is a toss-up whether we use Ben’s method or not; so let’s suppose we land in the non-requested area. The track can then be viewed as having three parts: pre-target, target, post-target. If we arrive at the track when blocks 2, 3, 4, 5, 6, 7, 8, 9, 10, 11 are appearing in order than {2,3,4,5,6} will be the “pre-target” and {10,11} will be post-target. Blocks 2 to 6 (the “pre-target” ) will be worthless to us: the sequential file is being read sequentially; reading of these blocks was already completed in the past. It is the reads of 10-11-… (the “post-target”) that may have value, but it isn’t clear how much of that data should be read before initiating the next seek to read data the host is _already waiting for._ In case anyone wants to treat the above preface as a final hint and take a minute now to guess the method, I’ve placed Ben’s Method inside spoiler tags. [SPOILER]Ben arranges the data blocks in the order 13, 12, 11, 10, 9, 8, 7, 6, 5, … The pre-target — data that can be read for free — is now precisely the data that we _want_ to pre-fetch speculatively. Once we read it and the target data there is no need to dawdle: the post-target data is worthless (as the pre-target is normally); we hurry the seek to next target, perhaps getting useful pre-target reads there as well. Instead of pre-target reading being useless and post-target speculative reads risking delay to a separate requested read, we “have our cake and eat it too!” The free pre-target reads are useful, and we initiate the system-requested seek ASAP, perhaps even getting more useful pre-target reads.[/SPOILER] It’s easy to find some cases where Ben’s Method might yield a loss rather win. (I’ve certainly conceded that Ben’s bosses did not adopt his Method and he never patented or published it.) _ **But it still seems novel, and nifty.** _ If thread participants were already familiar with it, they did an excellent job of keeping that knowledge secret. Or was it too trivial? I’m told in the thread, that this Method was probably well-known decades ago. Please link to such discussions. Was Ben, after all, a plagiarist? 😉 [Previous page](https://boards.straightdope.com/t/is-anyone-clever-enough-to-reinvent-bens-method-for-improving-disk-performance/819265.md?page=1) [Next page](https://boards.straightdope.com/t/is-anyone-clever-enough-to-reinvent-bens-method-for-improving-disk-performance/819265.md?page=3)