Do governments or security agencies ban certain computing books?
The reason I ask is for a long while I’ve had a dream of making my own operating system and I’ve got a lot of books on os’s.
I know that it’s the organisation of the pc’s I’d like information on and there isn’t a much documentation on things at a hardware level.
It seem to me that hardware manufacturers incentive would be to withhold information about how things work . I have some assembly & higher level experience.
But my main question is have any computing books been banned or not published because they were considered too factual?
I’ve never heard of any interference about data on hardware. If a book were restricted, if anything it would be on cryptography or the equivalent of the Anarchist Handbook, but for hacking.
I suspect you are simply looking in the wrong places for the information you seek. For instance the various Intel reference guides on the x86 family contain everything you need to know about how to access and use the lowest levels of the processor, including all the OS relevant descriptions. However the device you are going to need to work with is the bus interface system (in Intel parlance the Platform Controller Hub). Modern chips fully or partly integrate this functionality onto the same wafer as the processor. In reality the PCH is a mess of a huge number of functional units, many that once existed as individual chips. This is a world of device control registers, mapping registers DMA and interrupts. You will find everything you need in various scattered reference manuals. But the level of complexity is going to be overwhelming. Think hundreds of pages all alike with nothing but bit level descriptions of device registers. Luckily the BIOS component of mother boards is intended to provide some level of abstraction and initialisation, but there remains a huge and horrible cliff to negotiate if you want to start from scratch.
OTOH, relatively simple systems (say a Pi or Beagle Board) are not quite so terrible. Still pretty daunting however.
Rolling your own OS is not for the faint hearted. But as Linus proved, it can be done. However he was cloning Minix, and starting with Tannenbaum’s seminal book on the subject. He wasn’t inventing anything new in the world of operating systems, and indeed famously eschewed what even then were considered fairly basic innovations. (History has proved his approach correct.) If you were to try a hobby OS project, you would want to have a very clear idea about what it was you wanted to achieve. Another Unix clone is arguably a waste of effort. But by the same token you need development tools. The Gnu toolchain is probably the most critical element in the modern world for open innovation. (Which is why Stallman is so antsy about the name Gnu/Linux.) Building an OS kernel is only a beginning. Building the higher level abstractions - security, naming, persistence, IO, etc is a huge undertaking. What one has seen in the past is that people have adopted the Unix level abstractions as a fast way of getting something up and running (Mach, Chorus, Choices, Linux, etc) whilst only a few have gone down the path of a scratch start on abstractions - Amoeba, Lilith, Grasshopper, Mungi, Iguana for instance. You have to go back a very long way to see major innovations.
Thanks, that’s some good information for me, I have a couple of tanenbaum books and the minix book , I guess I’ve not fully realised how much of a huge task it may be. I will check out the intel reference guides , I understand buses and am also reading about FPGA but with the complexity of modern hardware I don’t know how much progress can be made. I would like to make my own render farm and have considered using ARM processors in a parallel fashion with some homemade PCB, yeah it’s maybe too ambitious as I’m no electronics genius, still I’ll continue reading about stuff and tinkering out of curiosity. Thanks anyway for the info 
A render farm will depend upon the rendering software you want to use. That means (unless you are writing this from scratch as well) that the software will expect an existing OS abstraction. Or you need to modify it to use whatever yours is. Minimally this is going to mean a file system interface. (Sadly, file systems are what everyone uses and expects, so although you don’t actually need one, your software will expect one.)
I would imagine that most rendering is now heavily geared to use either GPU accelerations or x86 SSE family acceleration. So whilst an Arm would be cute, the performance would be very disappointing.
But no, doubt. A pure compute server, very basic kernel structure is within a hobbyist’s grasp. A compute farm needs little of what a fully blown OS provides.
OTOH, you could do worse than to look at some of the very stripped own Linux builds to get your hands dirty. You can configure some to remove a huge amount of cruft, leaving only the bare bones you need. Then you could build a distributed render engine API on top of that. I wonder if you could get Hadoop to run on a very cut down Linux. (Hmm, it is written in Java, so maybe a bit of a trial.)
You could have a lot of fun. The overarching advice is to start with a task that is clearly achievable in a sensible time. OS projects like Mach succeeded because they identified the core goals, and implemented only those to start with, and they leveraged every last bit of help from existing sources for the rest. Mach was written by a few PhD students locked in a windowless cinder block room at CMU. Yet is now arguably one of the top 5 OS’s on the planet.
Incremental well defined achievable goals is the key.
As part of my job I maintain a custom proprietary operating system for an industrial controller. It was originally written on custom hardware, but back around 2001 I ported it to the PC architecture so that we could use off the shelf embedded PC type hardware instead of rolling our own.
There’s plenty of information out there about the PC architecture. Back when Intel used to print out hard copies of their manuals, the x86 development manual was 3 large volumes. The first two volumes contained things like the instruction set and a lot of architecture info, but the third volume contained the real guts of what you need to know if you are writing your own OS.
There are also some decent books available on the PC architecture. If you are still having trouble finding any, I’ll see what I can dig up. I learned the basic PC architecture back in the early DOS days so the books that I have on my shelf are very outdated.
If you go back to the original 8086 processor, it had two modes of operation, minimum and maximum modes. Minimum mode was designed for embedded controllers and uses a fairly minimal amount of chips (in those days - these days you can get an entire PC on a single chip). Maximum mode was designed for full-featured computers. The PC architecture is basically built around 8086 maximum mode. All of the extra stuff is built into the controller hub chips these days, and the controller hubs have just about everything except the kitchen sink in them. The data sheet for your typical controller hub these days is several thousand pages in length. Some fun afternoon reading…
I am very knowledgeable and experienced in this area, and it took me months of work just to port our existing operating system over to the PC architecture. Writing device drivers is also a huge time sink. I’ve spend months writing ethernet drivers.
If your eventual goal is a custom render farm, writing everything from scratch is going to take you decades of work and probably isn’t practical. You are better to start off with off-the-shelf hardware and as much off-the-shelf software as you can get, probably basing it around some flavor of embedded linux.
If you do want to continue down the path of making your own operating system, you are going to need a boot loader, stuff to initialize all of your hardware and software, you’ll need to switch into protected mode (which means setting up your own global descriptor tables and interrupt vector tables, and debugging the setup of those can be a real treat if something goes wrong), and once everything is set up you’ll need to run your own kernel core, which is typically set up off of the PC timer tick. You’ll need to handle task switching, resource blocking so that two things can’t access the same piece of hardware simultaneously (which then gets you into the whole topic of potential semaphore locks), resource allocation, all kinds of stuff.
If you just want to tinker, start with FreeDos, as this makes a great quick and easy boot loader. All you need to do is have it automatically load and execute your own custom program and you can go experimenting from there. Start with setting up your own vector tables and switching into protected mode, then do some other things like getting the list of PCI devices attached to your system. Then maybe come up with a simple task structure, then load and execute your own custom tasks, which of course means that you’ll need a functioning task switcher by then. Rouind-robin task switchers aren’t the best thing to use in the real world, but for basic experimenting they are at least simple and easy to implement.
By the way, the “poor man’s supercomputer” these days is basically using multiple graphics cards as vector processors. If you want the best bang for your buck for a render farm, this is going to be the way to go.
Depending upon what your compute task is it is also the very rich man’s supercomputer.
Real supercomputers tend to be built with a specific task in mind, and they have an architecture that is very carefully tuned to that one task. You will see some that have no GPU assistance at all. And you will see some that have a fleet of FPGAs bolted onto each processor. Then you will also see a very wide range of ratios of processors per node (ie processors sharing the same memory) and a range of possible interconnects, ranging from little more than fast Ethernet, right up to memory coherent very fast comms with very high bisection bandwidth. Many times these machines won’t rank especially high in the top500, as they don’t run Linpack as fast, but for the task at hand, they will often be devastatingly fast.
The problem with GPUs and any sort of custom OS is that you have zero chance of making anything work without the vendor’s compute library. And that will be available as a binary module that expects a subset of possible OS APIs. It will expect to be able to do useful things like its own DMA.
I find it a bid saddening that it has become so hard for anyone to try really novel ideas in OS design out. The current paradigms are well past their use by date, but the barriers to entry have never been higher. (And processor architecture novelty is even worse.)
At the hardware level, each device is different. You can’t write a generic textbook about it for most cases, so you won’t find such books on Amazon or in the library. You have to go to the hardware maker’s website and find their documentation.
But those weren’t books about computing (they were, at best, books that mentioned computing).
“I want to build a render farm” and “I want to start with my own OS” are about 30 levels of abstraction apart.
It’s about like “I’d like to replace my car’s stereo speakers” and “I’d like to mine my own metal and refine my own crude oil to build my own car from scratch.”
Yes, your scratch-built car could/would have a different speakers. But you’re doing about 1E10 times as much work as necessary to get there. And you won’t live long enough to get more than 1E5 (at best) of the way into that 1E10 pile of work. Leaving 9.99995E9 of the work undone.
There was a time, back in Ye Olden Dayes of the (WAG) early 1960s when there wasn’t quite so tall an abstraction stack and people could talk of projects like these.
Nowadays, as the experts above have said, you *can *do simple low-level OS-like things for fun and understanding. Provided you start on a very simple substrate such as a Raspberry Pi.
An OS is actually “just” a realization of several different well-understood algorithms for things like scheduling, file systems, memory management, etc. If your interest is at that level, there’s nothing preventing you from building a high level part-task OS simulator that runs as an ordinary app and schedules, files, allocates, or whatever for a batch of simulated problem tasks you feed it.
My bottom line:
Even today there are artisans who make adobe bricks from mud and straw. The important thing to remember is that they then use these to build small huts, not soaring skyscrapers. And after a year’s outdoor labor under the baking sun they have one small and homely mud hut. An *authentic *hut, but just a hut nonetheless.
As if the responses aren’t daunting enough, I mention something my son told me. He was working on the NT team in the 90s; his job was writing device drivers for various hardware devices. Every device came with a list of functions from its manufacturer. Put 00100110 into register A and the device will do x,y,z, that sort of thing. He quickly discovered that none of these spec sheets was worth a damn. The manufacturers kept changing things around, but not bothering with keeping the documentation up to date. So the first thing he had to do was test all the possibilities and build his own spec sheet. And this for each device. Maybe things have improved in 20 years, but maybe not.
No, things have not improved in 20 years. At least according to an open-source embedded systems hacker I ran into recently. Documentation is (quite often deliberately) obsolete or nonexistent, and the devices need to be painstakingly reverse-engineered in exactly the way Hari Seldon’s son described before an open-source device driver can be created.
On the subject of OS development, one popular approach is to develop a microkernel, which takes under 10000 lines of code. That does not include any device drivers, but if it just an exercise then maybe you do not need them, or you can somehow recycle BSD or Linux or Minix drivers.
Now, as for books being banned or not published due to interference by “governments or security agencies”, there definitely are known cases, like IBM’s failing to publish papers on differential cryptanalysis because the NSA leaned on them. What happens in such cases is that eventually someone else publishes their own independent research.
With things like critical zero-day hardware/software exploits, sometimes the hacker delays dissemination of the details for a short while as a courtesy to the manufacturer, giving them a chance to release a patch.
There are fields where people regularly work with secret classified information and are not so quick to publish it for public consumption, and companies that jealously guard their industrial trade secrets, but all of that hardly applies to general computer science concepts.
Maybe he doesn’t mean “render farm at same quality level as pixar”. Drawing an unshaded triangle on the screen is still technically rendering. If he only supports a single processor (that’s a layer removed), and heonly supports a single communication bus type (serial), and he just implements the bare minimum OS functions, and he just implements some crude triangle drawing snippet of code written in assembler to do the “rendering”, followed by sending the frame buffers to a single computer to be assembled into a final image - it could maybe work.
I’ve seen “OS” examples written in about 3 files. A single render operation isn’t too bad. The data sheet problems mentioned above aren’t as bad if you only need to access a single hardware device, a high speed serial port. Might take him 2-3 months.
But I wouldn’t work on it alone. Georgia Tech has courses, freely available to the public, and if you have a bachelor’s degree in STEM and a 3.0 they will almost certainly allow you to take them online for credit, in all the topics mentioned. They have an OS class, a class in some rendering techniques, a class in networks, and so on. Each of them will make you write functioning sections of code to implement the key concepts. Once you take about 5 different courses, you will have all the knowledge you need to start on the project and get it done.
See here : Current Courses | OMSCS | Georgia Institute of Technology | Atlanta, GA
You’d want to take Introduction to Operating Systems, Advanced Operating Systems, Intro to High Performance Computing, Computational Photography, High Performance Computer Architecture, Computer Networks, Embedded Systems, and maybe the courses in Software development process. At which point, assuming you took them for credit and made at least a B in them all, you’ll probably have enough practice to actually build what you want to do from scratch in a reasonable timeframe, or to realize it’s a bad idea and use your newly minted Master’s degree to get a job instead…
And that still wouldn’t be banned. In fact, back in the 1990s, when cryptographic software was heavily controlled as a munition under ITAR, Schneier got around those laws by publishing full source to multiple cryptographic algorithms in his book Applied Cryptography, because books were undeniably protected under the First Amendment at a time when similar protections for software were a lot murkier.
Sure, if you consider “cloning Minix” to mean “writing a completely different kind of OS with completely different technical underpinnings and a completely different fundamental design”, but most people in the computing world know that Linux and Minix were competing designs, that Tanenbaum thought Torvalds was going down the wrong path, and that Torvalds disproved Tanenbaum quite comprehensively. More information on that.
Eh, not really.
Part of the problem is that the spec sheet for something like a controller hub can be thousands of pages long. Some stuff is bound to be glossed over and not well documented. I found one bit of critical info buried in what was almost an aside style comment in a completely unrelated part of the documentation. If all you did was look through the relevant section of the spec sheet for that particular function of the chip, you would have never seen it.
We also had a fairly recent case where I was trying to turn on a fairly simple feature of one particular part of an interface chip. It seemed very simple and straightforward. Just twiddle these few bits in these couple of registers and you should be good to go. I did exactly that, and the chip completely ignored me. After days of beating against the thing, we found out that even though those bits were documented to do that, if you wanted to do that, you didn’t use those bits. Instead, you used some other bits to send a command to something else which then ended up doing that for you. There was absolutely nothing in the spec sheet that even hinted that this was how you had to go about it.
The worst spec sheets are the ones that look like they were hastily translated from another language by someone whose grasp of English was marginal at best. “All your base are belong to us” is funny in a video game. Imagine trying to decipher technical information in a document with that level of English throughout.
For those who aren’t aware, this all started with AT&T writing Unix. For many years, AT&T freely gave the source code for Unix away to universities so that they could use it for teaching operating system principles and such. Then AT&T decided they didn’t want to keep giving it away for free. In response to this, Tanenbaum created Minix, which was a Unix clone that could be used for education without containing any AT&T code in it at all, meaning that educators could use it freely without paying royalties to AT&T.
Almost from the beginning, there were two competing schools of thought. One group wanted to keep Minix current and keep adding features to keep up with AT&T. The other group wanted to keep Minix simple and stripped down for educational use. Tanenbaum was in the latter group.
Linus created Linux partly as a hobby and partly as his thesis for his Master’s Degree. You can see from his original announcement that it was originally created just for him (and others) to tinker with and that he never expected it to be a serious rival for major operating systems:
Linus didn’t originally write Linux to compete with Minix. He was just experimenting. Linus was part of the “let’s make it as full-featured as AT&T Unix” school of thought, though, and Linux quickly grew to rival Minix.
Minix is generally regarded as a Unix clone, and it contains no AT&T code whatsoever and was a complete re-write of Unix from the ground up. Linux completely re-wrote Minix from the ground up, but I wouldn’t take any real issue with someone calling it a Minix clone because that’s basically what it started as. Under the hood it was completely different, but from a user’s point of view it was just another Unix clone.
You’re missing a big point: Minix was Tanenbaum’s idea of how an OS should be designed, in that it was, and is, a microkernel design, whereas classic Unix and Linux are both monolithic kernels. Minix would likely exist even if AT&T never clamped down on Unix, because, after all, the BSD Unix variants were still freely available for any use, and Tanenbaum would certainly have known about them, given how much BSDs were used in colleges at the time, both as teaching models and as production systems.
Tanenbaum also wanted to ensure Minix would continue to run on 16-bit x86 systems, which were quite a bit cheaper at the time, whereas Torvalds had just bought a shiny new 386 system and had no intention of supporting 16-bit x86 chips.
(There were real-world production Unix variants which ran on 16-bit x86 chips, such as Xenix, which was the best-selling Unix variant in the world for a while, but the 16-bit x86 chips were never a good platform for Unix-variant OSes.)
Cite that it was ever an educational project. I’ve never heard that from anyone but you, and I doubt it.
Same as Linux.
It’s not a Minix clone. It’s a Unix clone. Torvalds explicitly didn’t copy Minix’s design, even though he’d seen it, and argued the point when Tanenbaum called him on it.
Minix played some role in Linux’s existence, but it wasn’t as large a role as you seem to want it to be, and I think you misstate history to magnify it.
From here: https://www.computer.org/web/awards/pioneer-linus-torvalds
Here is the actual thesis (warning, pdf):
ftp://ftp.polsl.pl/pub/linux/kernel/people/torvalds/thesis/torvalds97.pdf
I never said Linus copied Tanenbaum’s design. In fact, I would say he most definitely did not copy Tanenbaum’s design because, as you pointed out, they had drastically different opinions on how the kernel should be designed.
Linus started by playing around with Minix though. If you look at his original Linux announcement that I quoted, it was posted to the Minix newsgroup. 
I don’t see how I am magnifying Minix’s role in the history of Linux.
Thanks everyone for your replies so far, You’ve given me some good links and information. I’m still checking your replies even tho I have a few things taking keeping me busy at the moment.
I have been reading up on this for some time and understand what you have posted,my initial idea was that i could set up a data processing environment with some machine instructions and like Francis suggested it would be a pure compute serever which I could send jobs via a main machine . I will be back online later. Thanks
Apart from the errors, this is the case?
Back in the 90’s, you needed to sign non-disclosure agreements to get some of the Intel information, and even then some of the Intel information was on a need-to-know basis anyway.