It is magic in a sense. The same magic that runs human civilization: the gradual accumulation of prior solutions and approaches into something no individual or manageable group could accomplish from scratch in a reasonable time.
You already understand about the smal scale functional building blocks. These are well within the ability of a single person to design. In fact, there are many sets of solutions – “logic families” like TTL (Transistor-Transistor Logic), RTL (Resistor-Transistor Logic) and on a different level, CMOS, are basically sets of building blocks with a common approach, yielding a compatible set of building blocks, taking into account factors like substrate and doping, trace timing, signal thresholds, and timing (on this level, all digital circuits are analog) I don’t do this for a living, so I may list some of the properties in the wrong category below (and tests may occur in different stages depending on the specific tools you use), but I hope it’ll help you understand the process – to the extent I do, anyway.
Once you have a basic overall design, there are many tools that largely automate the process. First, you set up “design rules” to specify parameters related to the target design and IC technology design voltage thresholds, timing, propagation, and spacing. A program called a DRC or Design Rule Checker constantly checks your design as you lay in the circuit, and alerts you to issues interactively.
Similarly, there are “electrical rules” checked by an Electrical Rule Checker (ERC), which take into account the materials (e.g. substrate, doping, electrical properties at a given feature size like insulation and antenna properties of traces, etc.) and manufacturing constraints and properties (e.g. diffusion, breakdown, lithography limitations and imperfection tolerance, etc.) for a given technology and manufacturing facility. Actually, you usually choose a compromise: local personnel experience, conditions, suppliers and even climate can affect production – though this is controlled as tightly as possible.
Different facilities with the same equipment in the same company may have quite different yields. Factories tweak techniques and physical production parameters to improve yields over the course of a production yield, so generally a cutting edge chip aims for electrical rules that are expected to be achievable at low yield at present; Improvements over existing capabilities (within the theoretical capabilities of the underlying technology) are not just expected during the production run, they are almost demanded! For most chips, the Electrical Rules and Design Rules can be looked up in an in-house book, but for cutting edge chips, it’s a mix of art, marketing, and finger-crossing
Once the design is finally entered (with any changes required by the rule checkers) it is written in EDIF format and run through run through a circuit simulator (e.g. SPICE or Verilog) for extended and detailed checking. Some of the tests create virtual oscilloscope traces of expected waveform at various points in the circuit, so the design can be tweaked. Other tests are extended automated runs looking for glitches.
Routing is also an important task. Crossover, leakage, antenna properties and timing issues (e.g. often signals from different circuit block on different parts of a chip must arrive at their destination with tight synchrony) are crucial on the scale of modern CPUS. To do this, the tested circuit resulting formthe circuit simulations is converted to a network description, and tested in that paradigm. this si called Network Consistency Checking (NCC) or “Layout vs. Schematic” (LVS) testing. After it passes this stage, it is “disambiguated” (a process which I think I understood once, but can recall at the moment. “Disambiguated” comes from the root word “Ambiguity”) to create a workable design to be compacted for efficent packing.
Your VHDL compiler creates a link list between basic low-level building blocks, which can be compacted (usually by alternating vertical and horizontal compaction passes, aided by a little skilled human experience and intuition) and then run through yet another level of simulated testing (like RSIM) and eventually into a Silicon Compiler. Modern silicon compilers don’t just create a basic mask, but can also perform tests like “Logical Effort” which help them determine optimal feature size for a given target speed, layout and timing, etc., etc.
Doing a search for some of the key terms above, I found several manuals that seem to provide fairly clear but detailed descriptions fo the process and options for specific tool-sets. I think they should answer any question that you might have. The one for the Electric™ VLSI Design System looked pretty instructive to me, and would at least suggest the search terms for more specific topic reseach, If you don’t like that one, “VHDL compiler” seems to be a good single search term to find the manuals for other packages. Pretty much every IC design package either has a VHDL compiler (if they don’t, they’d have to explain what they use in its place)
I hope that helps, and isn’t too confusing. Dammit, Jim, I’m a doctor, not a semiconductor manufacturing engineer!