In very broad terms, your premise may be mistaken
Many very benign tumors --not usually considered cancer–create pathological effects by displacing and compressing adjacent structures. For example, a benign lipoma (ordinarily harmless tumor of fat cells) on the back may exert pressure against a nerve root, or muscle or the spinal column when a person sits, and cause a variety of problems.
Other benign (noninvasive, not very metabolically active) tumor types are usually considered dangerous because of the tissues where they occur: a relatively inert, sluggish mass in the brain [inside the closed skull] or in the muscle of the heart [especially near a valve] may disrupt essential anatomy and function, even if it would be harmless anywhere else. Even an air bubble or piece of fat or bone marrow can kill (e.g. comtimes a globule of fatty bone marrow from a fracture travels through the blood to the lungs, causing a pulmonary embolism.)
Moreover, when a cell converts to a tumor, it doesn’t just sit there and “tume” [if you’ll pardon the term}. It continues to have many or most of its usual metabolic functions, usually without the fine control seen in normal tissue cells of that type. You can imagine that this can really cause a mess with primary endocrine glands: a “thyroid storm” or “adrenal crisis” can kill in minutes if triggered. As you can imagine, tumors don’t like being poked or cut, so operating on endocrine tumors can be tricky.
Aside from endocrine tumors, many tissues have hormonal effects on surrounding tissues, with significant effects when disrupted. Zollinger-Ellison tumors of the duodenum (upper small intestine) or sometimes pancreas, were one famous example: they were usually seen clinically as ulcers that wouldn’t heal or respond to treatment (which was a pretty common clinical picture just 20 years ago) and not recognized before it was too late [Today, our diagnosis and treatment of ulcers has improved dramatically, and new drugs dramatically improveour ability to control the hormone induced acids of a Z-E tumor]
However, not all tumors are so benign. many are actively nasty: they invade adjacent tissues ferociusly, disrupting their function, they react or overreact violently to metabolic changes disrupting the body’s physiology, they suck up limited resources, starving adjacent cells or poisoning them with toxic waste products. Single cells may may spread promiscuously through the blood or lymph to grow in completely different tissues.
These latter, invasive, nasty tumors – “malignant”, as opposed to “benign” tumors that just take up space-- are what we usually call "cancer"