Well, we really don’t know the long-term effects of all the additives we put into food, either singly or in combination.
However, we really don’t know the long-term effects of the chemicals - including fats, proteins, and carbohydrates - that are in food, either singly or in combination.
And we sure don’t know how the two groups work together.
What we do know tells us pretty much what we might suspect: some of these chemicals are more harmful to some people than others of these chemicals.
Everything else is just violent debate. 
However, I can add some history. The Food Additives Amendment passed in 1958 still controls substances added to foods. They are put into three major classifications:
- Substances Generally Regarded as Safe (GRAS)
- Additives
- Substances that are prohibited
In 1969 the Select Committee on GRAS Substances (SCOGS) was charged with evaluating the status of additives, and finally produced a report in 1980 that covered 415. They were placed into five classes, 1 - GRAS with no limitations; 2 - GRAS with some limitations; 3 - GRAS with continued testing required; 4 - ensure GRAS status by establishing safe usage conditions or remove from food; and 5 - insufficient data. (Some of the 415 substances were additives and use the same classification scheme.)
Salt, BTW, was deemed a Class 4. The additives BHT and BHA were put in Class 3. Calcium cyclamate was prohibited. Coumarin in vanilla flavor was prohibited. Food colorers are not part of GRAS, but a few of those too have been prohibited.
Many newer ingredients have no GRAS status at all, like stevia. Soy protein has no GRAS status, and there is a debate on whether it should even be considered for GRAS.
GRAS status does change with new information. The nature of science is that things change. The FDA issues GRAS notices on a regular basis.
So is everything GRAS really GRAS? I don’t know, but I know more testing needs to be done on many items. Are all additives GRAS? See above. Can we say for sure that Class 2 items are limited in the proper way? See above. Will opinions on GRAS status change? Yes.
Should the OP be more worried about trans fats, empty sugars, too many calories and the other known problems of the average American diet than “chemicals”? My opinion is yes. Adhering to the classic diet that people with no diet books to pitch recommend - lean meats, whole grains, lots of fruits and vegetables, limited fats and sugars, small portions - along with regular exercise is going to be far healthier than worrying about nitrates or MSG. That is not the same thing as saying there is nothing to worry about concerning nitrates or MSG or any number of other additives. Just that one is, ahem, small potatoes next to the other.