The basic principle is that they are communicating verbally and via an agreed-upon code of hand signs to buy and sell securities or commodities from each other. On the floor of the NYSE, there are two fundamental types of people on the floor: Brokers and specialists.
Brokers are usually (I think self-employed brokers still exist, but are getting rarer) employed by banks or other institutions which wish to trade on the stock exchange. They can do so for different purposes: Either the institution the broker works for wants to buy or sell securities on its own behalf, or it is executing client orders. A client order is exactly what it sounds like: A client instructs his bank to buy or sell certain securities on the client’s behalf, and the bank provides that service, collecting a commission from the client for doing so. It is important to remember that the broker typically does not decide when and if to buy or sell a security; he takes orders from whomever he works for, and his job is to make sure that the order is carried out (the people who decide what to buy are situated elsewhere; they might be, for instance, portfolio managers employed by an institutional investor to run that investor’s holdings). Such orders can take many different forms; the two fundamental types are market orders, whereby the broker is instructed to conduct the transaction at the best terms available (i.e., lowest price for buying, highest price for selling), and limit orders, whereby the order comes with a price limit above which the broker is not supposed to buy, or below which he is not supposed to sell. Many different types of limits exist, however.
The other type of people on the floor are specialists. Each specialist is in charge of a particular security or category of securities; for each security there is only one specialist. The specialists have defined stalls called posts where they stand all of the time and to which they are assigned. The trading of the securities assigned to a given specialist takes place at that specialist’s post.
The shouting and gesticulating on the floor is, ultimately, communication going on between brokers and specialists about trades which people intend. Suppose a broker receives an order for a particular trade. The trade could be to buy 10,000 shares at, say, $45 each. The broker can communicate that order to a specialist, who will take it in his “book”, which used to be a physical book but is now kept electronically. It is a registry of all open order in a particular security.
The book will be sorted by buy and sell orders, and they are also sorted by the quality of the order. By quality I mean likelihood of being executed: For sell orders, the order with the lowest price is listed first (because the seller willing to sell at the lowest price will be the first to find a buyer); conversely, for buy orders the highest price will be listed first. If a specialist has matching buy and sell orders for a given security (matching meaning that the prices at which the buyer is willing to buy and the seller willing to sell match), he will execute the two against each other. The information that the order is executed is transmitted to those who placed the orders, and the information goes on to settlement, i.e., the actual delivery of the securities (which takes place outside of the stock exchange and is different from trading). The orders are then deleted from the book. As a result of this, there will typically be a bid-ask spread: The order with the highest “bid” price (that is the price the buyer is willing to pay) will be below the lowest “ask” price (the price at which a seller is willing to sell). The difference between the two is the bid-ask spread. In the case of our specialist, who is just receiving an order to buy 10,000 securities of that sort at $45 each, there could be a situation where the same specialist has an open sell order in his book for 10,000 securities of that type at $45.20. In that case, the security will be quoted at $45.00 x 10,000 / $45.20 x 10,000, meaning there’s a buy order (or sum of buy orders) over 10,000 securities at $45 and a sell order (or sum of several sell orders) for 10,000 at $45.20, and a bid-ask spread of twenty cents. This information can be communicated via hand signals but is also available electronically, e.g. on screens on the trading floor.
In this situation, there are several possibilities. If nothing else changes, no trade takes place because buyers and sellers don’t match. But it is possible that now a buyer who posted the $45 bid ups his bid and changes it to $45.20. Then the orders match, the transaction is executed, and a price of $45.20 is posted as the “last price”, i.e., the last price at which a transaction occurred. Complete price data on the stock exchange always includes all three components: The current highest bid price in the book, the current lowest ask price in the book, and the last price at which trading took place. Conversely, it is also possible that the seller who submitted the $45.20 sell order lowers his ask price to $45, and that becomes the last price. It’s also possible that buyer and seller meet somewhere in between, say at $45.10. Lastly, it is possible that the specialist himself jumps in: Specialists are allowed to execute orders against their own portfolio, and they have an incentive for doing so because it generates transactions and keeps the bid-ask spread low. Specialists are constantly evaluated against the bid-ask spread in the books they manage and try to keep it as low as possible. In that scenario, the specialist could jump in by submitting, on his own behalf, a buy order for $45.10, thereby reducing the bid-ask spread from twenty cents to ten cents. This is called market-making and is considered desirable, because it makes the market more “liquid”, which means that more trading takes place and it is easier for orders to be executed. Stock exchanges are competing against each other for a high degree of liquidity, since investors prefer to do their business on liquid markets.
Brokers are also allowed to trade directly with each other without recourse to a specialist. In that case, they are obliged to report the price at which they traded to the stock exchange (for this purposes there are papers lips available all over the trading floor; the brokers fill in the slip with the details of the trade, scan it at one of the electronic readers also available all over the floor, and the data is disseminated).
So in essence, the shouting and gesticulating is a coded form of communication whereby people look for other people who are willing to buy securities from them, or sell securities to them, and the associated price and quantity negotiations. It may look like chaos at first glance, but there’s actually a sophisticated code behind it that mkes it possible for everyone involved to unambiguously understand each other’s signals.