Claim: the NSA records and stores 80% of U.S. phone calls

[Martin_Hyde]

@Steve_MB

Yeah, I suppose the guy behind the Internet Archive wouldn’t have understood all those potential obstacles. :rolleyes:

The Internet archive just crawls sites that have a robots.txt file present, basically sites that want to be found. It used the same fundamental concept of how search engines found websites in the old Internet before they got more algorithmic.

There may be easy “tap points” where you could just have a few locations tapped and you magically get every phone call (I don’t know enough about the telecommunications system), but it’s fundamentally different from how the Internet archive works either way.

There was a discussion in Cafe Society recently about Terminator 1, where someone claimed the Terminator should have just hacked the phone system and waited until Sarah Connor made a phone call in order to track her (as opposed to the method that actually worked that he ended up using–it was a silly thread), and someone piped in with information on how the nature of how phone calls works makes that essentially and impossibility. Maybe now that the old POTS network is mostly (entirely?) replaced by fiber it’s a different beast, though.

But yeah, the guy who wrote a web crawler to crawl sites that specifically wanted crawled would in fact not impress me as an expert on how to record every phone call in the world that passes through the U.S. physical infrastructure (which is apparently 80% of all global calls.) And in fact, he doesn’t attempt to explain the collection at all, he simply estimates (using true “envelope” math of no particular validity, in which he assumes total calling time based on his personal family’s monthly calling minutes) the amount of storage it would require and then quotes standard pricing for that amount of storage.

So again, doesn’t address my point. But maybe if you try to repeat it a third time without addressing my point I’ll buy it at that point and agree with you, so go ahead and try that out.

[XT]

@Steve_MB

Er… how do you divide 2 billion by 10k (10000) and get 200 million? Are you using the “long billion” (and thereby assuming that each person in the US makes an average of twenty thousand phone calls per day)?

1 call= 96 KB/min…I estimated an average of 1 min calls. Seemed like a reasonable estimate, but I concede that MMV. If you have better numbers I’m game. It’s still an obscene amount of data.

That’s an average of twenty calls per person (each one counts twice, having a caller and a callee) per day. For a more realistic guesstimate, divide by five or thereabouts.

Are you sure that the 3 billion calls a day are counting each call twice (or 5 times)? What do you base that on?

Considering that a 2 terabyte drive will run $100 at Best Buy, this isn’t as big as it seems.

Yeah, I had this same argument with our state police chief. Sure, you could buy something like that from Best Buy, but it costs a bit more to do it professionally, to be able to have redundancy and throughput and fault tolerance. I’m fairly sure that the NSA doesn’t buy their storage from Best Buy that fits into a whole bunch of USB drives. Instead, they have a mega-SAN (probably a bunch of them), and the cost on that is going to be pretty big. The 40 TB SAN shelf we recently bought from NetApp, for example, costs us over $100k. Doesn’t count the core infrastructure…just spindles (actually, it’s a solid state drive system, which is why it costs so much, but you need that or better to actually sort that obscene amount of data the OP is taking about).

Quite eminently possible. As noted in one of my previous links, the analysis done Brewster Kahle (who set up the Internet Archive and thus has a pretty good track record for knowing what such an operation entails) is that it’s trivially easy for government-scale resources.

Well, granted I don’t work for the NSA, but it’s not trivially easy for us to do a fraction of what they are talking about here. It’s in fact very difficult and expensive, and you have to factor into whatever scale you are talking about how long you plan on archiving the data. Are we talking a day? Ok, I could see that. A week? Harder but possible. A month? Getting much more difficult. A year? At that point it’s off the scale as far as I’m concerned, though again I don’t work for the NSA so maybe they have magic tech and really deep pockets that the rest of us don’t.

Do they do anything else except store phone conversations? If so, they they are going to need money and infrastructure for all of that too.

I’m not an expert in psychological dysfunction. Try asking someone who deals with those sad souls who have houses full of decades-old newpapers and whatnot.

You believe that the NSA does this for no rational reason, but simply because they are horders?

What I could never believe is that anyone ever fell for the “it’s just metadata” lie, given the huge discrepancy between the trivial amounts of storage required for “just metadata” and the actual demands of the agency.

Well, based on your Best Buy USB drive solution, it’s hard to take you seriously on this, to be honest. Until I see something solid, I’m going to stick with my initial assessment.

[Capt.Ridley_s_Shooting_Party]

@Martin Hyde

There may be easy “tap points” where you could just have a few locations tapped and you magically get every phone call (I don’t know enough about the telecommunications system), but it’s fundamentally different from how the Internet archive works either way.

But the claim isn’t that the Internet Archive’s mode of operation is the same as what the intelligence agencies are doing. The claim is that, contrary to what one may expect, and based on some reasonable assumptions, the storage capacity needed to physically record every phone call in the world is not as outrageously outlandish in cost or storage space as one may expect, and well within the capabilities of the NSA and GCHQ, given what we know of the data centres these agencies are building and currently operating.

The Internet Archive is only mentioned to bolster the credibility of the guy making the claim: he is running a massive operation that needs to store large amounts of data. He can reasonably be called an expert on how much data storage at the scale we are talking about actually costs.

[Capt.Ridley_s_Shooting_Party]

@XT

Instead, they have a mega-SAN (probably a bunch of them), and the cost on that is going to be pretty big.

This is a strangely academic discussion seeing as we know the NSA have just spent $1.5 billion building a massive data centre in Utah with a further estimated budget of $2 billion for hardware, software and maintenance, consuming an estimated 65 MW of electricity per year. Yes, these things cost a lot, which is why GCHQ and NSA have large budgets.

A Forbes article estimates the entire data centre has between 3 and 12 exabytes of storage capacity (!) Hmm, I wonder what the NSA could possibly be doing with that much storage? Recording all of the internet activity of “extremists” who venture onto the well-known hive of terrorist activity, Linux Journal, perhaps?

[buddha_david]

@Capt.Ridley_s_Shooting_Party

A Forbes article estimates the entire data centre has between 3 and 12 exabytes of storage capacity (!) Hmm, I wonder what the NSA could possibly be doing with that much storage?

Watch CaptainSparklez videos?

[Johnny_Cashless]

I think you need a slight adjustment; the NSA are basically doing what they are told - the money is the public’s money, and a bunch of clowns in DC spent it for them so the clowns know what the people who paid for it are doing in their private lives.

Along the way, a bunch of friends of the DC guys are going to get seriously wealthy.

At least the Stasi / East Germany thing was kind of a non-profit deal In the US version, the public gets it in its ass twice.

[Capt.Ridley_s_Shooting_Party]

@Johnny_Cashless

I think you need a slight adjustment; the NSA are basically doing what they are told - the money is the public’s money, and a bunch of clowns in DC spent it for them so the clowns know what the people who paid for it are doing in their private lives.

Along the way, a bunch of friends of the DC guys are going to get seriously wealthy.

At least the Stasi / East Germany thing was kind of a non-profit deal In the US version, the public gets it in its ass twice.

My point was that XT seems unaware that the NSA et al have seriously deep pockets and likely have access to technology (c.f. the quote below), built in-house, that has not made it into the mainstream yet. We know, for example, that GCHQ knew about public-key cryptography much earlier than mainstream cryptographers did and shared their discovery with the NSA. Similarly, the NSA knew about differential cryptanalysis many years before it was rediscovered by mainstream cryptographers and used their knowledge to try to change the design of the DES cryptosystem to make it impermeable to said differential cryptanalyses (widely interpreted at the time as the NSA attempting to weaken the cypher). Cryptanalysis techniques developed as part of the effort to crack Enigma and Lorenz in the 1940s are still classified to this day. Given that they can outpace mainstream cryptographers with apparently some ease, why can they not do the same when it comes to storage technology? They are, after all, collectively the single largest employers of mathematicians on the planet and invest heavily in funding their own research arms.

Further, if anything, the Snowden leaks have demonstrated that there’s functionally no difference between the signals intelligence agencies of the “Five Eyes” powers. They’re in effect a single giant intelligence agency with a budget pooled amongst them, further increasing the depth of their collective pockets, as they receive funding from five governments.

@XT

At that point it’s off the scale as far as I’m concerned, though again I don’t work for the NSA so maybe they have magic tech and really deep pockets that the rest of us don’t.

[Steve_MB]

@Capt.Ridley_s_Shooting_Party

This is a strangely academic discussion seeing as we know the NSA have just spent $1.5 billion building a massive data centre in Utah with a further estimated budget of $2 billion for hardware, software and maintenance, consuming an estimated 65 MW of electricity per year.

Indeed. In view of these rather overwhelming facts, the assertion that the NSA couldn’t possibly be scooping up content on the scale posited in the OP has the feel of a JFK assassination conspiracy theorist raising a quibble about the trajectory of the “magic bullet”.

[Capt.Ridley_s_Shooting_Party]

To address this point too:

@Martin Hyde

There may be easy “tap points” where you could just have a few locations tapped and you magically get every phone call (I don’t know enough about the telecommunications system),

This is exactly how it is assumed to work. The NSA and GCHQ operate a joint facility in Cornwall, England, called GCHQ Bude. Conveniently, it lies less than 10km from where 8 transatlantic communication cables make landfall. If you looks at this map you’ll see that in addition to these eight cables Cornwall as a whole is a rather important nexus for subsea cables, with traffic from Africa, the rest of Europe, Ireland and the Americas all being routed through the county, or its neighbour, Devon.

Bude, of course, is just one listening post that we know of. The Register last month, for instance, revealed an above Top Secret, previously unknown, GCHQ facility in Northern Oman which is also tapping subsea cables that make landfall in the country, a joint operation between GCHQ, BT — the UK’s formerly nationalised telecoms company, and Vodafone/Cable and Wireless. Again, looking at the subsea map, the Oman station is well placed to receive any and all traffic from the Middle East, the Indian subcontinent, and further afield.

It doesn’t take that many listening posts to capture the vast majority of communications traffic being sent via subsea cables (Bude also has a number of satellite dishes conveniently pointing directly at INTELSAT and InmarSat satellites), especially if you consider that the Australians, Canadians and New Zealanders are all in on the act, and the UK uses its many overseas possessions as locations for listening posts (e.g. the sovereign base territory on Cyprus).

[Onyerbike]

What would the garden variety American say or think if he thought it plausible that Brazil or Russia was monitoring <80% of their phone traffic (or any old value)? They’d rail of the deep end about an invasion of privacy, that it was the same as an actual invasion and tantamount to war, that little billy and his soccer ball will never be safe etc. That is how some of the world (you know, the rest of the people on planet earth) feels about the nsa.

I don’t know 'em personally, but damn if I could throw them further than I trust them. Any reliable estimates on the total weight of the NSA, including its agents and assets, just by way of inquiry? Either way I guess I’d better hit the weights.

[Smapti]

@Onyerbike

What would the garden variety American say or think if he thought it plausible that Brazil or Russia was monitoring <80% of their phone traffic (or any old value)? They’d rail of the deep end about an invasion of privacy, that it was the same as an actual invasion and tantamount to war, that little billy and his soccer ball will never be safe etc. That is how some of the world (you know, the rest of the people on planet earth) feels about the nsa.

I don’t have the right to not be spied on by Brazil, and neither do you.

[Onyerbike]

[QUOTE=Smapti]
I don’t have the right to not be spied on by Brazil, and neither do you.
[/quote]

But you do have a right to be safe from whatever actions those brazilians might undertake based on the information about you that they gather. For a country like brazil without the means of global interdiction it is not such a hot issue, because big whoop, what are they ever going to be able to do about what they hear in your private conversations. But a few mistakes in the chain of an organisation that has no civilian oversight and the power to do something about that ‘Smapti problem’ it seems so sure exists … and well thats why for ill or gain people get upset about these kind of things.

[Steve_MB]

@Smapti

I don’t have the right to not be spied on by Brazil, and neither do you.

With the spread of end-to-end strong encryption, the issue is becoming one of concrete practicality rather than abstract philosophy. Rather than arguing their rights, people are simply claiming them.

[Smapti]

@Steve_MB

With the spread of end-to-end strong encryption, the issue is becoming one of concrete practicality rather than abstract philosophy. Rather than arguing their rights, people are simply claiming them.

And that road leads to anarchy.

[Steve_MB]

@Smapti

And that road leads to anarchy.

Letting the tech experts do their thing without interference from the nitwits in the political peanut gallery leads to “anarchy”?

So, is anarchy the desired end result of your political philosophy, or is it just one of those “contradictions” like Marx kept finding in the nature of capitalism?

[Smapti]

@Steve_MB

Letting the tech experts do their thing without interference from the nitwits in the political peanut gallery leads to “anarchy”?

Enabling people to ignore the law of the land without consequence is anarchy.

So, is anarchy the desired end result of your political philosophy, or is it just one of those “contradictions” like Marx kept finding in the nature of capitalism?

The desired end result of my philosophy is order, safety, happiness, and freedom from want.

[Steve_MB]

@Smapti

Enabling people to ignore the law of the land without consequence is anarchy.

Well, then, it’s good that we’re beginning to see some technical repair of the security sabotage done by the government to facilitate its lawlessness. Admittedly, it doesn’t send the perpetrators to jail where they belong, but I’ll compromise and settle for the building of an infrastructure that prevents ongoing abuse.

The desired end result of my philosophy is order, safety, happiness, and freedom from want.

@Smapti

It is in the people’s best interest to conceal information from them which would cause them to act against their best interest.

[QUOTE=George Orwell]
Once again this argument was unanswerable. Certainly the animals did not want Jones back; if the holding of debates on Sunday mornings was liable to bring him back, then the debates must stop.
[/QUOTE]

[Smapti]

@Steve_MB

Well, then, it’s good that we’re beginning to see some technical repair of the security sabotage done by the government to facilitate its lawlessness. Admittedly, it doesn’t send the perpetrators to jail where they belong, but I’ll compromise and settle for the building of an infrastructure that prevents ongoing abuse.

What lawlessness? And what abuse?

[VinylTurnip]

@Smapti

What lawlessness? And what abuse?

What “law of the land” do I break by preventing the government from reading my personal email? Is there some sort of law of perpetual exposure?

People use roads to get from place to place, and some of these people are criminals. Do we need a homing beacon on every car, and armed checkpoints at every intersection?

[Smapti]

@VinylTurnip

What “law of the land” do I break by preventing the government from reading my personal email? Is there some sort of law of perpetual exposure?

People use roads to get from place to place, and some of these people are criminals. Do we need a homing beacon on every car, and armed checkpoints at every intersection?

Obstruction of justice, for one? To use your traffic analogy, you no more have a right to sabotage government email monitoring than you do to shoot out the lens on a red light camera, jam a radar gun, or slash a cop car’s tires.