The END of RETENTION LIMITATIONS?

[ericab]

can you tell us a bit about the harddrives used ? brand ? how can these drives be accessed by so many people at the same time, with no slowdowns ?

[Beck38]

All depends; I've been 'out of the loop' on pretty much everything for several years, but a 'real' industrial strength array today would probably be built on SAS (serial attached scsi), whereas back in my 'day' the best (and most expensive) were scsi array's, and there were (cira 2000) a 2-3 different types of scsi interfaces. 'Single-ended' had ruled the roost for quite a while, but 'double-ended' was coming into it's own (better compatability, higher speed, longer and more stable cable lengths). If I remember correctly (!), HP had a type that never really caught on.

Today, SAS it 'top-line', but SATA isn't any slouch. But things are moving fast (don't get run over), both in the SATA/SAS world and in the USB world. We might be having this conversation a year from now, and the 'talk of the town' may be USB3 killing off everything else.

The high-grade drives today aren't much, if any, different that what you'd put in a destop PC. Seagate just started shipping their 'XT' line of drives, in the $3-400 price range, but still 2TB (perpendicular), 7200rpm, 64MB cache, but performance wise (like for databases and such) the SAS drives beat them hands down.

But we're talking mass, super-mass, storage here. We don't need 'fast' drives, just super-stability and super-capacity. The OS is all specialized UNIX, with super-nested RAID. Each array has several 'hot stand-by' drives, and the SAS cluster has redundant drivers/interfaces. Take a look at the top of the line Adaptec (literally hundreds of drives supported on the interface card).

In fact, it's really enough to blow your mind. I was specifying/building/installing systems for companies 10 years ago that were in the $20-200K range, that today could be done for 1/10th that with a 100-1000 time as much storage, and CPU speed 15-20 times as fast.

Where will it end? I've read several articles in the past few months that postulate that 'Moore's Law' is running out of steam... but I've heard that more than a couple of times in the last 30+ years.

Really, the last vestige of mechanical computing IS the rotating hard drive.... but one can see that SSD's are making inroads, 'netbooks' are really taking off, and it's only a matter of time before 'boot' drives will be standard on SSD's.

But, what's the next 'step'? There are lots of 'tricks' in the lab to take perpendicular magnetic surfaces (and their read/write heads) up a number of notches, but when they'll make it to market is unknown.

Like I said previously, I think most of the manufacturers are taking a 'pause' right now, working on yields and cost containment. 2TB drives (SATA) that were $3-400 3 years ago when they were first introduced, are hitting as low as $130 ('green' or not-green) today.

On the performance, remember that 'you' are not 'directly' attached, but through, what is for the server, a VERY SLOW connection. How fast is your line? 3Mb, 10Mb, 50Mb? The LAN's now in an industrial system are running 1Gb, with the main lines at 10Gb.

So, say you got 1000 customers leeching at 3Mb/s. That's 3Gb. Hardly breaking a sweat. Take a look at the specs on the SAS cards, 10,000 times that much is within their capability. I look at it from a telecom perspective (that's my field), but the amount of bits these systems can move around are way faster than the communications links they attach to, even up at the OC192 (10Gb/s) level (and multiples of those and faster). Shoveling bits is easy for computers, spewing them out across the planet is hard for the telecom infrastructure.

For 'fun', I've run the numbers on what would happen if, say, every internet user in the US had a FIOS 50Mb line. Things get REALLY FUNNY/INTERESTING, to me! Just think, you could (and those with it do so now, that's a thought!) transfer a Blu-Ray disc (50GB, 400Gb/400000Mb) in just over some 2 hours or thereabouts.

Oh well. H*ll would break loose! It'll take 20 years though....if we really put out minds to it.

[ericab]

thanks for taking the time for that as well as all the info beck38 !

[Beck38]

Most the time I get pretty d*m* long winded, but what the hey.

BTW, I sent off a 'complaint' to one of the indexing 'services' I use (and pay for, based at Giganews, BTW), and come to find out, they 'index' only back 300 days. What a crock! I'll see what they have to say when they read my return message, probably come Monday.

[tesco]

Thanks, really useful information there.

I'm wondering, is anything stored compressed and then uncompressed on-the-fly when someone requests it? Maybe the older, less-downloaded articles, in the less popular groups?

[Beck38]

I'm wondering, is anything stored compressed and then uncompressed on-the-fly when someone requests it? Maybe the older, less-downloaded articles, in the less popular groups?

Virtually all the binary groups, like anything video (mpeg2/4/x.264/etc) are HIGHLY compressed to begin with; you can do some test of your own and try to rar (at max compression) such things, but you'll quickly find out that, at best, you'll get .001% compression or something equally pathetic. And tons of time/CPU cycles to do so.

Which is why basically no one uses rar compression, but only 'store'.

Now, things like jpeg (a compressed picture format) can be a bit more effective (maybe 5% or so), and of course text can be.

But such material is a sliver of a percent of usenet. Simply not worth the effort.

[Cabalo]

What a great insight how server farms work.
It was a joy to read. I've learned more in those minutes it took me to read the post than in years around using usenet.

[Beck38]

As I've gone 'way back', I'm unfortunately reminded of a period of time (late 2008) where a lot of 'newbies' (or perhaps 'oldies') forgot (or didn't know in the first place) basic usenet posting routines.

Anyone who's been around for any time remembers back when 20% PARS was the standard. Even the largest server plants (like Giganews) had retention levels of only a few months, and 'propagation' between servers was fragile at best.

Slowly, folks 'forgot'. % pars became less and less. Rar parts became slowly bloated (50MB was the 'standard', I've seen postings where the parts exceed 10GB!). Added to that, was many folks access to 'hyper-speed' lines, where a 50GB BD disc could be posted in a couple of hours max (50Gb/s).

The servers really couldn't handle it. Many (if not all) of the parts contained 'skips', where a single RAR part had 5000+ sub-parts, and a good percentage of those simply weren't stored by the server they were posted to (even Giganews).

Then, the number of PARS were dropped to truly insanely low levels. Like .001%, where it wasn't even enough to recover a single part. People thought all they had to do is 'throw' the data at the servers, never take a look to see if it got there, or propagated from that server to any others.

In short, the thinking was they everything worked perfectly, all the time, because a hand didn't come out of their screen and slap them upside the face and yell 'wake up!'.

The server staffs have upgraded their plants, and things seem to have calmed down a bit lately, even as more and more folks have access to 'super-speed' lines. The 'insanely low' levels of pars seems to have gone away quite a bit, or the folks/groups using that 'technique' have faded away (probably due to so many complaints at the time).

Things were 'creaking' a bit, hopefully we're entering a 'calm' period.

[mesaman]

Astraweb is using duplicate Message-ID's which is pissing off Giganews.

[Beck38]

Astraweb is using duplicate Message-ID's which is pissing off Giganews.

I seem to 'dimly' remember that. It's all a lack of software in their plant, I'd suspect. But s/w folks are, IMHO, generally allocated to the 10th level of the hot place.

My big bitch right now are the 'search' servers; it's not like they all (from newzbin to nzbmatrix to whoever) didn't KNOW long time ago that retention would be.... 'forever' at some point. Yet most/all seem to hit a wall back xxx hundreds of days, far short of the actual retention of the servers they are 'indexing'.

Actually, it's kinda like IPv4 addresses. Hey, I was 'almost there' back when all this was set in stone (federal gov't worker) when DARPA came up with the 'internet' (late 70's early 80's). I'm sure nobody thought we'd need addresses for Refrigerator's, for gods sake. But it was fairly foreseeable that if the system indeed would have 'taken off', that the amount was woefully short.

This was a hot topic as far back as the late 80's! I remember talking at length about it at meetings of a computer club in Tulsa, OK, with the SysAdmins of both Tulsa University and Oral Roberts. But I'm a telecom engineer, the hand-writing was 'on the wall' in big fat letters well before then.

Message ID's for Usenet? The RFC for this is so old, it creaks. Giganews I think has been bitching about an expansion of this for a LONG time. They saw, again, the writing on the wall. Look, back in the early-mid 80's, when the fastest fiber networks across north America were running 140Mb/s (yes MEGA bits!), 2000 voice channels, even then it was foreseeable that links faster than 40Gb (GIGA bits, 80 million voice channels) were within possibility, and with frequency (color) multiplexing, 10's to 100 times that, on a single fiber.

Done today. Across Oceans. Remember, AT&T/Bell Labs/Concordia or whatever they're calling themselves today, SAT on this technology from around 1960 through 1980, so when it got started in earnest (due to Supreme Court decisions like AT&T v. MCI and others), it was already 25 years behind. I think it's 'caught up' now....

So, the changeover from IPv4 to IPv6 is going to be.... interesting. I know that there are draft committee's working on updates to RFC 1036 (usenet), but how much real progress is up for grabs. Progress (?!) will run over them, flatten like a pancake. Road Kill.

Like the telcos. Talking around the globe was $$$'s per minute. Now, a couple bucks a month UNLIMITED from anywhere TO anywhere. Anyone feeding on that old paradigm is... Road Kill.