Astraweb/US retention losses (skips/fades)

[Beck38]

Every couple of weeks, I do a 'look back' at how Astraweb is doing keeping it's retention up (kind of an early warning to any major gaps/skips). Those who've been around for awhile know that there have been some pretty major faults in the past (that took a while but did get repaired, mostly).

In doing my usual 'maintenance check' routine today, I noticed that 'stuff' that was 100% 2 or a bit more weeks ago, were getting a bit 'hazy', where the missing segments were starting to craw up a bit, as well as the complete files (NO missing segments) appeared to be going down. So, time to do a 'study' of things a bit.

Found that going back a month, and the missing rate was climbing through 10%. Go back a couple more weeks, and it jumped to 20-30%, and continued that trend until at 6-7 weeks it was at 60% complete. Wrong direction for sure.

Now, just a few days ago I had d/l'ed a couple of large chunks of data (read: BD50's so 100GB) that were 1-2.5 years old, and the number of errors were almost zero. So I 'assumed' the problem with the database had to be something 'closer' to the current date.

So, lots of work later, and yep, there appears to be a rather large 'gap' from around 50 days ago back some 200 days; newer, no problem. Older, no problem, Just 'in between, approx back to around May 1, 2013, or thereabouts. The percentage seems to be an almost straight line at 55-60% (means 35-40% loss)

Don't quote me exactly, your mileage may vary, but not by much. Fills? Well, I check Giganews, and it was virtually perfect on the very things Astra was 'skip city'. Blocknews, almost the same.

So, Astra is having one of their 'problems' for sure. Anything posted looks good for aroub 3-4 weeks, then it rapidly drops to that 60% complete level. Holds at that for around 200 days back. Further back, 100% (or almost) again.

I'll send them a nice note, but folks should know what's going on. It's very recent, happened some time in the last couple weeks plus. Be advised.

[Hypatia]

Sending them a note is a must.. but I wish they would take their business more serious..I have the impression that their farms are run by a single person each lol
US server used to be more reliable than EU one. I guess now it's not the case anymore and they both get screwed up. But in my experience, what I downloaded, EU server lacked here and there random articles, just a little somrthing was missing. whereas US server usually had them all intact..

PS I'm checking atm what I uploaded 50+ days ago and it is totally broken.Out of 163 parts close to 50 are missing from all files. Ridiculous and disappointing.And as u mentioned the gap is quite large after 50days - I kept checking files and so many missing articles.

[Beck38]

Well, I see that you're doing just about what I'm doing, in 'checking back' and keeping the nzb's to allow yourself to do it. I also keep all the nzb's of 'stuff' I've d/l'ed, the amount of mass storage I have means that anything up or down over the last x years is still there, nzb wise. Again, what I found was a drop-off at around 30 days back, then a major drop at around 50, then pretty constant (the 55-65% completion) to around 1 May 2013 or thereabouts, where it starts to push it's head above 98-99% completion. If you go forward even 5-6 days (6 May or thereabouts) you'll start to see the rate start down the 'slope' so that another 5 days or so (10 May) it starts to get really bad again, and by the 25th it's firmly back into 40% loss territory.

But yes, it's pretty bad. Got the usual from their so-called support, they want someone to do their work for them, they're concentrated on a single bad file-group or nzb, and not the 'big picture'. I don't think they have a system in place to really tell them what's going on. Back when I was still working for a living, designing and troubleshooting massive transmission systems (including the really big trans-oceanic ones) I'd walk into stations where they had a fair logging system for keeping their heads on the system, but how they were doing the error logging had serious gaps in how they were doing it, allowing major errors to be happening but would only show up in those logs if the entire system fell on ti's face. I showed them why, and they were usually dumbfounded (they WERE professionals!) and we changed their 'system'.

I'll figure that it'll take some serious beating on these folks to get them to understand, or at least look, at where their problem is. Again, they can't see the forest for the trees, apparently.

Sent them an 'update' with examples (not my postings but others). It's a 'forest/trees' problem. Get them looking at that forest, it's on fire.

BTW, thank the maker for NZBcc.

[Beck38]

I'll have to assume Astra/US is looking at this, since my ticket is still open and I did give them an 'example', although as I pointed out it appears to be everything between ~60-180+ days back with around 60% completion (40% loss).

I've been verifying on other plants (Giganews particularly, and NZBcc is for whatever reason slow as molasses on them) and things are all at 99+% there.

If I don't get a direct answer at the end of today (Pacific time) I'll shoot them another message asking what's up, but the ticket is still open and I'm hoping that they are looking into it.

The kind of major problem with their 'twin' (US/EU) sites is (and I pointed this out to them in my message) is that they have 'synced' them up so tightly over the past couple of years, that when problems occur it's probably really hard to tell if the US failed, then propagated that failure to the EU, or the other way around. Now, both plants (from my NZBcc testing) are exhibiting the same level of failure, hopefully there are logs within the system that will allow them to track down the original error. Maybe.

We shall see.

[Beck38]

Well, they closed my ticket, complaining that the nzb I sent was 'unusable', even though both major serach sites (Binsearch and nzbindex) had the particular posting fully indexed.

So I resent it, gave them more info, and a 'piece of my mind' on how to run things. But I suspect you're right on the number of personnel, and I wonder (as I stated in my note back) about their system for 'keeping tabs' on the running of the operation.

I seem to see some minor improvement in the percentage of skips, but it is 'minor' at best. I'll have to wait a few more days to see if there is actual movement; I commented that it looked like something 'automated' was at work (and that would be in any logs kept) but I haven't gotten a reply on that.

In previous 'melt-downs', it did take several weeks to repair the losses, and how they accomplished that no one really said. But I'll of course keep an eye on it, but I'm sure that all the folks over the years that just about everyone has said to have a secondary 'fills' account, that those accounts are being rapidly exhausted trying to fill the Astra Gaps.

Again, I'm wondering if so many folks have, at this point, either given up on Astra or on usenet in general.

[Hypatia]

eventually their revenues will diminish if not already, I mean with the whole dmca rampage, gaps in retention( and I mean all of them, especially giganews\supernews and aw) , leading to decrease in clients, Im wondering how they are going to support their operation at all while keeping pushing retention up. I think we already see consequences of this: giganews, once top tier provider with great completion rate is suffering from huge gaps in retention and claiming it to be due to dmca although its not dmca pattern at all( hey ,its cheaper than t buying and installing new drives so that articles wont get deleted to free space lol) and such like. Maybe some of them will switch their priorities from usenet to other services.

[Beck38]

With current posting levels, they need to be adding several terabytes of storage PER DAY; one would assume they are buying their drive storage 'in bulk' so even if getting top-tier drives the cost is probably around $500-1000/day.

At their subscription rates, that figures out to around 100 subs. Add on the cost of the human(s) to run things, then the really big bills (electricity and internet connections) and the cost of maintaining 'adds' is really a minor expense.

They are more than a bit lackadaisical about their maintenance, however, I think everyone sees that. They are probably hiring the lowest level of humans they can. Any automated system to keep and ey on things 24/7 is a good question; a lot of development in usenet 'technology' for plants was done around 10+ years ago, I haven't heard about much in the way of improving things since (and if those carbon-based lifeforms have even minimal, to me, intelligence, then it probably doesn't need to be).

Anyway, my ticket is still open, no real reply from the last 'update' I put into it. I'll continue re-checking later today, hopefully things haven't gotten worse.

[Beck38]

Believe it or not, repairs are being affected to Astra/US, and appears to be almost (note I said 'almost') complete back to around 1 May or so.

Actually, as I write this, it's about a month shy of that mark (so, 1 June or so looks good) so it may be done in another 12+ hours or so. I noted in a couple messages back that I thought I saw some minor improvement, it's now super-major in that things that were suffering from a good 40% loss just a few days ago are now reporting in the 98-99% completion range.

There is still a definite 'drop off' between in the May-June time-frame (sometime like 120-150 days back from today) but it's a huge improvement from the last couple days.

So, it looks like something definitely is being done (and my ticket is still open, but no actual 'response' as of yet), here's hoping that it continues and the last of the mess is cleaned up, like I said, in the next day or so.

[Beck38]

It appears from the way the US plant is responding this morning, that the maintenance they were doing may have either started ramping down to be ended or ended; I'll have to run some NZBcc tests to see if everything in the original 60-200 days back has fully 'recovered' the 'gap' I originally found.

I did get, believe it or not, a big 'thank you' from them this morning (my time, same time zone as that of the San Jose plant), but like I said, I'll be re-running some testing later on today to confirm the repairs.

Well, edit that top comment a bit, as soon as it got past around 7:30-8 they ramped up the rebuilding (must have finished the 'chore' they sent it on last evening) and they started up again on more 'work'. In short, it's still operational, just being a bit slow responding again. Well, if it actually fixed stuff....

And oh, took a bit of a look at Astra/EU, and it appears to be looking better as well, FYI.

[Beck38]

As of some 6 or so hours ago, everything I had originally reported (the 50-200 days 'gap') had been, to my testing, complete repaired, with everything I could 'test' coming in at 99+% completion.

It appeared as though they went further in their repairs (if one was vigilant, or perhaps in my case hyper-vigilant) one would tell the plant was still grinding away at some cpu-sucking task until about 2am this morning (San Jose time). They may still have some 'cleaning up' to do, but it appears to me they have probably already gone the 'extra mile' in fixing things.

That this could have, if I hadn't tripped across this rather large 'gap' and they hadn't responded (eventually) to my report, is something that could have grown into a really bad, year-defining, failure. As it was, there's a good possibility (especially as it didn't seem to affect things in the 0-50 day range) that most users never ran into much problems at all. I know I didn't when I d/l'ed a whole bunch of stuff in the 365-1500 day 'range' earlier in the week. It was only when I was doing 'final check' on some uploads before zapping them off my array storage that I noticed the completion percentage (thanks to NZBcc) was getting a bit low (it was actually around 95%, on something that's usually no worse than 98% some 40 days after posting).

It was enough to 'scout around' a bit on some older things, and the 'gap' was quickly identified. It's all been repaired as of at least 24 hours ago, and again, one could tell that the plant was under stress up until about an hour ago; and again, they may do some more work on it, but it appears to be all done.

Hopefully, this will also show them they need some kind of 'early warning' system or routine to give them a 'heads up' as to any gaps appearing in the future, as it has shown that whatever automated (or human) system they have in place now didn't give them that, and only a persistent user (yours truly) got them on the trail. Hopefully, they'll do some heavy thinking about it in the days ahead.

[Hypatia]

Beck, you did great.

Your knowledge on the matter helps you give them thorough hints and ur participation in making usenet a better place is highly appreciated. =)

Let's just hope they don't screw things up again soon. This whole situation is kinda sad, because when one screws up, everyone in the world seems to know, but when one fixes things and do good, everyone has moved on already and reputation is damaged anyways
=> less clients, less money=>less improvements, stability and more problems=>...

I think they should employ you as a consultant ;P

[Beck38]

If I still lived, quite literally, right down the road (okay, some 15 miles or so, 'just down the road by California standards!) I'd think about it. But I'm retired now, and am several states away. But hopefully, they may have learned something by this, and will make changes accordingly.

It appears today that they may be doing some 'clean up' as posting to the system has pretty much ground to a halt. But 'leeching' appears to be going at a 'normal' rate.

I think, despite all the problems, even the latest (the last 6+ months) of 'sky is falling' pronouncements, that the number (and quality) of totally non-scrambled, non-encrypted, totally 'in the open' postings continues apace, and haven't hardly been touched. Those that are what I consider 'open encrypted' (no silly website-based system for retrieval) also continues.

I, literally, was there (SF-area) when torrents were first 'launched'. I had (this was 1999!) a T1 connection at my home (several companies had rolled out service at pretty good prices) and, compared to Usenet, was pretty poor. Now, the system (torrents) has improved, and the availability of really fast internet/broadband service has spread pretty far and wide (actually, so far and wide that at least in the state I live in, farms and orchards here now have BETTER access to fiber (100Mb to 1Gb) than those of use in the 'big cities'). But this time next year, that may well be 'solved'. We'll see.

But as I was saying, even with all the silkiness, I vote for Usenet.