May Outage - whys and wherefores - summary
Update for website recovery work 24/5/2014
Again we thank you for your patience and support. It has been a very difficult time for us all - but we are getting there!
Please see the Summary of Progress just below and more detailed information after that for those who wish to read it
Summary of Progress
Where are we at?
We believe all sites we are able to restore are now visible on the web. If yours is not there please call or email.
They aren't all perfect yet but we have done our best to get almost everyone a web presence first and then work on refining function and appearance where it is needed.
A couple of sites running very old code are not active - can't be run on the newer server. It is important to keep your website up to date with regular updates when we propose them.
Why did it take so long?
it was not "just" a case of restoring from our backups .. much more complicated than that!
Once we abandoned our repeated efforts to recover sites on the original server, we had to push forward updates to ! all sites planned for the next 6-12 months.
The new more modern, more secure server we have been gradually upgrading some clients to required us to update every website to run on it - a process taking 1 to 10-20 hours a website
(depending on custom features for 200+ websites) that forced us to use our new software that was still undergoing development and testing for the future upgrade process.
Why is my website not showing?
Either: Your site was running program code too old to be able to be used on the new server,
Or: We needed your assistance to change the IP address to the new server and this has not been done.
Why has my site got some issues?
- We were forced to upgrade sites to software that had not been completely developed/tested as above to get sites going.
- We decided to get everyone some kind of web presence as quickly as we could and then refine/improve in a second/third cycle as needed. These days you are expected to have a website of some kind, at least!
- We did find another hosting service that could take some shops/catalogues in current form as an emergency way of getting more sites running quickly and we have a couple of configuration issues with that host.
We are working on it.
Where to from here?
During the next week we hope to get on top of the tuning/functional issues for websites, more serious problems will be done first.
And we hope to restart our regular work of developing/enhancing/upgrading websites as we normally do!
Please let us know of any outstanding issues with your site. We will process them as soon as we can.
Thanks for your support,
Margaret, Merv and Staff.
More Detailed Information - May 24, 2014
More Detailed Information
Review of Sites
Margaret went through the entire set of web sites and checked each home page was showing. In some cases 404 pages showed up.
Once upon a time we used to put little stories about what to do next. For these pages, however in more recent times we have been using a Reference module to direct people directly to the home page with no hint of a broken link.
This has been done where the 404 page was showing.
We have checked all the home pages and have noted obvious items such as menu styles and text issues on that page. We will be working steadily through that list.
If you have more than one web browser on your PC do take the time to check your site's appearance using both.
There appears to be more issues with sites viewed on IE.
Again, ch! eck your menus, backgrounds, your text fonts - size and visibility.
Are your stats visible? Many stats will start again on the new server.
Check your links. You know your site better than we do.
We have already received some requests for fixes from published sites and are dealing with them as promptly as we can. Thankyou for checking them thoroughly.
As said before keep the list brief as in bullet points - it makes it so much easier to work out the issues, but tell us where they occur in the site. Put your domain name- website issues in the Subject Line.
It will help us filter them out from other emails. Send them to email@example.com or piggyback them on this newsletter
We are still w! aiting for some who needed to change their DNS records to match the new server address and we contacted each of them on Wednesday to make sure they had received our emailed requests.
Without that your website cannot be made visible at all - not even to show a website outage page.
More about - Why has it taken so long?
A fair question about a very unfair situation!
We understand your angst, we share it and we are also heavily affected and feeling it too.
RS Net staff have worked 16-18 hour days since the 7th of May. It is now the 24th.
We first thought that it was a server failure and rebuilt it with specialist help. It was not just restoring your files - we first had to restore the server software with its millions of files, then the sites - more millions of files.
We were just, after 36 hours, congratulating ourselves on having achieved so much. We even sent out an email.
As this was relief was being shared, we incredulously watched the whole server being deleted again. Suspicions re hacking were aroused.
Measures were taken to shore up the server defences and the whole process was done again, and yet again a third time.
As soon as the situation became close to a solution, the hacker struck again.
After the third attack, the server was decommissioned and! all sites were moved - some to an external server at our cost, some were upgraded so they could be put elsewhere on our system.
All sites, however, needed a minimum of one hour to get ready to move and some custom sites took up to 20 hours and we are talking about moving more than 200 sites.
Our guys did their best to maintain your service. Their wives saw very little of them, and social events were limited to a break on Mothers' Day.
Like E Bay we were hacked. And if you are not aware and use E-bay , please login and change your account passwords ASAP!
The partial upgrade means that many of you have the new WYSIWYG. Here is a pdf (also found on the RS Net website- left block) to access for help.
There are several versions of it depending on your web presence - website, web brochure, catalogue or shop but all the symbols are the same.
We are making sure Search Engines know you are back.
All visible sites with paid search engine publishing have been republished today. Check any message from search engines as they may need confirmation. Those who don't use this service should publish themselves to Google.
Hosting dates moved ahead for one full month
All who have been affected have be! en given an extra month on their hosting at the next renewal, so if it is now due in January 2015, you have been put forward to February. Those still due this year will be billed a month later.
What have we learned from this?
- One of the most obvious things we can say is that RS Net has a very supportive team, and a very loyal and supportive set of clients! Thankyou All!
- A second is to be even more paranoid about server and site security.
This means all clients need to be more pro-active in updating their site software when we suggest it.
We believe the hacker exploited some older code on a website to get an entrance to the server and then buried themselves amongst the millions of files and countless folders there .. so we can't have older code on our sites ANY MORE!
- The other major thing is that no business can do without backups.
- Don't rely on things being backed up because you think it is being done!
- Take the time to check that they work, that you can restore a file and that file you most value is actually being backed up.
- Keep them off site, and backup in more than one way.
- Talk to us about backup solutions!
Where to from here?
- At RS Net, we ALL want to get back to "Normal", and move forward with our regular business helping your websites do better and better.
- In the next few months we are looking to moving sites to a new hosting system where we can still service them, but have greater uptime support from a third party. We will keep you informed.
We are researching options even now, but our main thrust is of course to get everyone going properly again.
Merv and his team have pulled out all stops to get this far in this time. The overtime bill is looking scary - one programmer has clocked over 130 hours of overtime including weekends - Merv even more, and he went down with a head cold but "soldiered on".
We are committed to doing our best by you and for you.
Again thank you for your supportive messages - they have really helped! We particularly enjoyed the one suggesting using viagra on the sites not yet up!
Margaret, Merv and Staff.
The following items reflect the progress made by RS Net after the deliberate, malicious and repeated hacking of their server.
May Outage - Progress report May 13 - 11.30pm
Thank you to all of you for your continued patience and messages of support.
Here is what we have achieved. Approximately 75% of sites are back and repaired where needed. Some Style and function fixes are still to be done.
Shops, catalogues and custom sites are being restored as quickly as possible, but these are taking a bit more time each to do.
If your website is up, please check it for style or functional issues and email us.
There are too many sites for us to check each in detail individually at this time - but with your help we can get more sites up as soon as we can.
If your site is back but not does not look fully restored?
Have you refreshed the browser or is it remembering the problems? Press <Ctrl> & <F5>
If your site still has errors on it, eg the background has changed, the menu text is not right, please let us know.
If you send us an email with Your domain name and the words 'website issues' in the Subject line, that will help.
Then bullet point your discoveries so we can deal with it.
Please be specific about your issues - where they are occurring, and what has happened.
Why have I got a Fedora Test page on my site?
This is an automatically server-produced page.
We are replacing these and generic site down pages steadily with Blue RS Net pages with your contact details on them
If you received an email about changing your DNS, have you followed it through?
This affects only a few sites and individual emails have been sent with the instructions of how to go about it!
Not everyone who got this information has acted on it yet. This is a case of where only you can help the situation.
It is really important as if we have your website ready to go today, it may still take another full day or longer to become visible while the DNS change is propagated through the Internet.
Your patience has been incredible in this very frustrating situation.
Merv is just over 4 weeks out from his hip replacement and is doing very well, but as you can imagine is tiring easily still.
He and others are working 18-20 hours daily and have been doing so since Tuesday 6th am
with just a bit of time off for part of Mothers' Day (even missing his physio sessions)
There is still a lot of work to do and we continue on as fast as we can.
We will keep you in touch.
Margaret and Merv Renton and Staff.
May Outage- Progress report May 11 - 7.30pm
Thankyou to all of you affected by our outage for your patience. We have received messages of support which we have greatly appreciated.
Where are we at?
We have continued to work 18-20 hours a day, except taking a bit more time out on Mother's Day.
We do need to celebrate our Mothers!
Over the weekend many sites have been moved to a different server and more security features have been developed.
We have about 60% of our websites working at present ... they are not all in perfect condition, but we are trying to get all sites up quickly first.
Those sites that are visible have been checked and problems noted and they will be attended to as soon as we can. You are welcome to email us with specific issues.
Please check your home pages, your menus and placement of text in both IE and Firefox if you have that on your PC.
My site is not showing
Owners of sites with their DNS at other ISPs will have been contacted individually by email and given an action list of things to be done. Please do these tasks urgently as until you do the sites cannot be seen and there may be a lag time involved before the sites are visible.
My site shows a White Page or Fedora Test Page
Your site transfer is still in progress.
More complex and custom sites are taking more time to transfer and reconfigure for the new somewhat different, more secure server environment.
We plan to add a "Down for Maintenance" page to these sites so there is at least something to show while we continue to work.
I can't edit?
Some of you may not be able to edit yet, and this is also being dealt with as promptly as possible. Probably an inconvenience to some of you but we are sure you realise that we need to get everyone visible first.
What about our data?
We reiterate that your data is safe. It is backed up fully monthly and incrementally every day, and it is kept for more than 6 months.
Some people have requested their data to be put on disk which we can do if need be but it is only a snapshot of the day the disk was made and won't reflect any changes you may make after that time.
However this will need to wait while other sites are made visible.
What about our emails?
During this process, your domain emails have continued to function as they were managed on a different server.
How can Clients help us?
We request that you send any questions through by email so that Merv and the technicians can keep working as uninterruptedly as possible, thus doing our best by all of you. We will call if circumstances require it.
Please put your domain name and website issues (eg RS Net-website issues) in the subject header and then just bullet point any issues in the body.
We can only apologise again for the interruption to your business and ours, and certainly share your frustration as all our businesses are affected by this unnecessary act of vandalism by some anonymous bad person.
Margaret and Merv
Websites lost again - Sad news...
After 24 hours of work which got us to the point where the first batch of websites was working on the original server, we saw them deleted before our eyes!
This means we have not been able to exclude the hacker successfully and we now have to reconsider our options and adopt a different approach to get websites going on a different server, which is a slower process.
We will work on doggedly until we have this problem resolved.
Again our sincere apologies for the trouble this has caused our clients.
We wish people did not get such perverse enjoyment out of destructive activities.
Merv and the RS Net Team
May Outage- further explanation
Our apologies for the events of the last several days.
We are painfully aware of the fact that many of our websites are down again!!
Having worked most of 2-days straight to get the affected web server system and its websites back, we lost it all again just as we were congratulating ourselves that we were almost 100% done.
It seems that a malicious hacker has found his way into one of our web servers by exploiting some weakness in one of over 250 websites. It may not even be our own code with the problem, as we host sites by other developers. The hacker waited till we were rolling again and exploited the weakness again to kill the server and websites. Simply, they are trying to kill our hosting business.
There are around 4,500,000 scripts/program files on the server so it is not possible to quickly detect and solve the weakness that was exploited. We checked all straight-forward possibilities immediately with no success.
Since then we have been working 18+ hours per day to recover sites in an alternative manner so we can reduce the chances of the hacker being successful and reduce the chances that all sites are lost if the hack re-occurs.
We know the problem is painful for your business but it is threatening the very existence of our business, so of course we will work continuously except for a bit of sleep to get things back on track.
We thank our clients for being understanding in these trying times.
The amount of files involved alone means it takes time to restore and reinstate a server, its systems, and its websites, and it needs to be done carefully and technically correct, so it will take time to achieve.
We expect to have the first of our affected sites rolling on another web server by later today, and we will keep going until we get them all back.
Thanks again for your patience and support.
It helps us keep going.
Merv Renton and the RS Net Team
PS: If you have already received this message, our apologies, we have been sending them manually.
RS Net Website Outage 6th/7th May 2014
Our apologies to those clients who were affected.
We believe all sites are working now. Please check yours to make sure all is OK.
The Outage affected some webshops and websites hosted by RS Net.
It was detected after midnight on the 6th and an initial assessment was made on site.
Unfortunately the damage was too great to be repaired immediately.
A vast amount of system and website data was lost.
With the help of a technician the server was tested- OK - and its System files were recovered from backup and so the base server software could be rebuilt.
Websites and shops were recovered next. This is a huge process involving millions of files.
There were some images that were not fully recovered on some sites in the first run and these sites will be further restored over the day today (7th).
Emails were not affected as email is managed by a different server.
We will be sourcing a replacement server as soon a possible to reduce the chances of this happening again.
As you know all computers and systems can have issues at times - there have been banks and phone carriers and major ISPs recently also who have experienced considerable downtime.
You will be pleased to know that RS Net always does a full back up of the web system on the first full weekend of each month and this has just passed.
RS Net does incremental backups throughout the month also, keeping data for over 6 months.
We were able to restore 99.9% of our systems and sites from the backups.
How well are you backed up?
As we have said repeatedly over the years a backup is only as good as what you get when you restore!
Please make sure that your backups are done, that they can be restored successfully and that they are backing up the correct files,
especially those most important files that if missing can stop your business in its tracks.
Don't rely on just one copy and don't keep it on the premises with the original.
Ask us about automated remote backup systems if this worries you. All businesses need it.
We thank you for your patience and understanding during this event.
Affected clients will be given a week's free hosting to make up for the 1.5 days you may have been offline.
Please call or email with any questions or information.
Margaret and Merv Renton
Margaret Renton firstname.lastname@example.org 0408 022 420 and 08 7324 3024
Merv Renton email@example.com 08 7324 3024
Garry Hinitt firstname.lastname@example.org 0419 641 322
Sean McLoughlin email@example.com 0488 505 840
Fax 08 8363 3398