Rob Boudrie

There will be additional movement within the next couple of days, as I will be turning on the "auto expire" that will remove people who have signed up for the waiting list but not paid the waitlist fee for at least 4 days. If you are one of those people, consider this a hint

100 slots to each match have been released to the waiting list.

There is a selfish reason to buy stuff from sponsors - there seems to be a strong correlation between companies that sponsor events and those that provide top notch customer support.

I have confirmed no slots have been released. I will ask HQ for a status update when I talk to Kim this week.

I believe the deadline for return of the earned slots that have been mailed out is May 23 - which means that USPSA still doesn't know exactly how many slots they will be able to offer on the waitlist. They are issued on a rolling basis as opening become available - slots issued but not returned; people on the waitlist declining a slot; registered competitors cancelling; etc. Contact USPSA HQ is you have a problem with your PIN.

Actually, I never tracked that detail - I just set the waitlist up for USPSA HQ and let them worry about processing it. No, it's instant, but you have to refresh the page. It could be a browser cache issue. Your record is written to the waitlist database before the system asks your for $$, so you can take your time typing in your credit card number. I checked on one "not on the classification page" report, and found the person was in the database, and on the classification page, correctly - so I assume it was stale browser data.

Incorrect. Your database index number/internal record locator is #684. The number started in the low 400's this year as I simply removed previous years entries, but did not delete and recreate the table, so the automatically created index (MySQL "integer auto_increment unique") continues to ascend over the previous high number. As of the time I write this, there are only 268 on both waiting lists combined. The only info you can derive from your record locator is that if your number is higher than someone else, who is waiting for the same match, that person is ahead of you. The only reason this number is even disclosed is so that international competitors without USPSA member numbers can use it to look up their waitlist status. If you go to www.uspsa.org and look up your classification you will see your position on the waiting list, which is going to be a number considerably lower than 684.

All looks well. 175 signups in the first 4 minutes and 5 seconds. I just kicked off the script to download a backup to my off-site development system. Remember, you must pay the $10 fee within one business day to lock in your position on the waitlist. We are allowing the "one business day" so that nobody will lose a slot due to problems with the automated credit card processing (please, please,try on-line - and only call Sedro if you have can't get it to work) rob

The waitlist should open itself up at exactly 10:00PM. I just synchronized the system clock with a network time server to correct for any drift. I'll be watching - if there are any problems, I'll resolve them as quickly as possible. And yes, I'll be making an off-site backup at around mignight to capture all of the early signups "just in case"

Palm support is a MUST for the official USPSA program. I could live without it on another approved program, but the burden would be upon the applicant to prove that the qualification and compatibility testing could be done without any load upon the USPSA office. It's not a matter of turf protection, but ANY scoring inaccuracy or process problem at a match, particularly a major one, is going to cause much wailing an gnashing of teeth for members and USPSA staff alike. Most importantly, I am NOT speaking for USPSA or the USPSA office; just stating what requirements I would recommend and insist upon if USPSA asked what I felt was necessary to "approve" a program. As to squadding - if you convinced USPSA HQ to create a match in the system like "ZZ TEST", it would be possible to sandbox it.

Funny thing.... the master plan was to rotate to a new server over the 2011/2012 winter. We don't want to get so many revs out of date on server software, and an update can take a long time, so we move to a new system every few years. My guess is the two failures in rapid succession were due to drive age. Anyhow, I've learned about a few deficiencies in our disaster recovery procedures (mainly missing FTP passwords and mail forwarders in the backups), and have made some improvements. In fact, the RAID array was so old the ISP told us "we don't even use those old ones any more; can we give you a newer model with bigger drives at the same price?". It should. If your DNS is slow to catch up, try using the Godaddy DNS servers 8.8.8.8 and 8.8.4.4. These seem to update pretty quickly. ---- Trust me - for a low budget operation done with consultants, volunteers and non-dedicated staff, this could have been a whole lot worse

No, it just is hidden I had to change the IP address to install a secure certificate (it's best for a SSL site to have its own IP). The IP for the site has change from 209.62.63.82 to 209.62.63.83. The DNS should propogate shortly or, if you are a hacker, you can change "/etc/hosts" on Linux or %systemroot%\system32\drivers\etc\hosts on Windows - but, my recommendation is don't fool around with that if you don't already know what I am talking about - as you can forget it and have problems when the IP changes again in the future. And yes, I know there is a "self signed" cert there now. I'll have a Godaddy cert up once I attend to the day job for a while .

I don't speak for USPSA, but I know what would be involved in approval: - Scoring several matches and comparing results, and all reports to make sure they are identical. - Interface to configure classifier / activity fees if they ever change - Classifier library matching USPSA's, with formal verification each one is identical - All web interfaces - scores up to the web; classification files; on-line squadding up and down 100% compatible with USPSA systems - Accomplish the above without using USPSA resources to assist in debugging, provide documentation other than perhaps the code that generates/reads transfer files; etc. - An agreement by all users that in the event of ANY incompatibility scores would not be accepted, and activity fees not refunded, if the sole reason they could not be entered into USPSA processes was a flaw in a non-USPSA program. - An agreement that USPSA staff will provide NO support for the third party program. It's not just about a working scoring program; but about one that fits smoothly into the USPSA process.

We are working on the SSL problem. As to redundancy - yes, it's possible but would require a bunch of coding to handle redundancy on the file part - not to mention the need to mirror email configurations, any new sites created on the server for USPSA use; etc. It's not like we don't know about these industry standard techniques. The one thing nobody is offering is an explanation as to how we will do this with minimal time and cost, including the custom apps and ongoing maintenance issues. Keep in mind that USPSA currently has *NOBODY* on the staff with this sort of sysadmin skill, which means hiring someone do to the work. Yes, we can do better - but I don't think we can get more per dollar than we do now. Put another way, it all depends on how much we are willing to spend to avoid downtime that has historically averaged less than one day per year. What does look promising is using a virtual system in a cloud environment, that way the entire system can be backed up on a daily basis as a "machine image" - and, in the event of a server failure, just restarted on another server with no need to do an OS and application reinstall.

OK, I'll bite. This would be trivial if our site were static, and pretty straightforward if the only dynamic changes were in the database. The server has many functions that upload files to the server. Some are user visible (upload classification results); some are via a content management admin panel. What procedure would you recommend to easily keep the files in sync, and also assure that a file is synced at the same time a database entry that references that file is updated (database replication is easy using a log shipping approach for continuous replication). It would need to be a bit fancier than a cron job with "rsync", since data could get out sync between file and database. It would also need to catch any change to a content file uploaded to the primary server. Now, explain how I do this all in an evening or two (but there are some cloud approaches that look promising) There are two primary metrics in the disaster recovery world: RTO - Recovery Time Objective. How long goes it take to get back up? We are at a couple of days for a full server loss. That is an inconvenience for USPSA, but would cost a brokerage house millions, hence the difference in resources. RPO - Recovery Point Objective. If you have a crash at time T(crash), and recover from a backup taken at Time(backup), how big is the difference between these times? This is about one day for a USPSA recovery from full server loss.

Status: - The data was restored from backup, and we are getting the server state as of May 4. Yes, a few squadding changes and results uploads could be lost. - I had all the user data and databases (the really important stuff), but missed some stuff that would speed up the configuration on restore - so I am spending a bit more time than I would like getting things back in operation. - The ONE thing I was not able to restore (missed in the backup config) was the main password for each web site (the one you use to FTP to your web site). Individual email passwords were restored. If you take care of a web site on the USPSA server, email me (rob at boudrie dot com) telling me what you would like the password reset to and I will do it. - I have restored many of the sites, but still have some to go - including some of the area sites and some USPSA sites. I'll post additional info as progress progresses.

It's risk/effort/reward. The budget is minimal. There is NOBODY with web server sysadmin skills on staff. I take care of the web stuff at home after a "day job". USPSA does some work on the site in-house, the staff helps out on some web design/maintenance, and we had a consulting company re-design the USPSA.ORG look and feel. Yes, USPSA HQ is aware of the need to bring expertise in-house. If I had been properly notified of a RAID 5 single drive failure, this would have been a non-event. Give me a budget and some staff and I can set up a system that will even survive uninterrupted if some folks representing a religion of peace take out the building hosting the server. Remember that day job I mentioned - it's all about reliable data storage . In some senses, this is an example of a setup for success: - We had a single preventive mechanism (RAID 5) in place to protect live data. The hardware did what is should; human processes failed. - We performed regular backups so we could survive any hardware failure. That worked!!! As a financial exercise, let's suppose the down time for this is 4 days. It would represent less than one day downtime per year based on our experience to date averaged out over the years. How much $$ should USPSA spend to protect against this? We are already at three nines (.999) uptime, or the server is down less than one hour out of a thousand of operation. Each additional "nine" of reliability costs. Dave Thomas and I discussed this years ago, with simple conclusions: 1. It is important not to lose a lot of data if we have a disaster 2. We are not going to spend a lot of $$ to totally eliminate the risk of a couple days downtime every few years when something happens. Naturally, we will be discussing the matter again as the degree to which USPSA members rely on web based services has increased significantly over the past few years.

WAITLIST ANNOUNCEMENT The opening of the nationals waitlist will be delayed exactly one week due to the server issues. There is a decent chance the server will be in operation by the originally scheduled time, however, Kim and I are concerned that someone may not get notice due to the outage and this additional week will give everyone time to plan for the waitlist opening. The notice on www.uspsa.org will be updated shortly after the site is back in operation.

Maybe not in Uruguay. The server is in Texas (Dallas or Houston; not sure which). The server farm provides support, but USPSA also subscribes to www.totalserversolutions.com, also known as "The Uruguayians". The folks at TSS are absolutely exceptional and well worth what USPSA pays to keep them "on call" (well under $100/month). "Shortly thereafter" was about 3 weeks - which really, really annoys me as timely notice via a call to the emergency contact # for the account would have avoided this problem. At least I am getting to test our restore procedures.

<update> Yippie Kay Yay - the server farm staff found and fixed the problem, data is moving back. There is work to be done after the data is back, but this is the most critical (and scariest) part of the process. I expect we will lose less than one day's worth of updates to the web site. </update> This is frustrating. The backups were configured correctly, but the restore program installed on the server errors out. I have escalated to the vendor, but this could take time as I don't know if the backup vendor offers weekend tech support at the level I need. As to the suggestions - yes, I know we could rewrite the CMS system to put the uploaded files in a holding area; configure multi-system synchronizations; cluster the database; etc. I know how do to all these things, but it takes one of two things - time or money. I don't have the time to dedicate all of my USPSA efforts to sysdmin work, and USPSA does not have tech staff on board to implement all the suggestions. Of course, if you're offering to write the code, make sure all current processes and features workl configure the systems, and monitor it to make sure it continues to operate, HQ would like to talk to you In other words, we do the best with the resources we have. It's kind of ironic since I spend my days making sure that a lot of very familiar corporate names have access to quick, reliable disk based backups.

Well said. I don't mind when businesses make decisions like this, but I find it a bit insulting to be BS'ed with a reason other than "market conditions do not require it". It's sort of like when I write a check a the store - I don't mind showing my driver's license, but don't insult by telling me you're requiring it for my protection. The good news is that even though parts are not free for a 1050, you can get them shipped very quickly and they always seem to be in stock.

Putting up a temp site is problematic since restore work is active on the server. Mirrored sites are not as simple as with a purely static site, since the web site "self modifies". A mirrored site for a static web site is easy, and also not that hard for a system where all dynamic content is database driven. It's a bit trickier with CMS and "self modification". Clustered systems either rely on duplication of data on multiple nodes, or use SAN or NAS storage so nodes can access the same datastore when they switch over. Classifier uploads go to files downloaded by USPSA HQ; content including photos is managed by a custom content management system; and certain other files are regularly updaed. It's easy to replicate to cluster a database; a bit trickier with the file system. I could use cloud based storage so that in the event of a hardware failure, I just get another server and configure the FS mount point to the cloud - but even that has a failure possibility. There was a big new item recently where Amazon had a cloud failure, with some loss of customer data. What I am interested from the server farm is if they even consider the failure to make a phone call in a situation like this a process failure, and if they are going to do something to improve their process. I really don't care about getting a bit of service fee back - I do care about having decent assurances that they can learn from this mistake ... which can only happen if they consider it a mistake. If they tell me "email is sufficient notice, we did exactly what we should, and exactly what we will do if it happens again", I may have some shopping to do.

I am not familiar with this particular RAID controller of if it was hot-swap - but the server farm asked for a bit of downtime. Too bad they didn't call when I didn't get the email. It was an Adaptec controller (we're getting a new raid controller as part of the repair since they want us on a newer model). Not all of the lower end raid controllers support things like hot spare (some just have 3 drive slots). RAID6 would have been just the trick. I live in the RAID world so I know more than a little bit about this sort of thing. Backups are performed daily, however, my fingers remain crossed until I do the restore and see for myself. The hardware is back up and the server farm finishing some config issues. Once that is done, a restore will be kicked off that should take a few hours, after which there will be some more config work (it's backup of the data, not a bare metal recovery backup). As to the waitlist: If I am not able to have the server up by Sunday, the waitlist opening will be delayed exactly one week. (This has been cleared with USPSA HQ, so I will be able to post info immediately if this delay is implemented). I'm hoping that won't be necessary, however, this is the best procedure to make sure everyone gets a good shot a the waitlist. I have always grabbed a copy of the database from the server to my PC about 2 hours after the waitlist opens, and then again the next day to give that particular data a bit extra protection. And yes, I most certainly am going to have a talk with the server farm folks. There is no excuse for me not getting a phone call (There are multiple emergency numbers on file, including my home # and Dave Thomas's cell # on file with the company). The only thing I haven't decided is if the overnight letter goes to the CEO or the VP of customer service and technology. I'm waiting until the issue is resolved and the server is back up before taking that step. Not with RAID6 (that allows you to lose two drives), however, RAID6 setups tend to do more in software and less in the drive itself, so there are often performance implications.

The USPSA web server went down at about 4AM UTC-04 today. I am really annoyed, since USPSA pays extra for RAID 5 storage. A drive failed on 4/20, and the server farm staff sent an email I never received asking for permission to replace the failed drive and reboot - which would have been a "non-event" and about 30 minutes (max) downtime. Unfortunately, although the server farm has my number as an emergency contact on file, as well as Dave Thomas' cell phone, neither of us were called. A second drive failed today resulting in the problem. We have daily backups, and are seeing if we can get the raid array to come up for a bit to extract the current data - if not, it's restore from backup time. USPSA subscribes to a service from a premium quality service that specializes in managing web servers (mytss.com), and they are already working this issue. Those folks are in Uruguay, are hard core nerds, and mean that the issue is being actively worked by experts so USPSA does not have to wait until I get home from the day job for action. This also means mail to @uspsa.org addresses won't go anywhere until this is solved.

I have not been able to find ANY law that makes a casino an exclusion zone for possession of firearms, or even for carry with a permit from a state recognized by NV (http://www.nvrepository.state.nv.us/ccw_changes.shtml). The key is to get lost in a crowd of one (a bit difficult with a long gun case, but I doubt you will have any problem if you don't linger by the tables with your luggage). The only hotel with a posted no guns sign have seen is the Stratosphere, and even the statute they cite is "iffy" since it's a trespassing law, not a "no guns" or "binding signage" law.