Timetastic downtime - 20th Feb

On the 20th February, 2018, at about 20:10, Timetastic had a problem, and was out of action until just after 22:00. We're very sorry about this, it's the last thing we want for our customers.

This blog post goes into detail about what happened, and what we are going to do to help reduce the liklihood of it happening again.

What happened?

We host Timetastic in Microsoft's Azure cloud. At 20:12, they started experiencing problems in their "UK South" region, which is what Timetastic uses. This took our database out of action. The following is taken from the Azure status page:

Summary of impact: Between 20:12 and 21:50 UTC on 20 Feb 2018, a subset of customers in UK South may have experienced difficulties connecting to resources hosted in this region. Impacted services included Storage, Virtual Machines, Azure Search and Backup. Some Virtual Machines may have experienced unexpected reboots.

Preliminary root cause: Engineers continue to investigate a potential power event that occurred in the region, impacting a single storage scale unit.

Mitigation: The impacted storage scale unit automatically recovered.

Next steps: Engineers will continue to investigate to establish the detailed root cause, and the full root cause analysis report will be posted on this Status History page and in the Azure Service Health blade of customers' management portals once completed.

After it was clear that the problem wasn't going to resolve very swiftly, we switched the Timetastic database to use our secondary failover. As soon as this operation was complete, we were able to bring everything back online again.

Mitigation Strategy / Lessons learned

We were able to sucessfully switch (failover) to our database backup hosted in another region. However, it took a while to a) Identify that the failover was going to be required, and b) for the failover to complete.

We're working with Microsoft to determine a more efficient solution to this - so that in the event that Azure has a similar issue, we can failover in a much smaller time window - ideally with no downtime at all!

Once again, we're sorry about this if anyone was affected.

Rate limiting the Timetastic API

Usage of the API has increased recently, so to keep things running nice and smooth for everyone we need to introduce a rate limit

What's the limit?

Quite simple - 5 per second per customer API key. Our logs show that most consumers should be fine with this, but if you've written code that fires multiple API calls at the same time, then you may want to change that to work sequentially, so the requests go out one at a time.

What happens if I hit the limit?

You'll get a 429 status code result and the request will fail. You'll also get some information in the body of the response telling you what the current rate limits are.

Might this change in the future?

It's possible that this limit may be reviewed and changed in the future. You can find the latest rate-limit information by logging into Timetastic and heading to the API page. You can also review the response headers we send back when you call the API - in particular, we pass back "X-Rate-Limit-Limit" and "X-Rate-Limit-Remaining" which detail the current rate limit period (5s for 5 seconds), and how many calls you have remaining for that period.

When does it come into force?

We'll be activating the API limit on the 20th February 2018.

Improved Work Schedule and 1/4 Hour Increments

Two Big Improvements for the Work Schedule on Timetastic Today

1) More Flexible Work Schedule

The work schedule is now split into 2 shifts; Morning and Afternoon. The gap between being the lunch break, obvious I guess. Seems basic but it's a vast improvement from the previous which had a fixed lunch time of 12pm.

It means you can now specify a more accurate weekly working schedule, and time off will be shown on the wall chart in the correct AM / PM slots.

2) 1/4 Hour Increments

You'll also notice on the work schedule the introduction of quarter past and quarter to the hour. So again, more accuracy in the work schedule.

You can now also book time off in those 1/4 hour increments, to align with your working hours.

Hopefully both improvements will mean your information on Timetastic will now be more closely aligned to your employment contracts.

Privacy Mode

Introducing Privacy Mode, an optional setting that makes your leave information more private.

Privacy is one area that divides opinion in our customer base. We regularly receive requests to hide departments, leave type info, even requests that employees can't see anything but their own calendar. (Although in many respects this latter case negates most of the planning benefits that come from the improved information available via Timetastic - and sharing it).

On the flip side we get just as many people checking that everyone can see everything - they want exactly the opposite of privacy, full open book. Everyone has their own policies and attitude towards privacy in the absence area.

Until today everything was open book, much like the old style wall planner, but we tend to agree that some things could, and maybe should be private. Maybe sickness, doctors appointments, early stage maternity leave for scans.

Maybe it's ok to know the person working next to you is off sick. But someone in a different department, is it right that you can see they are sick, maybe just knowing that they are off work is sufficient?

That's why we're introducing Privacy Mode, so you can choose to make your leave information more private.

With Privacy Mode On, the 'Leave Type' is only viewable by:

  • Your Approver
  • Department Boss
  • Your Timetastic Admin users

Users can see their co-workers are off, but not the reason why they are off. The coloured squares on the Wall Chart and Calendar are replaced by grey. Department Bosses can see the reasons his/her team are off, but not for another department.

Admins - of course, can see everything.

Privacy is a global setting, and can be found in SETTINGS > GENERAL, toggle it on or off as you see fit.

Here's some screen shots:

Without Privacy:

Timetastic Privacy Mode Off

And with Privacy On:

Timetastic Privacy Mode On

We're Hiring

.net Developer

Things are going well in the world of managing absence. More companies than ever signing up, more companies taking advantage of a really simple concept and feeling the benefits.

But we're not one to be complacent, we've got a backlog of tweaks, features, design improvements, usability and hosting enhancements. Tech doesn't stand still and we're not so arrogant as to think our app is perfection, not quite anyway ;)

'Onwards and upwards' as they say!

We're looking for a .net web developer with around 4 years+ experience building web apps in the real world. Ideally you'll have experience with:

  • ASP.NET MVC and Web API
  • C# (the more recent versions a big advantage)
  • Hosting web apps in azure
  • Mobile app development (Cordova, native, all good)
  • SQL Server
  • Javascript (the language, not the latest noun-js framework)
  • ReactJS, and other UI frameworks
  • GulpJS, WebPack, or similar
  • A passion for the web, and for building modern web apps

We're a small team so you'll also be on the front line. Customer service is provided through a help center with ticketing (Zendesk). We all work on this, it's a great way to get close to customers and understand how they use Timetastic and the problems they face. Being on the front line, understanding their pain, answering their questions, helps us all build a better product.

We're all homeworkers at Timetastic, and you will be too. Ideally though, as we like to meet up in Manchester on a regular basis, a reasonable proximity to Manchester is important.

If this sounds like your thing then send us an message and introduce yourself.