Company where this pilot fish works is having a bit of a problem with non-deterministic load times in the data warehouse. If the load runs normally, even with heavy volume, everything is done by about 9 a.m., which means management can see their reports just around the time they’re having their morning coffee.
But the load doesn’t always run normally. In fact, a system problem during the night can set things back by as much as six hours, which management finds unacceptable.
So various measures are taken to get the load to run in less time: Users are kicked off the system, priorities are rejiggered and so on.
Nothing works. The load processes just don’t seem to take advantage of the resources made available to them.
Then, a chance discussion with a member of the storage team lifts the veil. Turns out that the backup, which runs directly on the network storage system, kicks off automatically when drives are available after 9 a.m. There’s no direct visibility into this with the flavor of Unix on the warehouse server.
Now the solution is simple: Disable the automatic backup within the storage system’s scheduler and trigger it with a job that’s added to the end of the production stream. Once the load completes — and only then — the backup is triggered.
Result: Production delays — likely caused by users being on the system — levy about a 10% penalty rather than the 100%-plus penalty that had been hitting the process before.
