Modernizing IT Operations for System Reliability for a Big four Bank – Altimetrik

Challenge

Altimetrik’s SRE Transformation team engaged with the client to help define, deploy scalable site reliability engineering framework, policies, and procedures to modernize their IT operations. In our discovery phase we mapped their current way of working and identified certain challenges impacting their system reliability, such as:

1. Scattered dashboards with noisy reactive alerts (~1000 alerts/day).
2. No end-to-end traceability.
3. Tech deliverables were not up to the mark and did not match client’s business priorities.
4. Multiple tickets SLA breaches with delayed RCAs.
5. Fragmented SRE tool adoption that resulted in high maintenance cost.

Solution

Our team outlined a 12-month transformational roadmap and assisted in adopting SREfoundational services, optimizing observability capabilities and developed an automated suite to provide self-healing capabilities.

1. Cockpit Controller

We developed a self-service automation platform to support critical flow remediation and introduced system throttling to manage event queues and transition of payment methods.

2. ISDMS Integration

Simplified monitoring, and constructed dashboards to capture average request services on active nodes setup of Prometheus and HA Proxy Integration to highlight non-utilized / underutilized nodes.

3. Mission Control

Consolidated monitoring and logging services with single interface providing comprehensive view into live traffic. Also optimized logging and monitoring by distributed tracing to reduce MTTD.

Results

We optimized observability capabilities and developed automated suites to provide self-healing capabilities and reduced their operational toil.

1. 1. SLO based alerting.
2. Automated service management.
3. Eliminated development team overhead in supporting production events.
4. Reduced operational and maintenance cost for SRE tools.
5. Predictive capacity planning and management.