Migrating to Big Data platforms on GCP cloud.

Our client is a conglomerate of multinational companies with significant presence in sectors such as banking and insurance, retail, hospitality, restaurants, real estate, education, administration, lottery, cinemas and services, and is considered to be one of the three most valuable brands in its region. As part of its transformation plan, it has initiated the process of migrating its systems to cloud platforms and using big data technologies.

At Bluetab Peru we have collaborated with them in defining and implementing the corporate cloud strategy and guidelines and the subsequent deployment in each of its companies. This involved defining work architectures and best practice documents in Continuous Integration (CI) and Continuous Deployment (CD), as well as implementing a use case based on DataFusion and AirFlow technologies.

It is about the definition and implementation of a global model for one of the main Spanish Financial Institutions, which guarantees the security of access to the data according to the regulations that apply in each area of ​​the different entities of the group, in a homogeneous way throughout the business group. This implies the identification of sensitive data and diagnosing its physical location, trying to limit its access only to those people considered necessary and as such with access privileges to them.

For the purpose of defining the appropriate access criteria, the following types of data are determined:

One of the most common problems for telecommunications operators is to align its various business areas as far as possible with its support areas, especially with the technology area.

To meet this need, an operator has started work on the construction of a continuous integration platform to not only facilitate the solution developer’s task, but also to be able to integrate the complete solution life cycle, from the definition of requirements by the business to the test use cases, so that they themselves validate the response the technological solution provides for their business needs. It also enables automation of much of the testing and speeds up response times. Based technologically on YouTrack, GitLab, Jenkins and Kubernetes.

As the main data solutions partner for one of the leading financial institutions in the Spanish market, we participated with their architecture area on an innovative Business Intelligence as a Service project aimed at offering users of different areas of the bank the possibility of exploiting data using any market exploitation tool, by passing the cost of use to the area and deploying the BI tool on Public Cloud infrastructure while keeping the data governed in the Private Cloud.

For this purpose, the front end was adapted to include the service as a new resource in Sandbox Manager, performing back-end jobs for authentication of the new service with the current systems, integration with current platform monitoring and subsequent integration with AWS for provision of the Billing & monitoring service.

Encompassed within the collaboration framework with the technology area of one of the largest Spanish financial institutions is the reviewing and improvement of its Big Data processes, considering both the redesigning of the developments and reviewing of the execution environment.

To this end, a review of the production environment was carried out by performing a realignment of the parametric configuration used for the execution of the various Spark Jobs, as well as making a proposal for a redistribution of resources based on the analysis performed for the various Jobs. A review of the critical path review was performed, identifying the most critical processes in the system with the goal of optimising overall times by means of configuration, and redesigning the Big Data processes using best practices to improve their efficiency.

The project is based on Scala/Spark technology under Informatica’s Big Data Management suite, knowledge of which is a priority in making the various specific parametric configurations and modifications.