The ENTERPRISE Marathon: A Corporate Portal for Ukrtelecom

A successful project for a large ENTERPRISE client is not just a nice case study, a gold medal and glory at the finish line. It is a marathon distance of increased difficulty, with countless surprises along the way. Few teams make it to the finish. We did — and we are ready to tell you how it went.

What an ENTERPRISE marathon really is

Before we move on to the project itself, it is worth calling things by their names. Here is what a team faces on a distance like this.

• Complex business processes with non-standard logic — in the most unexpected places and in a truly indecent quantity.
• Non-standard, one might even say unexpected, behaviour of well-known software that had never behaved that way in typical projects.
• Rapid growth of additional requirements after the technical specification has already been approved.
• Integrations with the most exotic information and accounting systems.
• A trial by the strictest client-side control over information security conditions.
• A long list of formalities for approving and documenting changes.

And that is still far from a complete list.

About the client

Public Joint-Stock Company Ukrtelecom is one of the largest companies in Ukraine, providing the full range of telecommunications services in every region of the country. The company holds especially strong positions on the market of Internet access and fixed-line telephony services: Ukrtelecom is the market leader in high-speed fixed Internet access and holds leading positions on the fixed-line telephony market.

PJSC Ukrtelecom has built Ukraine's most powerful national backbone data transmission network, based on modern DWDM technology. It is this network that makes it possible to deliver modern telecommunications services to consumers in almost every populated locality in Ukraine.

Today PJSC Ukrtelecom comprises 33 branches, including 27 regional ones. The company owns a primary network, backbone and zone communication lines, and provides all types of basic and state-of-the-art telecommunications services — international, long-distance and local telephony, wired broadcasting, radio communications, radio broadcasting and television, documentary telecommunications, video conferencing, satellite communications, leasing of digital channels, ATM/Frame Relay, ISDN, and Internet access.

The project brief

In 2015 the ANIART team started work on implementing an internal corporate portal for PJSC Ukrtelecom — built on a corporate platform for internal portals, in the edition designed for holding structures.

The client formulated its top-level requirements across six areas.

Operational requirements

• Supporting simultaneous work by employees who have access to corporate information systems — up to 4000 people.
• Access to internal-use information about the company's employees and the ability to approve HR documents electronically: hiring, transfers, dismissals, remuneration, benefits, motivation, training, development, onboarding, sick leave, business trips and so on.
• Information about the company's projects, functional areas and departments.
• Designing the portal in line with the corporate identity of PJSC Ukrtelecom.
• Taking into account the user's territorial and structural affiliation.
• Multifunctional portal pages by purpose and access level: pages must simultaneously act as general-purpose pages, department pages, service pages and personal pages.
• Support for mobile devices.
• Link inheritance for portal resources: when resources are moved, links to documents must remain functional.

Stage one: the minimum sufficient functionality for launch

To avoid repeating the pattern of the previous portal, it was vital to involve employees in the new information space as much as possible. Communication has to be effective, and in our experience a clear, recognisable presentation of information about services, departments and employees provides a good foundation for that.

For the clearance and permission system to work, information about employees had to be supplemented with data from the security system. The task of regularly synchronising the company's HR system, plus merging data on roles and clearances, became decisively important for us.

The way data is stored in the HR system differs enormously from the "nice picture" of the structure of services and departments that a user sees in the company's organisational chart. The task was significantly complicated by the need to convert the company's organisational structure multiple times in order to produce that "nice picture".

For effective HR management we solved the task of bringing employee attendance information onto the portal: planned and unplanned absences, holidays, sick leave. This data had to be taken from the Parus accounting system. And since Parus was not used in small regional units, absence data for those employees had to be entered directly on the portal. This gave rise to the task of two-way synchronisation.

Moving large volumes of data during exchange and synchronisation required implementing a journal-based exchange method: only changed and new records take part in the exchange, and records older than two months are moved to an archive. We also had to change the system's standard interface — when the "Absence schedule" is displayed, only the records of the selected department are shown, without child departments; "cross-department mode" is disabled.

Stage two: system architecture and fault tolerance

The main difficulty in implementing an ENTERPRISE system designed for a large number of users (over 24,000) was ensuring an appropriate level of performance and fault tolerance. Since the portal was planned as the primary means of communication and interaction for a company with a large number of employees, a server solution with a high level of guaranteed fault tolerance was required.

The architecture of the cluster solution

To solve the problem of speed and fault tolerance, a server cluster was built. Several servers had the "web server" status with the nginx, apache and memcached services — they were united by a shared store of user sessions implemented on redis. Several servers had the "database server" status with the mysql service. The database servers ran in master-master replication mode, and replication was supported by a percona + galera stack together with the HAProxy proxying load balancer.

Portal users were dynamically distributed among the web servers by a hardware load balancer. Several servers were assigned the "NAS" role — network attached storage. A purpose-built monitoring system tracked the state of the cluster.

1. Request. The hardware load balancer dynamically routes the user to one of the web servers.
2. Failure. If, due to unforeseen circumstances, some server does not respond, the request returns to the load balancer.
3. Rerouting. The load balancer updates the availability matrix and passes the request on to proxy/HAProxy.
4. Recovery. HAProxy finds a working server instance and routes the request to it.

For the NAS we applied a solution based on a cluster of NFS servers, arranged as follows. Two NFS clusters work in a master-slave scheme. Data is written to the master node, and the slave synchronises at intervals. The master's availability is checked with the NFS HeartBeat service, and data synchronisation is handled by lsyncd. If an unforeseen situation occurs and the master becomes unavailable, the slave is automatically switched to master mode.

Headroom for growth

The architecture was designed with potential for further growth: Ukrtelecom is an actively developing company with a noticeable trend towards a year-on-year increase in the number of employees — and therefore of portal users. We understood that as the headcount grows, the number of HR events increases linearly, while the number of operational events increases exponentially.

The architecture we implemented allowed horizontal scaling by adding more servers. There were no limits on the number of physical servers.

While working through the system architecture and integration with external services, we had to solve dozens of tasks related to improving performance and finding bottlenecks at the "seams between systems". Profiling systems were used extensively: scripts with unusually long execution times were tracked, and profiling was enabled for them. The overall system log was analysed statistically for the frequency of "slow scripts" — to rule out situations where a rare slow script is optimised instead of a faster one that runs frequently.

Information security requirements and quality standards

A corporate portal is a system holding personal data, commercial and financial information. Comments on the required security level are superfluous here. Several scenarios of possible unauthorised intrusion were worked through; for each of them we carried out preventive measures and wrote up action procedures for contingency situations.

• Vulnerability of the servers' system environment.
• Gaining unauthorised access to the console of one or several cluster servers at once.
• Vulnerability at the application software level.
• Gaining unauthorised access to the CMS.

The centralised LDAP service was used for authentication. We also applied traffic encryption, firewalls and a structure for separated storage of personal data with denormalisation — this required changing the logic of the platform's base classes, but all the requirements were met.

Quality system standards

Large companies always use quality assurance system standards (ISO), one of which describes the requirements for a system that records user actions: it sets out clear requirements for such a system and a list of actions and events that must be logged. We had to solve the task of extending the platform's existing logging system to comply with the ISO requirements — that is, the task of recording a large stream of events, storing them and searching within large data arrays. Over the course of a day the portal generated up to a million events, and a database with a fast storage engine was used to record them.

As a result, we got a system that can search for events by type, source, user and time interval.

The second requirement we had to take into account was the need to obtain users' consent to the publication and processing of personal data. A dedicated interface was implemented that asked for consent to the processing and publication of personal data during the user's first portal session, and then stored all the timestamps of that event in a dedicated repository.

Stage three: refining the user interface

An ENTERPRISE client is distinguished by heightened quality requirements. The usage stories of typical scenarios in large organisations are thoroughly worked out, so the client demands that the solution being implemented fully matches the existing business processes. And the user interface is a particularly sensitive matter for corporate clients: it has to match the look users are accustomed to. At this stage many standard system interface solutions had to be reworked to meet the requirements of user cases — stories of real-world usage.

Employee photographs

To speed up adaptation to the new interfaces, we decided to import employee photographs from LDAP, while giving everyone the ability to change their own photo on the portal. Authors' photographs appeared in messages and tasks, and this matters a great deal in a large corporate information system: the perception of textual information (first name and surname) lags far behind the perception of easily recognisable photographs of colleagues. At the same time this solved the problem of confusion when people share the same combination of first name and surname — as a rule, few people know their colleagues' patronymics.

Confirmation of familiarisation

One of the goals of the implementation was to inform people about new processes and changes in the company. This called for a mechanism confirming that employees had familiarised themselves with materials, along with familiarisation statistics and the ability to obtain lists of employees with timestamps. The mechanism handled the publication of personal notifications and recorded the date and time of all employee actions on materials that require confirmation of familiarisation.

Limits on public messages

To keep the portal rollout from backfiring and turning the portal into a huge chat free-for-all with dozens of public messages addressed to "everyone" every minute, we faced the task of developing limits on the creation of such messages by ordinary users. Permission to create such publications was restricted with special roles for each department: at department level only one employee could hold the rights to create public messages. And to make sure this rule could not be circumvented, we introduced a limit on the number of message recipients.

Thanks and recognition

To make the portal a genuinely single information space, many HR department functions were moved onto it — in particular, publicly announcing employees' achievements and successes: around a dozen types of thanks and individual recognition awards were implemented. This immediately affected employee engagement with the portal and the development of team spirit — the rise in activity was reflected in the analytics straight away.

At the same time we refined the user profile — we implemented the display of managers and direct reports on employees' personal pages.

Internal career growth potential was implemented as well: all the vacancies available in the company began to be published on the portal, and employees gained the ability to put themselves forward. Since the information on each employee was integrated into the portal, there was no need to write a CV — it was enough to put yourself forward, as the HR department already had the information it needed for its assessment.

Documenting the project

Documentation is vital in ENTERPRISE-class projects. It provides an overall vision of the system when changes are being planned, and it makes it easy to onboard a new developer — the same specialists will not stay on one project for years.

It was a natural client requirement to receive user documentation, system maintenance instructions and action scenarios for special situations. As a result we did a large amount of work: we described and handed over to the client the portal's technical design and instructions for using it. The total volume of documentation produced came to around two hundred pages. The documentation also included materials on the acceptance testing suite — more than 100 tests.

An enormous amount of work was done, and now we can confidently say that the client is prepared to act in any situation — both standard and non-standard. The scenarios for using and maintaining the portal have been developed in detail and worked through in practice more than once.Leonid Sydorenko, QA engineer

Results and conclusions

What is good to know at the start

You should not blindly rely on the standard approaches described in application software vendors' materials, or on off-the-shelf solution options. Unfortunately, most of the time they are rather "theoretical" and very far from real cases. The most important source of practical information is real cases from similar projects. Here it is important to be in touch with the people who took part in those projects directly: they can share valuable details capable of changing the whole picture.Artem Volkov, ANIART developer

Large projects always mean complex architecture, high load and major financial responsibility. That is why all critical functionality must be covered by tests — this makes it possible to prevent potential problems in the future.

We want to note the exceptionally high level of technical competence of our colleagues at Ukrtelecom — they are professionals with extensive experience. We spoke the same language with them, both on business goals and on technical matters.Valentyn Kertychak, project manager on the ANIART side

We managed to solve every task the project faced. The difficulties that arose were overcome promptly and without delay. And we are launching the portal into planned operation with complete confidence.Yurii Huchek, project manager on the Ukrtelecom side

The project team

The team worked effectively for eight months and achieved an excellent result.