The Procuring Entity Index is a tool that, with the help of various groups of indicators, allows comprehensively assessing the quality of the procuring entity's procurement transactions. It was created and is administered by the DOZORRO project team of Transparency International Ukraine.

Purpose of the tool

Making procurement more professional is one of the most important tasks that will make the use of public funds more efficient. To do this, it is necessary to be able to assess both the individual procuring entity and the overall dynamics of procurement. However, there are more than 40,000 procuring entities in the public procurement system, and it is impossible to apply classical methods of direct process assessment to objectively analyze such an array of information. Therefore, it is currently difficult for procuring entities and their governing bodies to objectively assess the quality of organizing procurement, as well as to analyze their weaknesses and form a roadmap for their improvement.

For the most part, procuring entities use one or more assessment indicators, such as “savings” (in terms of the difference between the expected cost and the cost presented by the awarded participant), the average number of participants, and the success of procurement (percentage of successfully completed procurement transactions).

At the same time, these indicators do not allow forming a full understanding of how efficiently the processes take place. In addition, they may contain high risks of bias of such an assessment.

For example, the most popular indicator for determining a procuring entity's performance is “savings,” which is calculated as the difference between the expected cost and the cost of the winning bid. This indicator is not always an objective parameter. Determining the expected cost depends primarily on the procuring entity and their motivation. A tender with a high level of savings does not guarantee that it will have a lower price compared to other procurement transactions of other procuring entities where the “savings” are lower. Thus, this indicator is not objective enough to assess the procuring entity.

Since there is no single standardized approach to assessment, this does not allow focusing on the main problems of procurement and leads to the fact that the efficiency of the use of public funds is not systematically improved.

The concept

For a more objective and comprehensive assessment of procuring entities, the DOZORRO project of Transparency International Ukraine proposes to apply a set of indicators that complement and compensate for each other's risks.

How procuring entity behavior affects indicators in the Prozorro system

Let's consider several scenarios to see the impact of the procuring entity's behavior on certain indicators and determine the relationship between them.

Scenario 1

The procuring entity wants to conclude a contract with a certain supplier at an inflated cost, and for this, they overstated the expected cost and spelled out discriminatory requirements. In this case, we can observe a sufficiently high level of “savings” and the success of such a procurement transaction. At the same time, there is a high probability that such procurement will have a higher percentage of disqualifications, complaints, and identified violations based on the results of monitoring by the State Audit Service.

By analyzing all these indicators together, we can objectively assess the process, while the analysis of individual indicators will not give us a complete picture. If we assess the procuring entity only by the indicator “savings,” the conclusions may be false altogether.

Scenario 2

The procuring entity determines the requirements of the tender documentation for a specific product and limits the competition of participants. In this case, we will see low quantitative (number of participants) and qualitative (decrease in proposals in the bidding process) competition. An additional factor that will help to detect abuse may be numerous questions, requirements, and complaints about the procuring entity's tender documentation.

Scenario 3

The procuring entity creates obstacles to the conclusion of contracts with an “unnecessary” supplier. In this case, a high percentage of canceled procurement transactions, disqualifications, and complaints can be identified.

Consequently, different procuring entity behavior scenarios need different indicators to identify and record the problem.

How we selected indicators

Despite the fact that Prozorro stores a lot of procurement data, their volume is still limited. It does not allow evaluating the procurement preparation process, and information on contract performance

    is not always objective and timely.

    Therefore, our first task was to determine a list of indicators that were sufficiently objective and complemented each other, which allowed us to combine them into groups that could characterize the process.

    As a result, we formed the following groups of interrelated indicators:

    • Performance - how successfully the procurement transaction achieves its goal;
    • Openness - how open is the participants' access to the procuring entity's contracts;
  • Competitiveness - how actively participants compete for being awarded a contract;
  • Compliance with procedures (violations) - how well procuring entities comply with the requirements of the law.
  • We also added the “Contracts” group of indicators to the tool. However, at the moment, they do not affect the calculation, since the quality of data on contract implementation depends on the procuring entity. The procuring entity may not report on changes to the contract in a timely manner or not report on the expiration, which leads to distortion of information on the status of contract implementation.

Procuring Entity Index

Based on the result of the procuring entity's assessment in terms of different groups of indicators, a weighted average index of the procuring entity is calculated, which allows determining the overall quality of the procuring entity's procurement process.

The Procuring Entity Index is a top-level assessment of the overall result of their procurement. The analysis of specific indicators and groups of indicators can give a clearer idea of the main strengths and weaknesses of the procuring entity.

Approaches and methods used

Method for calculating the score by indicators

For all indicators used to determine the Procuring Entity Index, a single principle for calculating the assessment is applied. It is based on determining three parameters: the maximum, average, and minimum value of the indicator.

The minimum value is the minimum possible result in the Prozorro system, which is assigned 0 points in the assessment system. For example, for the indicator “percentage of successfully completed lots,” it will equal 0%.

The maximum value of the indicator is the maximum result, which is assigned 100 points. For the indicator “percentage of successfully completed lots,” it will equal 100%.

The average value is defined as the average value of the indicator in the previous year among all Prozorro procuring entities falling under the assessment criteria. For example, for 2022, this will be 67% of successful competitive lots completed. It is assigned a score of 50.

Determining these initial coordinates allows building a graph of the dependence of the assessment on the procuring entity's indicator:

So, if the procuring entity's result corresponds to the average value on Prozorro over the past year, then they receive 50 points out of 100 possible. If their result is higher than the average, they will have a proportionately higher score. If it is lower, they will get a proportionally lower score.

This model allows for solving the problem of different ranges of values for different indicators. For example, for such indicators as “savings,” the number of participants, and the percentage of disqualifications, the result in most cases will be quite low (2–5–10), whereas, indicators such as the percentage of successfully completed procurement transactions and the percentage of competitive ones have a high range of values (60–80%).

Converting them to a nonlinear graph of the dependence of the score on the base values eliminates the problem of different ranges of values. A value close to the average on Prozorro will be assigned the average indicator, regardless of what range of values it is in.

Sampling limitations

Periods

Most indicators can be objectively determined only at the stage when the procurement transaction is completed. Therefore, the division into periods occurs relative to the date of completion of the lot and not the date of the announcement, as provided for in the public analytics module.

That is, indicators in January will be calculated in relation to those procurement transactions that were designated as completed in the course of January. This indicator is determined after the end of the month for which the indicators are calculated.

The completed state refers to the following procurement statuses:

  • cancelled procurement transaction;
  • procurement transaction did not take place;
  • successful procurement transaction (ended with the conclusion of a contract).

For indicators that are tied to an event other than the procurement completion event (percentage of lots with confirmed violations by the State Audit Service, changes in contracts, etc.), the calculation takes place relative to the date of the event in this period (regardless of the date of announcement and completion of the procurement transaction).

Sampling

The calculation of all indicators, except for the percentage of competitive lots, takes place exclusively for competitive procurement. After all, most of the indicators are unavailable for non-competitive procedures.

Minimum amount of data (of procurement) for assessment

In order to objectively identify certain trends and behavioral factors, it is necessary to analyze the amount of data (lots) that is sufficient to eliminate the risk of exposure to a single anomalous value.p>

Thus, only procuring entities that have completed at least 10 competitive procurement transactions (lots) in the selected period are assigned a score. If fewer procurement transactions are assessed, there is a high risk that one anomalous value will have a disproportionate impact compared to the others.

For example, procuring entity X completed 3 lots during the period under study, in one of which the number of participants was 9, and the other two had the level of average Prozorro values (about 2–3). In this case, the value of the procuring entity will be twice as high as the average on Prozorro, while such an impact is due to only a single event. With an increase in the number of procurement transactions, the accuracy of measurements becomes higher (in accordance with the Student's method).

Accumulation of periods

Since there is a certain “pass mark” in the form of 10 competitive procurement transactions (lots), the calculation of indicators for each individual period can lead to the fact that, in certain months, the same procuring entity will have 10 competitive completed procurement transactions, and in others, there will be fewer of them. Accordingly, the score will not be determined for the entire period, which covers several months, and this leads to the loss of part of the data and an inaccurate calculation.

To eliminate this risk, procurement assessment takes place by accumulation. That is, the January period will contain completed procurement transactions of January, the February period will show completed procurement transactions of January–February, the March period will feature completed procurement transactions of January–March, and so on.

Thus, all procurement transactions will be accumulated, and even if the procuring entity was not assigned a score in January, they will get it in the following months, when the number of completed procurement transactions exceeds 10.