<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Publishing DTD v1.2 20190208//EN" "http://jats.nlm.nih.gov/publishing/1.2/JATS-journalpublishing1.dtd"><article xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" article-type="other" dtd-version="1.2" xml:lang="en">
    <front>
        <journal-meta>
            <journal-id journal-id-type="pmc">F1000Research</journal-id>
            <journal-title-group>
                <journal-title>F1000Research</journal-title>
            </journal-title-group>
            <issn pub-type="epub">2046-1402</issn>
            <publisher>
                <publisher-name>F1000 Research Limited</publisher-name>
                <publisher-loc>London, UK</publisher-loc>
            </publisher>
        </journal-meta>
        <article-meta>
            <article-id pub-id-type="doi">10.12688/f1000research.153917.1</article-id>
            <article-categories>
                <subj-group subj-group-type="heading">
                    <subject>Policy Brief</subject>
                </subj-group>
                <subj-group>
                    <subject>Articles</subject>
                </subj-group>
            </article-categories>
            <title-group>
                <article-title>Contract award criteria in public procurement procedures &#x2013; The possibility of improving the situation of society from the perspective of the European Union and Poland</article-title>
                <fn-group content-type="pub-status">
                    <fn>
                        <p>[version 1; peer review: 1 approved, 1 approved with reservations]</p>
                    </fn>
                </fn-group>
            </title-group>
            <contrib-group>
                <contrib contrib-type="author" corresp="yes">
                    <name>
                        <surname>Brzezi&#x0144;ski</surname>
                        <given-names>Mateusz</given-names>
                    </name>
                    <role content-type="http://credit.niso.org/">Conceptualization</role>
                    <role content-type="http://credit.niso.org/">Formal Analysis</role>
                    <role content-type="http://credit.niso.org/">Resources</role>
                    <role content-type="http://credit.niso.org/">Writing &#x2013; Original Draft Preparation</role>
                    <role content-type="http://credit.niso.org/">Writing &#x2013; Review &amp; Editing</role>
                    <uri content-type="orcid">https://orcid.org/0009-0009-9586-2490</uri>
                    <xref ref-type="corresp" rid="c1">a</xref>
                    <xref ref-type="aff" rid="a1">1</xref>
                </contrib>
                <aff id="a1">
                    <label>1</label>Master of Laws (LL.M. and magister), University of Warsaw, Warsaw, Masovian Voivodeship, Poland</aff>
            </contrib-group>
            <author-notes>
                <corresp id="c1">
                    <label>a</label>
                    <email xlink:href="mailto:mateusz.brzezinski@wp.pl">mateusz.brzezinski@wp.pl</email>
                </corresp>
                <fn fn-type="conflict">
                    <p>No competing interests were disclosed.</p>
                </fn>
            </author-notes>
            <pub-date pub-type="epub">
                <day>24</day>
                <month>7</month>
                <year>2024</year>
            </pub-date>
            <pub-date pub-type="collection">
                <year>2024</year>
            </pub-date>
            <volume>13</volume>
            <elocation-id>832</elocation-id>
            <history>
                <date date-type="accepted">
                    <day>15</day>
                    <month>7</month>
                    <year>2024</year>
                </date>
            </history>
            <permissions>
                <copyright-statement>Copyright: &#x00a9; 2024 Brzezi&#x0144;ski M</copyright-statement>
                <copyright-year>2024</copyright-year>
                <license xlink:href="https://creativecommons.org/licenses/by/4.0/">
                    <license-p>This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.</license-p>
                </license>
            </permissions>
            <self-uri content-type="pdf" xlink:href="https://f1000research.com/articles/13-832/pdf"/>
            <abstract>
                <sec>
                    <title>Background</title>
                    <p>Public tenders are vital for a country&#x2019;s GDP and citizens&#x2019; quality of life, enabling public administration to achieve various goals. Developing and developed countries allocate over 10% of their GDP to public procurement. This highlights the significant societal support public tenders can provide, making it important to consider how they can further benefit society.</p>
                </sec>
                <sec>
                    <title>Policy and implications</title>
                    <p>Public procurement can achieve policy objectives and benefit society by selecting tenders based on criteria beyond price, such as economic advantage and social benefits. This approach, endorsed by EU directives since 2014, encourages innovation and socially responsible practices. Contracting authorities in the EU can use social, environmental, and qualitative criteria to determine the most advantageous offers.</p>
                    <p>Poland&#x2019;s Public Procurement Law (PPL) allows contract award criteria based on quality and price, including social aspects. Contracting authorities can specify criteria like employing marginalized groups, though these criteria often face scrutiny and legal challenges. Non-price criteria aim to enhance competition and achieve social, environmental, and economic goals.</p>
                </sec>
                <sec>
                    <title>Recommendations</title>
                    <p>EU and Polish laws permit and encourage using social aspects as contract award criteria in public procurement. However, contracting authorities must analyze priorities, risk balancing, time constraints, and departmental coordination to effectively implement these criteria. This approach can improve the social situation and support specific groups.</p>
                </sec>
                <sec>
                    <title>Conclusions</title>
                    <p>Public procurement significantly influences a country&#x2019;s economy and quality of life, with EU and Polish laws allowing social criteria in contract awards. Directive 2014/24/EU supports tenders based on economic and social benefits. Poland&#x2019;s PPL aligns with this, emphasizing marginalized group employment. Effective implementation fosters job creation, social integration, and improved living standards.</p>
                </sec>
            </abstract>
            <kwd-group kwd-group-type="author">
                <kwd>public procurement</kwd>
                <kwd>tenders</kwd>
                <kwd>contract award criteria</kwd>
                <kwd>socially responsible public procurement</kwd>
                <kwd>social aspects</kwd>
            </kwd-group>
            <funding-group>
                <funding-statement>The author(s) declared that no grants were involved in supporting this work.</funding-statement>
            </funding-group>
        </article-meta>
    </front>
    <body>
        <sec id="sec5" sec-type="intro">
            <title>Introduction</title>
            <p>There should be no doubt that public tenders are one of the most important elements not only of generating the GDP of a given country, but also affect the quality of life of citizens. By means of public tenders, public administration bodies can meet various types of goals. OECD countries spend on average nearly 13% of annual GDP on public tenders (
                <xref ref-type="bibr" rid="ref7">OECD 2021</xref>). EU Member States spend on average around 14% of annual GDP on public tenders. Poland spends on average 10% of its annual GDP in public tenders. It means that countries can significantly support the functioning of society, using the tools that public tenders can provide. Therefore, it is worth considering how these public tenders can improve the situation of society.</p>
        </sec>
        <sec id="sec6">
            <title>Policy outcomes and implications</title>
            <sec id="sec7">
                <title>Legal basis in the EU</title>
                <p>According to Jolien Grandia and Joanne Meehan public procurement can be leveraged to accomplish specific policy objectives, thereby benefiting society. This approach often involves selecting a tender design that awards contracts based on criteria beyond just the price, such as choosing the tender that offers the greatest economic advantage. Numerous studies have found that public procurement can stimulate innovative solutions by procuring functions rather than predefined goods or services (
                    <xref ref-type="bibr" rid="ref4">Geroski 1990</xref>; 
                    <xref ref-type="bibr" rid="ref3">Elder and Georghiou 2007</xref>; 
                    <xref ref-type="bibr" rid="ref1">Aschhoff and Sofka 2009</xref>). Socially responsible public procurement is an approach in which contracting authorities use solutions that allow them to achieve additional social benefits when awarding public contracts. In practice, contracting authorities in European Union Member States have been using solutions to achieve additional social benefits for over 30 years, but it was only in 2004 that they were included in the EU directives on public procurement.</p>
                <p>One of such elements of socially responsible public procurement may be the contract award criteria. The essence of the contract award criteria, as emphasized in recital 92 of the preamble to Directive 2014/24/EU, is to enable a comparative assessment of the level of performance offered in each of the tenders for the subject of the contract. Bearing in mind also the principle that public tenders should be awarded on the basis of objective criteria ensuring compliance with the principles of transparency, non-discrimination and equal treatment, with a view to ensuring an objective comparison of the relative value of tenders in order to determine, in conditions of effective competition, which of the tenders is the most economically advantageous (see recital 90 of the preamble to Directive 2014/24/EU), all principles that the contracting authority intends to apply to the evaluation of tenders (i.e. criteria, weights and method of evaluation based on the criteria) must be specified in the procurement documents in an unambiguous and understandable manner. The description of the evaluation criteria and principles of evaluation of offers should therefore allow contractors to obtain full knowledge of how and based on what information their offers will be assessed, so that they can include appropriate data in the content of their offers.</p>
                <p>The EU legislator himself in article 67 para 2 of Directive 2014/24/EU indicates that the most advantageous offer may include the best value for money, which is estimated on the basis of criteria including qualitative, environmental or social aspects related to the subject of a given public procurement - depending on which decision is made. The contracting authority itself will decide on the criteria.</p>
                <p>It is important that the EU legislator advised contracting authorities that one of the criteria for evaluating offers may be social or environmental aspects. It is important because in the previously applicable Directive 2004/18/EC, in article 53 para 1 regarding contract award criteria, there was no indication at all that social aspects could be one of the possible contract award criteria. This also means that the EU legislator has noticed that it is actually possible to include this element in the contract award criteria and that they can not only help obtain the most advantageous offer in general, but it is also possible, using social aspects in public procurement, to meet the social goals of countries or local government units. Indeed, policies aimed at promoting societal objectives like social sustainability or innovation can impact the criteria (
                    <xref ref-type="bibr" rid="ref6">Kadefors et al. 2021</xref>).</p>
                <p>Nevertheless, it is noted that the lowest price is chosen when the goal is to maximize savings, while the best value approach is used for more complex projects (
                    <xref ref-type="bibr" rid="ref2">Ballesteros-Perez et al. 2017</xref>). However, each time it is the contracting authority who must decide which contract award criteria to choose for a given public tender. Importantly, the EU legislator does not indicate exemplary contract award criteria - it describes them rather in terms of categories in which contracting authorities can look for appropriate solutions. Therefore, the freedom of choice rests with the contracting authority both in terms of selecting the category of contract award criterion or criteria in general, as well as in terms of determining these criteria in detail.</p>
            </sec>
            <sec id="sec8">
                <title>Legal basis in Poland</title>
                <p>Due to the fact that the EU legislator decided to use directives to regulate the issue of public procurement in the EU, in Poland the issue regarding contract award criteria, at the level of principles and objectives, is the same as in the provisions of Directive 2014/24/EU. However, precisely because the issue was regulated by directives, EU Member States were able to clarify and detail certain issues. This also happened in the case of Poland, where the PPL regulations stated that the most advantageous tender may be selected on the basis of quality criteria and price or cost, according to article 242 para 1 point 1 of the PPL.</p>
                <p>In this respect, it is worth paying attention to article 242 para 2 point 2 of the PPL, where it is indicated that quality criteria may be, in particular, criteria relating to the social aspects, including the social and occupational integration of persons referred to in article 94 para 1 (persons with disabilities, the unemployed, jobseekers, who do not remain in employment or do not perform gainful employment, to-be self-reliant persons, persons deprived of liberty or released from prisons having difficulties in integration with the environment, persons with mental disorders, homeless persons, persons who have obtained refugee status or subsidiary protection in the Republic of Poland, persons under the age of 30 and after 50 years of age with job-seeker status, without employment, persons who are members of disadvantaged minorities, in particular members of national and ethnic minorities). Interestingly, under contract award criteria it is not possible to reward employment based on an employment contract. This is due to the fact that the legislator in article 95 para 1 of the PPL provides for the obligation to employ people on the basis of an employment contract - the contracting authority is obliged to indicate in the procurement documents the persons or the scope of activities to which the contractor&#x2019;s obligation to demonstrate the employment of persons under an employment contract applies.</p>
                <p>It is worth noting that while meeting the social aspects taken into account by the contracting authority in the conditions for participation in a procurement procedure or the terms of the contract (e.g. in terms of employment under an employment contract) is obligatory for the economic operator applying for a given contract, including them in the contract award criteria influences the selection of the most advantageous offer (the economic operator who meets the requirements regarding social aspects receives additional points in the offer evaluation), but failure to meet them does not, in principle, eliminate his offer from participation in the procedure. This is important because when determining the criteria for evaluating offers, these criteria can be defined more freely than when determining the conditions for participation in the procedure - because, in principle, they do not prevent the contractor from submitting an offer.</p>
                <p>Before applying contract award criteria related to social aspects, it is advisable for the contracting authority to research the situation on the labor market in terms of the validity of the requirements specified in the criterion. This will avoid a situation in which the contractor gains an advantage in points by declaring the employment of e.g. unemployed people, and at the stage of order execution the declaration submitted as part of the offer turns out to be impossible to implement due to the lack of unemployed people able to perform the order. Depending on the decision of the contracting authority, related to the assessment of the specificity of the specific subject of the contract, as part of the social criterion, the contracting authority may specify the requirement to employ one or more people from one or several marginalized groups indicated in article 94 para 1 of the PPL. In practice, contracting authorities have doubts about how the evaluation is carried out in the case of the contract award criterion regarding the employment of a certain number of marginalized people to perform the contract. It is assumed that it is based on the economic operator&#x2019;s only declaration regarding the employment of the indicated number of such people to perform the contract. The fulfillment of this obligation takes place during the implementation of the public procurement contract, and thus requires the contracting authority to determine the method of verifying the implementation of the obligation in question in the contractual provisions. In such a case, the contracting authority should specify the method of documenting and controlling the employment of marginalized people and the sanctions that will apply to the economic operator who does not implement the employment declared in the contract award criterion.</p>
                <p>Nevertheless, Polish contracting authorities are quite reluctant to use contract award criteria related to social aspects. One can even risk the thesis that Polish contracting authorities are not willing to establish interesting and innovative contract award criteria. In my opinion, this is due to the fact that they are afraid of the potential use of legal remedies - an appeal to the National Appeals Chamber by a potential economic operator, and they also have doubts as to how such contract award criteria would be verified in an audit in projects co-financed from EU funds. Most often, in such a situation, a violation of the principle of fair competition and equal treatment of contractors is alleged. Nevertheless, in one of its judgments, the National Appeals Chamber emphasized that the contract award criteria are not intended to guarantee an &#x201c;equal position of economic operators&#x201d;, but to make competition between these economic operators more realistic. If the position of economic operators were to be equal, non-price contract award criteria would be illusory and essentially pointless. However, it is important that non-price contract award criteria are objective and guarantee that economic operators can compete. The competitiveness of economic operators in terms of non-price contract award criteria should be related to the objectivity of these criteria, so that their meaning and assessment is unambiguous, which in turn allows for a clear assessment of a given economic operator. Selection of non-price contract award criteria is the right of the contracting authority, which may also make the purpose of the public procurement procedure more realistic using these criteria, as well as obtain the subject of the contract of the highest quality (case KIO 610/22). This means that contract award criteria by their nature - by promoting specific features or functionalities - lead to an increase in the chances of obtaining a contract by some economic operators, while limiting these chances for other economic operators.</p>
                <p>However, it should be noted that these contract award criteria may allow the fulfillment of the fundamental principles of public procurement, which are included in article 17 para 1 point of the PPL, i.e. that the contracting body shall award the contract in a manner ensuring the best results of the contract, including social, environmental and economic effects, insofar as any of these effects can be obtained in a given contract in relation to the expenditure incurred.</p>
            </sec>
            <sec id="sec9">
                <title>Actionable recommendations</title>
                <p>None of the provisions of EU law and PPL prevent the use of social aspects as contract award criteria in public procurement procedures. Moreover, the provisions of relevant legal acts even encourage promoting the social aspects of public tenders through contract award criteria. However, in order to apply the appropriate contract award criteria in this area, each contracting authority must perform appropriate analysis, including the following:
                    <list list-type="order">
                        <list-item>
                            <label>1)</label>
                            <p>a matter of mentality - moving away from the priority of price and the contract award criteria.</p>
                        </list-item>
                        <list-item>
                            <label>2)</label>
                            <p>the issue of balancing risks - challenging the contract award criteria criteria by potential economic operators vs. customer preferences.</p>
                        </list-item>
                        <list-item>
                            <label>3)</label>
                            <p>a matter of time - the urgent need to prepare documentation of the proceedings, limiting the possibility of reliably preparing the criteria.</p>
                        </list-item>
                        <list-item>
                            <label>4)</label>
                            <p>the issue of synergy of work - the need to coordinate the work of the substantive and administrative and legal departments.</p>
                        </list-item>
                    </list>
                </p>
                <p>This means that each contracting authority has a chance, through public procurement procedures and contract award criteria, to improve the social situation and enable greater activity of specific groups of people.</p>
            </sec>
        </sec>
        <sec id="sec10" sec-type="conclusion">
            <title>Conclusion</title>
            <p>Public procurement plays a vital role in shaping the economic and social landscape of a country, impacting GDP and the quality of life of citizens.</p>
            <p>The EU legal framework allows public procurement to support societal goals through the inclusion of social criteria in contract awards, as outlined in Directive 2014/24/EU, which enables contracting authorities to assess tenders based on more than just price, incorporating aspects that offer the greatest economic advantage and additional social benefits.</p>
            <p>In Poland, PPL align with EU directives, allowing the selection of the most advantageous tenders based on quality and price or cost. Notably, Polish regulations also emphasize the inclusion of social criteria, such as the employment and integration of marginalized groups. While Polish contracting authorities are hesitant to use these criteria due to concerns about legal challenges and verification in audits, there is a growing recognition of their potential benefits.</p>
            <p>Public procurement offers a powerful tool for achieving social goals. By carefully selecting and implementing contract award criteria, contracting authorities can foster significant social benefits, contributing to a more inclusive and equitable society. Legal frameworks provide a solid foundation for such practices, though their success depends on the effective and innovative application by contracting authorities.</p>
            <p>It should be noted that the solutions included in the EU and Polish legislation allow for achieving various social aspects and benefits, but they focus primarily on:
                <list list-type="order">
                    <list-item>
                        <label>1)</label>
                        <p>increasing employment stability and social security of employees,</p>
                    </list-item>
                    <list-item>
                        <label>2)</label>
                        <p>employment and professional and social integration of people from groups at risk of social exclusion,</p>
                    </list-item>
                    <list-item>
                        <label>3)</label>
                        <p>facilitating access to public procurement for social economy entities, and especially social enterprises conducting social and professional integration of people from groups at risk of social exclusion,</p>
                    </list-item>
                    <list-item>
                        <label>4)</label>
                        <p>facilitating access to the subject of procurement for all users, in particular for people with disabilities and other special needs,</p>
                    </list-item>
                    <list-item>
                        <label>5)</label>
                        <p>using products or services with social labels confirming compliance with standards regarding, among others, decent work or fair trade.</p>
                    </list-item>
                </list>
            </p>
            <p>In the long term, the use of social public procurement contributes to:
                <list list-type="order">
                    <list-item>
                        <label>1)</label>
                        <p>creating jobs,</p>
                    </list-item>
                    <list-item>
                        <label>2)</label>
                        <p>social and professional integration of people from groups at risk of social exclusion,</p>
                    </list-item>
                    <list-item>
                        <label>3)</label>
                        <p>dissemination of decent work and fair trade standards among economic operators and employees,</p>
                    </list-item>
                    <list-item>
                        <label>4)</label>
                        <p>development of social economy entities,</p>
                    </list-item>
                    <list-item>
                        <label>5)</label>
                        <p>socializing the provision of services important to citizens, e.g. care services or revitalization services,</p>
                    </list-item>
                    <list-item>
                        <label>6)</label>
                        <p>improving the quality of life, in particular of people with disabilities, by increasing their access to public spaces, buildings and services (adapting orders - services, construction works, purchased goods to the requirements of people with special needs).</p>
                    </list-item>
                </list>
            </p>
        </sec>
    </body>
    <back>
        <sec id="sec13" sec-type="data-availability">
            <title>Data availability</title>
            <p>No data are associated with this article.</p>
        </sec>
        <ref-list>
            <title>References</title>
            <ref id="ref1">
                <mixed-citation publication-type="journal">
                    <person-group person-group-type="author">

                        <name name-style="western">
                            <surname>Aschhoff</surname>
                            <given-names>B</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Sofka</surname>
                            <given-names>WEJ</given-names>
                        </name>
</person-group>:
                    <article-title>Innovation on demand? Can public procurement drive market success of innovations?</article-title>
                    <source>

                        <italic toggle="yes">Res. Policy.</italic>
</source>
                    <year>2009</year>;<volume>38</volume>(<issue>8</issue>):<fpage>1235</fpage>&#x2013;<lpage>1247</lpage>.
                    <pub-id pub-id-type="doi">10.1016/j.respol.2009.06.011</pub-id>
                </mixed-citation>
            </ref>
            <ref id="ref2">
                <mixed-citation publication-type="book">
                    <person-group person-group-type="author">

                        <name name-style="western">
                            <surname>Ballesteros-Perez</surname>
                            <given-names>P</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Pellicer</surname>
                            <given-names>E</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Gonz&#x00e1;lez-Cruz</surname>
                            <given-names>MC</given-names>
                        </name>
</person-group>:
                    <chapter-title>Multi- attribute construction auctions.</chapter-title>
                    <source>

                        <italic toggle="yes">Proc., 9th Int. Structural Engineering and Construction Conf.</italic>
</source>
                    <publisher-loc>Fargo, ND</publisher-loc>:
                    <publisher-name>International Structural Engineering and Construction Society</publisher-name>;<year>2017</year>; pp.<fpage>1</fpage>&#x2013;<lpage>6</lpage>.</mixed-citation>
            </ref>
            <ref id="ref3">
                <mixed-citation publication-type="journal">
                    <person-group person-group-type="author">

                        <name name-style="western">
                            <surname>Edler</surname>
                            <given-names>J</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Georghiou</surname>
                            <given-names>L</given-names>
                        </name>
</person-group>:
                    <article-title>Public procurement and innovation&#x2014;Resurrecting the demand side.</article-title>
                    <source>

                        <italic toggle="yes">Res. Policy.</italic>
</source>
                    <year>2007</year>;<volume>36</volume>(<issue>7</issue>):<fpage>949</fpage>&#x2013;<lpage>963</lpage>.
                    <pub-id pub-id-type="doi">10.1016/j.respol.2007.03.003</pub-id>
                </mixed-citation>
            </ref>
            <ref id="ref4">
                <mixed-citation publication-type="journal">
                    <person-group person-group-type="author">

                        <name name-style="western">
                            <surname>Geroski</surname>
                            <given-names>PA</given-names>
                        </name>
</person-group>:
                    <article-title>Procurement policy as a tool of industrial policy.</article-title>
                    <source>

                        <italic toggle="yes">Int. Rev. Appl. Econ.</italic>
</source>
                    <year>1990</year>;<volume>4</volume>(<issue>2</issue>):<fpage>S.182</fpage>&#x2013;<lpage>S.198</lpage>.</mixed-citation>
            </ref>
            <ref id="ref5">
                <mixed-citation publication-type="journal">
                    <person-group person-group-type="author">

                        <name name-style="western">
                            <surname>Grandia</surname>
                            <given-names>J</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Meehan</surname>
                            <given-names>J</given-names>
                        </name>
</person-group>:
                    <article-title>Public procurement as a policy tool: using procurement to reach desired outcomes in society.</article-title>
                    <source>

                        <italic toggle="yes">Int. J. Public Sect. Manag.</italic>
</source>
                    <year>2017</year>;<volume>30</volume>(<issue>4</issue>):<fpage>302</fpage>&#x2013;<lpage>309</lpage>.
                    <pub-id pub-id-type="doi">10.1108/IJPSM-03-2017-0066</pub-id>
                </mixed-citation>
            </ref>
            <ref id="ref6">
                <mixed-citation publication-type="journal">
                    <person-group person-group-type="author">

                        <name name-style="western">
                            <surname>Kadefors</surname>
                            <given-names>A</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Lingeg&#x00e5;rd</surname>
                            <given-names>S</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Uppenberg</surname>
                            <given-names>S</given-names>
                        </name>

                        <etal/>
</person-group>:
                    <article-title>Designing and implementing procurement requirements for carbon reduction in infrastructure construction &#x2013; international overview and experiences.</article-title>
                    <source>

                        <italic toggle="yes">J. Environ. Plan. Manag.</italic>
</source>
                    <year>2021</year>;<volume>64</volume>(<issue>4</issue>):<fpage>611</fpage>&#x2013;<lpage>634</lpage>.
                    <pub-id pub-id-type="doi">10.1080/09640568.2020.1778453</pub-id>
                </mixed-citation>
            </ref>
            <ref id="ref7">
                <mixed-citation publication-type="other">
                    <collab>OECD</collab>:
                    <article-title>Government at a Glance 2013: Procurement Data.</article-title>
                    <year>2021</year>.
                    <ext-link ext-link-type="uri" xlink:href="https://www.oecd-ilibrary.org/sites/3d5c5d31-en/1/3/7/1/index.html?itemId=/content/publication/3d5c5d31-en&amp;_csp_=6a6aa038f2496de10696e78b24488941&amp;itemIGO=oecd&amp;itemContentType=book">Reference Source</ext-link>
                </mixed-citation>
            </ref>
            <ref id="ref8">
                <mixed-citation publication-type="other">
                    <article-title>Public Procurement Law of 11 September 2019 (Journal of Laws of 2023, item 1605 as amended).</article-title>
                </mixed-citation>
            </ref>
            <ref id="ref9">
                <mixed-citation publication-type="other">
                    <article-title>Sprawozdanie Prezesa Urz&#x0119;du Zam&#x00f3;wie&#x0144; Publicznych z funkcjonowania system zam&#x00f3;wie&#x0144; publicznych w 2022 r.</article-title>
                </mixed-citation>
            </ref>
            <ref id="ref10">
                <mixed-citation publication-type="other">
                    <article-title>judgment of the National Appeals Chamber of March 25, 2022, case KIO 610/22.</article-title>
                </mixed-citation>
            </ref>
            <ref id="ref11">
                <mixed-citation publication-type="other">
                    <ext-link ext-link-type="uri" xlink:href="https://single-market-scoreboard.ec.europa.eu/business-framework-conditions/public-procurement_en">Reference Source</ext-link>
                </mixed-citation>
            </ref>
            <ref id="ref12">
                <mixed-citation publication-type="other">
                    <article-title>Directive 2014/24/EU of the European Parliament and of the Council of 26 February 2014 on public procurement and repealing Directive 2004/18/EC (OJ L 94), 28.3.</article-title>
                    <year>2014</year>; pp.<fpage>65</fpage>&#x2013;<lpage>242</lpage>.
                    <ext-link ext-link-type="uri" xlink:href="https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A32014L0024">Reference Source</ext-link>
                </mixed-citation>
            </ref>
        </ref-list>
    </back>
    <sub-article article-type="reviewer-report" id="report315637">
        <front-stub>
            <article-id pub-id-type="doi">10.5256/f1000research.168873.r315637</article-id>
            <title-group>
                <article-title>Reviewer response for version 1</article-title>
            </title-group>
            <contrib-group>
                <contrib contrib-type="author">
                    <name>
                        <surname>Ochrana</surname>
                        <given-names>Franti&#x0161;ek</given-names>
                    </name>
                    <xref ref-type="aff" rid="r315637a1">1</xref>
                    <role>Referee</role>
                </contrib>
                <aff id="r315637a1">
                    <label>1</label>Department of Public and Social Policy, Charles University, Prague, Czech Republic</aff>
            </contrib-group>
            <author-notes>
                <fn fn-type="conflict">
                    <p>
                        <bold>Competing interests: </bold>No competing interests were disclosed.</p>
                </fn>
            </author-notes>
            <pub-date pub-type="epub">
                <day>26</day>
                <month>8</month>
                <year>2024</year>
            </pub-date>
            <permissions>
                <copyright-statement>Copyright: &#x00a9; 2024 Ochrana F</copyright-statement>
                <copyright-year>2024</copyright-year>
                <license xlink:href="https://creativecommons.org/licenses/by/4.0/">
                    <license-p>This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.</license-p>
                </license>
            </permissions>
            <related-article ext-link-type="doi" id="relatedArticleReport315637" related-article-type="peer-reviewed-article" xlink:href="10.12688/f1000research.153917.1"/>
            <custom-meta-group>
                <custom-meta>
                    <meta-name>recommendation</meta-name>
                    <meta-value>approve</meta-value>
                </custom-meta>
            </custom-meta-group>
        </front-stub>
        <body>
            <p>I read with interest the text "Contract award criteria in public procurement procedures - The possibility of improving the situation of society from the perspective of the European Union and Poland." Thank you for this opportunity.</p>
            <p> The reviewed policy brief is a suitable contribution for the discussion of the issue of contract award criteria in public procurement procedures. The study contains a number of inspiring ideas on the evaluation criteria of public procurement. The analysis of this problem is carried out on the case of Poland. I accompany the assessment of the reviewed study with an additional comment. The comment contains topics for discussion as well as possible suggestions for the author for further research.</p>
            <p> Poland is an interesting CEE country that has a common historical footprint with the neighboring countries of the former Soviet bloc, such as the Czech Republic and the Slovak Republic. All three countries became new members of the EU on the same date (2004). They have adopted the same EU legislation for public procurement. We would therefore expect approximately similar behavior of bureaucrats in public procurement. In reality, however, when analyzing the contract award criteria in these countries, we see significant differences.</p>
            <p> Unlike other CEE countries, Poland has a low share of public procurement evaluations according to the lowest price. According to TED data (2018; 2024), Poland was one of the leading CEE countries in 2016, with about 16% of public contracts awarded on the basis of the lowest price. This is significantly different compared to the Czech Republic (82% in 2016; 64.9% in 2024) and Slovakia (85% in 2016; 89.6% in 2024) of orders evaluated according to the lowest price. It is an "odd strategy" in the selection of evaluation criteria (Pernica et al., 2024)(Ref-3).</p>
            <p> In the context of the theory of institutional changes (Mahoney and Thelen, 2010)(Ref-10) and historical institutionalism (Schmidt, 2008;(Ref-8),Schmidt, 2010(Ref-7); Pla&#x010d;ek et al., 2022)(Ref-4), the question is raised, what are the causes of this difference?</p>
            <p> It is undoubtedly more factors. One of the key ones is legal regulations. The author of the policy brief deals with these regulations in the section Policy outcomes and implications (see part Legal basis in the EU and part Legal basis in Poland). I will try to respond to the author's inspirational ideas with the following note. I believe that on the basis of historical institutionalism and retrograde analysis (Ochrana, 2022)(Ref-2), we can explain the differences between Poland, the Czech Republic and Slovakia.</p>
            <p> The author of the policy brief focuses on the role of legal regulations of public tenders. These are contained in the new Polish Public Procurement Act (
                <ext-link ext-link-type="uri" xlink:href="https://www.portalzp.pl/ustawy/ustawa-z-dnia-11-wrzesnia-2019-r.-prawo-zamowien-publicznych-tekst-jedn.-dz.u.-z-2023-r.-poz.-1605-17500.html">https://www.portalzp.pl/ustawy/ustawa-z-dnia-11-wrzesnia-2019-r.-prawo-zamowien-publicznych-tekst-jedn.-dz.u.-z-2023-r.-poz.-1605-17500.html</ext-link>). To this I note:</p>
            <p> Act of 11 September 2019 - The Public Procurement Act has been amended several times since its adoption. The latest consolidated version of the law was published on 14 August 2023. (See 
                <ext-link ext-link-type="uri" xlink:href="https://www.portalzp.pl/ustawy/ustawa-z-dnia-11-wrzesnia-2019-r.-prawo-zamowien-publicznych-tekst-jedn.-dz.u.-z-2023-r.-poz.-1605-17500.html">https://www.portalzp.pl/ustawy/ustawa-z-dnia-11-wrzesnia-2019-r.-prawo-zamowien-publicznych-tekst-jedn.-dz.u.-z-2023-r.-poz.-1605-17500.html</ext-link>). This is a significant institutional change in the legal regulations of public tenders. The importance of this change in the Public Procurement Act was characterized by the then Minister for Enterprise and Technology, Jadwiga Emilewicz, by saying that contracting authorities would not only focus on the price offered, but would also take into account the quality of goods and services and their importance for the environment.</p>
            <p> When analyzing the Public Procurement Act, it is necessary to bear in mind that this legal regulation also has an impact on the behavior of bureaucrats in public procurement. In this respect, too, the reviewed policy brief is inspirational for public policy. Public procurement bureaucrats carry out their activities in a certain social and institutional environment, while the framework and incentives for their behavior are set by the public tender regulations.(
                <ext-link ext-link-type="uri" xlink:href="https://ted.europa.eu/en/index">https://ted.europa.eu/en/index</ext-link> )</p>
            <p> The author makes an interesting point that Polish contracting authorities are rather reluctant to use award criteria related to social aspects and that they are reluctant to establish interesting and innovative award criteria. The author explains this behavior by fear of the possible use of legal remedies (and appeal to a higher authority).</p>
            <p> This conclusion is in line with similar attitudes of contracting authorities in the Czech Republic, which we examined in the study by(Schmidt et al., 2016)(Ref-9). Advertisers are behaving opportunistically. I therefore support the author's assertion that contracting authorities are concerned about the possible use of legal remedies. By opportunistic behaviour, contracting authorities try to avoid the risk of a public tender failure and to reduce the risk of a public procurement review as much as possible (see Schmidt et al., 2016). At the same time, the bureaucrat is pre-cautious of risk and uncertainty. This caution stems from the fear of failure of the public contract. From the author's analysis, we can conclude that the behavior of bureaucrats in public procurement is characterized by risk aversion (see Tversky and Kahneman, 1992)(Ref-5); Bajoori, et al., 2024)(Ref-9). The bearer of risk aversion is the bureaucrat responsible for the administration of public procurement and for the formal contract. Formal contracts cannot completely avoid opportunistic behaviors (Zhang et al., 2022)(Ref-6). This explains why, in an institutionally underdeveloped environment, contracting authorities prefer to evaluate according to the lowest price.</p>
            <p> The author of the policy brief also points to the significant impact of the choice of evaluation criteria on public procurement as a tool for achieving social goals. This part of the policy brief is also inspirational for public policy. It shows that contracting authorities can promote important social objectives and contribute to a more inclusive and fairer society by choosing their evaluation criteria. This is in line with the above-mentioned idea of the influence of legal regulations on the conduct of contracting authorities. In a suitably set legislative environment, public procurement can thus become an important tool for the effective achievement of public policy objectives. This is a problem that, according to the author of the policy brief, four fundamental assumptions and conditions must be met to implement. These are mainly:</p>
            <p> 1) a change of mentality that will lead to a change in behavior when selecting evaluation criteria ("moving away from the priority of price and the contract award criteria".)</p>
            <p> 2) the issue of balancing risks;</p>
            <p> 3) efficient administrative process (preparation of quality documentation);</p>
            <p> 4) synergy of actors preparing and implementing public procurement.</p>
            <p> The author of the policy brief sees in the implementation of these conditions a chance for each contracting authority to "through public procurement procedures and contract award criteria, to improve the social situation and enable greater activity of specific groups of people."</p>
            <p> We can agree with this conclusion.&#x00a0; In short, it is mainly changes in the institutional environment and changes in the mentality of contracting authorities that can induce the necessary changes in the behavior of public procurement actors. In this respect, the conclusions of the policy brief are inspiring for other countries that are faced with a similar question of how to effectively achieve public policy objectives through appropriately set evaluation criteria for public procurement.</p>
            <p> Conclusion:</p>
            <p> In conclusion to the reviewed text "Contract award criteria in public procurement procedures &#x2013; The possibility of improving the situation of society from the perspective of the European Union and Poland", I would like to state that it is an inspiring and stimulating policy brief. I recommend indexing it.&#x00a0; If the author finds in my evaluation some suggestions for improving his study, then I warmly welcome such an initiative.</p>
            <p>Does the paper provide a comprehensive overview of the policy and the context of its implementation in a way which is accessible to a general reader?</p>
            <p>Yes</p>
            <p>Is the discussion on the implications clearly and accurately presented and does it cite the current literature?</p>
            <p>Yes</p>
            <p>Are the recommendations made clear, balanced, and justified on the basis of the presented arguments?</p>
            <p>Yes</p>
            <p>Reviewer Expertise:</p>
            <p>public administration, public procurement</p>
            <p>I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.</p>
        </body>
        <back>
            <ref-list>
                <title>References</title>
                <ref id="rep-ref-315637-1">
                    <label>1</label>
                    <mixed-citation publication-type="journal">
                        <person-group person-group-type="author"/>:
                        <article-title>Risk aversion in first-price security-bid auctions</article-title>.
                        <source>
                            <italic>Operations Research Letters</italic>
                        </source>.<year>2024</year>;<volume>55</volume>:
                        <elocation-id>10.1016/j.orl.2024.107121</elocation-id>
                        <pub-id pub-id-type="doi">10.1016/j.orl.2024.107121</pub-id>
                    </mixed-citation>
                </ref>
                <ref id="rep-ref-315637-2">
                    <label>2</label>
                    <mixed-citation publication-type="journal">
                        <person-group person-group-type="author"/>:
                        <article-title>Historical-evolutionary and Retrograde Approach to the Study of Social Phenomena and Public Administration</article-title>.
                        <source>
                            <italic>NISPAcee Journal of Public Administration and Policy</italic>
                        </source>.<year>2022</year>;<volume>15</volume>(<issue>2</issue>) :
                        <elocation-id>10.2478/nispa-2022-0011</elocation-id>
                        <fpage>1</fpage>-<lpage>24</lpage>
                        <pub-id pub-id-type="doi">10.2478/nispa-2022-0011</pub-id>
                    </mixed-citation>
                </ref>
                <ref id="rep-ref-315637-3">
                    <label>3</label>
                    <mixed-citation publication-type="journal">
                        <person-group person-group-type="author"/>:
                        <article-title>Small arms procurement and corruption in small NATO countries</article-title>.
                        <source>
                            <italic>Journal of Public Procurement</italic>
                        </source>.<year>2024</year>;<volume>24</volume>(<issue>3</issue>) :
                        <elocation-id>10.1108/JOPP-04-2024-0045</elocation-id>
                        <fpage>348</fpage>-<lpage>370</lpage>
                        <pub-id pub-id-type="doi">10.1108/JOPP-04-2024-0045</pub-id>
                    </mixed-citation>
                </ref>
                <ref id="rep-ref-315637-4">
                    <label>4</label>
                    <mixed-citation publication-type="journal">
                        <person-group person-group-type="author"/>:
                        <article-title>Historical institutionalism: a tool for researching the nonprofit sector in times of pandemic</article-title>.
                        <source>
                            <italic>Innovation: The European Journal of Social Science Research</italic>
                        </source>.<year>2022</year>;
                        <elocation-id>10.1080/13511610.2022.2052027</elocation-id>
                        <fpage>1</fpage>-<lpage>15</lpage>
                        <pub-id pub-id-type="doi">10.1080/13511610.2022.2052027</pub-id>
                    </mixed-citation>
                </ref>
                <ref id="rep-ref-315637-5">
                    <label>5</label>
                    <mixed-citation publication-type="journal">
                        <person-group person-group-type="author"/>:
                        <article-title>Advances in prospect theory: Cumulative representation of uncertainty</article-title>.
                        <source>
                            <italic>Journal of Risk and Uncertainty</italic>
                        </source>.<year>1992</year>;<volume>5</volume>(<issue>4</issue>) :
                        <elocation-id>10.1007/BF00122574</elocation-id>
                        <fpage>297</fpage>-<lpage>323</lpage>
                        <pub-id pub-id-type="doi">10.1007/BF00122574</pub-id>
                    </mixed-citation>
                </ref>
                <ref id="rep-ref-315637-6">
                    <label>6</label>
                    <mixed-citation publication-type="journal">
                        <person-group person-group-type="author"/>:
                        <article-title>Can Psychological Contracts Decrease Opportunistic Behaviors?</article-title>.
                        <source>
                            <italic>Front Psychol</italic>
                        </source>.<year>2022</year>;<volume>13</volume>:
                        <elocation-id>10.3389/fpsyg.2022.911389</elocation-id>
                        <fpage>911389</fpage>
                        <pub-id pub-id-type="pmid">35719522</pub-id>
                        <pub-id pub-id-type="doi">10.3389/fpsyg.2022.911389</pub-id>
                    </mixed-citation>
                </ref>
                <ref id="rep-ref-315637-7">
                    <label>7</label>
                    <mixed-citation publication-type="journal">
                        <person-group person-group-type="author"/>:
                        <article-title>Taking ideas and discourse seriously: explaining change through discursive institutionalism as the fourth &#x2018;new institutionalism&#x2019;</article-title>.
                        <source>
                            <italic>European Political Science Review</italic>
                        </source>.<year>2010</year>;<volume>2</volume>(<issue>01</issue>) :
                        <elocation-id>10.1017/S175577390999021X</elocation-id>
                        <pub-id pub-id-type="doi">10.1017/S175577390999021X</pub-id>
                    </mixed-citation>
                </ref>
                <ref id="rep-ref-315637-8">
                    <label>8</label>
                    <mixed-citation publication-type="journal">
                        <person-group person-group-type="author"/>:
                        <article-title>Discursive Institutionalism: The Explanatory Power of Ideas and Discourse</article-title>.
                        <source>
                            <italic>Annual Review of Political Science</italic>
                        </source>.<year>2008</year>;<volume>11</volume>(<issue>1</issue>) :
                        <elocation-id>10.1146/annurev.polisci.11.060606.135342</elocation-id>
                        <fpage>303</fpage>-<lpage>326</lpage>
                        <pub-id pub-id-type="doi">10.1146/annurev.polisci.11.060606.135342</pub-id>
                    </mixed-citation>
                </ref>
                <ref id="rep-ref-315637-9">
                    <label>9</label>
                    <mixed-citation>
                        <person-group person-group-type="author"/>:
                        <article-title>An Empirical Analysis of Post-Contractual Behaviour in Public Works Contracts: The Czech Case as a Secondary Comparison with the Slovak Republic</article-title>.
                        <source>
                            <italic>https://www.researchgate.net/publication/310773774_An_Empirical_Analysis_of_Post-Contractual_Behaviour_in_Public_Works_Contracts_The_Czech_Case_as_a_Secondary_Comparison_with_the_Slovak_Republic/citation/download</italic>
                        </source>.<year>2016</year>;</mixed-citation>
                </ref>
                <ref id="rep-ref-315637-10">
                    <label>10</label>
                    <mixed-citation>
                        <person-group person-group-type="author"/>:
                        <article-title>Explaining Institutional Change</article-title>.
                        <source>
                            <italic>https://www.cambridge.org/in/universitypress/subjects/politics-international-relations/comparative-politics/explaining-institutional-change-ambiguity-agency-and-power?format</italic>
                        </source>.<year>2009</year>;</mixed-citation>
                </ref>
                <ref id="rep-ref-315637-11">
                    <label>11</label>
                    <mixed-citation publication-type="journal">
                        <person-group person-group-type="author"/>:
                        <article-title>Institutions, Institutional Change and Economic Performance</article-title>.<year>1990</year>;
                        <elocation-id>10.1017/CBO9780511808678</elocation-id>
                        <pub-id pub-id-type="doi">10.1017/CBO9780511808678</pub-id>
                    </mixed-citation>
                </ref>
            </ref-list>
        </back>
        <sub-article article-type="response" id="comment12337-315637">
            <front-stub>
                <contrib-group>
                    <contrib contrib-type="author">
                        <name>
                            <surname>Brzezi&#x0144;ski</surname>
                            <given-names>Mateusz</given-names>
                        </name>
                        <aff>Not Applicable, Poland</aff>
                    </contrib>
                </contrib-group>
                <author-notes>
                    <fn fn-type="conflict">
                        <p>
                            <bold>Competing interests: </bold>There are no competing interests</p>
                    </fn>
                </author-notes>
                <pub-date pub-type="epub">
                    <day>30</day>
                    <month>8</month>
                    <year>2024</year>
                </pub-date>
            </front-stub>
            <body>
                <p>Dear Professor,</p>
                <p> </p>
                <p> I would like to express my sincere gratitude for the insightful analysis and review of the article I prepared. I will incorporate the publications you mentioned in your review into my bibliography, as they indeed respond to the questions and issues highlighted in my article in an interesting and relevant manner.</p>
                <p> </p>
                <p> I hope that in the future we will have the opportunity to collaborate in the field of public procurement, and perhaps we can create a joint publication. It is extremely interesting to compare our countries in this regard &#x2013; the challenges we face and how we attempt (or rather, do not attempt) to address them.</p>
                <p> </p>
                <p> Yours respectfully,</p>
                <p> Mateusz Brzezi&#x0144;ski</p>
            </body>
        </sub-article>
    </sub-article>
    <sub-article article-type="reviewer-report" id="report307651">
        <front-stub>
            <article-id pub-id-type="doi">10.5256/f1000research.168873.r307651</article-id>
            <title-group>
                <article-title>Reviewer response for version 1</article-title>
            </title-group>
            <contrib-group>
                <contrib contrib-type="author">
                    <name>
                        <surname>Caranta</surname>
                        <given-names>Roberto</given-names>
                    </name>
                    <xref ref-type="aff" rid="r307651a1">1</xref>
                    <role>Referee</role>
                </contrib>
                <aff id="r307651a1">
                    <label>1</label>Turin University, Turin, Italy</aff>
            </contrib-group>
            <author-notes>
                <fn fn-type="conflict">
                    <p>
                        <bold>Competing interests: </bold>No competing interests were disclosed.</p>
                </fn>
            </author-notes>
            <pub-date pub-type="epub">
                <day>14</day>
                <month>8</month>
                <year>2024</year>
            </pub-date>
            <permissions>
                <copyright-statement>Copyright: &#x00a9; 2024 Caranta R</copyright-statement>
                <copyright-year>2024</copyright-year>
                <license xlink:href="https://creativecommons.org/licenses/by/4.0/">
                    <license-p>This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.</license-p>
                </license>
            </permissions>
            <related-article ext-link-type="doi" id="relatedArticleReport307651" related-article-type="peer-reviewed-article" xlink:href="10.12688/f1000research.153917.1"/>
            <custom-meta-group>
                <custom-meta>
                    <meta-name>recommendation</meta-name>
                    <meta-value>approve-with-reservations</meta-value>
                </custom-meta>
            </custom-meta-group>
        </front-stub>
        <body>
            <p>The article briefly highlights the framework for using award criteria in SRPP under EU and Polish law. It is clearly written, but for the phrase "Interestingly, under contract award criteria it is not possible to reward employment based on an employment contract. This is due&#x00a0; &#x2026;".&#x00a0;</p>
            <p> It would benefit from articulating what are the social benefits we are talking about.&#x00a0;</p>
            <p> The reference list might be beefed up.</p>
            <p>Does the paper provide a comprehensive overview of the policy and the context of its implementation in a way which is accessible to a general reader?</p>
            <p>Partly</p>
            <p>Is the discussion on the implications clearly and accurately presented and does it cite the current literature?</p>
            <p>Partly</p>
            <p>Are the recommendations made clear, balanced, and justified on the basis of the presented arguments?</p>
            <p>Yes</p>
            <p>Reviewer Expertise:</p>
            <p>Comparative administrative law, EU law, public procurement</p>
            <p>I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.</p>
        </body>
        <sub-article article-type="response" id="comment12280-307651">
            <front-stub>
                <contrib-group>
                    <contrib contrib-type="author">
                        <name>
                            <surname>Brzezi&#x0144;ski</surname>
                            <given-names>Mateusz</given-names>
                        </name>
                        <aff>Not Applicable, Poland</aff>
                    </contrib>
                </contrib-group>
                <author-notes>
                    <fn fn-type="conflict">
                        <p>
                            <bold>Competing interests: </bold>There are no competing interests</p>
                    </fn>
                </author-notes>
                <pub-date pub-type="epub">
                    <day>22</day>
                    <month>8</month>
                    <year>2024</year>
                </pub-date>
            </front-stub>
            <body>
                <p>Dear Professor,</p>
                <p> Thank you very much for reading the article, reviewing it and providing tips.</p>
                <p> </p>
                <p> I will try to provide an improved version as soon as possible.</p>
            </body>
        </sub-article>
    </sub-article>
</article>
