Search engine optimization: methods and techniques

Context and importance of search engine optimisation

The exponential growth of online content has led to significant challenges in information retrieval. Jaime Teevan et al. (2010) address in their article, titled “Slow Search: Information Retrieval without Time Constraints,” the significance of speed in online searching and how even a slight retrieval latency from search engines can negatively impact users’ perception of result quality. The authors delve into the concept of “slow search,” where traditional speed requirements are relaxed in favour of a high-quality search experience. They underline that while speed is crucial, there are scenarios where result quality takes precedence over speed. This presents opportunities for search systems to facilitate novel search experiences with in-depth, high-quality content that requires time to identify. This notion of slow search has the potential to revolutionise users’ online search behaviours, emphasising a more thorough approach.

This approach aligns with the notion that today, users have high expectations in terms of the relevance and speed of search results.³ An effective search engine must be able to sort and rank billions of web pages to deliver the most relevant information in the shortest possible time. Moreover, search engines must continually adapt to linguistic changes, emerging trends, and user preferences.³

This need for optimisation is exemplified by the dominance of certain players in the market. For instance, up to 2021, Google largely dominated the search engine market with a share of over 90% in many regions, notably in Europe.⁴ Other engines like Bing held a much smaller share, often less than 5% in numerous regions.⁵ However, some private search engines, such as DuckDuckGo and StartPage, have seen steady growth in popularity. Though they still represent a small fraction of the market compared to Google, their growth underscores a rising awareness of the importance of online privacy, with DuckDuckGo holding a share of around 2-3% in the US in 2021.⁶

Issues

Search engine optimisation aims to meet these challenges by improving the performance and functionality of the search engine. It encompasses a wide range of techniques and methods designed to improve the relevance of results, search speed and user experience. The main objective of optimisation is to enable users to find the information they need quickly and efficiently, while providing a pleasant and personalised search experience.

Hypothesis

The results of a query emanating from a search engine would certainly be relevant if the search engine integrated certain specificities linked to a generalised optimisation of the architecture and infrastructure of the said engine in terms of adaptability to the context of evolution and the number of requests for information per second.

In this case, we are referring to the thematic circumscription at the heart of this article, the optimisation of a search engine to facilitate the smooth running of the engine, and to adapt its infrastructure to current and future needs in terms of much greater use, taking into account the increasing number of users.

Objectives of the article

In this scientific paper we focus on search engine optimisation and provide a detailed study of the methods and techniques used in this field. The aim is to examine the different optimisation approaches for improving search engine performance and to provide practical recommendations based on case studies and concrete examples.

More specifically, this article sets the following objectives:

Key components of a search engine

To understand search engine optimisation, it is essential to understand the key components that make up a search engine. A typical search engine is made up of the following elements:

Search engine workflow

The search engine workflow generally follows the following steps:

Basic challenges and limitations

The design and optimisation of a search engine faces a number of challenges and limitations, including:

The issue of search engines has generated a fair amount of interest in the world of research. In his research, Belmond DJOMO presents the search engine as an assembly of algorithms and tools for responding to a request for information.⁷ He develops a theoretical approach that exploits scientific methods and empirical approaches to conceptualise the optimisation of the relevance of the results of a thematic search engine by aggregating existing tools. PICAROUGNE et al. have developed a generic search engine called “GénieMinier”.⁸ This search engine references all information relating to medical science. In implementing this engine, the authors have carried out a comparative study in which they show the important relationships that exist between statistical studies of Web properties, search engines based on agent-based approaches, and the techniques conventionally used in optimisation. Dirk Lewandowski et al. have focused instead on optimising the fundamental elements of quality and communication policy within search engines.⁹ They specify that the theoretical framework for analysing the quality of a search engine comprises four areas: the quality of indexing, the effectiveness of information retrieval, the quality of search functionalities and the usability of the engine in question. Urruty Thierry proposes two different approaches. The first uses a multidimensional indexing structure adapted to a heterogeneous distribution of data. In a second approach, he also proposes using a structural distance between two XML documents to obtain a preliminary classification of all the documents.¹⁰ Similarly, Agnès MAGRON, Mehdi Jaoua and Florent Verroust, in their work, have focused on optimisation and technical aspects enabling machines to access the contents of databases.^11–13 Agnes M. used a protocol to facilitate the collection of metadata. Its basic operation is based on client-to-server communication. The server here being HAL, then HAL can be harvested in its entirety or in sets, which we call OAI sets, whereas Mehdi has focused more on execution time as a function of document size. In the same optimisation context, Léa Laporte et al. proposed an approach for optimising search engine results using the learning-to-rank algorithm. The special feature of this mechanism is that it presents search results in the form of a geographical map or a list of index cards. These records contain the characteristics of the location (name, address, telephone number, etc.).¹⁴ Noelie Di Cesare et al. used the PageRank-PSO (PR-PSO) algorithm in a benchmark of common lattices. This particle swarm optimisation algorithm uses a stochastic Markov chain to redefine the influence of particles within the swarm in real time.¹⁵ Laurence Longo has developed a module for the automatic identification of RefGen reference chains, which has been evaluated using current co-reference metrics.¹⁶

Indexing and semantic analysis

Indexing is a key process in which web pages are analysed and relevant information is extracted to build the search engine index. Optimising indexing involves using advanced techniques such as semantic analysis to understand the meaning of keywords and phrases, and the relationship between different pieces of content.

Advanced search techniques

Search engines can use advanced search techniques to improve the relevance of results. This can include fuzzy search, which finds results even when the search terms do not match exactly, and contextual search, which takes into account the context of the query to provide more accurate results.

Improving the relevance of results through machine learning

Machine learning can play a key role in improving the relevance of results. Search engines can use machine learning algorithms to analyse user usage patterns, understand their preferences and personalise results accordingly. In addition, machine learning can help improve spam detection, filter out irrelevant results and adjust rankings based on user feedback.

Parallel indexing and searching

Parallel indexing and searching are effective approaches to speeding up the search process. The main idea is to spread the workload over several processing nodes or servers, enabling several search tasks to be processed simultaneously and optimising the overall performance of the search engine.

For example, suppose we have a search engine that processes vast amounts of data. By using parallel indexing, each processing node can concentrate on a specific part of the index, in parallel with the other nodes. In this way, indexing can be carried out more quickly, allowing the index to be updated more frequently. Similarly, when a query is submitted, parallel searching allows the workload to be distributed between the nodes, reducing the overall response time.

Tools and frameworks such as Apache Hadoop and Apache Spark are commonly used to implement parallel indexing and searching, providing a distributed infrastructure for efficient, simultaneous data processing.

Optimising queries and the search algorithm

Optimising queries and the search algorithm is another essential approach to improving search speed. The aim is to optimise the way in which queries are processed and results are retrieved from the index.

For example, by using efficient data structures, such as binary search trees or hash tables, the search engine can speed up the index search. These structures enable faster searching by reducing the number of operations required to locate relevant results.

In addition, the search algorithm can be optimised to reduce query processing time. Techniques such as the pruning of irrelevant results, the use of intelligent heuristics to guide the search and the parallelization of search operations can help to speed up the overall performance of the search engine.

Caching and pre-loading results

Caching and pre-loading results are effective techniques for improving search speed by reducing response times for frequent queries. Caching consists of storing search results already obtained for specific queries in memory, so that they can be quickly retrieved for subsequent queries.

For example, let’s assume that a user regularly carries out searches on specific subjects. The search engine can cache the results of these searches, associated with the corresponding queries, to avoid having to repeat the same process each time. So when the user submits a similar query, the results can be instantly retrieved from the cache, reducing the overall response time.

In addition, result pre-loading can be used to anticipate common queries and pre-load relevant results into the cache before the user even requests them. For example, if a specific query is frequently submitted at a certain time of day, the relevant results can be pre-loaded before that time, ensuring an instant response when the user actually makes the query.

These caching and prefetching techniques require efficient management of memory and storage space. Strategies such as cache size management, cached data expiry policies and the use of intelligent replacement mechanisms can ensure the efficiency and consistency of cached results.

By combining parallel indexing and search, query and search algorithm optimisation, as well as caching and pre-loading of results, search engines can dramatically improve search speed, delivering a faster, more responsive user experience.

These optimisation techniques are widely used in the search engine industry, and their judicious application can lead to significant improvements in overall search engine performance and user satisfaction.

User-friendly interface design

A well-designed user interface plays a crucial role in the overall search experience. The aim is to provide an intuitive, user-friendly and attractive interface that allows users to navigate easily and interact effectively with the search engine. Here are some key principles for designing a user-friendly interface:

Personalising search results

Personalisation of search results makes it possible to adapt results according to the specific preferences and interests of each user. The aim is to provide relevant, personalised results, taking into account factors such as geographical location, language preferences, search histories and user profiles. Here are some commonly used methods for personalising search results:

Analysis of user interactions for continuous improvement

User interaction analysis enables us to collect data on how users interact with the search engine. This information can be used to continuously improve the user experience and performance of the search engine. Here are a few examples of user interaction analysis:

The analysis of user interactions enables decisions to be made based on concrete data, leading to continuous improvements in the user experience. Using this information, search engines can respond to users’ needs and preferences, offering a more personalised and satisfying search experience.

By optimising the user experience through user-friendly interface design, personalisation of search results and analysis of user interactions for continuous improvement, search engines can provide a high-quality user experience and respond effectively to user needs and expectations.

Case study 1: Optimising a search engine for a specific website

Let’s take the example of a company that has a complex website with a large amount of content and wants to optimise its internal search engine to deliver a better user experience. To do this, the company can take advantage of cloud architecture services such as Microsoft Azure to implement optimisation solutions.

In this case study, the company begins by using Azure Search, a fully managed search service, to improve the relevance of results. It configures index fields, adjusts search term weights and uses semantic analysis to understand synonyms and relationships between keywords. Using Azure Search’s machine learning capabilities, the company can also refine results based on user behaviour and preferences.

To optimise search speed, the company can use Azure Cognitive Search, which offers parallel indexing and search functionalities. By distributing the workload across several processing nodes, the search engine can deliver results more quickly, even when processing large amounts of data.

The enterprise can also use Azure Cache for Redis to cache frequently requested search results, reducing overall response time. By using the cached data preload and expiry features, the search engine can ensure that the most relevant results are quickly available to users.

By applying these optimisation techniques with Microsoft Azure cloud architecture services, the company can significantly improve the relevance and speed of its internal search engine, delivering an enhanced user experience and more efficient search.

Figure 1. Optimisation of a search engine for managing training catalogues, training providers and learners.

Optimisation of a search engine for managing training catalogues, training providers and learners.

Web Apps

This example scenario highlights the advantage of using a dedicated search service to increase the relevance of search results.

Search is the primary mechanism by which users enter keywords to search for relevant results. It is therefore essential that the search results are relevant to the intent of the search query and that the end-to-end user search experience matches the requirements of the search by providing near-rapid results, linguistic and vertical analysis, geographic matching, filtering, facets, autocomplete, match highlighting, etc.

Let’s imagine a typical web application with data stored in a relational database. Search queries are typically managed in the database using LIKE queries or LIKE features. With Azure Cognitive Search, we unleash the full power of the operational database from query processing and can easily take advantage of these complex-to-implement features that deliver the best search experience. What’s more, because Azure Cognitive Search is a Platform as a Service (PaaS) component, we don’t have to worry about infrastructure management.

Appropriate use cases

Other appropriate uses are as follows:

Finally, it makes it easy for any application that includes any form of search functionality to take advantage of the dedicated search service.

So we opted to optimise the existing infrastructure.

Modelling

This scenario covers a search solution in which users can use catalogues to search for information on training courses, trainers, training organisations and learners…

Figure 2. New architecture proposed to optimise the infrastructure and the search engine.

Users can access the engine from any device. The training catalogue of training organisations is stored in an Azure SQL database, which forms the datamart for transactional processing. Cognitive search Azure uses a search indexer to automatically keep its search index up to date via integrated change tracking. User search queries are directed to the Azure Cognitive Search service, which processes the query and returns the most relevant results. In addition to the web-based search experience, users can also use a conversational bot in a social network or directly from digital assistants to search for training courses, external training organisations and incrementally refine their search query and results. We are also using the skills functionality to apply artificial intelligence for even more intelligent processing.

Components of the new architecture

We offer optimisation solutions based on cloud applications:

Which is a Microsoft Azure (Cloud) solution that will host web applications, enabling automatic scaling and high availability without on-site infrastructure management.

This is a general-purpose relational database management service from Microsoft Azure, which supports structures such as relational, JSON, spatial and XML data. It serves as our Datamart

It’s a search-as-a-service (SAAS) cloud solution that provides a rich search experience for private and heterogeneous content in web, mobile and enterprise applications. The solution is compatible with numerous data security and confidentiality standards. To secure access to the service, the solution uses two types of key: administration keys, which allow us to perform any task on the service, and query keys, which can only be used for read-only operations such as querying. In general, the application performing the search does not update the index. It should therefore only be configured with a query key and not an administration key (especially if the search is performed from an end-user device such as a script running in a web browser).

It’s an Azure cloud solution that provides tools for creating, testing, deploying and managing intelligent robots.

It’s a service that uses intelligent algorithms to see, hear, state, understand and interpret users’ needs using natural modes of communication.

Azure Synapse is an enterprise analytics service that shortens the time to analysis across data warehouses and Big Data systems. It also offers a high degree of integration with other Azure services such as Power BI, Cosmos DB and Azure ML.

Solutions

Search relevance based on infrastructure development and engine optimisation. The effectiveness of our search engine depends to a large extent on the relevance of search results for users. By carefully tuning our search service to deliver optimal results based on the user’s search query, or by using built-in features such as search traffic analysis to understand our users’ search patterns, we can make data-driven decisions. Typical methods of parameterising our search service include the following:

The evolution of the infrastructure has enabled us to resolve the problems of duplicate content, by using 301 redirects, replacing the use of canonical tags with a more suitable Azure cognitive services solution, and using Azure bot service instead of robots.txt. The use of Azure App service solved the problem of latency in loading web pages, which are supposed to return the results of user requests quickly.

Selecting performance indicators

Performance indicators are measures used to evaluate search engine results and performance. In our study, we selected the following performance indicators:

Recall (R): The number of correct positive results divided by the number of positive results that should have been returned (i.e., the proportion of actual positives that were correctly identified).

The F-measure is calculated as:

The F-measure returns a value between 0 and 1, where 1 indicates perfect precision and recall, and 0 indicates neither precision nor recall.

There are also variations of the F-measure, such as F2-score and F0.5-score, which weigh recall higher than precision or vice versa. The general formula for these is:

Collecting data and setting up the experimental environment

To conduct our experiments, we have collected representative data, such as sets of real search queries and corresponding document corpora. This data will enable us to assess the relevance of the results and measure the performance of the search engine.

We set up an experimental environment based on the Microsoft Azure architecture, using components such as Azure Cognitive Search, Azure Cache for Redis and other relevant services. We configured these components according to the needs of our study, adjusting the indexing, search and caching parameters for each optimisation technique evaluated.

Optimisation parameters and metrics used

For each optimisation technique evaluated, we defined specific parameters for configuring the search engine components. For example, for indexing and semantic analysis optimisation, we defined the appropriate indexing fields, term weights and semantic analysis parameters.

To evaluate the search engine’s performance, we used the metrics mentioned above. We calculated recall, precision and F-measure by comparing the results returned by the search engine with previously established reference results. We measured the average response time for each search query, recording individual response times and averaging them.

To evaluate the user experience, we have used surveys or usability evaluations to gather users’ impressions and reactions on the interface. We used rating scales and open-ended questions to gather subjective information on user satisfaction and ease of use.

To evaluate the user experience, we carried out surveys and usability evaluations targeting a diverse group of participants. Our participants primarily comprised two groups: members of the general public and academic faculty. We believed that capturing insights from both the general population and individuals with academic backgrounds would yield a more comprehensive understanding of the user experience across varying degrees of familiarity and expertise with similar interfaces.

Before launching our study, we meticulously reviewed the ethical guidelines stipulated by Laboratory AIDE of Universite Virtuelle de Cote d’Ivoire. We sought to ensure our study adhered to the highest standards of research ethics, particularly given the subjective nature of the evaluations.

Regarding the consent process, participants were provided with an information sheet detailing the purpose of the study, what would be expected of them, the nature of the questions they’d encounter, and assurances of data anonymity and confidentiality. They were informed that their participation was entirely voluntary and that they had the right to withdraw from the study at any point without facing any repercussions.

After being given adequate time to read the information sheet and ask any clarifying questions, participants were asked to give their consent. For the sake of clarity and record-keeping, we opted for written consent. Participants signed a consent form, which was then stored securely to maintain a record of their agreement to participate.

Once the consent was obtained, participants proceeded with the usability evaluations. Our approach combined rating scales for quantitative feedback and open-ended questions for qualitative insights, allowing us to gauge both specific and general sentiments about user satisfaction and the interface’s ease of use.

The experimental methodology we followed enabled us to evaluate the performance and results of different search engine optimisation techniques. The performance indicators selected, the data collection and the experimental environment set up enabled an in-depth evaluation of the relevance of the results, the search speed and the user experience. The optimisation parameters and metrics used provided quantitative and qualitative measures to assess the performance and improvements brought about by each optimisation technique.

Analysis of the results obtained for each optimisation technique

Relevance of results:

Search speed:

User experience:

We will outline the differences between the two sections:

Title & Focus:

First Section: It focuses on an analysis of the results obtained for each optimisation technique. Second Section: It zeroes in on comparing performance before and after optimisation.

Depth of details:

First Section: Provides an overview of the observed improvements post-optimisation without detailing specific figures from before the optimisation. Second Section: Delivers more specific insights, showcasing exact percentages and figures from both before and after the optimisation.

Metrics used:

First Section: Centers on recall, precision, average response time, and usability scores. Second Section: Expounds further on metrics, introducing exact figures like a 72% recall rate before optimisation.

Format:

First Section: Articulated in a more general manner, offering a panoramic view of improvements. Second Section: Takes on a more analytical stance, breaking down each metric in detail and delivering precise comparisons between pre- and post-optimisation performances.

Mention of the technology used:

First Section: Does not mention any specific technology or platform. Second Section: Refers to the utilisation of Microsoft Azure architecture and Azure components.

Nature of information provided:

First Section: Is more concise, perhaps tailored for an audience seeking a general overview without diving into technical specifics. Second Section: Is richer in detail and more technical, possibly geared towards a specialised audience or stakeholders keen on understanding the specific enhancements made.

In summary, the first section offers a general glimpse of the post-optimisation enhancements, while the second section delivers a detailed comparative analysis, clearly spotlighting the benefits of the optimisations with exact figures and percentages.

Comparison of performance before and after optimisation

Comparing performance before and after optimisation, we found a significant improvement in all aspects assessed. The relevance of results improved, with an increase in recall and precision. After making the necessary adjustments to our system, we observed an increase in recall from 72% to 89% and an increase in precision from 78% to 92%. Search speed also improved, with a significant reduction in average response time. Users expressed greater satisfaction and found the interface more user-friendly after the optimisation.

Comparing performance before and after optimisation, we observed a marked enhancement across all evaluated dimensions. Specifically:

Overall, these numerical outcomes underscore the tangible benefits derived from our optimisation efforts, leading to a more efficient, effective, and user-centric search experience.

This comparison demonstrates the effectiveness of the optimisation techniques used, exploiting the functionalities of the Microsoft Azure architecture and Azure components. Search engine performance has been significantly improved, providing a more satisfying search experience for users.

Discussion of the advantages and limitations of the different techniques

In the discussion, we will highlight the advantages and limitations of the different optimisation techniques used. Optimising indexing and semantic analysis has improved the relevance of results, but may require more complex data and algorithm management. Optimising search speed using parallel search and indexing, as well as caching and preloading results, has reduced response times, but may require additional resources and effective cache management.

It is important to note that the benefits and limitations of different techniques may vary depending on the context and specific objectives. Results and performance can also be influenced by other factors such as data size, complexity of search queries and user profiles.

The results obtained from optimising a search engine using Microsoft Azure architecture and Azure components have demonstrated significant improvements in the relevance of results, search speed and user experience. The various optimisation techniques have their specific advantages, but also their limitations. By considering these factors, practitioners can choose the best optimisation strategies to achieve their search engine objectives, providing an optimal search experience for end users.

Summary of the main contributions of the article

We embarked on a journey to delve deep into the intricacies of search engine optimisation techniques, placing special emphasis on the architecture offered by Microsoft Azure and its associated components. To assess the efficacy of these techniques, we introduced an experimental methodology tailored to evaluate the performance outcomes of various optimisation strategies. The results garnered from each technique were meticulously analysed. What emerged was a clear testament to the prowess of our approach - there was a marked enhancement in the relevance of search results, the speed of searches, and the overall user experience.

Delving into the horizon of future prospects in search engine optimisation, a plethora of exciting avenues unfolds. Notably, harnessing the potential of machine learning and artificial intelligence stands out as a pivotal measure to elevate the relevance of search results and further personalise the search experience. Additionally, there’s an untapped potential in exploring novel cloud architectures and components. Such exploration could revolutionise the performance and scalability of search engines. Furthermore, a contextual analysis of how optimisation techniques resonate in specific scenarios, be it retrieving multimedia information or probing into specialised fields, is imperative. A deeper dive into understanding user interactions and satisfaction metrics will pave the way for an enriched user experience that resonates with the evolving demands of users.

In essence, leveraging the Microsoft Azure architecture and its components for search engine optimisation presents a promising frontier. With a focus on amplifying the relevance of results, expediting searches, and crafting an unparalleled user experience, this domain beckons continuous innovation. The dynamic nature of this field ensures a perpetual evolution, providing ample scope for incessant enhancement in search engine optimisation.

Search engine optimisation using the Microsoft Azure architecture and Azure components offers opportunities to improve the relevance of results, search speed and user experience. This area of research continues to evolve, offering many opportunities for continuous improvement and innovation in search engine optimisation.