How Search Works

EventsHow Search WorksHome

Hey there, fellow developers and tech enthusiasts! Today, I want to take you behind the scenes of one of the most crucial features in many of my projects: the search functionality. If you’ve ever wondered how search algorithms work or how to approach implementing an efficient search in your own projects, you’re in for a treat. Let’s dive into the concepts and ideas that power my search function.

The Problem: Fuzzy Searching Across Multiple Data Types, Including Blog Posts

In many of my applications, I deal with various types of data - projects, social media links, achievements, and, importantly, blog posts! I needed a search function that could:

Handle different types of data efficiently, including full-text search of blog content
Perform fuzzy matching (to account for typos or partial matches)
Rank results based on relevance
Be fast and scalable
Ensure blog posts can compete with other data types

Let’s break down how I approached solving this problem.

The Solution: A Custom Fuzzy Search Algorithm

Step 1: Data Structure

First, I needed a flexible data structure to represent various types of content. Think of it as a blueprint that includes fields like:

Title
Description
Year
Keywords
And other relevant fields depending on the content type

This flexible structure allows the search to handle everything from project descriptions to social media links and blog post content.

Step 2: Fuzzy Matching

The core of the search functionality lies in fuzzy matching. I implemented two main algorithms for this:

Levenshtein Distance: This algorithm calculates how different two strings are by counting the minimum number of single-character edits required to change one word into another.

Pseudo-code idea:

function levenshteinDistance(string1, string2) {
  // Create a matrix of distances
  // Fill the matrix based on character comparisons
  // Return the bottom-right cell of the matrix
}

Jaccard Similarity: This algorithm compares the similarity of two sets of words by looking at the size of the intersection divided by the size of the union of the word sets.

Pseudo-code idea:

function jaccardSimilarity(string1, string2) {
  // Convert strings to sets of words
  // Calculate intersection and union of these sets
  // Return size of intersection / size of union
}

These algorithms allow the search to find matches even when the search query isn’t an exact match.

Step 3: Scoring System - Now with Blog Post Focus!

The heart of the search function is the scoring system. Each potential match is given a score based on various factors.

For general data items:

function calculateScore(searchString, item) {
  score = baseScore;
  if (exactMatch(item.title, searchString)) {
    score *= 100; // High bonus for exact title match
  }
  score += partialMatchScore(item.title, searchString);
  score += descriptionMatchScore(item.description, searchString);
  score += keywordMatchScore(item.keywords, searchString);
  // Add more scoring rules for other fields
  return score;
}

However, blog posts require special consideration:

Base Score: Blog posts start with a higher base score to ensure they’re not overshadowed by other data types.
Exact Phrase Matching: A significant bonus is awarded if the entire search query appears in the blog post content.
Relevance Scoring: Scores are increased based on how many search terms appear, with priority given to terms found in the title or earlier in the query.
Term Density: The more times search terms appear, the better the score.

Pseudo-code idea for blog posts:

function calculateBlogPostScore(searchString, blogPostContent) {
  score = 700; // Higher base score
 
  if (blogPostContent.includes(searchString)) {
    score += 1000; // Big bonus for exact match
  }
 
  searchTerms.forEach((term, index) => {
    if (blogPostContent.includes(term)) {
      score += (searchTerms.length - index) * 20; // Relevance boost
 
      if (blogPostTitle.includes(term)) {
        score += 50; // Title bonus
      }
    }
  });
 
  score += termDensity * 5; // Boost for density
  return score;
}

Higher weights are given to matches in more important fields (like title) and exact matches. This ensures that the most relevant results appear at the top.

Step 4: Searching Across Different Content Types

One of the challenges was searching across different types of content (projects, social links, etc.). The solution involves iterating through all content types:

Pseudo-code idea:

function searchAllContent(searchString, data) {
  results = [];
  for (const contentType in data) {
    for (const item of data[contentType]) {
      const score = calculateScore(searchString, item); //Or CalculateBlogPostScore
      results.push({ item: item, score: score, type: contentType });
    }
  }
  return results;
}

This approach allows the search to look through all content types efficiently.

Step 5: Ranking, Normalizing, and Returning Results

Finally, the results are sorted based on their scores, and the top N matches are returned. But there’s a crucial step in between: Normalization!

Score Normalization

Because blog posts and other content types use different scoring mechanisms (and potentially wildly different score ranges), it’s essential to normalize the scores before sorting.

Normalization involves scaling all scores to a common range (e.g., 1 to 100). This prevents one content type from dominating simply because its scoring system produces larger numbers.

Pseudo-code:

function normalizeScores(results, blogPosts) {
    // Find min and max scores across all results
    // Scale each score to the range of 1 to 100
    // Return normalized results
}

After normalization, results are sorted. The edge case when all scores are same is handled, and the sort uses the default value for score if the result item lacks score.

Top Results

Then the results are sorted and sliced:

Pseudo-code idea:

function getTopResults(results, n) {
  const normalized = normalizeScores(results,blogPosts);
  sortedResults = sortByScore(normalized);
  return topN(sortedResults, n);
}

Special consideration is given to always include social links and prioritize project results, as these are often the most relevant to users.

The API Endpoint

To make this search function accessible, an API endpoint is created:

Pseudo-code idea:

function handleSearchRequest(request) {
  query = getQueryParameter(request);
  n = getNumberOfResultsParameter(request);
  results = searchAllContent(query, globalData);
  blogPosts = searchBlogPosts(query, blogData); //New step
  topResults = getTopResults(results, blogPosts, n);  //blogposts also goes here
  return formatAsJsonResponse(topResults);
}

This allows other parts of the application (or even external services) to use this search functionality.

Conclusion

Implementing a custom search function was a challenging but rewarding experience. It allowed me to create a tailored solution that perfectly fits my needs. The combination of fuzzy matching algorithms, a custom scoring system (with blog post enhancements), and efficient data handling and score normalization results in a powerful and flexible search functionality.

Remember, this is just one approach to implementing search. Depending on your specific needs, you might want to explore other algorithms or even consider using dedicated search engines for larger-scale applications.

Happy coding!