Question

Performance issue with Manticore Search

Hello, the response time of my index on Manticore Search has increased a lot recently (some requests take several seconds). There are about 40 million documents currently indexed. How can I increase the performance of Manticore Search? Is there any configuration I can do that would improve it? I have put below my current configuration.

index my_index
{
    type                = rt
    path                = /mnt/.../manticore/data/my_index

    rt_field            = field1
    rt_field            = field2
    stored_fields       =

    rt_attr_timestamp   = attr1
    rt_attr_string      = attr2
    rt_attr_string      = attr3
    rt_attr_string      = attr4

    charset_table       = non_cjk
    ngram_len           = 1
    ngram_chars         = cjk

    rt_mem_limit        = 2048M
}

searchd
{
    listen              = 127.0.0.1:9312
    listen              = 127.0.0.1:9306:mysql41
    log                 = /var/log/manticore/searchd.log
    pid_file            = /var/run/manticore/searchd.pid
    seamless_rotate     = 1
    preopen_tables      = 1
    unlink_old          = 1
    net_workers         = 2
    binlog_path         = /mnt/.../manticore/data
    rt_flush_period     = 3600
}

Submit an answer


This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

KFSys
Site Moderator
Site Moderator badge
April 10, 2023

Hey @penguin3523,

Apart from has already been said, here are some general suggestions I give to others when they face such issues,

  1. Increase the value of the rt_mem_limit parameter: The rt_mem_limit parameter sets the amount of memory that can be used by the real-time index. Increasing this value can help improve the performance of the index, especially if you have a large number of documents. You can try increasing this value to 4096M or even higher to see if it improves the performance.

  2. Optimize your search queries: Complex search queries can take longer to execute and slow down the performance of your index. Try to simplify your search queries and avoid using unnecessary operators or conditions. Also, consider using query caching to speed up your queries.

  3. Optimize your indexing pipeline: The way you index your documents can also affect the performance of your index. Make sure you are using the most efficient indexing method for your data and consider using bulk indexing for large datasets. Also, try to avoid unnecessary indexing fields or attributes.

  4. Consider using distributed indexing and search: If you have a large dataset, you can consider using distributed indexing and search to distribute the workload across multiple nodes. This can help improve the performance of your index and reduce the response time for search queries.

  5. Tune your searchd settings: You can try tuning the searchd settings to optimize the performance of your index. For example, you can increase the number of worker threads (net_workers) to improve parallelism or adjust the rt_flush_period to reduce the frequency of index flushing.

  6. Monitor your system resources: Make sure you monitor the CPU, memory, and disk usage of your system to identify any potential bottlenecks or performance issues. You can use system monitoring tools like top or htop to monitor your system resources.

  7. Use a caching layer: Consider adding a caching layer to your application to reduce the number of queries hitting your Manticore Search index. This can help improve the response time of your application and reduce the load on your Manticore Search server.

alexdo
Site Moderator
Site Moderator badge
April 8, 2023

Hello @penguin3523

What you can do is to increase the mem_limit and the write_buffer An exact values for the config files will be.

  1. mem_limit = 3G # will consume max 2G
  2. write_buffer = 1G # will consume max 4*1 = 4G

Keep in mind that this is recommended only if your droplet has enough free memory at the moment. You can first analyse the performance of the droplet in terms of CPU and memory usage.

If the droplet is running low on memory or is having constant CPU spikes you can look into upgrading your droplet in order to boost the performance of your application.

The following articles will help to check the current resource usage on your droplet.

https://docs.digitalocean.com/support/why-does-my-droplet-have-high-cpu-or-ram-usage/

https://www.digitalocean.com/community/tutorials/how-to-monitor-cpu-use-on-digitalocean-droplets

Hope that this helps!

Try DigitalOcean for free

Click below to sign up and get $200 of credit to try our products over 60 days!

Sign up

Get our biweekly newsletter

Sign up for Infrastructure as a Newsletter.

Hollie's Hub for Good

Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.

Become a contributor

Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

Welcome to the developer cloud

DigitalOcean makes it simple to launch in the cloud and scale up as you grow — whether you're running one virtual machine or ten thousand.

Learn more
DigitalOcean Cloud Control Panel