Memory Leak?

Discussion in 'General' started by izrunas, Jun 16, 2009.

  1. izrunas

    Joined:
    Mar 4, 2009
    Messages:
    94
    Likes Received:
    0
    We are having one of our servers crash almost daily and it appears to correlate to high memory usage. Here's the chart:

    [​IMG]

    In the chart, the system locked up about 6 pm which was when the call center was stopping for the day. The system was rebooted and that's where the red memory line went back down low.

    We turned off openfire because of the threads talking about its issues, but that only seems to have slowed down the problem-- not ended it.

    Ideas?
     
  2. donhwyo

    Joined:
    Aug 8, 2008
    Messages:
    293
    Likes Received:
    0
    How much memory is in the machine? It only seems to get to 600 but we don't know how much you have. It should be able to use swap if it does fill up.

    Don
     
  3. izrunas

    Joined:
    Mar 4, 2009
    Messages:
    94
    Likes Received:
    0
    We built with 4Gb of physical memory in the machine-- and Elastix auto-configured it for 768Mb of Swap when it was installed.
     
  4. dicko

    Joined:
    Oct 24, 2008
    Messages:
    4,099
    Likes Received:
    0
  5. donhwyo

    Joined:
    Aug 8, 2008
    Messages:
    293
    Likes Received:
    0
    I would guess you have not run out of memory but are having other problems. You could run memtest off a linux install disk or live disk. I can't remember if it is available on the Elastix disk or not.

    You can run top to see what is using how much memory. It does show a steady increase in used memory but I think that is not abnormal.
     
  6. jgutierrez

    Joined:
    Feb 28, 2008
    Messages:
    5,737
    Likes Received:
    0
    Try setting a cronjob that clears cache:
    crontab -e
    0 */1 * * * sync; echo 3 > /proc/sys/vm/drop_caches
    That will clear your cache every hour... You wont have any problem, just a small amount of extra cpu processing
     
  7. izrunas

    Joined:
    Mar 4, 2009
    Messages:
    94
    Likes Received:
    0
    Thank you, it appears to be working well.
     
  8. izrunas

    Joined:
    Mar 4, 2009
    Messages:
    94
    Likes Received:
    0
    We continue to have to use this fix on a number of Elastix boxes. Is this indicative of some sort of bug? To be a little more specific, on any box we deploy which has more than a couple dozen extensions and one or more PRI connections, there seems to be a "memory leak" (slowly increasing memory usage) throughout a work day that causes servers to crash eventually. These are all high call volume systems (avg loading of 18-30 simul. trunk/channels during work hours).

    It seems silly to need to keep the machine from utilizing more than 1Gb of the 4Gb we typically install, but it seems to be necessary.
     
  9. dicko

    Joined:
    Oct 24, 2008
    Messages:
    4,099
    Likes Received:
    0
    I suggest you install sysstat and use sar (system activity report) to analyze memory usage, if you still only have 768M of swap and 1g of main, then it is likely that you are overloading your hardware for 30 simultaneous calls.
     
  10. izrunas

    Joined:
    Mar 4, 2009
    Messages:
    94
    Likes Received:
    0
    The lowest rated of the machines that need this "fix" have 4Gb of physical memory and at least 4Gb of swap space. All are now running 64bit version of Elastix and we use a variety of Rhino and Red-Fone equipment, along with IAX2 trunks for PSTN connectivity.
     
  11. dicko

    Joined:
    Oct 24, 2008
    Messages:
    4,099
    Likes Received:
    0
    so try the sar thing also vmstat, free (using the timer options) and of course ps aux

    I'm sorry I can't be more helpful but my similarly loaded machines don't exhibit this behavior
     
  12. Bob

    Bob

    Joined:
    Nov 4, 2007
    Messages:
    2,400
    Likes Received:
    1
    Similar to Dicko, in general none of our numerous machines exhibit this behaviour, especially of Elastix 1.5 upwards. 1.5 and 1.6 are based on later kernels which handle memory usage far more reliably.

    However, having said that, we have had issues with the standard Openfire included with the 1.5 and 1.6. which does definitely have a memory leak with the version included.

    Any client who wants to use openfire (e.g. they want to use presence etc), we upgrade the Openfire to the latest Openfire. This in all cases has prevented this issue occurring and based on the investigation that we did, it is to do with the Java memory space causing the issue.

    Hope this helps

    Bob
     
  13. dicko

    Joined:
    Oct 24, 2008
    Messages:
    4,099
    Likes Received:
    0
    As Bob says, it's something, Openfire the "Well Known" Culprit, others may qualify, but I doubt anything in core Elastix, normal memory diagnostics will show that "something" growing, so a specific diagnostic question might be, "what do you add to your servers?"
     
  14. izrunas

    Joined:
    Mar 4, 2009
    Messages:
    94
    Likes Received:
    0
    Actually, we neither add anything to the servers, nor do we run openfire. Actually, we generally disable or uninstall both Openfire and FOP.
     
  15. voya

    Joined:
    Apr 6, 2009
    Messages:
    15
    Likes Received:
    0
  16. ismed

    Joined:
    Oct 1, 2009
    Messages:
    120
    Likes Received:
    0
    I have the same problem with my elastix 1.6 (32bit) that installed on HP Proliant server G3 with 756MB RAM, the free RAM consistently decreasing 1MB per hour. When the free RAM remain under 100MB the respond from elastix is slowing down and it will totally crashed when the RAM is remaining only 8MB after several hours. So I have to reboot this server once a day to keep it alive for the last three days since I installed it.

    I do not face that problem on the other Elastix 1.6 (32bit) that installed on ordinary PC with mini atx intel mobo, intel core 2duo procie and 4GB RAM. This PC is already run 3 months without any problem or rebooting/restarting.

    Is there solution to this problem?

    Thanks
    Ismed
     

Share This Page