[PageOneX] [dev] PageOneX server is down

Mon Dec 16 18:56:31 EST 2013

Ed, did you had a chance to look into those errors?

I was looking at the server performance.
The only thing I see is that memory and swap usage values are usually very
heigh, even when there are no requests to the website.
Almost steady values for a while: Mem 734/1076MB and Swap 139/255MB

CPU load average looks ok, under any problematic threshold, I think.

On Tue, Dec 10, 2013 at 11:38 PM, pablo rey <pablo at basurama.org> wrote:

> I wrote the sad story of a two days long thread in PageOneX :)
> https://github.com/numeroteca/pageonex/issues/179#issuecomment-30275654
>
>
> On Tue, Dec 10, 2013 at 8:20 PM, pablo rey <pablo at basurama.org> wrote:
>
>> Ok, I will add the steps to reproduce these errors in the already open
>> issue https://github.com/numeroteca/pageonex/issues/179
>>
>> I don't think it is related to the cpu throttling, these problems were
>> happening before.
>>
>>
>> On Tue, Dec 10, 2013 at 8:11 PM, Edward Platt <elplatt at media.mit.edu>wrote:
>>
>>> Pablo can you create issues for these bugs in github and make sure they
>>> contain steps to reproduce?  Let me know if you think we should revert the
>>> cpu throttling.  I'll put a little time into fixing those errors.
>>>
>>> -Ed
>>>
>>>
>>> On Tue, Dec 10, 2013 at 12:49 PM, pablo rey <pablo at basurama.org> wrote:
>>>
>>>> My experience with those problems:
>>>>
>>>> For long threads: Almost every time I create or update a thread (add
>>>> one day) I get a "500 Internal Server Error", after "creating background
>>>> image grid" according to the logs (sometime after "creating overlays for
>>>> topics").
>>>> When I refresh the display view sometimes it works, but I guess many
>>>> users leave the page after that (I've found many broken threads with no
>>>> even a single area coded). Sometimes after 5 or 6 times of refreshing, the
>>>> display view works again.
>>>>
>>>> For short threads (one or two days): it returns the error for the
>>>> composite images http://pageonex.com/numeroteca/test-repeated/ It is
>>>> not able to generate them 88 newspapers, one day).
>>>>
>>>> I don't know if the problem for short threads is related to the big
>>>> images that imagemagick has to create. If that is the case, we should limit
>>>> the total height of those images: to avoid the tool from crashing AND to
>>>> provide a good overview of the news coverage without the need of zooming
>>>> out too much.
>>>>
>>>> thanks Rafael for the hints!
>>>>
>>>> ps: Pagination in the thread list is not working again<https://github.com/numeroteca/pageonex/issues/182>,
>>>> maybe related to lack of memory?
>>>>
>>>>
>>>> On Tue, Dec 10, 2013 at 6:46 PM, Rafael Porres Molina <
>>>> rporres at gmail.com> wrote:
>>>>
>>>>> Request timeouts can cause internal server errors. You may look into
>>>>> that.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Rafa
>>>>>
>>>>>
>>>>> 2013/12/10 Edward Platt <elplatt at media.mit.edu>
>>>>>
>>>>>> I haven't seen any cpu usage notifications, and the server hasn't
>>>>>> crashed, so the throttling seems to be helping on that front.
>>>>>>
>>>>>> The errors and broken images you see could be from the throttling or
>>>>>> they could be a coincidence.  Any idea how widespread/common those problems
>>>>>> are?  If it is throttling, the error likely occurs when the server would
>>>>>> have otherwise crashed.
>>>>>>
>>>>>> Obviously we want to avoid both, so there will still need to be some
>>>>>> work around limiting the image size.
>>>>>>
>>>>>> -Ed
>>>>>>
>>>>>>
>>>>>> On Tue, Dec 10, 2013 at 7:41 AM, pablo rey <pablo at basurama.org>wrote:
>>>>>>
>>>>>>> Great, Ed, how is this CPU throttling working?
>>>>>>>
>>>>>>> I added one extra day to the thread
>>>>>>> http://pageonex.com/numeroteca/corrupcion-espana-diciembre-2013/and updated:
>>>>>>>
>>>>>>> 1st time: internal server error.
>>>>>>>
>>>>>>> I reloaded:
>>>>>>> 2nd time: the newspaper images and overlays were broken.
>>>>>>>
>>>>>>> I went back to edit and "updated" the thread
>>>>>>> 3rd time: it worked fine.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Dec 9, 2013 at 11:12 PM, Edward Platt <elplatt at media.mit.edu
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> I've added cpu throttling to the imagemagick settings.  Hopefully
>>>>>>>> this will help stop the crashes, but keep an eye out for any weird behavior!
>>>>>>>>
>>>>>>>> -Ed
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Dec 8, 2013 at 12:14 PM, pablo rey <pablo at basurama.org>wrote:
>>>>>>>>
>>>>>>>>> Hi Ed,
>>>>>>>>> Prevent overloading would be great.
>>>>>>>>> I think the server just went down again when editing the same
>>>>>>>>> thread.
>>>>>>>>>
>>>>>>>>> Let's talk next week if there is something I can look into also.
>>>>>>>>> p
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Dec 6, 2013 at 11:19 PM, Edward Platt <
>>>>>>>>> elplatt at media.mit.edu> wrote:
>>>>>>>>>
>>>>>>>>>> Hey Pablo!
>>>>>>>>>>
>>>>>>>>>> The server's back up.  We've been getting notifications about cpu
>>>>>>>>>> and disk usage, probably from resizing images.  Next week I'll look into
>>>>>>>>>> whether there's an easy fix to prevent overloading the server on long/short
>>>>>>>>>> threads.
>>>>>>>>>>
>>>>>>>>>> -Ed
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Dec 6, 2013 at 5:30 AM, pablo rey <pablo at basurama.org>wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Ed and Rahul,
>>>>>>>>>>> I was adding a new topic to the thread
>>>>>>>>>>> http://pageonex.com/numeroteca/corrupcion-espana-diciembre-2013/and the server went down.
>>>>>>>>>>>
>>>>>>>>>>> I can not ssh.
>>>>>>>>>>>
>>>>>>>>>>> Apart from the problem of lack of memory  to calculate the
>>>>>>>>>>> background image grid for too many or too few images, I think there is
>>>>>>>>>>> another problem with too many overlays when it has to create the composites.
>>>>>>>>>>>
>>>>>>>>>>> Please reboot over-9000 server!
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Pageonexdev mailing list
>>>>>>>>>>> Pageonexdev at mit.edu
>>>>>>>>>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Edward L. Platt
>>>>>>>>>> Civic Technology Programmer
>>>>>>>>>> MIT Center for Civic Media
>>>>>>>>>> E15-348
>>>>>>>>>> http://civic.mit.edu
>>>>>>>>>> http://elplatt.com
>>>>>>>>>> @elplatt <http://twitter.com/elplatt>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Pageonexdev mailing list
>>>>>>>>>> Pageonexdev at mit.edu
>>>>>>>>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Edward L. Platt
>>>>>>>> Civic Technology Programmer
>>>>>>>> MIT Center for Civic Media
>>>>>>>> E15-348
>>>>>>>> http://civic.mit.edu
>>>>>>>> http://elplatt.com
>>>>>>>> @elplatt <http://twitter.com/elplatt>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Edward L. Platt
>>>>>> Civic Technology Programmer
>>>>>> MIT Center for Civic Media
>>>>>> E15-348
>>>>>> http://civic.mit.edu
>>>>>> http://elplatt.com
>>>>>> @elplatt <http://twitter.com/elplatt>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Pageonexdev mailing list
>>>>>> Pageonexdev at mit.edu
>>>>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Pageonexdev mailing list
>>>>> Pageonexdev at mit.edu
>>>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Pageonexdev mailing list
>>>> Pageonexdev at mit.edu
>>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>>
>>>>
>>>
>>>
>>> --
>>> Edward L. Platt
>>> Civic Technology Programmer
>>> MIT Center for Civic Media
>>> E15-348
>>> http://civic.mit.edu
>>> http://elplatt.com
>>> @elplatt <http://twitter.com/elplatt>
>>>
>>> _______________________________________________
>>> Pageonexdev mailing list
>>> Pageonexdev at mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/pageonexdev/attachments/20131217/8a27fb03/attachment.htm