[PageOneX] [dev] PageOneX server is down

Tue Dec 10 14:20:15 EST 2013

Ok, I will add the steps to reproduce these errors in the already open
issue https://github.com/numeroteca/pageonex/issues/179

I don't think it is related to the cpu throttling, these problems were
happening before.

On Tue, Dec 10, 2013 at 8:11 PM, Edward Platt <elplatt at media.mit.edu> wrote:

> Pablo can you create issues for these bugs in github and make sure they
> contain steps to reproduce?  Let me know if you think we should revert the
> cpu throttling.  I'll put a little time into fixing those errors.
>
> -Ed
>
>
> On Tue, Dec 10, 2013 at 12:49 PM, pablo rey <pablo at basurama.org> wrote:
>
>> My experience with those problems:
>>
>> For long threads: Almost every time I create or update a thread (add one
>> day) I get a "500 Internal Server Error", after "creating background image
>> grid" according to the logs (sometime after "creating overlays for
>> topics").
>> When I refresh the display view sometimes it works, but I guess many
>> users leave the page after that (I've found many broken threads with no
>> even a single area coded). Sometimes after 5 or 6 times of refreshing, the
>> display view works again.
>>
>> For short threads (one or two days): it returns the error for the
>> composite images http://pageonex.com/numeroteca/test-repeated/ It is not
>> able to generate them 88 newspapers, one day).
>>
>> I don't know if the problem for short threads is related to the big
>> images that imagemagick has to create. If that is the case, we should limit
>> the total height of those images: to avoid the tool from crashing AND to
>> provide a good overview of the news coverage without the need of zooming
>> out too much.
>>
>> thanks Rafael for the hints!
>>
>> ps: Pagination in the thread list is not working again<https://github.com/numeroteca/pageonex/issues/182>,
>> maybe related to lack of memory?
>>
>>
>> On Tue, Dec 10, 2013 at 6:46 PM, Rafael Porres Molina <rporres at gmail.com>wrote:
>>
>>> Request timeouts can cause internal server errors. You may look into
>>> that.
>>>
>>> Regards,
>>>
>>> Rafa
>>>
>>>
>>> 2013/12/10 Edward Platt <elplatt at media.mit.edu>
>>>
>>>> I haven't seen any cpu usage notifications, and the server hasn't
>>>> crashed, so the throttling seems to be helping on that front.
>>>>
>>>> The errors and broken images you see could be from the throttling or
>>>> they could be a coincidence.  Any idea how widespread/common those problems
>>>> are?  If it is throttling, the error likely occurs when the server would
>>>> have otherwise crashed.
>>>>
>>>> Obviously we want to avoid both, so there will still need to be some
>>>> work around limiting the image size.
>>>>
>>>> -Ed
>>>>
>>>>
>>>> On Tue, Dec 10, 2013 at 7:41 AM, pablo rey <pablo at basurama.org> wrote:
>>>>
>>>>> Great, Ed, how is this CPU throttling working?
>>>>>
>>>>> I added one extra day to the thread
>>>>> http://pageonex.com/numeroteca/corrupcion-espana-diciembre-2013/ and
>>>>> updated:
>>>>>
>>>>> 1st time: internal server error.
>>>>>
>>>>> I reloaded:
>>>>> 2nd time: the newspaper images and overlays were broken.
>>>>>
>>>>> I went back to edit and "updated" the thread
>>>>> 3rd time: it worked fine.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Dec 9, 2013 at 11:12 PM, Edward Platt <elplatt at media.mit.edu>wrote:
>>>>>
>>>>>> I've added cpu throttling to the imagemagick settings.  Hopefully
>>>>>> this will help stop the crashes, but keep an eye out for any weird behavior!
>>>>>>
>>>>>> -Ed
>>>>>>
>>>>>>
>>>>>> On Sun, Dec 8, 2013 at 12:14 PM, pablo rey <pablo at basurama.org>wrote:
>>>>>>
>>>>>>> Hi Ed,
>>>>>>> Prevent overloading would be great.
>>>>>>> I think the server just went down again when editing the same thread.
>>>>>>>
>>>>>>> Let's talk next week if there is something I can look into also.
>>>>>>> p
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Dec 6, 2013 at 11:19 PM, Edward Platt <elplatt at media.mit.edu
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Hey Pablo!
>>>>>>>>
>>>>>>>> The server's back up.  We've been getting notifications about cpu
>>>>>>>> and disk usage, probably from resizing images.  Next week I'll look into
>>>>>>>> whether there's an easy fix to prevent overloading the server on long/short
>>>>>>>> threads.
>>>>>>>>
>>>>>>>> -Ed
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Dec 6, 2013 at 5:30 AM, pablo rey <pablo at basurama.org>wrote:
>>>>>>>>
>>>>>>>>> Hi Ed and Rahul,
>>>>>>>>> I was adding a new topic to the thread
>>>>>>>>> http://pageonex.com/numeroteca/corrupcion-espana-diciembre-2013/and the server went down.
>>>>>>>>>
>>>>>>>>> I can not ssh.
>>>>>>>>>
>>>>>>>>> Apart from the problem of lack of memory  to calculate the
>>>>>>>>> background image grid for too many or too few images, I think there is
>>>>>>>>> another problem with too many overlays when it has to create the composites.
>>>>>>>>>
>>>>>>>>> Please reboot over-9000 server!
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Pageonexdev mailing list
>>>>>>>>> Pageonexdev at mit.edu
>>>>>>>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Edward L. Platt
>>>>>>>> Civic Technology Programmer
>>>>>>>> MIT Center for Civic Media
>>>>>>>> E15-348
>>>>>>>> http://civic.mit.edu
>>>>>>>> http://elplatt.com
>>>>>>>> @elplatt <http://twitter.com/elplatt>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Pageonexdev mailing list
>>>>>>>> Pageonexdev at mit.edu
>>>>>>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Edward L. Platt
>>>>>> Civic Technology Programmer
>>>>>> MIT Center for Civic Media
>>>>>> E15-348
>>>>>> http://civic.mit.edu
>>>>>> http://elplatt.com
>>>>>> @elplatt <http://twitter.com/elplatt>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Edward L. Platt
>>>> Civic Technology Programmer
>>>> MIT Center for Civic Media
>>>> E15-348
>>>> http://civic.mit.edu
>>>> http://elplatt.com
>>>> @elplatt <http://twitter.com/elplatt>
>>>>
>>>> _______________________________________________
>>>> Pageonexdev mailing list
>>>> Pageonexdev at mit.edu
>>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Pageonexdev mailing list
>>> Pageonexdev at mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>
>>>
>>
>> _______________________________________________
>> Pageonexdev mailing list
>> Pageonexdev at mit.edu
>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>
>>
>
>
> --
> Edward L. Platt
> Civic Technology Programmer
> MIT Center for Civic Media
> E15-348
> http://civic.mit.edu
> http://elplatt.com
> @elplatt <http://twitter.com/elplatt>
>
> _______________________________________________
> Pageonexdev mailing list
> Pageonexdev at mit.edu
> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/pageonexdev/attachments/20131210/16a40f1e/attachment-0001.htm