[PageOneX] [dev] PageOneX server is down

Tue Dec 10 14:11:16 EST 2013

Pablo can you create issues for these bugs in github and make sure they
contain steps to reproduce?  Let me know if you think we should revert the
cpu throttling.  I'll put a little time into fixing those errors.

-Ed

On Tue, Dec 10, 2013 at 12:49 PM, pablo rey <pablo at basurama.org> wrote:

> My experience with those problems:
>
> For long threads: Almost every time I create or update a thread (add one
> day) I get a "500 Internal Server Error", after "creating background image
> grid" according to the logs (sometime after "creating overlays for
> topics").
> When I refresh the display view sometimes it works, but I guess many users
> leave the page after that (I've found many broken threads with no even a
> single area coded). Sometimes after 5 or 6 times of refreshing, the display
> view works again.
>
> For short threads (one or two days): it returns the error for the
> composite images http://pageonex.com/numeroteca/test-repeated/ It is not
> able to generate them 88 newspapers, one day).
>
> I don't know if the problem for short threads is related to the big images
> that imagemagick has to create. If that is the case, we should limit the
> total height of those images: to avoid the tool from crashing AND to
> provide a good overview of the news coverage without the need of zooming
> out too much.
>
> thanks Rafael for the hints!
>
> ps: Pagination in the thread list is not working again<https://github.com/numeroteca/pageonex/issues/182>,
> maybe related to lack of memory?
>
>
> On Tue, Dec 10, 2013 at 6:46 PM, Rafael Porres Molina <rporres at gmail.com>wrote:
>
>> Request timeouts can cause internal server errors. You may look into that.
>>
>> Regards,
>>
>> Rafa
>>
>>
>> 2013/12/10 Edward Platt <elplatt at media.mit.edu>
>>
>>> I haven't seen any cpu usage notifications, and the server hasn't
>>> crashed, so the throttling seems to be helping on that front.
>>>
>>> The errors and broken images you see could be from the throttling or
>>> they could be a coincidence.  Any idea how widespread/common those problems
>>> are?  If it is throttling, the error likely occurs when the server would
>>> have otherwise crashed.
>>>
>>> Obviously we want to avoid both, so there will still need to be some
>>> work around limiting the image size.
>>>
>>> -Ed
>>>
>>>
>>> On Tue, Dec 10, 2013 at 7:41 AM, pablo rey <pablo at basurama.org> wrote:
>>>
>>>> Great, Ed, how is this CPU throttling working?
>>>>
>>>> I added one extra day to the thread
>>>> http://pageonex.com/numeroteca/corrupcion-espana-diciembre-2013/ and
>>>> updated:
>>>>
>>>> 1st time: internal server error.
>>>>
>>>> I reloaded:
>>>> 2nd time: the newspaper images and overlays were broken.
>>>>
>>>> I went back to edit and "updated" the thread
>>>> 3rd time: it worked fine.
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Dec 9, 2013 at 11:12 PM, Edward Platt <elplatt at media.mit.edu>wrote:
>>>>
>>>>> I've added cpu throttling to the imagemagick settings.  Hopefully this
>>>>> will help stop the crashes, but keep an eye out for any weird behavior!
>>>>>
>>>>> -Ed
>>>>>
>>>>>
>>>>> On Sun, Dec 8, 2013 at 12:14 PM, pablo rey <pablo at basurama.org> wrote:
>>>>>
>>>>>> Hi Ed,
>>>>>> Prevent overloading would be great.
>>>>>> I think the server just went down again when editing the same thread.
>>>>>>
>>>>>> Let's talk next week if there is something I can look into also.
>>>>>> p
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Dec 6, 2013 at 11:19 PM, Edward Platt <elplatt at media.mit.edu>wrote:
>>>>>>
>>>>>>> Hey Pablo!
>>>>>>>
>>>>>>> The server's back up.  We've been getting notifications about cpu
>>>>>>> and disk usage, probably from resizing images.  Next week I'll look into
>>>>>>> whether there's an easy fix to prevent overloading the server on long/short
>>>>>>> threads.
>>>>>>>
>>>>>>> -Ed
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Dec 6, 2013 at 5:30 AM, pablo rey <pablo at basurama.org>wrote:
>>>>>>>
>>>>>>>> Hi Ed and Rahul,
>>>>>>>> I was adding a new topic to the thread
>>>>>>>> http://pageonex.com/numeroteca/corrupcion-espana-diciembre-2013/and the server went down.
>>>>>>>>
>>>>>>>> I can not ssh.
>>>>>>>>
>>>>>>>> Apart from the problem of lack of memory  to calculate the
>>>>>>>> background image grid for too many or too few images, I think there is
>>>>>>>> another problem with too many overlays when it has to create the composites.
>>>>>>>>
>>>>>>>> Please reboot over-9000 server!
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Pageonexdev mailing list
>>>>>>>> Pageonexdev at mit.edu
>>>>>>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Edward L. Platt
>>>>>>> Civic Technology Programmer
>>>>>>> MIT Center for Civic Media
>>>>>>> E15-348
>>>>>>> http://civic.mit.edu
>>>>>>> http://elplatt.com
>>>>>>> @elplatt <http://twitter.com/elplatt>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Pageonexdev mailing list
>>>>>>> Pageonexdev at mit.edu
>>>>>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Edward L. Platt
>>>>> Civic Technology Programmer
>>>>> MIT Center for Civic Media
>>>>> E15-348
>>>>> http://civic.mit.edu
>>>>> http://elplatt.com
>>>>> @elplatt <http://twitter.com/elplatt>
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Edward L. Platt
>>> Civic Technology Programmer
>>> MIT Center for Civic Media
>>> E15-348
>>> http://civic.mit.edu
>>> http://elplatt.com
>>> @elplatt <http://twitter.com/elplatt>
>>>
>>> _______________________________________________
>>> Pageonexdev mailing list
>>> Pageonexdev at mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>
>>>
>>
>> _______________________________________________
>> Pageonexdev mailing list
>> Pageonexdev at mit.edu
>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>
>>
>
> _______________________________________________
> Pageonexdev mailing list
> Pageonexdev at mit.edu
> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>
>

-- 
Edward L. Platt
Civic Technology Programmer
MIT Center for Civic Media
E15-348
http://civic.mit.edu
http://elplatt.com
@elplatt <http://twitter.com/elplatt>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/pageonexdev/attachments/20131210/d1b9d2e5/attachment.htm