[PageOneX] [dev] PageOneX server is down

Tue Dec 10 17:38:54 EST 2013

I wrote the sad story of a two days long thread in PageOneX :)
https://github.com/numeroteca/pageonex/issues/179#issuecomment-30275654

On Tue, Dec 10, 2013 at 8:20 PM, pablo rey <pablo at basurama.org> wrote:

> Ok, I will add the steps to reproduce these errors in the already open
> issue https://github.com/numeroteca/pageonex/issues/179
>
> I don't think it is related to the cpu throttling, these problems were
> happening before.
>
>
> On Tue, Dec 10, 2013 at 8:11 PM, Edward Platt <elplatt at media.mit.edu>wrote:
>
>> Pablo can you create issues for these bugs in github and make sure they
>> contain steps to reproduce?  Let me know if you think we should revert the
>> cpu throttling.  I'll put a little time into fixing those errors.
>>
>> -Ed
>>
>>
>> On Tue, Dec 10, 2013 at 12:49 PM, pablo rey <pablo at basurama.org> wrote:
>>
>>> My experience with those problems:
>>>
>>> For long threads: Almost every time I create or update a thread (add one
>>> day) I get a "500 Internal Server Error", after "creating background image
>>> grid" according to the logs (sometime after "creating overlays for
>>> topics").
>>> When I refresh the display view sometimes it works, but I guess many
>>> users leave the page after that (I've found many broken threads with no
>>> even a single area coded). Sometimes after 5 or 6 times of refreshing, the
>>> display view works again.
>>>
>>> For short threads (one or two days): it returns the error for the
>>> composite images http://pageonex.com/numeroteca/test-repeated/ It is
>>> not able to generate them 88 newspapers, one day).
>>>
>>> I don't know if the problem for short threads is related to the big
>>> images that imagemagick has to create. If that is the case, we should limit
>>> the total height of those images: to avoid the tool from crashing AND to
>>> provide a good overview of the news coverage without the need of zooming
>>> out too much.
>>>
>>> thanks Rafael for the hints!
>>>
>>> ps: Pagination in the thread list is not working again<https://github.com/numeroteca/pageonex/issues/182>,
>>> maybe related to lack of memory?
>>>
>>>
>>> On Tue, Dec 10, 2013 at 6:46 PM, Rafael Porres Molina <rporres at gmail.com
>>> > wrote:
>>>
>>>> Request timeouts can cause internal server errors. You may look into
>>>> that.
>>>>
>>>> Regards,
>>>>
>>>> Rafa
>>>>
>>>>
>>>> 2013/12/10 Edward Platt <elplatt at media.mit.edu>
>>>>
>>>>> I haven't seen any cpu usage notifications, and the server hasn't
>>>>> crashed, so the throttling seems to be helping on that front.
>>>>>
>>>>> The errors and broken images you see could be from the throttling or
>>>>> they could be a coincidence.  Any idea how widespread/common those problems
>>>>> are?  If it is throttling, the error likely occurs when the server would
>>>>> have otherwise crashed.
>>>>>
>>>>> Obviously we want to avoid both, so there will still need to be some
>>>>> work around limiting the image size.
>>>>>
>>>>> -Ed
>>>>>
>>>>>
>>>>> On Tue, Dec 10, 2013 at 7:41 AM, pablo rey <pablo at basurama.org> wrote:
>>>>>
>>>>>> Great, Ed, how is this CPU throttling working?
>>>>>>
>>>>>> I added one extra day to the thread
>>>>>> http://pageonex.com/numeroteca/corrupcion-espana-diciembre-2013/ and
>>>>>> updated:
>>>>>>
>>>>>> 1st time: internal server error.
>>>>>>
>>>>>> I reloaded:
>>>>>> 2nd time: the newspaper images and overlays were broken.
>>>>>>
>>>>>> I went back to edit and "updated" the thread
>>>>>> 3rd time: it worked fine.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Dec 9, 2013 at 11:12 PM, Edward Platt <elplatt at media.mit.edu>wrote:
>>>>>>
>>>>>>> I've added cpu throttling to the imagemagick settings.  Hopefully
>>>>>>> this will help stop the crashes, but keep an eye out for any weird behavior!
>>>>>>>
>>>>>>> -Ed
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Dec 8, 2013 at 12:14 PM, pablo rey <pablo at basurama.org>wrote:
>>>>>>>
>>>>>>>> Hi Ed,
>>>>>>>> Prevent overloading would be great.
>>>>>>>> I think the server just went down again when editing the same
>>>>>>>> thread.
>>>>>>>>
>>>>>>>> Let's talk next week if there is something I can look into also.
>>>>>>>> p
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Dec 6, 2013 at 11:19 PM, Edward Platt <
>>>>>>>> elplatt at media.mit.edu> wrote:
>>>>>>>>
>>>>>>>>> Hey Pablo!
>>>>>>>>>
>>>>>>>>> The server's back up.  We've been getting notifications about cpu
>>>>>>>>> and disk usage, probably from resizing images.  Next week I'll look into
>>>>>>>>> whether there's an easy fix to prevent overloading the server on long/short
>>>>>>>>> threads.
>>>>>>>>>
>>>>>>>>> -Ed
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Dec 6, 2013 at 5:30 AM, pablo rey <pablo at basurama.org>wrote:
>>>>>>>>>
>>>>>>>>>> Hi Ed and Rahul,
>>>>>>>>>> I was adding a new topic to the thread
>>>>>>>>>> http://pageonex.com/numeroteca/corrupcion-espana-diciembre-2013/and the server went down.
>>>>>>>>>>
>>>>>>>>>> I can not ssh.
>>>>>>>>>>
>>>>>>>>>> Apart from the problem of lack of memory  to calculate the
>>>>>>>>>> background image grid for too many or too few images, I think there is
>>>>>>>>>> another problem with too many overlays when it has to create the composites.
>>>>>>>>>>
>>>>>>>>>> Please reboot over-9000 server!
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Pageonexdev mailing list
>>>>>>>>>> Pageonexdev at mit.edu
>>>>>>>>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Edward L. Platt
>>>>>>>>> Civic Technology Programmer
>>>>>>>>> MIT Center for Civic Media
>>>>>>>>> E15-348
>>>>>>>>> http://civic.mit.edu
>>>>>>>>> http://elplatt.com
>>>>>>>>> @elplatt <http://twitter.com/elplatt>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Pageonexdev mailing list
>>>>>>>>> Pageonexdev at mit.edu
>>>>>>>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Edward L. Platt
>>>>>>> Civic Technology Programmer
>>>>>>> MIT Center for Civic Media
>>>>>>> E15-348
>>>>>>> http://civic.mit.edu
>>>>>>> http://elplatt.com
>>>>>>> @elplatt <http://twitter.com/elplatt>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Edward L. Platt
>>>>> Civic Technology Programmer
>>>>> MIT Center for Civic Media
>>>>> E15-348
>>>>> http://civic.mit.edu
>>>>> http://elplatt.com
>>>>> @elplatt <http://twitter.com/elplatt>
>>>>>
>>>>> _______________________________________________
>>>>> Pageonexdev mailing list
>>>>> Pageonexdev at mit.edu
>>>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Pageonexdev mailing list
>>>> Pageonexdev at mit.edu
>>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Pageonexdev mailing list
>>> Pageonexdev at mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>
>>>
>>
>>
>> --
>> Edward L. Platt
>> Civic Technology Programmer
>> MIT Center for Civic Media
>> E15-348
>> http://civic.mit.edu
>> http://elplatt.com
>> @elplatt <http://twitter.com/elplatt>
>>
>> _______________________________________________
>> Pageonexdev mailing list
>> Pageonexdev at mit.edu
>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/pageonexdev/attachments/20131210/ad7529bc/attachment.htm