<div dir="ltr">Hi,<div><br></div><div>I don't know if I'm going to be able to make tonight.</div><div><br></div><div>Anyway this is what I saw talking with Pablo the other day.</div><div><br></div><div>- The instance where Rails and MySQL is running seems undersized memory-wise.</div>
<div><br></div><div><div>numeroteca@li563-7:~$ head -1 /proc/meminfo </div><div>MemTotal: 1104816 kB</div></div><div><br></div><div>which is around 1 Gb of memory.</div><div><br></div><div>- Just before hitting the error in the video Pablo sent, you can see that CPU I/O wait spiked, and the kernel swapping thread was most consuming process that top was showing. That indicates that the system was trying to page out memory pages to swap. As to why the browser returns a 500 error, it could be that the ruby process dies or it hits a response timeout for the Apache request (I see that Rails runs behind Apache, but I haven't had the time to take a look to the configuration).</div>
<div><br></div><div>- As for the MySQL crashes, they also seem related to memory. Mysql error logs show errors like the following:</div><div><br></div><div><div>140116 9:40:01 [Note] Plugin 'FEDERATED' is disabled.</div>
<div>140116 9:40:01 InnoDB: The InnoDB memory heap is disabled</div><div>140116 9:40:01 InnoDB: Mutexes and rw_locks use GCC atomic builtins</div><div>140116 9:40:01 InnoDB: Compressed tables use zlib 1.2.3.4</div><div>
140116 9:40:01 InnoDB: Initializing buffer pool, size = 128.0M</div><div>InnoDB: mmap(137363456 bytes) failed; errno 12</div><div>140116 9:40:01 InnoDB: Completed initialization of buffer pool</div><div>140116 9:40:01 InnoDB: Fatal error: cannot allocate memory for the buffer pool</div>
<div>140116 9:40:01 [ERROR] Plugin 'InnoDB' init function returned error.</div><div>140116 9:40:01 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.</div><div>140116 9:40:01 [ERROR] Unknown/unsupported storage engine: InnoDB</div>
<div>140116 9:40:01 [ERROR] Aborting</div></div><div><br></div><div>that indicates that it crashed while trying to allocate memory for the innodb buffer pool. As it is not explicitly configured, in mysql 5.5 the default is 128Mb, which is not much unless the instance where MySQL is running is so short in memory.</div>
<div><br></div><div><a href="http://dev.mysql.com/doc/refman/5.5/en/innodb-parameters.html#sysvar_innodb_buffer_pool_size">http://dev.mysql.com/doc/refman/5.5/en/innodb-parameters.html#sysvar_innodb_buffer_pool_size</a><br>
</div><div><br></div><div>The easiest thing to do is to try to get more memory for the instance where pageonex is running. How much? Well, as much as you can afford :)</div><div><br></div><div>Regards,</div><div><br></div>
<div>Rafa</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2014/1/23 pablo rey <span dir="ltr"><<a href="mailto:pablo@basurama.org" target="_blank">pablo@basurama.org</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div>Hi there,<br>Reminder that we are meeting in 2 and a half hours (4.30pm EST) today! <br>
Send me your gmail account so that I can add you to the hangout.<br><br></div>The idea is to brainstorm and understand why PageOneX is crashing when making long and short threads (500 internal server error) <a href="https://github.com/numeroteca/pageonex/issues/179" target="_blank">#179</a> + why is the mySQL crashing as well.<br>
<br></div>It would be great to understand how the server is built. <br>After a short talk with Rafa Porres, we figured out that is probably related to a lack of memory (now we have 1Gb).<br><br></div>Folks in Mexico made their <a href="http://estadodelperiodismo.articulo19.org/" target="_blank">own PageOnex deployment and research</a> and they also reported related issues, please, Alecs, post your problems in Github and share them through this list!<br>
<br></div>See you in a while,<br></div>pablo<br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Jan 21, 2014 at 6:27 PM, pablo rey <span dir="ltr"><<a href="mailto:pablo@basurama.org" target="_blank">pablo@basurama.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Reminder for everyone interested: this Thursday Jan 23rd 4.30pm EST (10.30pm CET) we have the session to try to solve different problems with the server crashing at PageOneX.<br>
<br>I made a video showing how it goes to "500 internal server error" when editing a thread and adding one extra day, including logs and top:<br>
<a href="http://www.youtube.com/watch?v=wuBMPu7Hty0" target="_blank">http://www.youtube.com/watch?v=wuBMPu7Hty0</a><br></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div>On Thu, Jan 9, 2014 at 12:08 AM, pablo rey <span dir="ltr"><<a href="mailto:pablo@basurama.org" target="_blank">pablo@basurama.org</a>></span> wrote:<br>
</div><div><div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Great Rafa, I'll write you off the list. If anyone is interested to join any of the meetings, shoot me an email!<br>
</div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jan 8, 2014 at 11:55 PM, Rafael Porres Molina <span dir="ltr"><<a href="mailto:rporres@gmail.com" target="_blank">rporres@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>I'd be happy to join if I can be of any help.</div><div><br></div><div>Pablo: We could arrange a previous hangout between you and me so that you can give me some context to be more helpful. </div>
<div><br></div><div>Regards,</div><div><br></div><div>Rafa</div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">2014/1/8 Edward Platt <span dir="ltr"><<a href="mailto:elplatt@media.mit.edu" target="_blank">elplatt@media.mit.edu</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Works for me, talk then!</div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Wed, Jan 8, 2014 at 5:21 PM, pablo rey <span dir="ltr"><<a href="mailto:pablo@basurama.org" target="_blank">pablo@basurama.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Given the 6 hours of difference Jan 23rd 4.30pm EST (10.30pm CET, Spain) is good for me.<br></div><div>
<div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jan 8, 2014 at 11:17 PM, Edward Platt <span dir="ltr"><<a href="mailto:elplatt@media.mit.edu" target="_blank">elplatt@media.mit.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Woks for me, my day should be open so far. What time's good for you?</div><div><div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jan 8, 2014 at 5:13 PM, pablo rey <span dir="ltr"><<a href="mailto:pablo@basurama.org" target="_blank">pablo@basurama.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Let's set another date, then. Thursday Jan 23rd?<br></div><div><div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Wed, Jan 8, 2014 at 9:05 PM, Edward Platt <span dir="ltr"><<a href="mailto:elplatt@media.mit.edu" target="_blank">elplatt@media.mit.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Touching base sounds good. After Jan 22 would be best.<div><br></div><div>-Ed</div></div><div>
<div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jan 8, 2014 at 2:58 PM, pablo rey <span dir="ltr"><<a href="mailto:pablo@basurama.org" target="_blank">pablo@basurama.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Thanks Ed,<br>I had to restart mysql again minutes ago :-/<br>The problem with this issue and specially with <a href="https://github.com/numeroteca/pageonex/issues/179" target="_blank">#179</a> is that early adopters do not come back after a first failed attempt to create a thread.<br>
<br></div>I would suggest to set up a google hangout during IAP (=January for non MIT folks!) to brainstorm together on this. <br>What do you think? anyone interested? <br>Which other sites are hosted in that server?<br>
<br>
</div>How about next Monday Jan 13th at 4.30pm EST (10.30pm CET, Spain)? <br><br></div>Best,<br>p<br><div><div><br><br></div></div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Thu, Dec 26, 2013 at 2:31 AM, Edward Platt <span dir="ltr"><<a href="mailto:elplatt@media.mit.edu" target="_blank">elplatt@media.mit.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks for the update Pablo. I don't see anything telling in the logs you've sent. The mysql problems may be unrelated to PageOneX as the server is shared. It's also possible that the performance problems we were seeing before are somehow causing this. I've been working out how to best fix those, but there doesn't seem to be a quick fix. It's on our radar!<div>
<br></div><div>-Ed</div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Dec 25, 2013 at 4:06 PM, pablo rey <span dir="ltr"><<a href="mailto:pablo@basurama.org" target="_blank">pablo@basurama.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>I had to restart the mysql again today!<br><br></div>This time it crashed after:<br><div style="margin-left:40px">
Started GET "/about" for 91.206.178.40 at 2013-12-24 23:20:01 +0000<br>Processing by HomeController#about as HTML<br>
Rendered home/about.html.erb within layouts/application (39.1ms)<br>Completed 200 OK in 340ms (Views: 268.5ms | ActiveRecord: 0.0ms)<br>Started GET "/numeroteca/corrupcion-espana-diciembre-2013/" for 91.206.178.40 at 2013-12-24 23:20:02 +0000<br>
Processing by CodingController#display as HTML<br> Parameters: {"username"=>"numeroteca", "thread_name"=>"corrupcion-espana-diciembre-2013"}<br> Rendered coding/_coding_footer.html.erb (0.3ms)<br>
Rendered coding/display.html.erb within layouts/application (5064.0ms)<br>Completed 200 OK in 5614ms (Views: 2025.2ms | ActiveRecord: 3365.2ms)<br>Started GET "/users/sign_up" for 91.206.178.40 at 2013-12-24 23:20:09 +0000<br>
Processing by Devise::RegistrationsController#new as HTML<br> Rendered devise/_links.erb (9758.9ms)<br> Rendered devise/registrations/new.html.erb within layouts/application (18132.6ms)<br>Completed 200 OK in 18877ms (Views: 18569.0ms | ActiveRecord: 0.0ms)<br>
Started GET "/users/sign_in" for 91.206.178.40 at 2013-12-24 23:20:29 +0000<br>Processing by Devise::SessionsController#new as HTML<br> Rendered devise/_links.erb (1.2ms)<br> Rendered devise/sessions/new.html.erb within layouts/application (102.2ms)<br>
Completed 200 OK in 477ms (Views: 272.0ms | ActiveRecord: 0.0ms)<div><br>Connecting to database specified by database.yml<br>Connecting to database specified by database.yml<br></div></div><br></div><div>
<div><div class="gmail_extra"><br><br>
<div class="gmail_quote">On Sun, Dec 22, 2013 at 11:57 PM, pablo rey <span dir="ltr"><<a href="mailto:pablo@basurama.org" target="_blank">pablo@basurama.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div><div><div>I had to restart mysql again minutes ago.<br><br></div>This is the rails log before crashing:<br><div style="margin-left:40px">Started GET "/numeroteca/local-national-world-news/coding" for 5.10.83.101 at 2013-12-22 19:11:02 +0000<br>
Processing by CodingController#process_images as */*<br> Parameters: {"username"=>"numeroteca", "thread_name"=>"local-national-world-news"}<br> Rendered coding/_coding_footer.html.erb (0.1ms)<br>
Rendered coding/process_images.html.erb within layouts/application (3323.2ms) <br>Completed 200 OK in 3331ms (Views: 1658.8ms | ActiveRecord: 1668.8ms)<br>Started GET "/" for 67.23.226.7 at 2013-12-22 19:11:10 +0000<br>
Processing by HomeController#index as HTML<br> Rendered home/index.html.erb within layouts/application (184.1ms)<br>Completed 200 OK in 284ms (Views: 163.3ms | ActiveRecord: 120.0ms)<br>Started GET "/assets/application.css" for 67.23.226.7 at 2013-12-22 19:11:16 +0000<br>
Served asset /application.css - 304 Not Modified (522ms)<br>Started GET "/Donna/juicio-marita-veron-cronica/export.ods" for 5.10.83.105 at 2013-12-22 19:11:17 +0000<br>Processing by ThreadsController#export as ODS<br>
Parameters: {"username"=>"Donna", "thread_name"=>"juicio-marita-veron-cronica"}<br>Sent file /tmp/export20131222-12134-1228f0a.ods (0.1ms)<br>Completed 200 OK in 167ms (ActiveRecord: 95.3ms)<br>
Started GET "/davidmbusto/dragado-en-bahia-blanca/export.json" for 180.76.5.75 at 2013-12-22 19:11:17 +0000<br>Processing by ThreadsController#export as JSON<br> Parameters: {"username"=>"davidmbusto", "thread_name"=>"davidmbusto"}<br>
Completed 200 OK in 93ms (Views: 0.2ms | ActiveRecord: 4.6ms)<br>Started GET "/threads/" for 67.23.226.7 at 2013-12-22 19:11:19 +0000<br>Processing by ThreadsController#index as HTML<br>Connecting to database specified by database.yml<br>
Connecting to database specified by database.yml<br></div><br>After restarting <a href="https://github.com/numeroteca/pageonex/issues/182" target="_blank">pagination in the thread list is not working again</a>. I mention it as pageonex crashed after a GET "/threads/" request. What could be the cause for making mysql die? Listing the threads doesn't seem very problematic.<br>
<br><br></div>Besides, I was trying to get a sense of the usage of PageOneX web site using the Piwik statistics. According to Piwik only 4 visits to the site today. According to the logs many many more. I assume many people have the stats blocked but the difference isway too high... guesses?<br>
<br></div>Happy holidays!<span><font color="#888888"><br>p<br><div><div><br><div><br><br></div></div></div></font></span></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Sun, Dec 15, 2013 at 1:53 AM, Edward Platt <span dir="ltr"><<a href="mailto:elplatt@media.mit.edu" target="_blank">elplatt@media.mit.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Should be fixed now. Mysql wasn't running, not sure why and there are no relevant log entries. If you ever feel the need to restart mysql, please go ahead.<div>
<br></div><div>-Ed</div></div><div class="gmail_extra">
<br><br><div class="gmail_quote"><div><div>On Sat, Dec 14, 2013 at 3:52 PM, pablo rey <span dir="ltr"><<a href="mailto:pablo@basurama.org" target="_blank">pablo@basurama.org</a>></span> wrote:<br></div></div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>
<div dir="ltr">Should I just restart the mysql server?<br> sudo service mysql restart<br></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Dec 14, 2013 at 9:09 PM, pablo rey <span dir="ltr"><<a href="mailto:pablo@basurama.org" target="_blank">pablo@basurama.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><dl><dt>Rahul and Ed!</dt><dt>The home page returns a passenger message: Ruby (Rack) application could not be started<br>
</dt><dd><span style="font-family:courier new,monospace"><br></span></dd><dt><span style="font-family:courier new,monospace">Error message:</span></dt>
<dd><span style="font-family:courier new,monospace">Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111) (Mysql2::Error)</span></dd><dt><span style="font-family:courier new,monospace">Exception class:</span></dt>
<dd><span style="font-family:courier new,monospace">PhusionPassenger::UnknownError</span></dd></dl><p><br></p><p>Which is the same message I was getting and reported at <a href="https://github.com/numeroteca/pageonex/issues/179#issuecomment-30275654" target="_blank">https://github.com/numeroteca/pageonex/issues/179#issuecomment-30275654</a> (but didn't make the app crashed).</p>
<p></p><p>It looks like it's been down since yesterday.</p><p>Thanks,<br></p><p>p<br></p></div>
</blockquote></div><br></div>
</div></div><br></div></div>_______________________________________________<br>
Pageonexdev mailing list<br>
<a href="mailto:Pageonexdev@mit.edu" target="_blank">Pageonexdev@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/pageonexdev" target="_blank">http://mailman.mit.edu/mailman/listinfo/pageonexdev</a><br>
<br></blockquote></div><span><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><div>Edward L. Platt<br></div><div>Civic Technology Programmer</div><div>MIT Center for Civic Media</div>
<div>E15-348</div><div><a href="http://civic.mit.edu" target="_blank">http://civic.mit.edu</a></div>
<div><a href="http://elplatt.com" target="_blank">http://elplatt.com</a></div><div><a href="http://twitter.com/elplatt" target="_blank">@elplatt</a></div></div>
</font></span></div>
<br>_______________________________________________<br>
Pageonexdev mailing list<br>
<a href="mailto:Pageonexdev@mit.edu" target="_blank">Pageonexdev@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/pageonexdev" target="_blank">http://mailman.mit.edu/mailman/listinfo/pageonexdev</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><div>Edward L. Platt<br></div><div>Civic Technology Programmer</div><div>MIT Center for Civic Media</div><div>E15-348</div><div><a href="http://civic.mit.edu" target="_blank">http://civic.mit.edu</a></div>
<div><a href="http://elplatt.com" target="_blank">http://elplatt.com</a></div><div><a href="http://twitter.com/elplatt" target="_blank">@elplatt</a></div></div>
</div>
</div></div><br>_______________________________________________<br>
Pageonexdev mailing list<br>
<a href="mailto:Pageonexdev@mit.edu" target="_blank">Pageonexdev@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/pageonexdev" target="_blank">http://mailman.mit.edu/mailman/listinfo/pageonexdev</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><div>Edward L. Platt<br></div><div>Civic Technology Programmer</div><div>MIT Center for Civic Media</div><div>E15-348</div><div><a href="http://civic.mit.edu" target="_blank">http://civic.mit.edu</a></div>
<div><a href="http://elplatt.com" target="_blank">http://elplatt.com</a></div><div><a href="http://twitter.com/elplatt" target="_blank">@elplatt</a></div></div>
</div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><div>Edward L. Platt<br></div><div>Civic Technology Programmer</div><div>MIT Center for Civic Media</div><div>E15-348</div><div><a href="http://civic.mit.edu" target="_blank">http://civic.mit.edu</a></div>
<div><a href="http://elplatt.com" target="_blank">http://elplatt.com</a></div><div><a href="http://twitter.com/elplatt" target="_blank">@elplatt</a></div></div>
</div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><div>Edward L. Platt<br></div><div>Civic Technology Programmer</div><div>MIT Center for Civic Media</div><div>E15-348</div><div><a href="http://civic.mit.edu" target="_blank">http://civic.mit.edu</a></div>
<div><a href="http://elplatt.com" target="_blank">http://elplatt.com</a></div><div><a href="http://twitter.com/elplatt" target="_blank">@elplatt</a></div></div>
</div>
</div></div><br>_______________________________________________<br>
Pageonexdev mailing list<br>
<a href="mailto:Pageonexdev@mit.edu" target="_blank">Pageonexdev@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/pageonexdev" target="_blank">http://mailman.mit.edu/mailman/listinfo/pageonexdev</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Pageonexdev mailing list<br>
<a href="mailto:Pageonexdev@mit.edu" target="_blank">Pageonexdev@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/pageonexdev" target="_blank">http://mailman.mit.edu/mailman/listinfo/pageonexdev</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div></div></div><br></div>
</blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Pageonexdev mailing list<br>
<a href="mailto:Pageonexdev@mit.edu">Pageonexdev@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/pageonexdev" target="_blank">http://mailman.mit.edu/mailman/listinfo/pageonexdev</a><br>
<br></blockquote></div><br></div>