[PageOneX] [dev] New round of development in PageOneX

Thu Aug 9 12:50:36 EDT 2018

Hi Rafa,

That sounds spot on.  I don't have a lot of Docker experience. Would that
Docker mount configuration be in the docker image setup script?

Rahul

On Mon, Aug 6, 2018 at 1:32 PM Rafael Porres Molina <rporres at gmail.com>
wrote:

> Hi Rahul,
>
> I've been taking a look at the images storage change and it is not a
> trivial change. It is not difficult to move the place where images are
> downloaded but since they're served under '/assets' there would be a fair
> amount of work if we were to move them into a custom location.
>
> I still don't understand the need for a configuration parameter for that.
> The current location in the docker image where dynamically downloaded
> images/metadata are located are:
>
> /workspace/app/assets/images/kiosko -> for the newspaper images
> /workspace/app/assets/images/threads -> for the threads images and metadata
>
> If we added a custom location for that we could change where they are
> stored but that shouldn't affect how docker works, as docker volume mount
> syntax allows for different local/image path. You can always tell docker to
> mount
>
> /your/local/volume/pageonex/images/kiosko:/workspace/app/assets/images/
> kiosko
>
> and that would work the same way as if we configured the kiosko assets to
> be stored in, e.g. /images/kiosko and we told docker to mount it using
>
> /your/local/volume/pageonex/images/kiosko:/images/kiosko
>
> So unless there's some constraint in Dokku that it is not present in
> vanilla Docker, I don't understand the need for a config variable to store
> images.
>
> I do agree that dynamically generated images should not be stored in the
> same place as the static assets that come with the application, but that's
> a different subject :-D
>
> Best regards,
>
> Rafa
>
>
> El lun., 6 ago. 2018 a las 13:46, Rafael Porres Molina (<rporres at gmail.com>)
> escribió:
>
>> Hi Rahul,
>>
>> Could you share the apache production file to work on replicating that
>> setup?
>>
>> Regards,
>>
>> Rafa
>>
>> El mié., 25 jul. 2018 a las 17:28, Rahul B (<rahulb at media.mit.edu>)
>> escribió:
>>
>>> OK - it is going to go down for a few days while we wipe it clean and
>>> update it to the latest Ubuntu.  It is the dev server, so I'm assuming
>>> there is no data the needs preserving on it - correct?
>>>
>>> After that it will be ready for you to test out containerized deployment
>>> on.  That'll probably happen around 8/6.
>>>
>>> Rahul
>>>
>>> On Wed, Jul 25, 2018 at 10:52 AM pablo rey <pablo at basurama.org> wrote:
>>>
>>>> I have not used it for a while, but I plan to use it for the last
>>>> updates we have developed recently.
>>>>
>>>> On 25 July 2018 at 16:46, Rahul B <rahulb at media.mit.edu> wrote:
>>>>
>>>>> Pablo do you currently use the dev installation of PageOneX on our
>>>>> civicdev server?
>>>>>
>>>>> Rahul
>>>>>
>>>>>
>>>>> On Mon, Jul 23, 2018 at 11:19 AM Rahul B <rahulb at media.mit.edu> wrote:
>>>>>
>>>>>> We are planning to deploy to production as a Docker image under Dokku
>>>>>> (details
>>>>>> <http://dokku.viewdocs.io/dokku/deployment/methods/dockerfiles/>).
>>>>>>
>>>>>> I believe we run under Apache with Passenger right now. Since this
>>>>>> will now be a Docker image the setup script can choose whatever it wants.
>>>>>>
>>>>>> Rahul
>>>>>>
>>>>>> On Mon, Jul 23, 2018 at 11:11 AM Rafael Porres Molina <
>>>>>> rporres at gmail.com> wrote:
>>>>>>
>>>>>>> Hi everybody,
>>>>>>>
>>>>>>> I have a good part of the Dockerize feature to be reviewed. Please
>>>>>>> take a look: https://github.com/numeroteca/pageonex/pull/213
>>>>>>>
>>>>>>> What's missing:
>>>>>>>
>>>>>>> * Image mount. I will try to figure out how to control that part
>>>>>>> once I get to know better the app
>>>>>>> * CI: I will need to know how we plan to deploy this once it is
>>>>>>> containerized to see if something like Travis fits here to create the
>>>>>>> docker images under certain conditions, e.g tag
>>>>>>> * Run under a proper server. What are we currently using: Unicorn?
>>>>>>> Puma? Passenger?
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Rafa
>>>>>>>
>>>>>>> El mié., 11 jul. 2018 a las 0:40, pablo rey (<pablo at basurama.org>)
>>>>>>> escribió:
>>>>>>>
>>>>>>>> I will review in a fresh install (without docker) if the migration
>>>>>>>> works and let you know if I have the same problem with the migration.
>>>>>>>>
>>>>>>>> The changes in config/database.yml will modify something from the
>>>>>>>> current way the tool works?
>>>>>>>>
>>>>>>>> On 9 July 2018 at 18:36, Rafael Porres Molina <rporres at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> See https://github.com/numeroteca/pageonex/issues/209 if you want
>>>>>>>>> to take a look to what I've done so far. There's a problem with a migration
>>>>>>>>> so I haven't been able to test it properly. I will continue working at some
>>>>>>>>> point in the following days.
>>>>>>>>>
>>>>>>>>> It is not finished, so I haven't done a pull request. But if you
>>>>>>>>> prefer that I do it so that you can take a look more easily, please let me
>>>>>>>>> know and I will create one (tagged with DO NOT MERGE)
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Rafa
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> El lun., 9 jul. 2018 a las 10:22, Rafael Porres Molina (<
>>>>>>>>> rporres at gmail.com>) escribió:
>>>>>>>>>
>>>>>>>>>> I'll start taking a look and keep you posted.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>>
>>>>>>>>>> Rafa
>>>>>>>>>>
>>>>>>>>>> El vie., 6 jul. 2018 a las 19:14, Rahul B (<rahulb at media.mit.edu>)
>>>>>>>>>> escribió:
>>>>>>>>>>
>>>>>>>>>>> Yes, a Docker container would work fine. Of course the problem
>>>>>>>>>>> there is we need persistent storage for the database and the images.  Both
>>>>>>>>>>> of these should be read from environment variables.
>>>>>>>>>>>
>>>>>>>>>>> I think this requires a change in the PageOneX configuration
>>>>>>>>>>> code.  Rails already supports specifying a database via a
>>>>>>>>>>> `DATABASE_URL` environment variable
>>>>>>>>>>> <http://guides.rubyonrails.org/configuring.html#configuring-a-database>
>>>>>>>>>>> .
>>>>>>>>>>>
>>>>>>>>>>> For docker it looks like you can specific environment variables
>>>>>>>>>>> either on the command line individually or via a file (docs
>>>>>>>>>>> <https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e---env---env-file>
>>>>>>>>>>> ).
>>>>>>>>>>>
>>>>>>>>>>> With that in mind, I think the main changes to make are probably
>>>>>>>>>>> to
>>>>>>>>>>> a) Dockerize it
>>>>>>>>>>> b) change the image uploading to save images to the path the
>>>>>>>>>>> environment variable specifies
>>>>>>>>>>>
>>>>>>>>>>> Rahul
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jul 6, 2018 at 12:38 PM Rafael Porres Molina <
>>>>>>>>>>> rporres at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Rahul,
>>>>>>>>>>>>
>>>>>>>>>>>> I'm unsure if Pablo will be able to make it this late in the
>>>>>>>>>>>> spanish afternoon, so I will ask some quesitons here just to understand
>>>>>>>>>>>> better what's needed.
>>>>>>>>>>>>
>>>>>>>>>>>> I guess what it is needed now is to be able to run pageonex in
>>>>>>>>>>>> a Docker container, am I right?
>>>>>>>>>>>>
>>>>>>>>>>>> The other thing that is important is to be able to configure
>>>>>>>>>>>> the root of the image storage through an env variable so that you can do
>>>>>>>>>>>> something like
>>>>>>>>>>>>
>>>>>>>>>>>> docker run --env ROOT_IMAGE_STORAGE=/blah -v
>>>>>>>>>>>> /dokku/persistent/storage/pageonex:/blah ....
>>>>>>>>>>>>
>>>>>>>>>>>> I don't know how pageonex works internally so I don't know how
>>>>>>>>>>>> easy is to fix that. But would the following work?
>>>>>>>>>>>>
>>>>>>>>>>>> docker run -v  /dokku/persistent/storage/pageonex:/
>>>>>>>>>>>> assets/images/kiosko...
>>>>>>>>>>>>
>>>>>>>>>>>> or whatever combination around this? I guess that will depend
>>>>>>>>>>>> on where the application writes the assets once is running inside the
>>>>>>>>>>>> container...
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Rafa
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> El vie., 6 jul. 2018 a las 18:12, Rahul B (<
>>>>>>>>>>>> rahulb at media.mit.edu>) escribió:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi - I have 20 mins now, or in about an hour.  If you are
>>>>>>>>>>>>> still available you can find me on Skype at civic-rahulb.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Rahul
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Jul 6, 2018 at 6:36 AM pablo rey <pablo at basurama.org>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hey Rahul, Let's chat when you are available today. Rafa said
>>>>>>>>>>>>>> that he will join. He can better help with the containers and image storage
>>>>>>>>>>>>>> location thing better than me.
>>>>>>>>>>>>>> Ping me.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 5 July 2018 at 15:53, Rahul B <rahulb at media.mit.edu>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yes - you can deploy a Docker image via Dokku (details
>>>>>>>>>>>>>>> <http://dokku.viewdocs.io/dokku/deployment/methods/dockerfiles/>).
>>>>>>>>>>>>>>> That said, deploying as a Heroic build pack is a little simpler.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regarding the images, the feature suggestion I made (to
>>>>>>>>>>>>>>> specify the storage location via an environment variable), will let us move
>>>>>>>>>>>>>>> the image storage to the larger hard drive.  That is precisely the goal.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If you'd like to chat I have time today or tomorrow morning
>>>>>>>>>>>>>>> (Boston time).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Rahul
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Jul 4, 2018 at 7:07 AM pablo rey <pablo at basurama.org>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hey Rafa, that'd be great. Thanks!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Let's see what Rahul says.
>>>>>>>>>>>>>>>> Are Docker and Dokku interchangeable?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 4 July 2018 at 12:06, Rafael Porres Molina <
>>>>>>>>>>>>>>>> rporres at gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hey Pablo,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This next Monday I can have some hours helping you if you
>>>>>>>>>>>>>>>>> need to move the application to Docker. Prior to that, if you want we can
>>>>>>>>>>>>>>>>> set a meeting with Rahul so that the time that I can work on the project is
>>>>>>>>>>>>>>>>> more focused.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Let me know if this works for you.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Rafa
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> El mié., 4 jul. 2018 a las 0:25, pablo rey (<
>>>>>>>>>>>>>>>>> pablo at basurama.org>) escribió:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Rahul,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for the good news, happy to continue at civic
>>>>>>>>>>>>>>>>>> media servers.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We have modest plans for this round of development, let's
>>>>>>>>>>>>>>>>>> see how we do.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I am not familiarized with Dokku and containerized, how
>>>>>>>>>>>>>>>>>> difficult is to change to that mode of deployment?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regarding image storage, remember you tried in
>>>>>>>>>>>>>>>>>> November'17.  You wrote me off the list "I'm going to copy the giant folder
>>>>>>>>>>>>>>>>>> of kiosko images to the bigger drive I mentioned again, but this time with
>>>>>>>>>>>>>>>>>> a flag to maintain all the permissions. Then perhaps we can schedule a time
>>>>>>>>>>>>>>>>>> where I can flip it to use the larger hard drive and you can make a thread
>>>>>>>>>>>>>>>>>> and test some old ones to tell me if it works.  If it does, all good and
>>>>>>>>>>>>>>>>>> I'll delete the giant folder on the main drive that is running low on
>>>>>>>>>>>>>>>>>> space.  If it doesn't work then I can just quickly put it back the way it
>>>>>>>>>>>>>>>>>> was and think harder about it." I think that caused some
>>>>>>>>>>>>>>>>>> problems of image missing
>>>>>>>>>>>>>>>>>> <https://github.com/numeroteca/pageonex/issues/203#issuecomment-361261644>,
>>>>>>>>>>>>>>>>>> still unsolved.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> You suggest that Dokku's persistent storage will enable
>>>>>>>>>>>>>>>>>> that change in storage. We'll need your help to make that shift.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>> pablo
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 3 July 2018 at 15:29, Rahul B <rahulb at media.mit.edu>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Pleasure to meet you Juanjo.  Glad to see some new life
>>>>>>>>>>>>>>>>>>> breathing into the project.  We host it here and the civic media servers,
>>>>>>>>>>>>>>>>>>> and I confirmed with Ethan that he is willing to continue to do so.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> * We are using RVM, but are switching to a containerized
>>>>>>>>>>>>>>>>>>> deployment system soon.  That means it'd be best to think about
>>>>>>>>>>>>>>>>>>> Heroku-style deployment (like PageOneX originally did).
>>>>>>>>>>>>>>>>>>> * We will be using Dokku
>>>>>>>>>>>>>>>>>>> <http://dokku.viewdocs.io/dokku/> for containerized
>>>>>>>>>>>>>>>>>>> deployment to our servers, we you will be able to deploy with a `git push`
>>>>>>>>>>>>>>>>>>> (once we set keys up). We are migrating our hosted applications to this
>>>>>>>>>>>>>>>>>>> mode now, so this is super good timing!
>>>>>>>>>>>>>>>>>>> * Image storage is the only complicated thing to think
>>>>>>>>>>>>>>>>>>> about... right now the `assets/images/kiosko` folder is 115GB (!).  It'd be
>>>>>>>>>>>>>>>>>>> great if you could prioritize a feature change that lets us specify the
>>>>>>>>>>>>>>>>>>> root image storage path via a environment variable.  That would allow us to
>>>>>>>>>>>>>>>>>>> use the Dokku persistent storage path
>>>>>>>>>>>>>>>>>>> <https://github.com/dokku/dokku/blob/master/docs/advanced-usage/persistent-storage.md> to
>>>>>>>>>>>>>>>>>>> specify a storage point on our giant 10TB network drive while still
>>>>>>>>>>>>>>>>>>> supporting containerized deployment, rather than having it fill up our root
>>>>>>>>>>>>>>>>>>> drive (as it does now)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Sound good?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Rahul
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Tue, Jul 3, 2018 at 4:03 AM pablo rey <
>>>>>>>>>>>>>>>>>>> pablo at basurama.org> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hey!
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> We are back after a long time. We have a few
>>>>>>>>>>>>>>>>>>>> enhancements and things to fix in our road map (
>>>>>>>>>>>>>>>>>>>> https://github.com/numeroteca/pageonex/projects/1).
>>>>>>>>>>>>>>>>>>>> they are basic features to extend the use for an easier and deeper analysis
>>>>>>>>>>>>>>>>>>>> (drawn areas would be able to store more data, not only the topic). Juanjo
>>>>>>>>>>>>>>>>>>>> Bazán will be developing in RoR.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Rahul, we have a few doubts to solve regarding
>>>>>>>>>>>>>>>>>>>> deployment process:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    - We are planning to update the ruby version that
>>>>>>>>>>>>>>>>>>>>    it's used. I guess that should be no problem if we are using RVM. Could you
>>>>>>>>>>>>>>>>>>>>    confirm?
>>>>>>>>>>>>>>>>>>>>    - Another related question to the gems. Which is
>>>>>>>>>>>>>>>>>>>>    the deployment process in dev and production servers? is there an script
>>>>>>>>>>>>>>>>>>>>    that executes the bundle/migrations of the database or everything is
>>>>>>>>>>>>>>>>>>>>    installed "manually" or  something like capistrano is used?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Down the road, but not yet in the listed in the
>>>>>>>>>>>>>>>>>>>> project, we are planning to update ruby, gems and bootstrap.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> If you want to join, just tell!
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> p
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> ------------------------------------------
>>>>>>>>>>>>>>>>>>> Rahul Bhargava
>>>>>>>>>>>>>>>>>>> http://rahul.connectionlab.org
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>>> Pageonexdev mailing list
>>>>>>>>>>>>>>>>>>> Pageonexdev at mit.edu
>>>>>>>>>>>>>>>>>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>> Pageonexdev mailing list
>>>>>>>>>>>>>>>>>> Pageonexdev at mit.edu
>>>>>>>>>>>>>>>>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>> Pageonexdev mailing list
>>>>>>>>>>>>>>>>> Pageonexdev at mit.edu
>>>>>>>>>>>>>>>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>> Pageonexdev mailing list
>>>>>>>>>>>>>>>> Pageonexdev at mit.edu
>>>>>>>>>>>>>>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> ------------------------------------------
>>>>>>>>>>>>>>> Rahul Bhargava
>>>>>>>>>>>>>>> http://rahul.connectionlab.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>> ------------------------------------------
>>>>>>>>>>>>> Rahul Bhargava
>>>>>>>>>>>>> http://rahul.connectionlab.org
>>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>> ------------------------------------------
>>>>>>>>>>> Rahul Bhargava
>>>>>>>>>>> http://rahul.connectionlab.org
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>> --
>>>>>> ------------------------------------------
>>>>>> Rahul Bhargava
>>>>>> http://rahul.connectionlab.org
>>>>>>
>>>>> --
>>>>> ------------------------------------------
>>>>> Rahul Bhargava
>>>>> http://rahul.connectionlab.org
>>>>>
>>>>
>>>> --
>>> ------------------------------------------
>>> Rahul Bhargava
>>> http://rahul.connectionlab.org
>>>
>> --
------------------------------------------
Rahul Bhargava
http://rahul.connectionlab.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/pageonexdev/attachments/20180809/a9fb6ff7/attachment-0001.html