[PageOneX] [dev] New round of development in PageOneX

Rafael Porres Molina rporres at gmail.com
Mon Aug 6 13:32:24 EDT 2018 

Hi Rahul,

I've been taking a look at the images storage change and it is not a
trivial change. It is not difficult to move the place where images are
downloaded but since they're served under '/assets' there would be a fair
amount of work if we were to move them into a custom location.

I still don't understand the need for a configuration parameter for that.
The current location in the docker image where dynamically downloaded
images/metadata are located are:

/workspace/app/assets/images/kiosko -> for the newspaper images
/workspace/app/assets/images/threads -> for the threads images and metadata

If we added a custom location for that we could change where they are
stored but that shouldn't affect how docker works, as docker volume mount
syntax allows for different local/image path. You can always tell docker to
mount

/your/local/volume/pageonex/images/kiosko:/workspace/app/assets/images/
kiosko

and that would work the same way as if we configured the kiosko assets to
be stored in, e.g. /images/kiosko and we told docker to mount it using

/your/local/volume/pageonex/images/kiosko:/images/kiosko

So unless there's some constraint in Dokku that it is not present in
vanilla Docker, I don't understand the need for a config variable to store
images.

I do agree that dynamically generated images should not be stored in the
same place as the static assets that come with the application, but that's
a different subject :-D

Best regards,

Rafa


El lun., 6 ago. 2018 a las 13:46, Rafael Porres Molina (<rporres at gmail.com>)
escribió:

> Hi Rahul,
>
> Could you share the apache production file to work on replicating that
> setup?
>
> Regards,
>
> Rafa
>
> El mié., 25 jul. 2018 a las 17:28, Rahul B (<rahulb at media.mit.edu>)
> escribió:
>
>> OK - it is going to go down for a few days while we wipe it clean and
>> update it to the latest Ubuntu.  It is the dev server, so I'm assuming
>> there is no data the needs preserving on it - correct?
>>
>> After that it will be ready for you to test out containerized deployment
>> on.  That'll probably happen around 8/6.
>>
>> Rahul
>>
>> On Wed, Jul 25, 2018 at 10:52 AM pablo rey <pablo at basurama.org> wrote:
>>
>>> I have not used it for a while, but I plan to use it for the last
>>> updates we have developed recently.
>>>
>>> On 25 July 2018 at 16:46, Rahul B <rahulb at media.mit.edu> wrote:
>>>
>>>> Pablo do you currently use the dev installation of PageOneX on our
>>>> civicdev server?
>>>>
>>>> Rahul
>>>>
>>>>
>>>> On Mon, Jul 23, 2018 at 11:19 AM Rahul B <rahulb at media.mit.edu> wrote:
>>>>
>>>>> We are planning to deploy to production as a Docker image under Dokku (
>>>>> details
>>>>> <http://dokku.viewdocs.io/dokku/deployment/methods/dockerfiles/>).
>>>>>
>>>>> I believe we run under Apache with Passenger right now. Since this
>>>>> will now be a Docker image the setup script can choose whatever it wants.
>>>>>
>>>>> Rahul
>>>>>
>>>>> On Mon, Jul 23, 2018 at 11:11 AM Rafael Porres Molina <
>>>>> rporres at gmail.com> wrote:
>>>>>
>>>>>> Hi everybody,
>>>>>>
>>>>>> I have a good part of the Dockerize feature to be reviewed. Please
>>>>>> take a look: https://github.com/numeroteca/pageonex/pull/213
>>>>>>
>>>>>> What's missing:
>>>>>>
>>>>>> * Image mount. I will try to figure out how to control that part once
>>>>>> I get to know better the app
>>>>>> * CI: I will need to know how we plan to deploy this once it is
>>>>>> containerized to see if something like Travis fits here to create the
>>>>>> docker images under certain conditions, e.g tag
>>>>>> * Run under a proper server. What are we currently using: Unicorn?
>>>>>> Puma? Passenger?
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Rafa
>>>>>>
>>>>>> El mié., 11 jul. 2018 a las 0:40, pablo rey (<pablo at basurama.org>)
>>>>>> escribió:
>>>>>>
>>>>>>> I will review in a fresh install (without docker) if the migration
>>>>>>> works and let you know if I have the same problem with the migration.
>>>>>>>
>>>>>>> The changes in config/database.yml will modify something from the
>>>>>>> current way the tool works?
>>>>>>>
>>>>>>> On 9 July 2018 at 18:36, Rafael Porres Molina <rporres at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> See https://github.com/numeroteca/pageonex/issues/209 if you want
>>>>>>>> to take a look to what I've done so far. There's a problem with a migration
>>>>>>>> so I haven't been able to test it properly. I will continue working at some
>>>>>>>> point in the following days.
>>>>>>>>
>>>>>>>> It is not finished, so I haven't done a pull request. But if you
>>>>>>>> prefer that I do it so that you can take a look more easily, please let me
>>>>>>>> know and I will create one (tagged with DO NOT MERGE)
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Rafa
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> El lun., 9 jul. 2018 a las 10:22, Rafael Porres Molina (<
>>>>>>>> rporres at gmail.com>) escribió:
>>>>>>>>
>>>>>>>>> I'll start taking a look and keep you posted.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Rafa
>>>>>>>>>
>>>>>>>>> El vie., 6 jul. 2018 a las 19:14, Rahul B (<rahulb at media.mit.edu>)
>>>>>>>>> escribió:
>>>>>>>>>
>>>>>>>>>> Yes, a Docker container would work fine. Of course the problem
>>>>>>>>>> there is we need persistent storage for the database and the images.  Both
>>>>>>>>>> of these should be read from environment variables.
>>>>>>>>>>
>>>>>>>>>> I think this requires a change in the PageOneX configuration
>>>>>>>>>> code.  Rails already supports specifying a database via a
>>>>>>>>>> `DATABASE_URL` environment variable
>>>>>>>>>> <http://guides.rubyonrails.org/configuring.html#configuring-a-database>
>>>>>>>>>> .
>>>>>>>>>>
>>>>>>>>>> For docker it looks like you can specific environment variables
>>>>>>>>>> either on the command line individually or via a file (docs
>>>>>>>>>> <https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e---env---env-file>
>>>>>>>>>> ).
>>>>>>>>>>
>>>>>>>>>> With that in mind, I think the main changes to make are probably
>>>>>>>>>> to
>>>>>>>>>> a) Dockerize it
>>>>>>>>>> b) change the image uploading to save images to the path the
>>>>>>>>>> environment variable specifies
>>>>>>>>>>
>>>>>>>>>> Rahul
>>>>>>>>>>
>>>>>>>>>> On Fri, Jul 6, 2018 at 12:38 PM Rafael Porres Molina <
>>>>>>>>>> rporres at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Rahul,
>>>>>>>>>>>
>>>>>>>>>>> I'm unsure if Pablo will be able to make it this late in the
>>>>>>>>>>> spanish afternoon, so I will ask some quesitons here just to understand
>>>>>>>>>>> better what's needed.
>>>>>>>>>>>
>>>>>>>>>>> I guess what it is needed now is to be able to run pageonex in a
>>>>>>>>>>> Docker container, am I right?
>>>>>>>>>>>
>>>>>>>>>>> The other thing that is important is to be able to configure the
>>>>>>>>>>> root of the image storage through an env variable so that you can do
>>>>>>>>>>> something like
>>>>>>>>>>>
>>>>>>>>>>> docker run --env ROOT_IMAGE_STORAGE=/blah -v
>>>>>>>>>>> /dokku/persistent/storage/pageonex:/blah ....
>>>>>>>>>>>
>>>>>>>>>>> I don't know how pageonex works internally so I don't know how
>>>>>>>>>>> easy is to fix that. But would the following work?
>>>>>>>>>>>
>>>>>>>>>>> docker run -v  /dokku/persistent/storage/pageonex:/
>>>>>>>>>>> assets/images/kiosko...
>>>>>>>>>>>
>>>>>>>>>>> or whatever combination around this? I guess that will depend on
>>>>>>>>>>> where the application writes the assets once is running inside the
>>>>>>>>>>> container...
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>>
>>>>>>>>>>> Rafa
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> El vie., 6 jul. 2018 a las 18:12, Rahul B (<rahulb at media.mit.edu>)
>>>>>>>>>>> escribió:
>>>>>>>>>>>
>>>>>>>>>>>> Hi - I have 20 mins now, or in about an hour.  If you are still
>>>>>>>>>>>> available you can find me on Skype at civic-rahulb.
>>>>>>>>>>>>
>>>>>>>>>>>> Rahul
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Jul 6, 2018 at 6:36 AM pablo rey <pablo at basurama.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hey Rahul, Let's chat when you are available today. Rafa said
>>>>>>>>>>>>> that he will join. He can better help with the containers and image storage
>>>>>>>>>>>>> location thing better than me.
>>>>>>>>>>>>> Ping me.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 5 July 2018 at 15:53, Rahul B <rahulb at media.mit.edu> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes - you can deploy a Docker image via Dokku (details
>>>>>>>>>>>>>> <http://dokku.viewdocs.io/dokku/deployment/methods/dockerfiles/>).
>>>>>>>>>>>>>> That said, deploying as a Heroic build pack is a little simpler.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regarding the images, the feature suggestion I made (to
>>>>>>>>>>>>>> specify the storage location via an environment variable), will let us move
>>>>>>>>>>>>>> the image storage to the larger hard drive.  That is precisely the goal.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If you'd like to chat I have time today or tomorrow morning
>>>>>>>>>>>>>> (Boston time).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Rahul
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Jul 4, 2018 at 7:07 AM pablo rey <pablo at basurama.org>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hey Rafa, that'd be great. Thanks!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Let's see what Rahul says.
>>>>>>>>>>>>>>> Are Docker and Dokku interchangeable?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 4 July 2018 at 12:06, Rafael Porres Molina <
>>>>>>>>>>>>>>> rporres at gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hey Pablo,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This next Monday I can have some hours helping you if you
>>>>>>>>>>>>>>>> need to move the application to Docker. Prior to that, if you want we can
>>>>>>>>>>>>>>>> set a meeting with Rahul so that the time that I can work on the project is
>>>>>>>>>>>>>>>> more focused.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Let me know if this works for you.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Rafa
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> El mié., 4 jul. 2018 a las 0:25, pablo rey (<
>>>>>>>>>>>>>>>> pablo at basurama.org>) escribió:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Rahul,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for the good news, happy to continue at civic media
>>>>>>>>>>>>>>>>> servers.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We have modest plans for this round of development, let's
>>>>>>>>>>>>>>>>> see how we do.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I am not familiarized with Dokku and containerized, how
>>>>>>>>>>>>>>>>> difficult is to change to that mode of deployment?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regarding image storage, remember you tried in
>>>>>>>>>>>>>>>>> November'17.  You wrote me off the list "I'm going to copy the giant folder
>>>>>>>>>>>>>>>>> of kiosko images to the bigger drive I mentioned again, but this time with
>>>>>>>>>>>>>>>>> a flag to maintain all the permissions. Then perhaps we can schedule a time
>>>>>>>>>>>>>>>>> where I can flip it to use the larger hard drive and you can make a thread
>>>>>>>>>>>>>>>>> and test some old ones to tell me if it works.  If it does, all good and
>>>>>>>>>>>>>>>>> I'll delete the giant folder on the main drive that is running low on
>>>>>>>>>>>>>>>>> space.  If it doesn't work then I can just quickly put it back the way it
>>>>>>>>>>>>>>>>> was and think harder about it." I think that caused some
>>>>>>>>>>>>>>>>> problems of image missing
>>>>>>>>>>>>>>>>> <https://github.com/numeroteca/pageonex/issues/203#issuecomment-361261644>,
>>>>>>>>>>>>>>>>> still unsolved.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> You suggest that Dokku's persistent storage will enable
>>>>>>>>>>>>>>>>> that change in storage. We'll need your help to make that shift.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>> pablo
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 3 July 2018 at 15:29, Rahul B <rahulb at media.mit.edu>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Pleasure to meet you Juanjo.  Glad to see some new life
>>>>>>>>>>>>>>>>>> breathing into the project.  We host it here and the civic media servers,
>>>>>>>>>>>>>>>>>> and I confirmed with Ethan that he is willing to continue to do so.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> * We are using RVM, but are switching to a containerized
>>>>>>>>>>>>>>>>>> deployment system soon.  That means it'd be best to think about
>>>>>>>>>>>>>>>>>> Heroku-style deployment (like PageOneX originally did).
>>>>>>>>>>>>>>>>>> * We will be using Dokku
>>>>>>>>>>>>>>>>>> <http://dokku.viewdocs.io/dokku/> for containerized
>>>>>>>>>>>>>>>>>> deployment to our servers, we you will be able to deploy with a `git push`
>>>>>>>>>>>>>>>>>> (once we set keys up). We are migrating our hosted applications to this
>>>>>>>>>>>>>>>>>> mode now, so this is super good timing!
>>>>>>>>>>>>>>>>>> * Image storage is the only complicated thing to think
>>>>>>>>>>>>>>>>>> about... right now the `assets/images/kiosko` folder is 115GB (!).  It'd be
>>>>>>>>>>>>>>>>>> great if you could prioritize a feature change that lets us specify the
>>>>>>>>>>>>>>>>>> root image storage path via a environment variable.  That would allow us to
>>>>>>>>>>>>>>>>>> use the Dokku persistent storage path
>>>>>>>>>>>>>>>>>> <https://github.com/dokku/dokku/blob/master/docs/advanced-usage/persistent-storage.md> to
>>>>>>>>>>>>>>>>>> specify a storage point on our giant 10TB network drive while still
>>>>>>>>>>>>>>>>>> supporting containerized deployment, rather than having it fill up our root
>>>>>>>>>>>>>>>>>> drive (as it does now)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Sound good?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Rahul
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Tue, Jul 3, 2018 at 4:03 AM pablo rey <
>>>>>>>>>>>>>>>>>> pablo at basurama.org> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hey!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> We are back after a long time. We have a few
>>>>>>>>>>>>>>>>>>> enhancements and things to fix in our road map (
>>>>>>>>>>>>>>>>>>> https://github.com/numeroteca/pageonex/projects/1).
>>>>>>>>>>>>>>>>>>> they are basic features to extend the use for an easier and deeper analysis
>>>>>>>>>>>>>>>>>>> (drawn areas would be able to store more data, not only the topic). Juanjo
>>>>>>>>>>>>>>>>>>> Bazán will be developing in RoR.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Rahul, we have a few doubts to solve regarding
>>>>>>>>>>>>>>>>>>> deployment process:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>    - We are planning to update the ruby version that
>>>>>>>>>>>>>>>>>>>    it's used. I guess that should be no problem if we are using RVM. Could you
>>>>>>>>>>>>>>>>>>>    confirm?
>>>>>>>>>>>>>>>>>>>    - Another related question to the gems. Which is the
>>>>>>>>>>>>>>>>>>>    deployment process in dev and production servers? is there an script that
>>>>>>>>>>>>>>>>>>>    executes the bundle/migrations of the database or everything is installed
>>>>>>>>>>>>>>>>>>>    "manually" or  something like capistrano is used?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Down the road, but not yet in the listed in the project,
>>>>>>>>>>>>>>>>>>> we are planning to update ruby, gems and bootstrap.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> If you want to join, just tell!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> p
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> ------------------------------------------
>>>>>>>>>>>>>>>>>> Rahul Bhargava
>>>>>>>>>>>>>>>>>> http://rahul.connectionlab.org
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>> Pageonexdev mailing list
>>>>>>>>>>>>>>>>>> Pageonexdev at mit.edu
>>>>>>>>>>>>>>>>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>> Pageonexdev mailing list
>>>>>>>>>>>>>>>>> Pageonexdev at mit.edu
>>>>>>>>>>>>>>>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>> Pageonexdev mailing list
>>>>>>>>>>>>>>>> Pageonexdev at mit.edu
>>>>>>>>>>>>>>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> Pageonexdev mailing list
>>>>>>>>>>>>>>> Pageonexdev at mit.edu
>>>>>>>>>>>>>>> http://mailman.mit.edu/mailman/listinfo/pageonexdev
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> ------------------------------------------
>>>>>>>>>>>>>> Rahul Bhargava
>>>>>>>>>>>>>> http://rahul.connectionlab.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>> ------------------------------------------
>>>>>>>>>>>> Rahul Bhargava
>>>>>>>>>>>> http://rahul.connectionlab.org
>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>> ------------------------------------------
>>>>>>>>>> Rahul Bhargava
>>>>>>>>>> http://rahul.connectionlab.org
>>>>>>>>>>
>>>>>>>>>
>>>>>>> --
>>>>> ------------------------------------------
>>>>> Rahul Bhargava
>>>>> http://rahul.connectionlab.org
>>>>>
>>>> --
>>>> ------------------------------------------
>>>> Rahul Bhargava
>>>> http://rahul.connectionlab.org
>>>>
>>>
>>> --
>> ------------------------------------------
>> Rahul Bhargava
>> http://rahul.connectionlab.org
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/pageonexdev/attachments/20180806/6133fa4c/attachment-0001.html

More information about the Pageonexdev mailing list