<p dir="ltr">I had a niche case where ebs were attached to master and used as GlusterFS bricks and than we shared that GlusterFS volume across starcluster nodes. </p>
<p dir="ltr">So the ebs is encrypted now. But the connection between clients to glusterfs nodes is not secured yet. </p>
<p dir="ltr">But it achieved my immediate goal for this test. </p>
<p dir="ltr">Thanks for all your help. </p>
<p dir="ltr">Ronen</p>
<div class="gmail_quote">On Jul 3, 2014 8:25 PM, "Rayson Ho" <<a href="mailto:raysonlogin@gmail.com">raysonlogin@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Thu, Jul 3, 2014 at 1:55 AM, ronen artzi <span dir="ltr"><<a href="mailto:ronen.artzi@gmail.com" target="_blank">ronen.artzi@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><span style="font-family:arial,sans-serif;font-size:12.7273px"><br></span></div>
<div><font face="arial, sans-serif">This one does not solve (?) the protocol encryption ( hence the encryption on the fly challenge but provide encryption at rest ). </font></div></div></div></blockquote><div><br><br></div>
<div>EBS provides <font face="arial, sans-serif">encryption at rest, but according to Amazon data is already encrypted when it leaves the instance.<br><br> "... </font><font face="arial, sans-serif">your data is encrypted before it leaves the
EC2 instance. In order to be able to do this efficiently and with low
latency, the EBS encryption feature is only available on EC2's M3, C3, R3,
CR1, G2, and I2 instances":<br><br><a href="http://aws.amazon.com/blogs/aws/protect-your-data-with-new-ebs-encryption/" target="_blank">http://aws.amazon.com/blogs/aws/protect-your-data-with-new-ebs-encryption/</a><br>
</font></div>
<div><br></div><div>However, I do get what you mean -- you want to encrypt the data before it reaches EBS or AWS, but then you most likely don't need to change the StarCluster code to pass encrypted=True to boto's create_volume(), as EBS encryption is not that useful in your use case.<br>
</div><div><br clear="all"><div>Rayson<br><br>==================================================<br>Open Grid Scheduler - The Official Open Source Grid Engine<br><a href="http://gridscheduler.sourceforge.net/" target="_blank">http://gridscheduler.sourceforge.net/</a><br>
<a href="http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html" target="_blank">http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html</a><br><br><br></div><br><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><div>
<div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif"><br></font></div></div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jul 2, 2014 at 11:58 PM, Rayson Ho <span dir="ltr"><<a href="mailto:raysonlogin@gmail.com" target="_blank">raysonlogin@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hmm, I was not very clear in my previous email...<br><br>create_volume() called in volume.py is defined in the StarCluster's EasyAWS class (awsutils.py). So if you want to pass encypted=True to boto's create_volume(), you will need to add the extra parameter in awsutils.py:<br>
<br> def create_volume(self, size, zone, snapshot_id=None):<br> ...<br> return self.conn.create_volume(size, zone, snapshot_id, encypted=True)<br><br></div>So you just need to work your way up the call chain so that you can determine the value passed in by the user from the command line.<br>
<div><br>(When your code is general purpose enough to be included in the StarCluster repo, create a pull request in github.)<br><div><div class="gmail_extra"><div><br clear="all"><div>Rayson<br><br>==================================================<br>
Open Grid Scheduler - The Official Open Source Grid Engine<br><a href="http://gridscheduler.sourceforge.net/" target="_blank">http://gridscheduler.sourceforge.net/</a><br><a href="http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html" target="_blank">http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html</a></div>
<br><br></div><div><div><div class="gmail_quote">On Wed, Jul 2, 2014 at 11:07 PM, ronen artzi <span dir="ltr"><<a href="mailto:ronen.artzi@gmail.com" target="_blank">ronen.artzi@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Thanks Rayson for your prompt response. <div><br></div><div> I am running on the "Cluster Launcher/controller" node with Boto 2.30.0 installed.<br></div><div>in my code I am using (encrypt is the option I am getting from the user : True/False): </div>
<div><br></div><div><div> <a href="http://log.info" target="_blank">log.info</a>(">>>>>>>>>>>> Creating new Volume with encrypt option : %s" % encrypt)</div>
<div> vol = self.ec2.create_volume(size, zone, None,None,None,encrypt,None)</div>
<div> <a href="http://log.info" target="_blank">log.info</a>("New volume id: %s" % <a href="http://vol.id" target="_blank">vol.id</a>)</div></div><div><br></div><div>The error I am getting : </div>
<div><br></div><div><div> <font color="#cc0000">File "/home/ubuntu/.starcluster/plugins/glusterfsencrypt.py", line 110, in run</font></div>
<div><font color="#cc0000"> volume = self._create_ec2_volume(<a href="http://self.name" target="_blank">self.name</a>, self.ebs_size, zone , self.encrypt)</font></div><div><font color="#cc0000"> </font></div><div><font color="#cc0000">File "/home/ubuntu/.starcluster/plugins/glusterfsencrypt.py", line 363, in _create_ec2_volume</font></div>
<div><font color="#cc0000"> vol = self.ec2.create_volume(size, zone, None,None,None,encrypt)</font></div><div><font color="#cc0000"><br></font></div><div><font style="background-color:rgb(255,255,255)" color="#cc0000">TypeError: create_volume() takes at most 4 arguments (8 given)</font></div>
</div><div><font style="background-color:rgb(241,194,50)" color="#cc0000"><br></font></div><div><br></div><div><br></div><div>On another note : when I am calling the create_function with encypted=encrypt </div><div><br></div>
<div>i am getting : </div><div><br></div><div><div>vol = self.ec2.create_volume(size, zone, None,encrypted=encrypt)</div><div>TypeError: create_volume() got an unexpected keyword argument 'encrypted'</div></div><div>
<br></div><div><br></div></div><div class="gmail_extra"><div><div><br><br><div class="gmail_quote">On Wed, Jul 2, 2014 at 6:35 PM, Rayson Ho <span dir="ltr"><<a href="mailto:raysonlogin@gmail.com" target="_blank">raysonlogin@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">StarCluster only needs the first 3 function parameters, thus it relies on the default parameters for boto's create_volume() function. If all you want is to pass in the encrypted boolean, then you can add encrypted=True to the list of parameters for create_volume().<div>
<br></div><div>For you code to be general purpose enough to be included, you will need to add logic so that is does not always pass the encrypted=True bool into the function all the time, and instead read the user's choice from the command line (see createvolume.py).</div>
<div class="gmail_extra"><br clear="all"><div>Rayson<br><br>==================================================<br>Open Grid Scheduler - The Official Open Source Grid Engine<br><a href="http://gridscheduler.sourceforge.net/" target="_blank">http://gridscheduler.sourceforge.net/</a><br>
<a href="http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html" target="_blank">http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html</a></div>
<br><br><div class="gmail_quote"><div><div>On Wed, Jul 2, 2014 at 5:42 PM, ronen artzi <span dir="ltr"><<a href="mailto:ronen.artzi@gmail.com" target="_blank">ronen.artzi@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div>
<div dir="ltr">Hello,<div><br></div><div>I am trying to extend a glusterfs plugin we have , and would like to introduce the options of encrypting the ESB bricks.</div><div><br></div><div>I'd like to be able to use the full parameter list coming with ec2 fro volume creation : </div>
<div><br></div><div><tt style="padding:1px 2px;font-size:14px;font-weight:bold;line-height:21.6px">create_volume</tt><big style="color:rgb(62,67,73);font-family:'Lucida Grande','Lucida Sans Unicode',Geneva,Verdana,Arial,sans-serif;line-height:21.6px">(</big><i style="color:rgb(62,67,73);font-family:'Lucida Grande','Lucida Sans Unicode',Geneva,Verdana,Arial,sans-serif;font-size:14px;line-height:21.6px">size</i><span style="color:rgb(62,67,73);font-family:'Lucida Grande','Lucida Sans Unicode',Geneva,Verdana,Arial,sans-serif;font-size:14px;line-height:21.6px">, </span><i style="color:rgb(62,67,73);font-family:'Lucida Grande','Lucida Sans Unicode',Geneva,Verdana,Arial,sans-serif;font-size:14px;line-height:21.6px">zone</i><span style="color:rgb(62,67,73);font-family:'Lucida Grande','Lucida Sans Unicode',Geneva,Verdana,Arial,sans-serif;font-size:14px;line-height:21.6px">, </span><i style="color:rgb(62,67,73);font-family:'Lucida Grande','Lucida Sans Unicode',Geneva,Verdana,Arial,sans-serif;font-size:14px;line-height:21.6px">snapshot=None</i><span style="color:rgb(62,67,73);font-family:'Lucida Grande','Lucida Sans Unicode',Geneva,Verdana,Arial,sans-serif;font-size:14px;line-height:21.6px">, </span><i style="color:rgb(62,67,73);font-family:'Lucida Grande','Lucida Sans Unicode',Geneva,Verdana,Arial,sans-serif;font-size:14px;line-height:21.6px">volume_type=None</i><span style="color:rgb(62,67,73);font-family:'Lucida Grande','Lucida Sans Unicode',Geneva,Verdana,Arial,sans-serif;font-size:14px;line-height:21.6px">, </span><i style="color:rgb(62,67,73);font-family:'Lucida Grande','Lucida Sans Unicode',Geneva,Verdana,Arial,sans-serif;font-size:14px;line-height:21.6px">iops=None</i><span style="color:rgb(62,67,73);font-family:'Lucida Grande','Lucida Sans Unicode',Geneva,Verdana,Arial,sans-serif;font-size:14px;line-height:21.6px">, </span><i style="color:rgb(62,67,73);font-family:'Lucida Grande','Lucida Sans Unicode',Geneva,Verdana,Arial,sans-serif;font-size:14px;line-height:21.6px">encrypted=False</i><span style="color:rgb(62,67,73);font-family:'Lucida Grande','Lucida Sans Unicode',Geneva,Verdana,Arial,sans-serif;font-size:14px;line-height:21.6px">, </span><i style="color:rgb(62,67,73);font-family:'Lucida Grande','Lucida Sans Unicode',Geneva,Verdana,Arial,sans-serif;font-size:14px;line-height:21.6px">dry_run=False</i><big style="color:rgb(62,67,73);font-family:'Lucida Grande','Lucida Sans Unicode',Geneva,Verdana,Arial,sans-serif;line-height:21.6px">)</big><br>
</div><div><br></div><div>Being very new to the starcluster code base I am not sure what areas are dealign with such volume creation. </div><div><br></div><div><font color="#3e4349" face="Lucida Grande, Lucida Sans Unicode, Geneva, Verdana, Arial, sans-serif" size="3"><span style="line-height:21.6px">it seems that under star cluster python packages area , volume.py support only the first 4 parameters. </span></font></div>
<div><div><br></div><div>Any insights here are appreciated. </div><span><font color="#888888">-- <br> Ronen<div><br></div>
</font></span></div></div>
<br></div></div>_______________________________________________<br>
StarCluster mailing list<br>
<a href="mailto:StarCluster@mit.edu" target="_blank">StarCluster@mit.edu</a><br>
<a href="http://mailman.mit.edu/mailman/listinfo/starcluster" target="_blank">http://mailman.mit.edu/mailman/listinfo/starcluster</a><br>
<br></blockquote></div><br></div></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br> - Ronen<br><div><span style="font-family:Verdana,Arial,Helvetica,sans-serif;font-size:13px"><span style="color:gray"> Cell: <a href="tel:%2B1-508-308-2417" value="+15083082417" target="_blank">+1-508-308-2417</a></span></span></div>
<div><span style="font-family:Verdana,Arial,Helvetica,sans-serif;font-size:13px"><span style="color:gray"> My profiles:</span> <a href="http://www.linkedin.com/pub/ronen-artzi/0/30/4b9" style="text-decoration:underline" target="_blank"><img alt="LinkedIn" style="padding:0px 0px 5px;vertical-align:middle" border="0" height="16" width="16"></a> <a href="http://tungle.me/ronenartzi" style="text-decoration:underline" target="_blank"><img alt="Tungle.me" style="padding:0px 0px 5px;vertical-align:middle" border="0" height="16" width="16"></a></span></div>
<div><br></div>
</font></span></div>
</blockquote></div><br></div></div></div></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br> - Ronen<br><div><span style="font-family:Verdana,Arial,Helvetica,sans-serif;font-size:13px"><span style="color:gray"> Cell: <a href="tel:%2B1-508-308-2417" value="+15083082417" target="_blank">+1-508-308-2417</a></span></span></div>
<div><span style="font-family:Verdana,Arial,Helvetica,sans-serif;font-size:13px"><span style="color:gray"> My profiles:</span> <a href="http://www.linkedin.com/pub/ronen-artzi/0/30/4b9" style="text-decoration:underline" target="_blank"><img alt="LinkedIn" src="http://images.wisestamp.com/linkedin.png" style="padding:0px 0px 5px;vertical-align:middle" border="0" height="16" width="16"></a> <a href="http://tungle.me/ronenartzi" style="text-decoration:underline" target="_blank"><img alt="Tungle.me" src="http://images.wisestamp.com/tungle.png" style="padding:0px 0px 5px;vertical-align:middle" border="0" height="16" width="16"></a></span></div>
<div><br></div>
</div>
</div></div></blockquote></div><br></div></div>
</blockquote></div>