• Sparkfun - data storage

    image

    SparkFun, everyone’s favourite electronics internet retailer, has released a really cool new service called Phant. Well, Phant is the engine behind https://data.sparkfun.com/, a website that allows the “Internet of Things” to store simple data online.

    For example, you’ve got a little thermometer connected to an ethernet enabled Arduino, once you’ve connected it up to the data service, you can send and store the sensor readings in Sparkfun’s cloud.

    As SparkFun says:

    What types of projects would benefit from this service?

    Almost anything. Your weather station in the backyard. A classroom of kids working on science experiments. A community concerned with pollution and crowdsourcing data collection. The weight of your pug’s food dish. You get the idea. If your project can send an http request, you can push data out to our servers.

    I have started writing a simple PHP class that connects to the Phant system.

    It’s called PHPePhant, and it’s hosted on GitHub. It’s still in its early days (it only POSTs data so far), but I hope to build it out soon. I haven’t written an open sourced PHP class before, so if you have any ideas or suggestions let me know. Better still, just dive right in and submit a pull request!

    I can already think of heaps of ways to use this lovely little (and really simple) service, so thanks SparkFun!

    Currently I’m collecting a few parts to create little sensors (which I’ll connect via a really simple cheap 433MHz radio system) that can detect the temperature in my house, if the light is on or not, when the door was opened, etc. All is fed back to the master Arduino, which will feed the data right into Phant. Perfect.

  • LFTP - some commands and references

    Great little FTP client, but a steeper learning curve from normal GUI based ones. Here are a few commands/references I occasionally use. I’ll try to keep this post up to date as I go.

    Connect to server:

    (If wanting to connect via SFTP, add it at the start like below)

    lftp
    open sftp://host
    user USER
    

    It should then ask for a password. You can then ls and you should see the files.

    Download a directory:

    (A mirror command downloads multiple files at once and uses segmentation:)

    queue mirror -P 5 --use-pget-n=5 Directory
    

    One file

    pget -n 5 File.dmg
    

    Queue

    To delete a transfer from your queue, add the -d switch:

    queue -d #
    

    To stop and delete a running transfer, use the kill command.

    If you set lftp to allow multiple transfers simultaneously, kill #, where # is the number listed by jobs.

    kill all will stop and delete all transfers.

    Passwords

    If you want your password to be saved when you create a bookmark, use the command:

    set bmk:save-passwords true
    

    Sample aliases

    Edit MAC OS X at: /usr/local/etc/lftp.conf

    ##MyMods 
    set bmk:save-passwords true 
    alias dir !ls -Fsh --group-directories-first
    

    And a list of aliases:

    alias j jobs
    alias jv jobs -v
    alias q queue
    alias qs queue stop
    alias k kill
    alias ka kill all
    alias m mirror -c -P 2 --use-pget-n=5
    alias p pget -c -n 6
    

    Adding bookmarks

    To do this, use the “bookmark” command.

    Then these:

    add <name> [<loc>] 
    del <name> 
    list
    

    Full example of the process

    1. Navigate in Terminal to the folder you want, then connect to FTP server from that terminal window
    2. After it connects, it should say cd ok
    3. List all files with ls
    4. To download a file from the FTP, type: q p FILENAME
    5. NOTE: You don’t have to type the whole file name. Just the start, then you can fill the rest by pushing the tab key.

    It’s an easy way to use the FTP server to upload and download files.

  • The Martian

    I finished reading Andy Weir’s The Martian last night.

    I started the day before. I love Mars!

    It’s an easy read. Check it out.

  • Moving from Wordpress to GitHub Pages

    Bye bye Wordpress. Hello GitHub Pages!

    I just decided to move my blogging platform away from the tried-and-true Wordpress to GitHub.

    Why?

    Well, for a few reasons.

    • I can write posts in Markdown
    • I don’t have to worry about zero-day Wordpress vulnerabilities
    • Version tracking of blog posts!
    • Why not?

    It wasn’t as hard as I thought it would be.

    I followed this guide, which was really helpful in showing how to export comments, move them to Disqus, etc.

    Now I’m just waiting for the DNS changes to take place, and it’ll be all done (apart from learning the way Jekyll works!)

  • Deleting Amazon AWS Glacier vaults and archives

    I once used Arq – a backup program for Mac – to backup all my documents to Amazon’s AWS cloud. I decided to use Glacier, a low-cost and long-term option to do this.

    However, it all got out of sync and Arq wanted to re-back everything up when I reinstalled my Mac OS X, so I decided to ditch it.

    That left me with many vaults on Amazon Glacier that I couldn’t get rid of – but was still paying for. The AWS Console only allows you to delete vaults if they are empty.

    There’s this error message. How annoying!

    Screen Shot 2014-06-25 at 11.01.08 am

    There is a way to delete the archives and vaults, but it takes a few steps. Here’s how I did it.

    1) Find a VPS provider and spin up a new server

    I used Digital Ocean. It’s quick and cheap and easy. I used a VPS because at the end, I can nuke it and not worry about having programs I will no longer need installed on my computer.

    Screen Shot 2014-06-25 at 10.34.48 am

    I installed Ubuntu 12.04.4 x64

    2) SSH into your new VPS

    Once you’ve SSHed into the server, you’ll have to install a bit of software.

    I used this “Perl Multithreaded multipart sync to Amazon Glacier service.

    To install, I ran this commands:

    sudo apt-get update
    sudo apt-get install software-properties-common python-software-properties
    sudo add-apt-repository ppa:vsespb/mt-aws-glacier
    sudo apt-get update
    sudo apt-get install libapp-mtaws-perl
    

    That will install the software needed to interface with Glacier.

    From there, you’ll need to get your Amazon security tokens to the software can connect to the cloud.

    You’ll need to put it into a config file. Here’s an example of what it should look like:

    key=YOURKEY
    secret=YOURSECRET
    #region: eu-west-1, us-east-1 etc
    region=us-east-1
    #protocol=http (default) or https
    protocol=http
    

    I called my file: glacier.cfg

    Once you’ve done that, you’ll need to get the “inventory” of each vault you want to delete.

    To do that, log into the AWS Console and go to the Glacier section. Find the region the vault is stored.

    You’ll see a list of the vaults:

    Screen Shot 2014-06-25 at 10.51.21 am

    Then run this command for each of the vaults:

    mtglacier retrieve-inventory --config glacier.cfg --vault vault-name
    

    Replace “vault-name” with the name of the vault you see in the list.

    If you’ve done it correctly, you should see something like:

    MT-AWS-Glacier, Copyright 2012-2014 Victor Efimov http://mt-aws.com/ Version 1.115
    
    
    PID 2551 Started worker  
    PID 2551 Retrieved Inventory, job id XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  
    OK DONE
    

    Do it for each of the vaults you want to delete.

    Once you’ve done that, you’ll need to wait a while. Possibly a few hours. I waited about eight or so.

    After you’ve waited, there are a few more commands to run:

    Firstly, you’ll need to download the inventory. Replace “vault-name” twice. Once for the name of the vault and once so the vault’s name matches the log file.

    mtglacier download-inventory --config glacier.cfg --vault vault-name --new-journal vault-name.log
    

    You’ll see a similar message as above.

    Then, you need to purge the vault. Here’s the command. Don’t forget to replace the “vault-name” and make sure the .log file matches what you set it to in the previous step.

    mtglacier purge-vault --config glacier.cfg --vault vault-name --journal vault-name.log
    

    It should then run through the vault and delete the archives. Depending on how many archives are in the vault, it may take a while. It took me about 10-15 minutes to delete 20,000 archives – so don’t get impatient. Just let it run.

    Once that’s done, you may have to wait a while again. Then the vaults will be empty. You can then delete from the AWS Glacier Console!