My Life, Universe and Everything

Tuesday, October 14, 2014

iPhone 6 Plus vs Nokia 3310 Brick

I had a good chuckle over this joke, found making the rounds on Facebook. My kids have a little brick phone and they are indestructible.

Tuesday, October 7, 2014

Blood Moon

Blood Moon (image from Google Images)

Tomorrow morning (Wednesday, October 8th, 2014), there will be a "Blood Moon" total lunar eclipse. The eclipse will appear from 4:20 am to 5:20 am, with the peak maximum total eclipse appearing at around 4:50 am (for Salt Lake City).

Local viewing times can be found on the Eclipse Calculator.

Viewing conditions as reported by AccuWeather

UPDATE: I posted the photo I took of the Blood Moon on my 365 project.

Friday, October 3, 2014

Introduction to the Google Calendar API (HOWTO)

I have used Google Calendars to schedule and control a number of projects (eg. sprinkler system, alarm clock, etc) . The following How To will get you started. You will of course need a google account.

Create Project

1) Start by visiting the Google Developers Console

and create a new project.

2) Select the project and navigate on the left menu to APIs & auth then APIs and enable Calendar API for this project. You can disabled all other APIs if you only need Calendar access.

3) Next, select the Consent screen menu option from the APIs & auth menu. Enter a Product Name and select an Email Address. If you do not do this step, you will get a Error: invalid_client error later on.

4) Next, select the Credentials menu option from the APIs & auth menu. Under OAuth select Create new Client ID.

5) For the Create Client ID select Installed application for Application Type and select Other for Installed Application Type and finally click the Create Client ID button.

6) After the ID has finished being created, click the Download JSON button. Save and rename the file as something simple like client_secret.json.

This json file contains your API credentials needed to access the Google Calendar APIs.

Install Google API Libraries

1) Install the Google API Libraries using Python's PIP installer:

$ sudo pip install --upgrade google-api-python-client

gflags may or may not be needed, depending on what code you use: (may be optional)

$ sudo pip install --upgrade python-gflags

If you would prefer alternatives, view the Google APIs Client Library for Python page.

Authorize Application

Next we will need to run our application and authorize it against the desired account.

1) Clone my sample code:

# git clone https://github.com/oeey/gcalendar.git

The sample code is just a slight modification from the Getting Started Sample Code. The Google sample code has some outdated code that will throw some obsoleted warnings.

2) The application has not been authorized to an account yet. Run the application once and you will be asked to paste a validation URL into your browser. Login to your desired target account (with the calendars you want to access) and then paste the validation URL into your browser.

For convenience I have a first_auth.py script that is the same script as the gcalendar.py script, but terminates after authorization. You can run any of the scripts to complete this authorization step.

The first_auth.py is pretty simple:

from apiclient.discovery import build
from oauth2client.file import Storage
from oauth2client.client import AccessTokenRefreshError
from oauth2client.client import OAuth2WebServerFlow
from oauth2client.tools import run_flow
from oauth2client.client import flow_from_clientsecrets

def main():
    scope = 'https://www.googleapis.com/auth/calendar'
    flow = flow_from_clientsecrets('client_secret.json', scope=scope)

    storage = Storage('credentials.dat')
    credentials = storage.get()

    class fakeargparse(object):  # fake argparse.Namespace
        noauth_local_webserver = True
        logging_level = "ERROR"
    flags = fakeargparse()

    if credentials is None or credentials.invalid:
        credentials = run_flow(flow, storage, flags)

if __name__ == '__main__':
    main()

You may notice the "fakeargparse" code. The run_flow() call wants the flags to be set from the parameters pulled from argparse. I think that is overkill for what I needed, so I just created a fake container so run_flow() wouldn't complain.

Run the first_auth.py script to collect the application authorization.

$ python first_auth.py
Go to the following link in your browser:

    https://accounts.google.com/o/oauth2/auth?scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcalendar&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&client_id=1039XXXXXXXXXXXXXXXXXXXXXXXXXXcs46gdj2.apps.googleusercontent.com&access_type=offline

Enter verification code:

3) Copy the URL into the browser and accept the permissions.

4) You will be presented with a code, to which you will then enter back into the prompt of the first_auth.py application. The authorization will be stored in the credentials.dat file for future requests.

$ python first_auth.py
Go to the following link in your browser:

    https://accounts.google.com/o/oauth2/auth?scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcalendar&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&client_id=46XXXXXXXXXX2-bXXXXXXXXXXXXXusvh6.apps.googleusercontent.com&access_type=offline

Enter verification code: 4/WzAQfXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX2vw2M2Pl7OykQI
Authentication successful.

Now that we have our API credentials and are authorized to access an account, we can begin to play with the Google Calendars.

View Upcoming Events

The upcoming.py script builds off of the first_auth.py script, cycles through the next few upcoming calendar events and displays the event titles.

...
    if credentials is None or credentials.invalid:
        credentials = run_flow(flow, storage, flags)

    http = httplib2.Http()
    http = credentials.authorize(http)
    service = build('calendar', 'v3', http=http)

    print "Upcoming Events:"
    request = service.events().list(calendarId='primary')
    while request != None:
        response = request.execute()
        for event in response.get('items', []):
            print event.get('summary', 'NO SUMMARY')
        request = service.events().list_next(request, response)

This script defaults to the primary calendar associated with the account.

Calendar ID

The previous script defaults to the primary calendar associated with the account. If you wish to specify an alternate calendar, you will need the Calendar ID. A calendar's ID can be found on the Calendar Details setting page (same page you can change a calendar's name on). Look for the Calendar Address line, and the Calendar ID will be in the parenthesis. It will look something like "a3sd5221ap2qe5ksbev3ip4@group.calendar.google.com".

Next 12 Hours of Events

Finally, to specify a time range for events, I use the following code in my gcalendar.py script. This code will collect the next 12 hours worth of events.

  ...

    service = build('calendar', 'v3', http=http)

    # get the next 12 hours of events
    epoch_time = time.time()
    start_time = epoch_time - 3600  # 1 hour ago
    end_time = epoch_time + 12 * 3600  # 12 hours in the future
    tz_offset = - time.altzone / 3600
    if tz_offset < 0:
        tz_offset_str = "-%02d00" % abs(tz_offset)
    else:
        tz_offset_str = "+%02d00" % abs(tz_offset)
    start_time = datetime.datetime.fromtimestamp(start_time).strftime("%Y-%m-%dT%H:%M:%S") + tz_offset_str
    end_time = datetime.datetime.fromtimestamp(end_time).strftime("%Y-%m-%dT%H:%M:%S") + tz_offset_str

    print "Getting calendar events between: " + start_time + " and " + end_time

    events = service.events().list(calendarId='primary', timeMin=start_time, timeMax=end_time, singleEvents=True).execute()singleEvents=True).execute()
    #pprint.pprint(events)
    for event in events['items']:
        print event["summary]

And this is the basis for the code I use to schedule my sprinkler system with.

Shellshock Bash Vulnerability

System vulnerable to Shellshock

What type of systems are vulnerable? "Apache HTTP Servers that use CGI scripts (via mod_cgi and mod_cgid) that are written in Bash or launch to Bash subshells"

Quick test to see if your system is vulnerable:

env 'VAR=() { :;}; echo Bash is vulnerable!' 'FUNCTION()=() { :;}; echo Bash is vulnerable!' bash -c "echo Bash Test"

Example:

[root@testvm ~]# env 'VAR=() { :;}; echo Bash is vulnerable!' 'FUNCTION()=() { :;}; echo Bash is vulnerable!' bash -c "echo Bash Test"
Bash is vulnerable!
bash: FUNCTION(): line 0: syntax error near unexpected token `)'
bash: FUNCTION(): line 0: `FUNCTION() () { :;}; echo Bash is vulnerable!'
bash: error importing function definition for `FUNCTION'
Bash Test

To test if your web server is vulnerable, you can use the 'ShellShock' Bash Vulnerability Test Tool.

To secure your system, simply upgrade the Bash package (should be available on most distributions):

$  ## Debian Based
$ sudo apt-get update && sudo apt-get install --only-upgrade bash

$  ## RedHat Based
$ sudo yum update bash

References:

Danny Macaskill: The Ridge

Danny Macaskill takes on a death-defying mountain bike ride along the notorious Cuillin Ridgeline at the Isle of Skye in Scotland

Danny Macaskill: The Ridge - YouTube

Blowing Stuff Up with a Quadcopter

Destruction by Quadcopter

And now for something completely different...

What do you get if you attach a machine gun to a quadcopter?

Prototype Quadrotor with Machine Gun!

And if you fear the drones, here's a fun way to defend:

Johnny Dronehunter: Defender of Privacy - Official Trailer feat. Salvo 12 Shotgun Silencer

Wednesday, October 1, 2014

Invalid Argument When Trying to Write Direct I/O to a 4K Native Sector Size Device

Direct I/O Invalid Argument

When trying to write Direct I/O to a particular block device, I ran into this ugly error:

# dd if=/dev/zero of=/dev/sdb oflag=direct
dd: writing to ‘/dev/sdb’: Invalid argument
1+0 records in
0+0 records out
0 bytes (0 B) copied, 0.000202393 s, 0.0 kB/s

After some investigation, it turns out the cause was the device was actually configured and low level formatted to the newer 4K Native Sector Size (aka 4Kn Advanced Format). Older devices are all 512 byte sector size.

The problem is that the default block size for many applications is still 512 bytes. The solution is easy, once you know the cause. You just simply write blocks in 4K byte size.

# dd if=/dev/zero of=/dev/sdb oflag=direct bs=4k
^C
73063+0 records in
73063+0 records out
299266048 bytes (299 MB) copied, 4.83917 s, 61.8 MB/s

Is My Device 4K Sector Size?

Many newer storage devices may be configured for 512, 512e or 4K. So how can you tell if you have a disk device that is in 4K native sector size? There are several options:

1) fdisk will tell you:

# fdisk /dev/sdb
Note: sector size is 4096 (not 512)
...

2) blockdev can tell you:

# blockdev --help
 --getss                   get logical block (sector) size
 --getpbsz                 get physical block (sector) size

# ## 512 mode:
# blockdev --getss /dev/sda
512
# blockdev --getpbsz /dev/sda
512

# ## 4Kn mode:
# blockdev --getss /dev/sdb
4096
# blockdev --getpbsz /dev/sdb
4096

# ## 512e mode:
# blockdev --getss /dev/sdc
512
# blockdev --getpbsz /dev/sdc
4096

3) Your storage vendor's tools should tell you. Many times the storage vendor will also provide tools to low level format to either 512, 512e or 4K

Why 4Kn?

So why are some storage vendors providing newer storage devices in 4K sector size? Efficiency, and Data Integrity.

"Larger sectors use the storage surface area more efficiently for large files but less efficiently for smaller files, and enable the integration of stronger error correction algorithms to maintain data integrity at higher storage densities." (source)

Summary:

Greater storage efficiency for larger files (but less efficient for smaller files)

Better bandwidth performance (large block read/writes), but IOPS (small block read/writes) will suffer

Improved error correction algorithms to maintain data integrity at higher storage densities

512e Alternative?

For Operating Systems and applications that may not support 4K native sector size, vendors may also provide a 512e mode (512 emulation). 512e mode is a 512 byte local sector size front-end emulation with 4K byte physical sector size back-end.

The Operating System would hand down a 512 byte request as usual, but then the storage controller would convert the request to a 4K byte request. This provides the efficiency and data integrity benefits for the physical storage, but may come with a performance hit, due to the emulation and conversion required (especially if not aligned properly).

This mode also allows a stepping stone for storage vendors to move forward with 4K technology, while still supporting older Operating Systems and applications.

Pages