Category Archives: Tech

Severe Weather Alerts API – JSON Web Service

I’ve had a couple requests over the past few months to add a webservice to my WeatherAlerts project, the code for this is now functional (and working quite well) but I’m doing some testing with it before I release the new version. If you’d like to experiment with the webservice you can either run an instance yourself (three lines of code to get up and running) or use my reference/test implementation. The new feature is on the development branch of WeatherAlerts on github and you can find the documentation for it here.

If you’d like to try out a live instance, you can experiment with mine which is located at http://wxalertws.zebpalmer.com/  take a look at the documentation for how it works, but currently there are only two endpoints. First hitting http://wxalertws.zebpalmer.com/all will give you json output of all active alerts in the US. If you want to get just your own area’s alerts, look up your SAMECODE  and query the API like so: http://wxalertws.zebpalmer.com/samecodes/016027,016001   replacing those samecodes with one or more samecodes that you care about.

The data is downloaded from the National Weather Service CAP Emergency Alerts Feed via my WeatherAlerts python package and it’s updated every minute. I’ll leave this running for my own experimentation and use in my own automation projects (instead of running multiple instances of WeatherAlerts) so feel free to use if for tinkering. But standard disclaimer applies, DO NOT DEPEND ON THIS FOR IMPORTANT STUFF (including but not limited to your continued existence). Your primary source of data and alerting should be the National Weather Service and NWS Radio. 

You may ask why I’m adding a webservice to WeatherAlerts when the NWS already provides a XML/CAP feed….  1) because I am already parsing that feed via WeatherAlerts, adding a simple queryable WS API was easy, I’ll be adding other queryable fields soon 2) I hate XML 3) I’ve had several requests for it

Enjoy.

Tagged , , |

Cortana: Home Automation A.I. (Intro)

House Temps

We’ll dive more into this later, but for now here’s an old screenshot of our house temps dashboard.

Meet Cortana…

This post has been a long time coming… a few of my friends and people I’ve met in my travels have been asking a lot of questions about a little project of mine, but thus far I’ve only mentioned it in passing on Google+ and twitter. This post will largely be a teaser, an intro, but will be the first in a series on Home Automation (HA), specifically, my take on it. In this series, I’ll try to outline my background in HA, the current state of my HA, a few of my goals and the general scope and direction of my project. I’ll start with an overview of the architecture, hardware and software then follow up with a deep dive into specific components of my setup including the monitoring networks, the chatbot user interface, some of the logic in my software, problems I’ve run into, a couple areas where I wish I had more expertise and hardware I wish existed. We’ll see where we go from there.

First, let me introduce you to my project, I call it “Cortana”… ok, many of you will recognize the name from HALO. My wife and I are both HALO fans and when I was trying to decide on a persona for my project after pouring through lists of AI names in Sci-Fi past and present, Cortana was the obvious winner. More on the need for a persona in later posts. I also refer to it as an “AI”. My project is pretty smart and it does learn from past conditions and results, but sadly there is no high level ground breaking artificial intelligence work going on here. Everything that happens is programmed to happen based on various conditions, commands or other input, as I said, there is learning involved, but it’s highly programmed. I use the term “Automated Intelligence.”

A little context

It might help to have a little background and context. I’ve always been interested in home automation, it started with switching a light remotely, this is where many home automation projects start, and promptly end. But two years ago we bought our ‘dream’ home on the outskirts of town; three bedrooms upstairs, one downstairs (my photo studio), front room, living room, a large kitchen and dining room, sizable garden, large back yard, three car garage, the works. For us it’s a crazy big house and we love it… We live in southwest Idaho, which is mostly high desert. Two months out of the year it’s freakishly cold. Two months out of the year it’s freakishly hot. But for 6-8 months of the year it’s nice enough outside that we’ll have windows open at some point during the day/night. Typically, in spring and fall we’ll have the windows open 24/7 for a month or so. You’ll see how these points matter as we dive further into my HA implementation.

After buying the house, I started in on some basic home automation (lights and such) but some manual tasks started to annoy me and caused me to take a step back, one big driver for the scope of my project is energy efficiency. Thankfully we have plenty of electrons here in Idaho and they’re fairly cheap, but I had inefficient processes/systems, so it ranks pretty high. We’ll dive deeper into my Home Automation Manifesto in the next post. But as my goals started to grow and the complexity of what I wanted to manage grew, I realized what I was whiteboarding was far different than your average HA project.

Off the Shelf?

There are a ton of off the shelf home automation systems out there. The vast majority of them suck. Ok, some of them are pretty impressive, but impressive carries a large price tag. And actually, even the really cool, crazy expensive systems don’t do some of the core requirements of my system. After searching various commercial systems, I knew it would have to be DIY, of course that’d make it much more fun too. I also spent a lot of time looking into the existing Open Source HA platforms, none of them quite fit what I was looking for. The better ones hadn’t been touched in ages and showed no signs of being resurrected. I wasn’t about to pull one out of the grave that was written in (insert any non python language here). So apologies in advance for not using (insert your favorite HA platform here). Generally the problems were 1) to simple 2) to complex for what they did 3) not extensible 4) horribly documented 5) Written in perl 6) Written in java 7) bad architecture 8) w. t. f. were you thinking?

That’s not to say that my project is the most amazing thing since sliced bread, but it works for me. Hopefully though, even if you don’t ever use a single line of code I’ve written, my thoughts, trials and errors as noted in this series will help you design and build your HA system.

Got Source?

Those that know me, or see my twitter bio, or have ever talked to me for more than 5 minutes, know that I’m an open source junkie. I love the open source community and what we are able to accomplish. Currently, my project is not open source, it will be one day. Once a component of it reaches some level of maturity, I will release it. Take WeatherAlerts for instance it’s part of my HA system and I have packaged it and released it via PyPI with the source is on GitHub. There are a couple other packages and snippets I have released to the wild and I have plans on releasing more over the next few months. It’ll probably be later this year before any core code is released. The reason I haven’t, is largely due to the fact that my core code is currently VERY specific to my implementation. 

Alright, enough rambling for now, I’ll try to post the first 3-4 parts of this every over the next 3-4 weeks, after that I’ll start into some deeper technical details and such every couple of weeks or so or as I add new components. I hope this series gives you some ideas. Please let me know if you have any questions, comments, ideas or such.

Tagged , , , , , , , , |

WeatherAlerts Use Metrics

A couple years ago, I was needing a python script that would grab realtime Severe Weather Alerts and integrate that data into my Home Automation project. I couldn’t find anything (in any non-horrible programming language) that did what I wanted. So, I wrote a script to do it. It stayed hidden in the depths of my home automation project for several months until an acquaintance asked for a copy. I decided to go ahead and open source script, throwing it in a public github repo and packaging it for inclusion in PyPI. It’s called “WeatherAlerts.”  

Let me be completely clear, this package was a mess. It was my first foray into python packaging and it showed. It was also written much earlier in my python days, but, it worked (mostly). I decided that since it worked, if someone else could find use for it, or at least give them a starting point toward making something better… in the spirit of Open Source, I’d throw it out there. It’s had a few improvements over the past two years and recently it’s gotten much better but it’s still a functional mess though I’m working to fix that (the mess part).

Last year, my WeatherAlerts python package had a total of 3,574 downloads on PyPI. This year, it’s already at 2,409 with 1,287 so far this month… Since these numbers are just PyPI download counts, I don’t have any analytics on which to ponder, which drives me nuts. I do have analytics on the documentation I’ve posted and it has show a similar uptick recently. But it’s rather difficult to equate downloads to actual users considering bot downloads, redownloads, installs that are never used, cosmic events, etc. I’d love to know how many people actually use it…. When I decided to open source it, I assumed it might help as many as four or five users 🙂  Even if the PyPI numbers are way off, it is still being looked at a lot more than I would have expected.

Over the past few months, I’ve had a number of emails and social media contacts from users which has resulted in some great feedback and conversations. This makes me want to spend a few minutes some day to clean it up and make it presentable. 🙂 Regardless it has fulfilled the goal I had when I decided to open source it, that being to help more than 0 people, in the process I’ve learned more about python packaging, sphinx documentation and python as a whole.  

I have four public packages on PyPI and WeatherAlerts isn’t the most popular. NagParser is a package a Friend/Coworker and I open sourced shortly after I released WeatherAlerts. I just pulled the metrics for NagParser and it had over 17,300 downloads last year and is already sitting at 8,117 this year. The other two packages I have on PyPI are barely touched and getting less popular by the day (if that’s even possible) but I’m not offended, they’re very niche packages that are hardware or API subscription specific. 

I also have run my own PyPI/pip compatible server at home, for stuff that’s not quite open source ready or stuff very custom to me.

Tagged , , , , |

Python Continuous Integration and Deployment (with Jenkins)

I’ve only been in the python community for about a year, so I’m still new, but I really do love coding in python. Anyway, I decided to open source a library (‘WeatherAlerts’) I started writing a few weeks ago.  I’ll post about it at some point in the future….  At first it was just a code repo on github, but after a bit of time working with it I wanted to package it and put it on pypi so it could be installed via easy_install or pip. With the state of change that python is in right now figuring out how to write the installer, have it support both python 2 and python 3 and get the package available on pypi wasn’t trivial. But now that it’s behind me, I have a much better understanding of python package management.

I had a simple test script that I was manually running every few commits and before pushing any changes out to the public repo. It worked, but being a Linux Systems Engineer though, this wasn’t going to stay that way long term. A lot of what I do in my day job is systems automation. Deploy servers, mange changes to large clusters, Continuous Integration and Continuous Delivery at the system level… So while my simple little python project was perfectly fine with being a manual build and deploy process, I couldn’t stand it.

So, I searched out a pythonic way to automate these tasks.

Continue reading »

Also posted in Python Tagged , , , |

Fedora 12 Alpha Release

Booting the Fedora 12 Alpha CD for an install

Screenshot: Booting the Fedora 12 Alpha CD for an install

I don’t “review” alpha releases, but I’m very much looking forward to Fedora 12, it’s alpha spin was released today. Currently scheduled for early November, even with some pushback I would expect it out by Christmas, probably thanksgiving. Fedora 12 has a large changelog and feature list, a few big ones based on upstream changes to Gnome, KDE, etc. Last night I downloaded the 64bit live CD and installed it on a virtual machine for some testing. Sometimes working with beta software I forget it’s beta software; so many stable products and distros are beta. Alpha is a whole different ball game. I don’t suggest installing any alpha release on your primary desktop. It’s best left as a testing release for those of us who file bugs. If you want a preview, download the live CD and either boot it live on your computer (without installing) or install it on a virtual box. For my testing I’m using virtualbox on an AMD quad core with 8gb of ram and a nvidia video card. The Virtual machine is set to 2GB of ram, 2 processors, and 64mb video ram (3d acceleration turned on). My base settings for testing Linux distros.

You can view the full release announcement, but I’ll point out a few of the interesting bits. Continue reading »

Tagged , , , , |

KDE 4.3 on Fedora Linux – Screenshot & Mini review

KDE 4.3 as installed on my primary desktop

KDE 4.3 as installed on my primary desktop (click image for full size). Widgets in use include microblog (twitter), weather forcast, Google Calendar Gadget, moon phase, color picker, and post-it note.

KDE 4.3 was recently released and should be hitting the Fedora Stable repos very soon as a system update. I installed if from the updates-testing repo to get a look at what was coming. You may have seen the release announcement and screenshots on the KDE website.  When I first saw them I was quite impressed. But the story goes back a bit for me. When I first started in Linux (ages and ages ago) I played with the KDE desktop but always went to Gnome. KDE had too many things that could get screwed up and it was near impossible to achieve and maintain the eyecandy screen shots we all saw and even posted on the web. You could get everything set just right, reboot or change one font setting and everything was screwed up. That hasn’t been keeping me out of KDE all this time, but it’s what made me chose the Gnome Desktop in the first place. Since then and up until KDE 4.2 I have been a avid supporter of the Gnome Desktop. Actually, I still am. I’ve used KDE on testing platforms thought the years, but never on my primary desktop. When Fedora 11 launched I wanted to do a clean install and make everything ext4, among other changes. I sat at the ‘KDE or Gnome?’ prompt for what seemed like hours even though I knew I was going to pick KDE. I have a lot of time and energy in using Gnome, there’s history there. But Gnome wasn’t doing what I wanted on my desktop and I’d been seeing all the eye candy and functionality of recent KDE so I went with it. And I’m very, very pleased. I still love Gnome and it’s installed on every other computer I own and it will stay my choice for any production systems. But KDE has won my heart on my main computer. KDE version 4.2 was the first version that I really liked. It still has some quirks and wasn’t quite as stable as I would have liked, but over all, I loved it.

KDE 4.3 brings a new default theme, “Air,” and I have say it’s one of the best looking themes I’ve ever seen. Its simple, clean and very easy on the eyes. The screen shot I posted above is the default theme, though with a different background color. I tried the animated backgrounds and I like them, but animated backgrounds aren’t for me. This was where I found the only bug I’ve found so far. I could switch to any animated background and they worked. Once I switched to “virus” however, I couldn’t switch away from it. KDE would freeze and I had CTRL + ALT + BACKSPACE to restart X. Even after restart, I couldn’t switch to anything other than “virus” without freezing. I had to manually switch away from the virus wallpaper. To do this open ~/.kde/share/config/plasma-desktop-appletsrc and find the line wallpaperplugin=virus and change “virus” to “color” save and reboot (or restart X). So far this has been the only bug I’ve found.

The Plasma Desktop Shell is a large part of KDE and this update.

The Plasma Desktop Shell introduces a new default theme, Air. Air looks much lighter and fits better with the default application theme. Plasma also has seen large performance improvements. Memory usage has been reduced, and animations are smoother. Activities can now be tied to virtual desktops, allowing users to have different widgets on each of their desktops. Furthermore, Plasma has improved upon its job and notification management. Running jobs are grouped in a single progress bar to prevent the popup of too many dialogs. Animations are used to signify that jobs are still running by smoothly sliding dialogs into the systemtray and animating the notification icon. Smaller changes in Plasma include fully configurable keyboard shortcuts and more extensive keyboard navigation, the ability to create a plasma widget when you drag or copy content on the desktop and many new and improved Plasma widgets. (snippet from KDE 4.3 Release Announcement via KDE.org)

There is a lot of information in the rest of the release announcement: I’d encourage you to visit KDE.org and learn about other new features of KDE. If you interested in trying KDE 4.3 on Fedora before it hits the stable repo. Temporarily enable the updates-testing yum repo and do a group update. At that point, you’ll have KDE 4.3 running and will receive further KDE updates once it hits stable.

Run the following command as root to install it then you’ll need to reboot the system.

yum --enablerepo=updates-testing groupupdate "KDE (K Desktop Environment)" updated: kde 4.3 is now in fedora stable

If you don’t see the pretty new look, you may need to right click on the desktop, chose “Desktop settings” and set the theme to Default “Air”.

If you’re not a power user, I’d suggest you wait till KDE 4.3 hits the stable repo. I’ve only had the one issue with it (others have documented the same issue) but there is a reason it sits in the testing repo before hitting stable. I would expect to see it as a system update in the next few days. Enjoy.

UPDATE: Looks like KDE 4.3 Has dropped into the Fedora stable repo. So you can do a normal update to install it. Feel free to read my comments below or head over and read about what’s headed to Fedora as I comment on the Fedora 12 Alpha Release

Tagged , , , , , |

Twitter: how not to miss important posts

twitterQuick departure from the “Tech Tuesday” schedule and the new Linux series for a tech post about twitter networking.

I hit 500 followers on twitter today. No, I don’t care so much about that,  more importantly, I’m now following over 650 people. I do my best to manage the accounts I follow to keep it relevant to what I want to see. I have also tried to keep the ratio of followers to following about 1:1.  This does get thrown off by a few people, companies and organizations that I keep tabs on. That’s not to say if I add you on twitter and you don’t follow me back that I’ll unfollow you. If I add you, you have something interesting in your tweets or bio that made me want to follow you, and you’ll stay. Unless you put “social media expert” in your bio or have more than one automated post per day or have ads for your “give me $5 and I’ll show you how to…” website. Those are instant unfollows.

As the number of people I was following hit 300 or so, I started missing too many tweets that I wanted to see. I tried several different ways to filter out the junk posts without filtering random tweets I might want to see. It all got easier when I started using TweetDeck

TweetDeck is a great little program that will manage multiple twitter programs, has amazing organizational abilities, and is cross platform, running on Adobe Air. With TweetDeck you can easily create groups of people you follow. I have six columns running in TweetDeck. The left is the default “All Friends.” Then I have three columns that search all of twitter for keywords that I tweet about. One is a photography (photography, photographer…) related search, next is Linux related (linux, redhat, fedora), next a local search (boise, idaho), and then I have the column for mentions and @replies, basically a search for “zebpalmer”. Finally, and most importantly, is my “VIPs” group. It’d probably give the users in the column too much of an ego if they knew they were in a group titled “VIPs”, but I created this group with very little sleep or forethought; it was the first random string I could type into the “Please name the group” box. Anytime I really like a twitter user or find a news feed I really like I add them to the group.

This setup is perfect for me as I can see whatever is scrolling by in “all friends” and may randomly see something I like. I also get to find new users and keep tabs on the twitterverse in the keyword searches. And I don’t miss tweets from high value twitter feeds and friends.

Tagged , |

How Far We’ve Come with Linux – Part 1

Last week my wife’s laptop started dying; for the record, it still is. In the interest of full disclosure, I love Linux. I don’t have a computer in this house running windows natively, though I do use windows on a virtual box to run Photoshop CS4  and Adobe Camera Raw. I also have countless virtual machines of windows (and other linux distros) for testing purposes–as a consultant I work a lot with windows, but it’s not the OS for me. I’m not a apple fanboy either, I’m a red hat guy: servers run CentOS, desktops run Fedora.

My wife doesn’t mind Fedora, but she’s switched to Ubuntu for the last few years on her laptop. She’s an English teacher at a state college and uses Linux and Open office, how great is that? How many College English teachers do you know that accept papers in Open Document format? Oh and the other day when I was watching the trailers for the new Halo 3 ODST, she may have been more excited then I was. I think we’ll end up playing ALL of the other Halo games through again before ODST comes out. For those of you who were wondering, no, my wife playing Halo has nothing to do with any of this… My wife chooses to use Linux and Open Office, she LOVES Halo… Just wanted you to know how lucky I am. If you were in my shoes wouldn’t you tell everyone?

Anyway, her part time job is online tutoring. So when her laptop started dying I needed to get her a desktop up and running as a backup. She had the choice between two computers I also gave her the choice of Windows XP, Ubuntu or Fedora and assuming she chose Linux the choice between Gnome and KDE. Continue reading »

Tagged , , , , , , , , , |

ServerDensity.com – Hosted Server Monitoring

Ever find something online and you have no idea how you found out about it? Well, this is sorta the case. I think I found out about ServerDensity.com via some well placed marketing on twitter. Regardless of how I found it, I did.

There have been and probably still are a million hosted remote server monitoring services; years ago I tried most of them. All I needed at the time was a remote service to monitor my Nagios monitoring server. Lets face it–when the network connection to the server that monitors all your other servers fails, it can’t exactly send you an email. But an hour later you’ll see an email telling it is back up, implying you know it WAS down :). I tested a couple services that were basically ping monitoring… Didn’t like any of them. Either they were too expensive, overly complex or unstable. One I liked didn’t notify me a server hiccup and when I went to the site I found they were no longer in business. I finally stuck a simple Nagios configuration on a server I run for my Dad’s Office. It’s one of the servers I monitor from my network and since I give him all that free computer work, I let Nagios have a couple CPU cycles every 5 minutes to make sure it can talk back to my monitoring server.

Load Monitoring my sever via ServerDensity.com

Load Monitoring my sever via ServerDensity.com

So today I find ServerDensity.com. I make a quick trip through the site and like what I see, so I decided to try it. I should mention at this point, the service is beta, but it looks like they are set to launch later this month. Any linux geek worth their salt should be able to set this up in less than five minutes. Anyone who can follow instructions can set it up in 15.

I pulled up an ssh connection to a test server I have here that doesn’t do much. AMD 64bit, couple gigs of ram, nothing fancy… it runs Xen with one virtual server running on it 24/7 (my wife’s file server) and occasionally a virtual server that is used as a second test platform for server images.  If that server goes down, my wife will tell me pretty quick, so I don’t monitor it via Nagios. Literally 3 minutes after entering my ssh password I had the script downloaded, configured, running and was seeing the updates on the website.

The agent is nothing fancy, a small python script that runs as a daemon and doesn’t even need root permissions to work (which I very much appreciate). It pushes load stats to the server every minute or so, no holes to punch in a firewall, no further configuration needed. I did script it to automagically start at boot for simplicity.

The website is very simple and polished and as you see in the screen capture above, the stats are nicely graphed for you with noted min/avg/max. There are plenty of alert options you can configure to let you know when your server is acting up, down or being slashdotted. I’m still playing with the alerts a bit, but they seem straightforward.

I will point out that I haven’t had a chance to test the Apache monitoring yet. I don’t have Apache running on that box so I’ll put it off for another day.

All in all I’m pretty impressed–looking forward to the public launch of the service and updates to the agent. It won’t replace Nagios or (wait, is there anything other than Nagios? what’s that other one called? hmm… nevermind), but for those that don’t need as many options and need a hosted solution or just a solution to monitor their Nagios box, you might give ServerDensity.com a look. I’ll keep an eye on them for a while, but I do believe I’d recommend this option to clients when it fits their needs.

Tagged , , , |

Zune Fail – programming bug on 30GB units

Many of you probably heard about the Zune leap year bug. I saw a couple of Zunes with the problem before Microsoft released the official “fix.” On December 31st anyone with a 30GB Zune was annoyed to find it would freeze during start up and become totally unresponsive.  Late in the day Microsoft released a “fix” I hesitate to use that word. They suggested leaving the Zune on the frozen screen (you didn’t have much choice) and letting the battery run dead. Once it ran dead, wait till January 1st to turn it back on and sync it with your computer.

The source for the offending code showed up on several websites last night and I pulled out the function that has the problem. Unless your a programmer (I’m not) or can at least read most code (like me) then it won’t mean anything to you (just smile and nod). But basically on December 31st in a leap year there are 366 days, this code gets stuck in a loop when days equal 366 and there isn’t a graceful error exit. It just keeps telling itself that days equal 366 and it doesn’t know what to do.

This code is for a chip made by a third party, not Microsoft. I keep wondering if other devices have had the same problem.

//----------------------------------------------------------
//
// Function: ConvertDays
//
// Local helper function that split total days since Jan 1, ORIGINYEAR into 
// year, month and day
//
// Parameters:
//
// Returns:
//      Returns TRUE if successful, otherwise returns FALSE.
//
//----------------------------------------------------------
BOOL ConvertDays(UINT32 days, SYSTEMTIME* lpTime)
{
    int dayofweek, month, year;
    UINT8 *month_tab;

    //Calculate current day of the week
    dayofweek = GetDayOfWeek(days);

    year = ORIGINYEAR;

    while (days > 365)
    {
        if (IsLeapYear(year))
        {
            if (days > 366)
            {
                days -= 366;
                year += 1;
            }
        }
        else
        {
            days -= 365;
            year += 1;
        }
    }
Tagged , , , , |